Challenge
During project development, our team encountered several challenges.
Medical complexities
As radiology is a complex medical area that requires a deep understanding, our team had to navigate numerous intricacies. For example, organizing raw data from CT and MRI machines, assembling this data into 3D models, determining the final image quality, and more.
Recognizing the need for expert guidance, we invited an experienced radiologist to provide insights into radiology principles, guide us through the diagnostic process, and review our results. By combining the expert’s industry knowledge with our technical expertise, we were able to create a solution that aligns with real-world medical practice.
Visualization
Every study consists of numerous 2D scans from which we construct 3D models in Virtual Reality. The more scans the study has, the more complicated the construction is. So, the challenge was to ensure that the 3D models match the original flat images and are informative and detailed.
It was also important to maintain optimal lighting in the scenario room. We need to create volume and strike a balance between shadows and contrast to avoid issues like backlighting or excessively dark areas.
Data formats
To visualize studies in Virtual Reality, DICOM files should be converted to a suitable format. Since there are many DICOM variations that complicated building a custom converter, we first experimented with a ready-made one. However, it failed to meet all our requirements and made us dependent on a vendor.
So, we conducted additional research to develop our own converter that takes into account different file formats.
Measurements
The 3D models in Virtual Reality are smaller than those in reality. So, we configured the scale to ensure that users could obtain informative data reflecting the real-life dimensions when using a measuring tool.
Stomatological scans
Collaboration with a manufacturer of stomatological equipment introduced the need to visualize teeth scans, which differed from the formats we had previously worked with. With additional research and the assistance of stomatological specialists, users can now create teeth models in Virtual Reality.
During project development, our team encountered several challenges.
Medical complexities
As radiology is a complex medical area that requires a deep understanding, our team had to navigate numerous intricacies. For example, organizing raw data from CT and MRI machines, assembling this data into 3D models, determining the final image quality, and more.
Recognizing the need for expert guidance, we invited an experienced radiologist to provide insights into radiology principles, guide us through the diagnostic process, and review our results. By combining the expert’s industry knowledge with our technical expertise, we were able to create a solution that aligns with real-world medical practice.
Visualization
Every study consists of numerous 2D scans from which we construct 3D models in Virtual Reality. The more scans the study has, the more complicated the construction is. So, the challenge was to ensure that the 3D models match the original flat images and are informative and detailed.
It was also important to maintain optimal lighting in the scenario room. We need to create volume and strike a balance between shadows and contrast to avoid issues like backlighting or excessively dark areas.
Data formats
To visualize studies in Virtual Reality, DICOM files should be converted to a suitable format. Since there are many DICOM variations that complicated building a custom converter, we first experimented with a ready-made one. However, it failed to meet all our requirements and made us dependent on a vendor.
So, we conducted additional research to develop our own converter that takes into account different file formats.
Measurements
The 3D models in Virtual Reality are smaller than those in reality. So, we configured the scale to ensure that users could obtain informative data reflecting the real-life dimensions when using a measuring tool.
Stomatological scans
Collaboration with a manufacturer of stomatological equipment introduced the need to visualize teeth scans, which differed from the formats we had previously worked with. With additional research and the assistance of stomatological specialists, users can now create teeth models in Virtual Reality.
Solution
With our Virtual Reality app, users can upload any series of CT, MRI, or PET/CT images and reconstruct them in 3D. To view and interact with 3D models, users need Oculus Quest 2 VR headsets.
The system includes three components:
a desktop app for converting DICOM files into a format suitable for VR;
a VR app built on Unreal Engine 5;
an admin panel for creating user accounts for multi-user sessions.
To start the experience, a user needs to upload MRI or CT images to the converter, which prepares them for viewing in Virtual Reality. In this step, we’ve implemented the ability to download files anonymously, ensuring data privacy — a crucial consideration in the medical industry. This allows medical professors to present real case studies to students without compromising personal information.
Within the VR app, users can not only view and rotate 3D models but also:
scale models to explore specific body parts in detail;
apply a set of filters, separate for CT and MRI images, to visualize specific tissues by density;
slice the model at any angle to remove interfering structures;
measure objects using polygonal or regular measuring tools;
enhance visibility with a lamp.
With our Virtual Reality app, users can upload any series of CT, MRI, or PET/CT images and reconstruct them in 3D. To view and interact with 3D models, users need Oculus Quest 2 VR headsets.
The system includes three components:
a desktop app for converting DICOM files into a format suitable for VR;
a VR app built on Unreal Engine 5;
an admin panel for creating user accounts for multi-user sessions.
To start the experience, a user needs to upload MRI or CT images to the converter, which prepares them for viewing in Virtual Reality. In this step, we’ve implemented the ability to download files anonymously, ensuring data privacy — a crucial consideration in the medical industry. This allows medical professors to present real case studies to students without compromising personal information.
Within the VR app, users can not only view and rotate 3D models but also:
scale models to explore specific body parts in detail;
apply a set of filters, separate for CT and MRI images, to visualize specific tissues by density;
slice the model at any angle to remove interfering structures;
measure objects using polygonal or regular measuring tools;
enhance visibility with a lamp.
Results
Working closely with medical experts in radiology, our team created a comprehensive VR platform for recreating CT and MRI scans in volume and interacting with them. The solution has already been tested in several hospitals as a supplementary tool for educating young doctors, receiving positive feedback. A transplant surgeon noted that in VR, details are more clearly visible, making pathology identification and diagnosis easier.
We are proceeding with the project. Our future plans include enhancing collaborative sessions and delving deeper into multiplanar projection imaging.
Working closely with medical experts in radiology, our team created a comprehensive VR platform for recreating CT and MRI scans in volume and interacting with them. The solution has already been tested in several hospitals as a supplementary tool for educating young doctors, receiving positive feedback. A transplant surgeon noted that in VR, details are more clearly visible, making pathology identification and diagnosis easier.
We are proceeding with the project. Our future plans include enhancing collaborative sessions and delving deeper into multiplanar projection imaging.
