The goal of the project was to create a realistic system where future doctors can learn to do the amniocentesis procedure non-invasively. The whole project can be broken down into two subsystems that must work together. This relates to the virtual reality (VR) application and the mechanisms that constitute the actual test bed.
The programme was made using Unity, an integrated environment for creating 3D computer games and other interactive materials. The engine was also extended through the use of soft-body simulation plug-ins and tools to integrate the programme with VR goggles.
Since I was responsible for the part related to the operation of the application in the Unity 3D engine, the whole project will be presented from this perspective.
Features
USG Image: The generation is done by capturing a projection of a slice of the three-dimensional model defined by the ultrasound transducer. The texture is then filtered by a synchronised system of shading units.
Extended Reality: This application was not only to be run through the VR Goggles, but was still to match the VR image with the actual positioning of the patient model (phantom). Various computer vision and pattern recognition solutions in the camera images were used to achieve this.
Connecting to Devices: The app used custom devices such as a plunger to simulate skin and uterine resistance during operation. The app needed to connect these devices so that they could send two-way messages to each other.
Scenario System: The scenario system implemented in the application is intended for educational purposes, so that young doctors performing a given procedure for the first time are instructed step by step what to do. Additionally, the scenario system monitors whether the person performing the procedure will make any mistakes during the procedure, such as stabbing the fetus with a needle.
Using Your Own Hands: In order to obtain the greatest possible realism in the amniocentesis procedure, we gave up using controllers and chose the option of controlling our hands during the procedure.
