TU Dublin have developed AI assisted hardware/software package

About

Researchers at TU Dublin have developed a machine learning (AI) assisted hardware/software package or ecosystem that is run locally or via wireless or physically tethered connection on an Augmented/Virtual Reality Head Mounted Device (HMD) and worn by a neurosurgeon or training neurosurgeon, with networking capability to facilitate cloud computing and database retrieval. This ecosystem is intended for neurosurgical applications or training/education in the neurosurgery space.

A surgeon wearing a head-mounted device (HMD) is able to visualise automatically overlaid medical brain scan information on a patient (or patient analogue such as a dummy/manikin) so that structures appearing on the medical scan information overlap/register onto real world positions of said structures. Surgeons are able to toggle through different scan types, and to request a pretrained machine learning model’s recommendations for potential best entry routes for keyhole surgery, minimising potential damage to surrounding brain structures. In addition, they are able to view relevant medical information as well as virtual representations of surgical tools where said tools are embedded or partially hidden. Surgeon’s are also able to communicate with remote viewers/advisers and view annotations made by said advisers.

The ecosystem utilises a combination of HMD sensor-derived information (point cloud data, video/infrared/time of flight sensors etc) machine learning models and manual adjustment to detect real-world surfaces and align medical information based on image targets in the real world and/or patient facial features. This allows medical scan information to be overlaid via Augmented Reality onto the patient, so that they correlate with real world organic structures. Overlay alignment may be using a third party image recognition plugin that tracks real-world image markers, as well as open source computing algorithms or third party cloud computing solutions.. This overlay can also be corrected manually by a user. Conversion of MRI/CAT data into 3D volumes may be through external programs, or within the ecosystem itself.

Advantages

1. The use of both brute force algorithms and Artificial Intelligence to inform surgeon routes and predict surgery results, including potential link to database of previous surgery information.
2. Self-contained processing capability within one Head Mounted Device system.
3. No requirement for external controls – voice and/or gesture activated
4. Image target recognition for alignment of medical scans accurately to patient or patient analogue

Opportunites

1. Allowing for overlays of medical scan information directly spatially correlated with their real-world counterpart and reducing the need for neurosurgeons to refer to separate screens to view scan information. Networked environments will also allow remote viewers to supervise and advise on the operation in realtime, marking and annotating areas of interest.
2. Training future neurosurgeons using either patient analogue dummies (manikins) or fully virtual/augmented overlay training, with simulated patient head and tumour
3. Allowing patients to view representations of their own MRI scans, and enabling doctors to explain upcoming procedures visually in greater detail than previously possible

Current development status

Others

TU Dublin is seeking commercial partners to assist in bringing this technology to market.

Desired business relationship

Technology selling

Joint ventures

Technology development