A new wave of medical imaging is upon us

Monash University’s Faculty of Information Technology (IT) Digital Health team are bringing scans to life, by building 3D digital bodies that will provide medical professionals with unparalleled opportunities for exploration and interaction.

The aim? Enable doctors, surgeons and forensic specialists to explore traditional medical images, such as magnetic resonance imaging (MRI) and computed tomography (CT) scans, in ways never thought possible.

The pioneering projects range from creating 3D models of organs that appear almost like a hologram and using Artificial Intelligence (AI) to help diagnose COVID-19, to tracking bullet trajectories in virtual autopsies.

A new tool pioneered by the Digital Health team in the Faculty of IT, will provide health professionals with unparalleled insights and enhanced interaction with the human body.

The first-of-its-kind device will amplify existing medical scans to provide medical practitioners with a non-invasive insight into a patient’s illness.

Currently, scans are used to diagnose, treat and monitor patients. Even when the images are three dimensional (3D) most medical professionals view them on flat computer screens, or printed media, as a series of cross sections.

But what if these same scans could be viewed in 3D? They would appear in the office, and you could interact with them using a master controller, allowing you to peer around an organ, or peel through the layers of the head.

Enter the Embodied Axes. The device looks deceptively simple. With three arms, each with a series of sliders and knobs, it’s small enough to sit on your office desk.

Pioneering the device, in conjunction with industry and medical experts, is Dr Maxime Cordeil, a Monash University researcher and lecturer in the Faculty of IT.

He said the easiest way to understand how the Embodied Axes works was to describe it in action.

A radiologist is planning a procedure where a needle will be inserted into a patient’s pancreas.

Usually, he would use CT scans on a computer to work out the best needle placement, by considering the organ depth and needle angle.

But today he puts on an augmented reality (AR) headset.

Unlike virtual reality (VR), which blocks out the physical world and fully immerses the user in a new environment, when he puts on the headset nothing in his surroundings changes.

It's only when his computer connects to the Embodied Axes device that a 3D model of the patient’s pancreas, which has been created by the CT scans, appears before him. It looks like it is ‘floating’ within the device.

“The Embodied Axes allows the medical professional to see the data in 3D, as if it was a real part of the body,” Dr Cordeil said.

Aside from the visualisation advantage, one of the unique aspects of the device is that it also offers unparalleled interactivity.

The control panel allows the radiologist to directly manipulate the images with precision.

“They can physically interact with the 3D object in their own environment,” he said.

“They can peer around the side of the object, slice it and explore a different view of it, which is important when you are dealing with small objects, such as a tiny lesion on a liver.”

Another point of difference is the way you use the control panel. Usually interacting with an object or data in augmented reality involves using your hands.

However, this can lead to the ‘gorilla-arm effect’: too many movements and the hand begins to shake, reducing precision.

“Our control panel overcomes this problem, by allowing the user to make tiny micro-movements of approximately 100 slices per millimetre with the sliders and knobs.”

In a recent trial, presented at the prestigious ACM Conference on Human Factors in Computing Systems (CHI), the Embodied Axes was found to be more accurate than conventional controllers for selecting specific pieces of data, primarily because it enhanced the navigation speed.

The potential applications of the device will be extensive. For instance, it could be used as a presentation tool of lawyers in the courtroom to supplement the information provided to the jury about the way a person died, or help specialists prepare for complex operations.

The tool will also enhance collaboration between medical professionals, fast tracking diagnosis and enhancing research efforts.

Dr Cordeil said the Embodied Axes could be particularly beneficial for rural communities in our sparsely populated country.

“We tested whether two users, who both had the device, but were 1,000 kilometres apart, could explore the data together remotely.

“It was a successful proof-of-concept trial. We found that the system potentially provided a valuable cue for increasing the sense of presence for the remote collaborators.”

As part of the Human-in-the-loop design process, Dr Cordeil and his team consulted with potential users, including: medical professionals, forensic pathologists and data science engineers.

The current prototype is low cost, built with off-the-shelf electronics, and a Monash Faculty of IT PhD student is currently refining the device to enable larger trials.

Fast-tracked COVID-19 diagnosis

Another Digital Health project that enhances the use of medical images uses advanced AI to read lung CT images and generate medical reports to help diagnose COVID-19.

The virus can be detected via a blood test, but using CT scans results in rapid results, explains Dr Xiaojun Chang, a Senior Lecturer at Monash University’s Faculty of IT. He also has the ARC Discovery Early Career Researcher Award (DECRA).

In the world-first model, complex medical visual information in the lung scans is analysed by AI and a medical report is automatically generated.

“The technology can not only identify potential COVID-19 cases, but it also provides doctors with a complete picture of the diagnosis, including whether fever, coughing or breathing difficulties are present,” Dr Chang said.

“This is the world’s first COVID-19 public CT report dataset. It also demonstrates how AI can produce accurate and robust medical reports.”

The project is a collaboration between Monash University, Sun Yat-sen University and The First Affiliated Hospital of Harbin Medical University.

Information overload?

The AI-generated medical report could also be used to help health professionals diagnose patients with a range of other health conditions.

Dr Chang said many billions of gigabytes of healthcare data is being produced.

“It’s a tsunami of information, but it will be of limited value if we do not give healthcare providers accessible, quality information at the point that they are delivering patient care,” Dr Chang said.

“Data is just data unless it is logically compared and transformed into information.”

In the case of the CT scans for COVID-19, researchers have applied Machine Learning algorithms to mine the components and patterns from the data and obtain the actionable insights.

Currently the interpretation of the medical images is conducted by specialised health professionals, who write reports on their findings. This includes information, such as whether the area of the body scanned was found to be normal, or potentially abnormal.

“Even for experienced radiologists, writing imaging reports can be tedious and time-consuming,” Dr Chang said.

The Monash IT solution designs to mimic the process that a medical professional performs when they analyse a scan.

This usually involves scanning the overall image, carefully inspecting the area of concern and writing a report drawing upon medical information and their experience.

To mirror this approach, Dr Chang designed a new Auxiliary Signal-Guided Knowledge Encoder-Decoder (ASGK) that uses complex and inventive approaches to guide medical knowledge transfer and learning. He has even built in a medical textbook.

To test the effectiveness of ASGK, he and his collaborators sourced images from more than 700 patients who had COVID-19 (the largest known data set) and 40,000 images of the lungs.

They found the AI-generated medical reports out-performed all existing models. Dr Chang said their approach was about using AI to help guide decision making, not about replacing it.

Digital autopsies are the future of forensic medicine

Healthcare is not the only field set to be transformed by digital health.

The Victorian Institute of Forensic Medicine (VIFM) performs thousands of CT scans on deceased people each year.

Finding new ways to examine and explore the scans could reveal new information about how to prevent death and reduce the need for invasive post-mortems.

The Digital Health team is working with global IT company Leidos, VIFM and the State Coroner, to use Machine Learning and CT scans to build 3D models of the human anatomy.

It could be used to map bullet trajectories in shooting victims and determine the calibre and range from which a gun was fired.

The project may also pave the way for virtual autopsies.

In addition to the potential time and financial savings, digital transformations like these could provide grieving families with closure faster and make it easier to accommodate cultural rituals around funerals.

Professor of Practice in Digital Health, Professor Chris Bain, said these projects highlighted how the intersection between medicine and technology could help solve some of the world’s biggest health challenges.