AI regulation in healthcare: Are we ready for the changes?

Artificial intelligence (AI) is rapidly transforming healthcare - from diagnostics to digital mental health tools - but this brings significant challenges in ensuring patient safety and regulatory compliance. Australia’s Therapeutic Goods Administration (TGA), the authority responsible for overseeing medical devices, is actively reassessing its legislative framework to address these emerging challenges.

Currently, not all AI products used in healthcare fall under TGA regulations. Only AI systems directly involved in diagnosis, prevention, monitoring, treatment, or alleviation of health conditions require regulatory approval. These systems must demonstrate substantial evidence of safety, effectiveness, and reliability, proportional to their assessed risk level.

But what exactly could change?

Firstly, the TGA is exploring the reclassification of AI-driven medical software into higher risk categories. Notably, AI products specifically used for clinical prediction or prognosis - previously classified as lower risk - could now face stricter classification.

Secondly, regulatory oversight may soon extend to software previously exempt from TGA scrutiny, specifically digital mental health tools, advanced software calculators, and laboratory information management systems. This adjustment aims to ensure comprehensive regulation across all significant medical AI applications, particularly as their complexity and potential risk have increased.

Thirdly, the TGA proposes enhancing the essential principles governing clinical software. This includes addressing emerging complexities from adaptive, generative, and machine learning AI, ensuring ongoing performance, and introducing clearer labelling requirements to identify AI usage within medical devices.

These proposed regulatory changes closely align with the Australian government's recently articulated ‘guardrails’ for high-risk AI, encompassing ten principles. These principles emphasise accountability, transparency, rigorous risk management, human oversight, and clear avenues to challenge AI-driven outcomes.

Why does this matter?

The implications of these adjustments are profound. AI technologies classified as high-risk may require more rigorous evidence and validation prior to clinical use, potentially slowing their integration into healthcare practice. Healthcare professionals will also need enhanced training to understand and navigate their evolving legal responsibilities in AI utilisation.

Notably, AI applications like clinical note summarisation or patient record management may remain outside TGA jurisdiction, instead regulated by bodies such as the Australian Health Practitioner Regulation Agency (AHPRA) and the Australian Commission on Safety and Quality in Health Care.

Finally, while the TGA supports international regulatory alignment, addressing biases inherent in AI systems trained on international data remains a significant challenge. Such biases could affect the accuracy and fairness of AI outcomes for Australia's diverse population.

What’s still up for discussion?

  • Should AI products used for prediction and prognosis be classified at a higher risk level?
  • Are the current software exclusions still appropriate?
  • How should the responsibilities for AI models and systems be clarified within the existing legal framework?
  • What measures can ensure that AI systems in healthcare remain transparent and accountable?
  • What safeguards should be put in place to handle evolving AI technologies?

Contribute to the conversation!

We encourage healthcare professionals, researchers, technologists, and policy-makers to engage in this ongoing dialogue. As we strive for a balanced and effective regulatory approach, your insights, experiences, and suggestions are invaluable in shaping a future where AI enhances healthcare while safeguarding patient welfare.

Written by Khoa Cao, Monash Institute of Medical Engineering and Paula Chen, Monash Medical Technology Lab

Khoa Cao is an Associate Professor of Medicine and Engineering at Monash University, Associate Director of the Monash Institute of Medical Engineering and the Director of the Monash Medical Technology Lab.

Paula Chen is a third-year Monash medical student and a research associate at the Monash Medical Technology Lab.