Governing the next generation of clinical intelligence
Artificial intelligence is rapidly moving from research laboratories into everyday clinical practice. Algorithms now assist clinicians in interpreting radiological images, predicting patient deterioration, guiding treatment planning and optimising hospital workflows.
Recognising the scale of this transformation, the European Union has introduced the EU Artificial Intelligence Act, the first comprehensive legal framework regulating artificial intelligence across a major economic region.
For healthcare systems the implications are significant. Many clinical AI systems fall into the Act’s high-risk category, meaning they must meet strict regulatory requirements before entering the European market.
Yet the AI Act is not simply another compliance exercise. It represents a broader attempt to create a governance infrastructure for digital medicine, ensuring that artificial intelligence in healthcare is safe, transparent and accountable.
A risk-based regulatory architecture
The defining principle of the AI Act is its risk-based model of regulation. Rather than applying uniform rules to every AI system, the Act classifies technologies according to the level of risk they pose to individuals or society.
The regulation defines four broad categories.
Prohibited AI systems
Some applications are banned entirely because they are considered incompatible with fundamental rights. Examples include social scoring systems or technologies designed to manipulate human behaviour.
High-risk AI systems
Applications deployed in critical sectors such as healthcare, transport, education and law enforcement fall into this category. Most AI systems used in medical diagnosis or decision support are classified as high risk.
Limited-risk AI systems
These systems must meet transparency requirements. For example, users must be informed when they are interacting with AI.
Minimal-risk AI systems
Low-risk applications such as recommendation engines or spam filters face minimal regulatory obligations.
From a healthcare perspective, the high-risk category is the most consequential, as it determines how clinical AI systems must be developed, evaluated and monitored.
Why healthcare AI is classified as high risk
Artificial intelligence increasingly influences clinical decision-making.
Examples include:
- radiology image interpretation algorithms
- predictive models for sepsis or cardiac risk
- AI-assisted oncology decision support systems
- automated triage tools in emergency departments.
Because these technologies can directly affect diagnosis or treatment decisions, regulators consider them to pose potential risks to patient safety.
For this reason, many medical AI applications are classified as high-risk systems under the AI Act.
Functional requirements for high-risk AI systems
The AI Act introduces several operational obligations for developers of high-risk AI systems.
Lifecycle risk management
Developers must implement a continuous risk-management framework covering the entire lifecycle of the system, from design and development to post-market monitoring.
Data governance and dataset quality
Training, validation and testing datasets must be relevant, representative and free from systematic bias. This requirement addresses concerns that AI systems trained on incomplete datasets may produce inequitable outcomes.
Transparency and documentation
Developers must maintain detailed technical documentation describing how the algorithm operates, how it was trained and how performance was evaluated.
Human oversight
AI systems must support human decision-making rather than replace it entirely. Clinicians must retain the ability to interpret and override automated recommendations.
Post-market monitoring
AI systems must be monitored after deployment to detect performance drift or emerging safety risks.
This lifecycle approach reflects a key insight from academic research: machine-learning systems behave dynamically in real-world environments, requiring ongoing evaluation rather than one-time certification.
Conformity assessment and medical device regulation
Healthcare AI does not exist in a regulatory vacuum.
Many clinical AI systems already fall under the EU Medical Device Regulation, which governs software used for medical purposes.
In practice, developers must therefore satisfy two regulatory frameworks simultaneously:
- Medical Device Regulation (MDR) – assessing clinical safety and performance
- EU Artificial Intelligence Act – governing algorithm transparency, data governance and lifecycle monitoring.
Once compliance is demonstrated, the system receives CE marking, allowing it to be marketed across the European Union.
Academic perspectives: governing algorithmic medicine
Legal scholars and policy researchers view the AI Act as a landmark in global technology governance.
Several academic insights have emerged in recent literature.
AI regulation as a continuation of patient safety law
Some researchers argue that the AI Act should be interpreted as an extension of existing safety frameworks rather than an entirely new regulatory model.
From this perspective, the Act simply adapts medical device governance to the realities of algorithmic medicine.
The challenge of regulating adaptive systems
A major issue discussed in the literature concerns adaptive machine-learning models, which evolve over time as new data are introduced.
Regulators must determine how such systems should be evaluated when their behaviour changes after deployment.
Transparency versus technical complexity
Deep learning systems can be difficult to interpret. Policymakers therefore face the challenge of balancing demands for explainability with the technical limitations of modern AI architectures.
Economic implications: reimbursement and health technology assessment
The AI Act also has important implications for health economics and reimbursement policy.
Historically, many AI-based healthcare technologies struggled to gain reimbursement because payers lacked confidence in their clinical evidence or transparency.
The new regulatory framework could change this dynamic.
By requiring detailed documentation, robust data governance and post-market monitoring, the AI Act creates conditions that allow health technology assessment bodies to evaluate AI systems more rigorously.
This alignment becomes particularly important in light of the EU HTA Regulation, which introduces joint clinical assessments across the European Union.
Together these frameworks may enable a more structured pathway from algorithm development to reimbursement approval.
AI regulation and the European health data ecosystem
Another crucial policy interaction involves the European Health Data Space Regulation.
While the AI Act regulates algorithms, the EHDS aims to create a continental infrastructure for health data access.
This relationship is increasingly recognised in academic literature.
AI systems require large, high-quality datasets for training and validation. The EHDS could provide precisely this type of infrastructure through federated networks connecting hospitals, registries and national health databases.
In this sense, EHDS functions as the data engine, while the AI Act functions as the governance framework for algorithm deployment.
Europe versus the United States: two regulatory philosophies
The European approach differs significantly from the regulatory model used in the United States.
In the US, AI systems used in healthcare are regulated primarily through the U.S. Food and Drug Administration.
The FDA framework focuses on software as a medical device (SaMD) and emphasises iterative algorithm updates and clinical validation.
By contrast, the EU model introduces a broader governance framework addressing transparency, data quality and societal impact.
These two approaches reflect differing regulatory philosophies:
| European Union | United States |
| risk-based governance | product-based regulation |
| algorithm transparency | performance validation |
| societal oversight | regulatory agility |
Over time, global AI governance may emerge through interaction between these two models.
Toward a new infrastructure for clinical intelligence
Artificial intelligence is increasingly embedded within healthcare decision-making.
The EU Artificial Intelligence Act represents the first attempt to regulate this emerging layer of clinical intelligence infrastructure at continental scale.
By establishing requirements for transparency, data quality, risk management and human oversight, the Act seeks to ensure that AI systems deployed in healthcare are trustworthy and safe.
Its ultimate success will depend not only on legal text but on implementation: how regulators interpret its provisions, how developers design compliant systems and how healthcare organisations integrate AI responsibly into clinical workflows.
What is clear is that the governance of medicine is entering a new era one in which algorithms become part of the regulatory landscape itself.
Frequently Asked Questions (FAQ)
1. What types of healthcare AI systems are classified as “high-risk” under the EU Artificial Intelligence Act?
Under the EU Artificial Intelligence Act, AI systems are considered high-risk when they are intended to influence decisions that could affect health, safety, or fundamental rights.
In healthcare this typically includes AI systems used for:
- diagnostic image interpretation (e.g. radiology, pathology)
- clinical decision-support systems
- predictive models for patient deterioration or disease risk
- automated triage or treatment recommendation systems
- software classified as medical devices under the Medical Device Regulation (MDR)
High-risk classification triggers strict regulatory requirements including risk management frameworks, dataset governance standards, human oversight mechanisms, and post-market monitoring. These obligations are intended to ensure that algorithmic systems used in clinical contexts meet comparable safety standards to other regulated medical technologies.
2. How does the EU Artificial Intelligence Act interact with the Medical Device Regulation (MDR)?
Many clinical AI applications already fall within the scope of the EU Medical Device Regulation (MDR) because they function as Software as a Medical Device (SaMD).
The AI Act does not replace MDR but instead introduces additional algorithm-specific governance requirements.
In practice:
- MDR evaluates clinical safety, performance and risk classification of the device.
- AI Act focuses on algorithm transparency, training data governance, lifecycle monitoring and human oversight.
Developers of AI-based medical technologies must therefore comply with both regulatory frameworks simultaneously before obtaining CE marking and market access in the European Union.
3. Why does the AI Act place strong emphasis on dataset governance and bias mitigation?
Machine learning systems derive their predictive performance from the data used during training and validation.
If datasets are incomplete, unrepresentative or biased, AI models may generate systematically inaccurate outputs for certain patient groups, potentially leading to unequal clinical outcomes.
The AI Act therefore requires developers to demonstrate that training and testing datasets are:
- relevant to the intended clinical application
- sufficiently large and representative
- free from known sources of bias where possible.
These requirements reflect growing concerns within biomedical research that algorithmic bias could exacerbate existing health disparities if not carefully controlled.
4. How could the AI Act influence health technology assessment and reimbursement decisions?
Health technology assessment (HTA) agencies evaluate whether medical technologies deliver sufficient clinical benefit and cost-effectiveness to justify reimbursement.
Historically, AI technologies have faced challenges in this process due to limited transparency around algorithm design and performance.
By requiring:
- detailed technical documentation
- robust dataset governance
- post-market monitoring of real-world performance
the AI Act provides a regulatory framework that may improve the evidence base available to HTA agencies.
In combination with the EU HTA Regulation and the European Health Data Space, the AI Act could therefore facilitate more systematic evaluation of AI-based healthcare technologies and potentially support clearer reimbursement pathways.
5. How does the EU approach to regulating medical AI differ from the United States?
The European Union and the United States regulate healthcare AI through different policy frameworks.
In the United States, the Food and Drug Administration (FDA) regulates AI-based technologies primarily as Software as a Medical Device (SaMD), focusing on clinical safety and performance.
The EU approach is broader. The AI Act introduces a horizontal regulatory framework that applies across multiple sectors and emphasises:
- risk classification of AI systems
- transparency and explainability
- governance of training datasets
- human oversight and accountability.
While both systems aim to ensure patient safety, the EU model places greater emphasis on algorithm governance and societal oversight, whereas the US framework prioritises product-level regulatory agility and iterative updates.
References
European Union Legislation
European Parliament and Council of the European Union.
Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act).
Official Journal of the European Union. Brussels: European Union; 2024.
European Parliament and Council of the European Union.
Regulation (EU) 2017/745 on Medical Devices (MDR).
Official Journal of the European Union. Brussels: European Union; 2017.
European Parliament and Council of the European Union.
Regulation (EU) 2021/2282 on Health Technology Assessment.
Official Journal of the European Union. Brussels: European Union; 2021.
European Commission.
Proposal for a Regulation Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act).
COM(2021)206 final. Brussels; 2021.
Academic Literature on the EU AI Act
Floridi L., Cowls J., Beltrametti M., et al.
AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations.
Minds and Machines. 2018.
Veale M., Borgesius F.Z.
Demystifying the Draft EU Artificial Intelligence Act.
Computer Law Review International. 2021.
Edwards L.
Regulating AI in Europe: Four Problems and Four Solutions.
European Journal of Risk Regulation. 2022.
Veale M., Zuiderveen Borgesius F.
The EU Artificial Intelligence Act: A Risk-Based Approach to Regulating AI.
Common Market Law Review. 2021.
Cath C., Wachter S., Mittelstadt B., Taddeo M., Floridi L.
Artificial Intelligence and the ‘Good Society’: The US, EU, and UK Approach.
Science and Engineering Ethics. 2018.
Mittelstadt B.
Principles Alone Cannot Guarantee Ethical AI.
Nature Machine Intelligence. 2019.
Medical AI Regulation and Clinical Governance
Topol E.
High-Performance Medicine: The Convergence of Human and Artificial Intelligence.
Nature Medicine. 2019.
Kelly C.J., Karthikesalingam A., Suleyman M., Corrado G., King D.
Key Challenges for Delivering Clinical Impact with Artificial Intelligence.
BMC Medicine. 2019.
Benjamens S., Dhunnoo P., Meskó B.
The State of Artificial Intelligence-Based FDA-Approved Medical Devices and Algorithms.
npj Digital Medicine. 2020.
Jiang F., Jiang Y., Zhi H., et al.
Artificial Intelligence in Healthcare: Past, Present and Future.
Stroke and Vascular Neurology. 2017.
He J., Baxter S.L., Xu J., Xu J., Zhou X., Zhang K.
The Practical Implementation of Artificial Intelligence Technologies in Medicine.
Nature Medicine. 2019.
AI Safety, Data Governance and Algorithmic Transparency
Rudin C.
Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead.
Nature Machine Intelligence. 2019.
Doshi-Velez F., Kim B.
Towards a Rigorous Science of Interpretable Machine Learning.
arXiv preprint. 2017.
Wachter S., Mittelstadt B., Russell C.
Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.
Harvard Journal of Law & Technology. 2018.
Mittelstadt B., Allo P., Taddeo M., Wachter S., Floridi L.
The Ethics of Algorithms: Mapping the Debate.
Big Data & Society. 2016.
United States Regulatory Context
U.S. Food and Drug Administration.
Artificial Intelligence and Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan.
Silver Spring, Maryland: FDA; 2021.
U.S. Food and Drug Administration.
Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning-Based Software as a Medical Device.
Discussion Paper; 2019.
International Medical Device Regulators Forum.
Software as a Medical Device: Clinical Evaluation Guidance.
IMDRF/SaMD WG/N41FINAL; 2017.
Digital Health Policy Context
European Commission.
A European Strategy for Data.
Brussels; 2020.
European Commission Directorate-General for Health and Food Safety.
Shaping Europe’s Digital Future in Healthcare.
Brussels; 2022.
European Commission.
Ethics Guidelines for Trustworthy Artificial Intelligence.
High-Level Expert Group on AI; 2019.