In 2019 a team led by the physician and economist Ziad Obermeyer took apart an algorithm that American hospitals and insurers were using to decide which patients needed extra help. The software reached into the records of roughly 200 million people a year, and on its face it looked neutral. It predicted who would run up the biggest medical bills the following year, then flagged those patients for more attention. The trouble was the mechanism. Black patients with the very same chronic conditions as white patients generated about $1,800 less in spending a year, not because they were healthier but because they got worse care to begin with, and the algorithm read the smaller bill as a smaller need. Correcting that bias would have raised the share of Black patients enrolled in the extra-care program from 17.7 percent to 46.5 percent, more than double. The tool had been running quietly across the health system for years before anyone went looking.
We have seen this pattern before, and we are about to see a great deal more of it. The tools have multiplied, the regulators have finally written a rulebook, and a new commentary in the Journal of the Royal Society of Medicine argues the rulebook is built to protect the system rather than the person lying on the table. Regulators looked at medical AI and decided it was dangerous enough to supervise. They did not decide it was intrusive enough for a patient to refuse.
The commentary is “The need for patient rights in AI-driven healthcare: risk-based regulation is not enough.” Its lead author is Thomas Ploug, a professor of data and AI ethics at Aalborg University in Denmark, writing with four colleagues. Their target is the dominant model regulators have settled on across the Western world: classify a medical AI by how risky it is, then load safety obligations onto the companies that build and deploy it. The European Union’s AI Act, the flagship of this approach, sweeps medical AI into its high-risk tier and demands conformity controls before such a system can go to market.
That sounds protective, and Ploug’s team says it has three holes you could drive a hospital through. It ignores individual patient preferences, treating “the patient” as one statistical body rather than as people who weigh accuracy, bias, and the role of a machine in their own care very differently. It overlooks the systemic and long-term effects of letting these tools spread through a health system. And it leaves patients out of the regulatory process entirely, deciding what is safe for them without a seat for them at the table. “Regulation must move beyond system-level safety and account for individual rights and participation,” Ploug argues. “AI is transforming healthcare, but it must not do so at the expense of patient autonomy and trust.”
To see why that gap matters, look at the scale of what is already deployed. In the United States the Food and Drug Administration has now authorized more than a thousand AI and machine-learning-enabled medical devices, the large majority cleared through the 510(k) pathway, which lets a new device reach the market by showing it resembles one already cleared. It is a route built for speed and lineage, not for asking whether the patient understood that an algorithm was reading their scan, or whether they agreed to it. Risk-based regulation, in both Brussels and Washington, checks the box at the level of the system. It has very little to say to you.
And the system-level check is not the safety net it is sold as. Take the sepsis-prediction model Epic built into the electronic health records used at hundreds of American hospitals. When researchers at Michigan Medicine finally validated it against tens of thousands of patient encounters and published the result in 2021, they found the model missed roughly two-thirds of sepsis cases while burying clinicians under false alarms. It had been switched on at the bedside long before anyone outside the vendor tested whether it worked. The Optum algorithm and the Epic model are not exotic failures. They are what happens when a tool clears a system-level bar and then meets a real patient population the bar never accounted for.
So what would patient-centered regulation add? The commentary proposes four concrete rights, and the authors say none of them exist as patient rights in the current risk-based frameworks. A right to an explanation of an AI-generated decision: the patient gets told a machine weighed in, and what it concluded. A right to give or withdraw consent to AI in your care: you can say no. A right to a second opinion: when the algorithm decides, a human looks again. And a right to refuse a diagnosis or screening built on your data when that data was used without your consent. None of this is radical. It is roughly what you already expect from a human doctor, ported to the machine that is increasingly standing next to one.
Why are those rights missing? Follow the incentives. The risk-based frameworks place their duties on the parties who build, sell, and clear these systems, and frictionless deployment suits all of them. The vendors get a fast, predictable route to market. The health systems get tools that promise to cut cost and workload. The patient, whose body the algorithm acts on, has no duty assigned in their favor and no recourse written down, which is how you arrive at a “high risk” label that generates paperwork for providers and silence for the people the risk lands on. When even an academic warning in a Royal Society journal, hardly a populist broadside, reaches that conclusion, the imbalance is not a fringe complaint.
The EU’s high-risk obligations for medical AI do not fully bite until 2027 and 2028, and across the Channel Britain is drafting its own framework: the MHRA’s National Commission into the Regulation of AI in Healthcare ran a public call for evidence that closed in February and published its first findings on 11 June 2026, with new rules promised this year. So the question worth watching is narrow and concrete. Will any of these regulators convert even one of Ploug’s four rights into binding law before the framework hardens, or will the next thousand devices clear the predicate pathway first while the rights stay an academic wish list? The last time an algorithm decided who deserved care, it took an outside team years to catch it. The rules being written now decide whether the next one gets caught at all.
Sources
- RSM media release – “AI in healthcare needs patient-centred regulation to avoid discrimination” (Ploug et al., Journal of the Royal Society of Medicine)
- Science – Obermeyer et al., “Dissecting racial bias in an algorithm used to manage the health of populations” (2019)
- JAMA Internal Medicine – Wong et al., external validation of the Epic Sepsis Model (2021)
- npj Digital Medicine – taxonomy of AI use across 1,016 FDA-authorized medical devices
- EU Artificial Intelligence Act – Article 6, classification rules for high-risk AI systems
- GOV.UK – MHRA National Commission into the Regulation of AI in Healthcare, call for evidence
- News-Medical – New commentary urges patient-centered AI regulation in healthcare systems