Medical Chatbots Reached African Patients Before the Benchmark Meant to Grade Them

When a chatbot is said to have passed a medical licensing exam, the exam has usually been the American one, built around American patients, American disease patterns and an American hospital waiting at the end of every referral. A language model learns medicine from what it is fed, and when a case falls outside that diet it does not go quiet. It answers anyway, fluently, in whatever language the question arrived in, which is why the most useful thing anyone can do with these systems is test them far from home.

Researchers did that with AfriMed-QA, a benchmark presented at the ACL conference in 2025. It put 15,000 questions from more than 60 medical schools in 16 countries to 30 language models. The models’ multiple-choice performance “clearly lags” their showing on the US licensing exam, the authors reported, and the smaller, edge-friendly models built for modest local hardware struggled to reach a passing score at all. A second finding was more unsettling: when consumers compared the machines’ answers and explanations with those of clinicians, they consistently preferred the machines. AfriMed-QA was written in English, so it measured only the geography gap. Whatever gap exists in Swahili or Amharic would come on top of it.

That is the background to an announcement in Nature Medicine on 6 October introducing MAGIC, which describes itself as “an international network dedicated to building a global evaluation ecosystem for generative AI in health care.” Its listed contact is Nan Liu at Duke-NUS Medical School in Singapore. It promises two things: a benchmark, and a reporting standard meant to “clarify the scope of application and limitations” of generative AI before it is deployed. The benchmark grades models on three axes: languages “not yet widely included, especially low-resource languages”; “region-specific disease spectrums and comorbidity patterns”; and “whether AI can produce reliable responses within local sociocultural contexts.” Put that list next to AfriMed-QA and it reads as a map of where the models are expected to break. Building the yardstick around the weak points is an honest admission of where the technology stands.

The trouble is the calendar. In 2024, Jacaranda Health’s PROMPTS service in Kenya, a two-way text line for pregnant and postpartum patients that answers with AI-generated replies in Swahili and English, reached more than 500,000 users. It cut response times from hours or days to minutes and flagged serious cases for a human to follow up.

The evidence money came later. Also in 2024, the Gates Foundation, the Novo Nordisk Foundation and Wellcome launched a US$300 million global-health research partnership. Its second investment, announced in February 2026, was US$60 million for an initiative called Evidence for AI in Health, or EVAH. The initiative will evaluate machine-learning, computer-vision and large-language-model tools that help frontline workers with “triage, diagnosis, and referral” in primary and community care across sub-Saharan Africa, South Asia and Southeast Asia, with J-PAL and the African Population and Health Research Center running the work. The global benchmark for grading such tools arrived eight months after that, last week, from a network still inviting collaborators to help build it.

The funders’ own words suggest which way they expect the evaluation to come out. “We are already seeing the positive impact AI can have in global health,” said Trevor Mundel, the Gates Foundation’s president of global health, in the same release that announced the money to find out. He described safety testing as a way to speed things up: making sure the tools are “safe and effective can significantly reduce the time it takes to turn promising ideas into scalable innovations.” Lene Oddershede, chief scientific officer at the Novo Nordisk Foundation, made the admission that sits underneath the optimism: “more robust evidence is needed to guide investments and implementation locally.” Put plainly, the people paying for this field are saying in their own press release that they don’t yet have the evidence to decide where the money and the tools should go.

Run the same sequence past an American hospital board. Put a diagnostic aid in front of the nursing staff, pay for studies of whether it works once it’s there, and finish writing the grading rubric while both are under way. Most boards would ask for the evidence first. In Kenya the order ran the other way, and the people carrying the experiment have the least room to absorb a confident mistake: a patient at a rural clinic who can’t easily get a second opinion, asking a question in a language the models have barely been tested in.

Eleni Linos, who directs the Stanford Center for Digital Health, has described the reasoning that makes this feel acceptable. When a trained doctor is available, she said, “your standards for that chatbot may be pretty high,” but when the alternative is a long wait, “what counts as good enough is different.” As a description of triage, that’s honest. It’s also how a lower bar gets normalized for the poor and then defended as compassion. Linos made the harder point herself: a correct answer does nothing “if there’s no surgeon in your community, or no antibiotics.” PROMPTS shows what a defensible version looks like. The model drafts the reply, and a person picks up the dangerous cases. Its numbers count users, not patient outcomes, but the design keeps a clinician in the loop rather than replacing one.

Then there’s the question of who writes the test. MAGIC lists “industry stakeholders” among the groups it brings together, alongside health care professionals and AI researchers, and says its participants will “collaborate in constructing evaluation resources and continuously refine them.” That puts industry in the room where the yardstick is built. The network’s website says nothing about who pays for it. None of this is hidden, and none of it is disqualifying by itself. It is a reason to judge a benchmark that calls itself “globally representative” by the results it publishes, not by what it says it intends.

MAGIC and EVAH will show whether the evidence arrives before the scale does, and whether it is published where a skeptic can check it rather than in a funder’s progress update. The first EVAH evaluations, run by J-PAL and the African Population and Health Research Center, are supposed to produce the data Oddershede says decision-makers lack: efficacy, economic value and acceptability, measured where the tools are already in use. MAGIC’s test is whether it publishes its benchmark scores openly, broken out by language and region, before the systems it is meant to judge reach millions more patients. If the grading keeps trailing the rollout, the benchmark will end up recording a deployment instead of governing one, and the patients who were promised a safeguard will have served as its proof of concept.

Sources

1. Nature Medicine: “MAGIC: an international network for evaluating generative artificial intelligence in global health” (6 October 2026)
2. MAGIC network: scope, benchmark dimensions and reporting standard
3. Gates Foundation: Evidence for AI in Health (February 2026)
4. AfriMed-QA, ACL 2025
5. Stanford Medicine: “Can generative AI tackle global health problems?” (2025)