October 2026
Language models now score above the passing mark on medical licensing questions. In 2023, GPT-4 cleared the passing threshold on official USMLE practice materials by more than 20 points [1], and a version of Med-PaLM 2 tuned for MedQA, a benchmark of licensing-style questions, scored 86.5 percent [2]. It is tempting to read these numbers as evidence of clinical skill. They measure something narrower.
An exam question is a one-step prediction problem. The model receives a fixed, curated description of the patient, gives one answer, and is scored. Nothing it does changes what it sees next.
A patient is a sequential decision problem, and a hard one. The true state, what is actually wrong and how fast it is getting worse, is hidden. Most of what the clinician learns has to be bought with an action: a question, an exam, a test that takes time to come back. Actions can change the patient, what is known about her and how much time is left, and the patient keeps changing when nobody acts. Formally this is a partially observable Markov decision process. An exam asks whether you can name the answer once the information is in front of you. A patient asks whether you can get the information, in the right order, before it is too late to use.
Sequential problems punish small errors in a way one-step problems do not. A classic result in imitation learning makes this precise. A policy that copies an expert, with error rate ε on the situations the expert visits, can incur extra cost that grows with the square of the horizon, on the order of T²ε over T steps, because each early error moves it into situations the expert never showed it [3]. A model that is usually right on single questions can still drift badly over a long emergency visit with dozens of decisions. Its first mistake changes the patient it sees next.
Recent studies test this directly. A 2024 study in Nature Medicine built a simulated workup from 2,400 real emergency cases of abdominal pain [4]. Given the case up front, the models were less accurate than physicians and did not follow diagnostic or treatment guidelines. When they had to request the history, exam and test results themselves, they did worse still. A 2025 study in the same journal turned exam-style cases into conversations with a simulated patient, and every model it tested lost diagnostic accuracy [5]. The knowledge did not disappear between the two formats. What changed is that the model had to decide what to ask, and when.
Take a man in his late sixties with sudden flank pain and blood in his urine. On an exam, the question is written so the answer can be found. In practice, a review of published studies found that ruptured aortic aneurysms were first misdiagnosed in about four cases in ten, most often as a kidney stone or a heart attack [6]. Any model that passes the boards knows what an aneurysm is. The case shows what the skill actually is: keeping the dangerous alternative in view while gathering evidence, and acting before time runs out.
If the skill is sequential, it has to be measured and trained sequentially. That takes an environment where the patient’s state is hidden and changing, where information costs time, and where the score attaches to decisions, not only to the final answer. MedGym is a simulated emergency department for testing and training medical AI agents. Patients get better or worse over hours, based on what the agent does, and every decision is scored against an expert reference. Passing the boards is the entry requirement. The real work starts after it.