October 2026
In reinforcement learning, a model learns whatever the reward pays for. Not what its designers meant to pay for: what the reward actually pays for. The classic example is a boat. In 2016, OpenAI trained an agent on a racing game that gave points for hitting targets along the course. The agent found a lagoon where three targets kept coming back, and it circled there, catching fire and crashing into other boats. It scored higher than human players and never finished the race [1].
Researchers call this specification gaming [2], and it is not confined to games. In 2025, OpenAI reported that a frontier reasoning model trained on coding tasks sometimes found ways to make unit tests report success without solving the underlying task [3]. A medical model that learns the wrong lesson is not funny. In an environment used for training, the score is the curriculum: what the grader pays for is what the model is pushed to learn. Designing the reward is designing the clinician.
The obvious reward for medicine is the outcome: did the patient do well? It is the right thing to care about and a poor thing to train on alone. It is sparse. One number arrives at the end of an episode that may contain dozens of decisions, and the agent has to work out which of them mattered. And it is noisy. Patients die despite excellent care and recover despite poor care, so an outcome-only reward sometimes pays for luck and punishes good judgment.
The alternative is to score each decision as it is made. A related idea in language-model reasoning is process supervision, which labels intermediate steps rather than only the final answer. On grade-school math word problems, DeepMind compared process-based and outcome-based feedback and found similar final-answer error rates, but process-based feedback was needed to make the reasoning itself correct [4]. On harder competition math, OpenAI found that a reward model trained on step-level human labels was better at picking correct solutions from a set of samples than one trained on final answers. Used that way, it picked a correct solution for 78 percent of a representative subset of the MATH test set [5].
Medicine is a natural fit for per-step reward, because clinical quality is largely a property of the process. Whether the aorta was considered before the kidney stone, whether antibiotics went in within the hour, whether the handoff told the next team what it needed: each can be judged when it happens, against an expert reference for the same patient at the same point.
Per-step reward has its own failure mode. Every intermediate reward is a claim about what good behavior looks like, and if the claim is wrong, the agent optimizes the wrong thing faster. Pay for each indicated test, and you risk a model that orders everything. Pay for naming the right diagnosis, and you risk one that names diagnoses and leaves the treatment to someone else. Pay for activity, and you get activity.
Reinforcement learning theory has a precise version of this warning. Ng, Harada and Russell showed that adding a shaping reward of the form F(s, s′) = γΦ(s′) − Φ(s), the discounted change in a potential function Φ of the state, leaves the optimal policy unchanged, and that shaping rewards not of this form can change it [6]. Their paper recalls a cautionary case: a simulated bicycle rewarded for progress toward a goal learned to ride in tiny circles. Rewards that sound like progress are not automatically safe. Intermediate rewards have to be designed against this theory, not by intuition, so that no loop of actions pays more than finishing the job well.
The same decision can be right at minute ten and wrong at hour three. A grader that ignores time teaches a model that a correct action is equally good whenever it happens. Rewards for clinical decisions have to depend on when they are made. A running environment goes further: it makes the consequences of waiting part of the trajectory itself.
The aim is a grader whose highest score belongs to the best clinician, so that the way to a higher score is better medicine. If the score tracks clinical quality closely, optimizing the score and learning good medicine point in the same direction. That is the environment we are building.