Medical Evals
We benchmarked today's leading AI models on MedXpertQA, a dataset of expert-level medical exam questions spanning diagnosis, treatment, and basic science — including a multimodal subset with real clinical images. Every model ran zero-shot with chain-of-thought reasoning, no fine-tuning or few-shot examples.
Multimodal medical reasoning
2,000 questions with clinical images · 5 answer choices · Higher is better
Text-only medical reasoning
2,450 questions · 10 answer choices · Higher is better
Not every model was run on both subsets. Scores are single runs at each provider's default settings (no reasoning-effort tuning), graded by exact-match on the model's final answer letter. GPT-4o's score is from the original MedXpertQA paper's published leaderboard rather than a run in this project — it's directly comparable because it used the same evaluation harness as every other score here.
Where models struggle
Breaking multimodal accuracy down by organ system and by task type shows the gap between models isn't uniform — some organ systems are hard for every model, while the ranking between models holds fairly steady across task types.
Accuracy by organ system
Multimodal subset · darker = higher accuracy
| Lymphatic | Nervous | Urinary | Muscular | Reproductive | Integumentary | Endocrine | Skeletal | Digestive | Other / NA | Cardiovascular | Respiratory | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | 94 | 92 | 91 | 91 | 89 | 88 | 88 | 87 | 86 | 83 | 81 | 81 |
| Gemini 3.1 Pro Preview | 90 | 84 | 82 | 89 | 85 | 83 | 82 | 82 | 80 | 67 | 77 | 81 |
| gpt-5.6-sol | 80 | 74 | 70 | 87 | 77 | 80 | 77 | 79 | 71 | 67 | 74 | 78 |
| Claude Opus 5 | 89 | 77 | 80 | 80 | 74 | 80 | 78 | 75 | 74 | 61 | 73 | 75 |
| Claude Sonnet 5 | 69 | 57 | 63 | 65 | 61 | 70 | 62 | 56 | 55 | 44 | 57 | 51 |
Accuracy by task type
Multimodal subset · Higher is better
Limitations
- The dataset itself has some construction issues. MedXpertQA expands each question's original answer choices up to 10 using an LLM-assisted process, which occasionally produces two choices describing the same underlying fact, a handful of internally contradictory questions, and — in the multimodal subset — the occasional low-resolution or mismatched image. In manual review, roughly 17–21% of the answers models got “wrong” had a real, defensible argument for the model's answer. True accuracy for every model here is likely a few points higher than the raw numbers shown.
- Single run, default settings. Each score is one pass per model at the provider's default reasoning effort, not an average across repeated trials or a best-of-N result.
- Organ-system and task-type breakdowns use smaller samples. Some organ systems have as few as 18 questions, so those individual cells carry more statistical noise than the headline scores above.
- Grading is exact-match on the model's stated letter. A correctly-reasoned answer expressed in an unexpected format can, in rare cases, be mis-scored by the automated grader.