Stanford HAI released MedHELM, PhysicianBench, and HealthAdminBench, finding that medical AI agents' real-world completion rate is below half
Measuring What Medical AI Can Actually Do
AI BriefingThe Stanford team summarized three years of work building medical AI benchmarks. On PhysicianBench, the best model, GPT-5.5, completed only 46% of tasks in a single attempt; on HealthAdminBench, Claude Opus 4.6 completed 36.3%. MedHELM covers 121 clinical tasks across 5 major categories. Evaluations showed the lowest scores in clinical decision support and administrative workflows; failures mainly resulted from lost context when transferring information across documents and systems. Both benchmarks are open with synthetic data, public code, and leaderboards.
Source: Stanford HAI News · hai.stanford.edu