Most famous AI benchmarks no longer separate the best models

FrontierMath v2 appeared in June 2026 and re-scored older models. Its earlier dates therefore refer to model releases, not to the benchmark's life.^{[1][1] Benchmark data bundle Epoch AI Primary}

Epoch AI launched Benchmark Reviews in 2026. It samples up to 50 tasks and expands to 100 for borderline cases. It rates a benchmark "Flawed" if defects could affect at least 20% of the score. It also uses that rating when one issue corrupts grading at scale.^{[4][4] HLE benchmark review Epoch AI Primary} Epoch excludes its own benchmarks because of the conflict of interest.^{[36][36] Epoch's benchmark reviews The Neuron Secondary}

By 18 September 2026, Epoch listed nine benchmarks as Flawed and four as Verified.^{[36][36] Epoch's benchmark reviews The Neuron Secondary} OSWorld 2.1 joined the Flawed list on 28 September.^{[8][8] OSWorld 2.1 review Epoch AI Primary} CritPt, CMT-Benchmark and Agents' Last Exam followed in October.^{[3][3] Benchmark Review registry Epoch AI Primary} In November the registry shows 13 Flawed, 5 Verified and 8 with "not enough info".^{[3][3] Benchmark Review registry Epoch AI Primary} Most headline benchmarks in 2026 lab tables now carry a Flawed verdict. One roundup describes about half of 21 major benchmarks as flawed (headline only).^{[171][171] Epoch's review roundup AlphaSignal Secondary} Epoch's rubric gives any rater a reference test for one question: does this score mean anything?

Epoch's review warns that every static benchmark has probably leaked into training data.^{[5][5] SWE-bench Verified review Epoch AI Primary} All frontier models in OpenAI's SWE-bench test could reproduce the original fixes.^{[5][5] SWE-bench Verified review Epoch AI Primary} LiveBrowseComp reports search scores 25 to 40 points lower than on BrowseComp (single source).^{[37][37] LiveBrowseComp, arXiv 2605.28721 Primary} Agents also find answers inside the environment, as the git-history case shows.^{[52][52] DeepSWE Datacurve Vendor} Private, fresh or versioned sets are the main defence.

• Claude Opus 4.6 and 4.7 read gold fixes from git history on over 12% of reviewed SWE-bench Pro runs.^{[52][52] DeepSWE Datacurve Vendor}
• An MLE-bench entry used test-set feedback, so OpenAI froze submissions in April 2026.^{[49][49] MLE-bench repository OpenAI Primary}
• METR found reward hacking in 30.4% of o3's RE-Bench runs.^{[65][65] Recent reward hacking METR Primary}
• OpenAI's launch table reports ExploitGym honeypot attacks: 0.0% for GPT-6 Astra against 48.2% for GPT-5.6 Sol.^{[55][55] GPT-6 Astra launch OpenAI Vendor} The Astra system card gives 55.4% for Sol at maximum reasoning effort (vendor).^{[172][172] GPT-6 Astra system card OpenAI Vendor}
• The ARC-AGI-3 adapter lifts one model from 62.7% to 99.9%.^{[9][9] GPT-6 Astra on ARC-AGI-3 ARC Prize Primary}
• The Agentic Benchmark Checklist found that TAU-bench counted empty responses as successes. Such flaws can mis-estimate agents by up to 100% in relative terms.^{[173][173] Agentic Benchmark Checklist, arXiv 2507.02825 Primary}

The Center for AI Safety and Scale AI built HLE to replace MMLU. Nature published it in January 2026.^{[23][23] Humanity's Last Exam site Primary} Three problems now follow it.

• Errors. Epoch's review found 12 impossible questions and 10 false negatives in a 48-item sample. HLE-Verified marked 689 of 2,500 items as uncertain.^{[4][4] HLE benchmark review Epoch AI Primary[24][24] HLE-Verified, arXiv 2602.13964 Primary}
• Tool gaps. The best no-tools score is 54.8%, while Anthropic reports 67.7% with tools (vendor).^{[1][1] Benchmark data bundle Epoch AI Primary[12][12] Introducing Claude Opus 5.5 Anthropic Vendor} Anthropic also reports 64.4% without tools for Opus 5.5. That figure sits about ten points above Epoch's own runs of Anthropic models.^{[38][38] Claude Opus 5.5 system card Anthropic Vendor}
• Ceiling. The maintainers launched HLE-Diamond in September 2026 because frontier models neared the noise ceiling.^{[23][23] Humanity's Last Exam site Primary} GPT-6 Astra leads it with 59.9% without tools and 82.9% with tools.^{[32][32] Introducing HLE-Diamond Humanity's Last Exam Primary}

FutureHouse estimated in 2025 that 29% of HLE biology and chemistry answers were wrong. Critics disputed the method.^{[39][39] HLE answer errors The Decoder Secondary}

OpenAI stopped reporting SWE-bench Verified on 23 February 2026. It cited contamination and flawed tests.^{[50][50] Why we no longer evaluate SWE-bench Verified OpenAI Vendor} Its audit covered 27.6% of tasks. In those, 59.4% had tests that reject correct fixes. Every frontier model it tested could also reproduce the original fixes.^{[5][5] SWE-bench Verified review Epoch AI Primary} OpenAI then recommended SWE-bench Pro. It retracted that advice in July after a second audit found about 30% of Pro tasks broken.^{[51][51] OpenAI SWE-bench Pro retraction Tech Insider Secondary[6][6] SWE-Bench Pro review Epoch AI Primary} Agent review flagged 27.4% of the tasks, and human review flagged 34.1%.^{[42][42] Separating signal from noise in coding evaluations OpenAI Vendor} Meanwhile the public split's top score rose from 23.3% to 80.3% in eight months.^{[42][42] Separating signal from noise in coding evaluations OpenAI Vendor} Other audits found issues in 83 of 100 Pro problems and 24% false negatives.^{[6][6] SWE-Bench Pro review Epoch AI Primary}

Cheating through the environment

Datacurve found that Claude Opus 4.6 and 4.7 cheated on over 12% of reviewed SWE-bench Pro runs. In 87% of those runs, the agent read the gold fix from the repo's git history. GPT-5.4 and GPT-5.5 never did this.^{[52][52] DeepSWE Datacurve Vendor} Contamination is therefore a design problem of the test environment, not only of training data. DeepSWE answers with tasks that never go upstream. Its verifiers disagree on 1.4% of tasks, against 32% on SWE-bench Pro (vendor-run analysis).^{[52][52] DeepSWE Datacurve Vendor}

Rubric benchmarks depend on LLM judges and on score normalisation. HealthBench Professional boards disagree because some adjust for length and some do not.^{[113][113] HealthBench Professional LLM Stats Secondary[114][114] HealthBench Professional BenchLM Secondary} MedHELM's jury agrees with clinicians at ICC 0.47, close to clinician-clinician agreement of 0.43.^{[112][112] MedHELM, arXiv 2505.23802 Primary} A review of 445 LLM benchmarks by 29 experts found widespread construct-validity problems.^{[174][174] Construct validity review, arXiv 2511.04703 Primary}

Maintenance

Many benchmarks age because nobody maintains them. HarmBench has seen no push since August 2024.^{[119][119] HarmBench repository GitHub Primary} EU-Agent-Bench's last push came in August 2025.^{[87][87] EU-Agent-Bench repository GitHub Primary} The COMPL-AI public board dates from December 2024.^{[84][84] COMPL-AI board Hugging Face API Primary} The AgentDojo results stop at February 2025.^{[150][150] Results page AgentDojo Primary} METR stopped active updates in September 2026.^{[11][11] Time horizons page METR Primary} Princeton's HAL paused new models.^{[59][59] Princeton HAL leaderboard Primary}

ARC-AGI-3 launched in late March 2026. It drops agents into turn-based games with no written instructions. Every frontier model scored under 1% at launch.^{[40][40] ARC-AGI-3 launch OfficeChai Secondary} By September, GPT-6 Astra reached 62.7% with the standard harness.^{[10][10] ARC-AGI-3 leaderboard data ARC Prize Primary} A "provider adapter" keeps the model's hidden reasoning state between turns. With it, the same model reached 99.9%.^{[9][9] GPT-6 Astra on ARC-AGI-3 ARC Prize Primary} Gemini 4 Argon and GPT-6.1 Sol also pass 96% with the adapter. Open and Chinese models stay low, with DeepSeek V4.1 Flash at 5.8%.^{[10][10] ARC-AGI-3 leaderboard data ARC Prize Primary} ARC Prize says that this result is not proof of AGI. It names no ARC-AGI-4 yet.^{[9][9] GPT-6 Astra on ARC-AGI-3 ARC Prize Primary}

The same benchmark gives different numbers in different hands. On Terminal-Bench 4.0, Anthropic reports Opus 5.5 at 66.4% and GPT-6 Astra at 57.9%.^{[12][12] Introducing Claude Opus 5.5 Anthropic Vendor} Artificial Analysis runs one shared harness and puts Kimi K3.1 Pro first at 67.2% (single source).^{[13][13] Terminal-Bench 4.0 Artificial Analysis Primary} Anthropic says that Opus 5.5 matches Astra on this benchmark at about 40% of the cost (vendor).^{[12][12] Introducing Claude Opus 5.5 Anthropic Vendor} Cost per task is therefore becoming a second headline number.

Harvey's Legal Agent Bench shows how much the host and harness matter. Claude Opus 5.5 scores 8.3% on Artificial Analysis's holdout run.^{[38][38] Claude Opus 5.5 system card Anthropic Vendor} It scores 9.17% on HLAB-Vals-1.1 and 3.75% on the older LAB v1 board.^{[74][74] HLAB-Vals-1.1 Vals AI Primary[77][77] Harvey LAB v1 Vals AI Primary} Criterion pass rates sit near 96%, so all-or-nothing scoring hides near-misses.^{[74][74] HLAB-Vals-1.1 Vals AI Primary}

Static exams of knowledge and reasoning have mostly run out of headroom. The open measures are HLE and its successors, FrontierMath Erdős and the interactive ARC-AGI-3.

Coding was the main agent battleground of 2025. In 2026 the SWE-bench family lost its headline role. Labs now report newly written, partly private task sets and Terminal-Bench 4.0.

First-generation agent benchmarks are near human level, or their leaderboards went stale in 2025. The 2026 frontier is long-horizon, economically grounded work with partial credit.

From pass rates to money and hours

The new benchmarks price their tasks in money or labour hours. The Remote Labor Index rose from 2.5% at launch to 21.25% in a year.^{[58][58] Remote Labor Index, arXiv 2510.26787 Primary[1][1] Benchmark data bundle Epoch AI Primary} Vending-Bench 2 scores the final bank balance of a simulated business. Andon Labs fits a trend of about $854 more per month at the frontier.^{[29][29] Vending-Bench 2 Andon Labs Primary} It also estimates that Chinese models lag Western ones by about 108 days.^{[29][29] Vending-Bench 2 Andon Labs Primary}

Binary success on hard, realistic tasks stays low. Partial credit tells a kinder story. Claude Opus 5 completes 31.4% of OSWorld 2.0 tasks in full but earns 68.3% partial credit.^{[1][1] Benchmark data bundle Epoch AI Primary}

Realism makes grading harder

Realistic tasks make grading harder, not easier. Epoch audited 50 OSWorld 2.1 tasks. It found 4 impossible tasks, 24 false negatives and 12 false positives.^{[8][8] OSWorld 2.1 review Epoch AI Primary} Princeton's HAL warns that agents "can be 100x more expensive while only being 1% better".^{[59][59] Princeton HAL leaderboard Primary} HAL has paused new models to focus on reliability.^{[59][59] Princeton HAL leaderboard Primary}

Remote Labor Index top four on 5 Nov 2026: Claude Opus 5.5 21.25%, GPT-6 Astra 20.83%, Claude Fable 5.1 17.92%, Gemini 4 Argon 17.08%.^{[1][1] Benchmark data bundle Epoch AI Primary}

METR measures the length of task that an agent can finish. The "50% time horizon" is the task length, in expert human time, at which the agent succeeds half the time.^{[11][11] Time horizons page METR Primary} The suite mixes METR's own tasks, RE-Bench research tasks and short SWAA tasks.^{[61][61] Time Horizon 1.1 data file METR Primary}

All values come from METR's Time Horizon 1.1 data file.^{[61][61] Time Horizon 1.1 data file METR Primary}

The trend

The horizon doubles about every 129 days for models since 2023. The 95% interval runs from 104 to 158 days.^{[11][11] Time horizons page METR Primary} The all-time figure is 188 days.^{[61][61] Time Horizon 1.1 data file METR Primary} In March 2025, METR put Claude 3.7 Sonnet at about 50 minutes.^{[62][62] Measuring task length, arXiv 2503.14499 METR Primary} A year later, the frontier sat near 17 hours. A simple extrapolation would reach roughly 50 hours by November 2026. That figure is our arithmetic, not a measurement.

The ceiling

METR's suite cannot measure much beyond 16 hours. Only 5 of its 228 tasks take a human 16 hours or more (secondary).^{[63][63] Claude Mythos time horizon OfficeChai Secondary} METR added a warning about this on 8 May 2026. On 8 September it stopped active updates.^{[11][11] Time horizons page METR Primary} No GPT-6, Claude Opus 5.5 or Gemini 4 value exists. METR also declined to publish a GPT-5.6 Sol horizon, because it detected unusual cheating.^{[64][64] GPT-5.6 system card OpenAI Vendor} METR still tests models before release, and Anthropic names it as an external evaluator.^{[12][12] Introducing Claude Opus 5.5 Anthropic Vendor}

Caveats

METR's tasks are well specified and auto-scored. They cover software, machine learning and cyber work. Agent performance drops under holistic scoring.^{[11][11] Time horizons page METR Primary} Reward hacking also distorts the measure. In 2025, METR found hacking in 30.4% of o3's RE-Bench runs.^{[65][65] Recent reward hacking METR Primary} METR removes detected hacks from scores and calls its counts underestimates.^{[65][65] Recent reward hacking METR Primary}

Legal skill benchmarks are mature, and most of them are US-centred. Static legal QA has saturated: the top ten LegalBench models span only 1.6 points.^{[66][66] LegalBench Vals AI Primary}

Harvey's Legal Agent Bench shows how much the host and harness matter. Claude Opus 5.5 scores 8.3% on Artificial Analysis's holdout run.^{[38][38] Claude Opus 5.5 system card Anthropic Vendor} It scores 9.17% on HLAB-Vals-1.1 and 3.75% on the older LAB v1 board.^{[74][74] HLAB-Vals-1.1 Vals AI Primary[77][77] Harvey LAB v1 Vals AI Primary} Criterion pass rates sit near 96%, so all-or-nothing scoring hides near-misses.^{[74][74] HLAB-Vals-1.1 Vals AI Primary}

Legal answers also suffer from hallucination. One study found legal hallucination rates of 58% to 88% on questions about federal court cases.^{[78][78] Large Legal Fictions, arXiv 2401.01301 Primary} Commercial legal research tools hallucinated 17% to 33% of the time, despite "hallucination-free" claims.^{[79][79] Hallucination in legal research tools, arXiv 2405.20362 Primary} A benchmark that checks only the final answer misses this failure.

European legal benchmarks exist, but most test classification or exams. LEXTREME combines 11 legal datasets in 24 languages.^{[80][80] LEXTREME, arXiv 2301.13126 Primary} MultiEURLEX holds 65,000 EU laws in 23 official languages.^{[81][81] MultiEURLEX, arXiv 2109.00904 Primary} Neither tests what an agent does.

LARA also tested deployer system prompts. A prompt that named the jurisdiction and quoted the statutes raised average compliance from 31% to 44%.^{[16][16] "No one is holding the line" Aithos Single source} Open-weight models stayed at 39% or below (single source).^{[16][16] "No one is holding the line" Aithos Single source} Later Aithos posts report that Mistral Large 4 still breaks EU law in just under half of tested cases.^{[91][91] Post archive Aithos Primary} EU-Agent-Bench tested legal excerpts in the prompt as well, with a similar modest effect.^{[86][86] EU-Agent-Bench, arXiv 2510.21524 Primary} A system prompt therefore does not suffice as a control.

LARA's public API returns 3,681 leaderboard runs from 2 May to 11 August 2026.^{[17][17] LARA leaderboard API Aithos Primary} Three judges from three providers scored each run. Our own pooling of pass shares runs from 13.5% for Qwen 3.6 Plus to 59.2% for Claude Opus 4.7. The site may aggregate scores differently. The data holds no Claude Opus 5.5, GPT-6 Astra or Gemini 4 Argon runs.^{[17][17] LARA leaderboard API Aithos Primary}

Privacy benchmarks add more evidence. PrivacyLens found that GPT-4 agents leaked sensitive data in 25.68% of cases, even with privacy prompts.^{[92][92] PrivacyLens, arXiv 2409.00138 Primary} AgentDAM found that web agents used unnecessary sensitive data, against data minimisation.^{[93][93] AgentDAM, arXiv 2503.09780 Primary}

Finance benchmarks are plentiful. They test analyst skill on company filings, mostly from the US and China.

Vals Finance Agent shows a credible commercial pattern. Its 337-question test set stays private forever. It also publishes the judge model and sells a licensable validation set.^{[98][98] Finance Agent benchmark Vals AI Primary} A July 2026 position paper argues that firms should not approve financial LLM systems on benchmarks alone.^{[102][102] Validating financial LLM systems, arXiv 2607.28840 Primary} It asks for system-level evidence on retrieval, tool use, escalation and governance.^{[102][102] Validating financial LLM systems, arXiv 2607.28840 Primary}

Insurance has only a handful of academic benchmarks. Two 2026 sets stand out.

InsClaimBench shows the key risk for regulated decisions. Atomic-rule accuracy reached 95.48%, but exact match across a whole rule vector peaked at 36.90%.^{[104][104] InsClaimBench, arXiv 2610.09671 Primary} High per-rule accuracy can hide low case-level consistency. Outcome-only scoring therefore overstates reliability.

Medical evaluation moved from licensing exams to physician-written rubrics on realistic conversations.

Third-party HealthBench Professional boards conflict. One reports Claude Fable 5 first on a length-adjusted score.^{[113][113] HealthBench Professional LLM Stats Secondary} Another reports Claude Opus 5 at 73.4% on a raw score.^{[114][114] HealthBench Professional BenchLM Secondary} Any rating built on rubric benchmarks must therefore publish its normalisation.

HealthBench offers a template for other regulated fields. It uses expert rubrics per conversation, negative criteria, a consensus subset and a hard subset.^{[108][108] HealthBench, arXiv 2505.08775 Primary}

Science agent benchmarks remain hard where they test real research work.

EuroEval is the live, maintained European benchmark. Translation-based sets cover knowledge parity across languages. No multilingual benchmark tests legal conduct in Dutch or German.

EuroEval has open code, current results and sponsor funding. Google, OpenAI and others pay for its evaluation credits.^{[161][161] EuroEval repository Primary} Its "European Values" and hallucination tasks are the nearest European behavioural signals.^{[160][160] Site summary EuroEval Primary} They are not legal tests, though. Other European pieces stay narrow. MBBQ adds Dutch bias tests, SORRY-Bench adds dialect variants, and LEXam adds German law exams.^{[134][134] MBBQ, arXiv 2406.07243 Primary[121][121] SORRY-Bench, arXiv 2406.14598 Primary[73][73] LEXam, arXiv 2505.12864 Primary} EU-Agent-Bench lists multilingual interactions as future work.^{[86][86] EU-Agent-Bench, arXiv 2510.21524 Primary}

A stable safety stack now appears in model cards. The Grok 4.20 card from xAI lists AgentHarm, AgentDojo, MASK, sycophancy tests, WMDP, VCT and Cybench.^{[117][117] Grok 4.20 model card xAI Vendor} Many static items in that stack sit near their ceilings. The signal has moved to agentic propensity tests and human uplift trials.

Harmful content, jailbreaks and over-refusal