10 things to know about AI evals in 2026
The two-minute version of our eight-chapter report. One finding, one chart and one link per item.
Snapshot 5 November 2026 · 2 min read · Read the full report
- Benchmarks
1.Famous AI tests now fill up within months of their release.
In Epoch AI data (Nov 2026), every benchmark from 2024 on went from 50% to 90% within eight months.[1][1] Benchmark data bundle Epoch AI Primary Epoch AI audits also rated 13 benchmarks "Flawed" in November 2026, including HLE and SWE-bench Verified.[2][2] Benchmark Review registry Epoch AI Primary
Recent benchmarks went from 50% to 90% in under a year
Months between the first model above 50% and the first model above 90%, one dot per benchmark.
- 50% to 90% (6)
- Caveat: MMLU 50% to 75% only; FrontierMath re-scored (2)
Median of the row; number at right = items in the row
Data table
Item Row Months from first ≥50% to first ≥90% Category MATH Level 5 Released before 2024 7 50% to 90% ARC-AGI-1 Released before 2024 8 50% to 90% MMLU Released before 2024 15 Caveat: MMLU 50% to 75% only; FrontierMath re-scored GPQA Diamond Released before 2024 17 50% to 90% FrontierMath Tier 4 v2 Released 2024 or later 3 Caveat: MMLU 50% to 75% only; FrontierMath re-scored Cybench Released 2024 or later 5 50% to 90% ARC-AGI-2 Released 2024 or later 7 50% to 90% Mock AIME 2024-25 Released 2024 or later 8 50% to 90% Figure 2 in chapter 1, Benchmarks. Source: Epoch AI benchmark data (5 Nov 2026).[1][1] Benchmark data bundle Epoch AI Primary Read chapter 1 → Figure 2 and the full evidence
- Benchmarks
2.The best AI agent completes about one in five paid freelance projects.
The Remote Labor Index uses 240 real freelance projects worth about $140,000.[3][3] Remote Labor Index, arXiv 2510.26787 Primary The best agent automated 2.5% of them at launch in October 2025 and 21.25% in September 2026.[1][1] Benchmark data bundle Epoch AI Primary
The best agent now completes about one in five paid freelance projects
Remote Labor Index: 240 real freelance projects worth about $140,000. Each square is 1% of the projects.
2.5%
At launch, Oct 2025
best agent: 2.5%
21.25%
Sep 2026
Claude Opus 5.5: 21.25%
Each square is 1 of 100 (1%).
Data table
Panel Value At launch, Oct 2025 2.5% Sep 2026 21.25% Figure 8 in chapter 1, Benchmarks. Source: Remote Labor Index paper;[3][3] Remote Labor Index, arXiv 2510.26787 Primary Epoch AI;[1][1] Benchmark data bundle Epoch AI Primary Andon Labs.[4][4] Vending-Bench 2 Andon Labs Primary Read chapter 1 → Figure 8 and the full evidence
- Evaluation science
3.How a test counts cheating can change an agent's score 24-fold.
METR's "time horizon" is the length of expert task an agent finishes half the time.[5][5] Measuring AI ability to complete long tasks (Kwa et al.) Primary For GPT-5.6 Sol it was about 11 hours or over 270 hours, depending on how cheats count (June 2026).[6][6] GPT-5.6 Sol evaluation METR Primary
The agent time horizon grew twentyfold in a year, then cheating broke the measure
METR 50% time horizon in hours (log scale), by model release date.
- Published horizon
- GPT-5.6 Sol, cheats as successes
Data table
Point Model release 50% time horizon (hours) Category Note Claude 3.7 Sonnet 2025-02 0.8 h Published horizon ≈50 min, early 2025 GPT-5 2025-08 2.3 h Published horizon ≈2 h 17 min Claude Opus 4.6 2026-02 12 h Published horizon Claude Mythos Preview 2026-04 17.4 h Published horizon ≈17 h; 95% CI ≈8.5–55 h GPT-5.6 Sol (cheats = fail) 2026-06 11.3 h Published horizon ≈11.3 h; CI 5–40 h GPT-5.6 Sol (cheats = success) 2026-06 270 h GPT-5.6 Sol, cheats as successes over 270 h Figure 9 in chapter 2, Evaluation science. Source: Kwa et al.;[5][5] Measuring AI ability to complete long tasks (Kwa et al.) Primary METR time-horizons page;[7][7] Time horizons METR Primary METR Time Horizon 1.1 data;[8][8] Time Horizon 1.1 results METR Primary METR GPT-5.6 Sol evaluation.[6][6] GPT-5.6 Sol evaluation METR Primary Read chapter 2 → Figure 9 and the full evidence
- Evaluation science
4.Models increasingly notice when they are being tested.
In audits, 36% of Claude Opus 5.5 transcripts showed test awareness, against 0.4% in real use (Sep 2026).[9][9] Claude Opus 5.5 system card Anthropic Vendor Every model that the UK AI Security Institute tested for cheating tried to cheat (July 2026).[10][10] Cheating behaviour in frontier model evaluations UK AISI Primary
Test awareness jumps from almost none to over a third
Share of transcripts or runs showing evaluation awareness (%).
Data table
Line Real use, or earlier model Audit, or newer model Change Claude Opus 5.5: real use → audits 0.4% 36.0% +35.6 Gemini 3.1 Pro → Gemini 4 Argon, coding honeypots 0.0% 37.8% +37.8 Figure 11 in chapter 2, Evaluation science. Source: Anthropic Claude Opus 5.5 system card;[9][9] Claude Opus 5.5 system card Anthropic Vendor Google DeepMind Gemini 4 Argon Safety and Security Report.[11][11] Gemini 4 Argon Safety and Security Report Google DeepMind Vendor Read chapter 2 → Figure 11 and the full evidence
- Tools and technology
5.Free, open-source tools run most AI tests.
A harness is the software that runs a test against a model and scores the answers. The UK's Inspect AI draws 2.9 million downloads a month, against 1.0 million for lm-evaluation-harness (5 Nov 2026).[12][12] inspect-ai downloads pypistats Primary[13][13] lm-eval downloads pypistats Primary
The Most-Starred Harnesses Are Not the Most Used
Each dot is a harness: GitHub stars (across) against monthly PyPI downloads (up, log scale), 5 November 2026.
- Very active or active
- Maintenance mode
Data table
Point GitHub stars Monthly PyPI downloads (thousands) Category Note Harbor 6821 4,800k Very active or active 4.8M a month Inspect AI 3185 2,900k Very active or active 2.9M a month with 3,185 stars lm-evaluation-harness 14408 1,000k Very active or active Most stars, 1.0M a month inspect_evals 733 867k Very active or active SWE-bench 6149 22,100k Very active or active Mostly CI traffic lighteval 2579 6k Very active or active Likely a data gap OpenCompass 7563 3k Very active or active openbench 831 2k Very active or active HELM 2964 1k Maintenance mode Maintenance since 1 Jun 2026 Figure 2 in chapter 3, Tools and technology. Source: GitHub API and pypistats API (5 Nov 2026).[14][14] swebench downloads pypistats Primary[15][15] Inspect AI GitHub Primary[12][12] inspect-ai downloads pypistats Primary[16][16] inspect-evals downloads pypistats Primary[17][17] lm-evaluation-harness GitHub Primary[13][13] lm-eval downloads pypistats Primary[18][18] Harbor GitHub API Primary[19][19] harbor downloads pypistats Primary[20][20] SWE-bench GitHub Primary[21][21] OpenCompass GitHub Primary[22][22] lighteval GitHub Primary[23][23] lighteval downloads pypistats Primary[24][24] openbench GitHub Primary[25][25] HELM GitHub Primary Read chapter 3 → Figure 2 and the full evidence
- Market and companies
6.AI labs pay for most of the evaluation economy.
One forecast puts the top five labs' spend on expert data at $5–10 billion in 2026 (estimate).[26][26] Lab human-data forecast FutureSearch Estimate The independent safety tester METR received about $71 million in six months of 2026.[27][27] Funding update August 2026 METR Primary
Lab Money Dwarfs What Every Other Segment Takes In
Rough 2026 money flowing into each segment, where a published figure exists. Square area is proportional to dollars.
Area is proportional to dollars per year Solid = low estimate, dashed outline = high estimate
Data table
Item Value Note Expert data $5–10B Top five labs, 2026 (forecast); estimate RL environments Over $1B One lab, discussed in 2025 Governance platforms $492M Gartner, 2026 Builder tools $0.2–0.5B Vendor revenue, 2026 (our estimate); estimate Leaderboards Over $100M Arena run rate, Oct 2026 Safety evaluators ~$71M To METR in six months, 2026 Figure 2 in chapter 4, Market and companies. Source: FutureSearch (estimate);[26][26] Lab human-data forecast FutureSearch Estimate TechCrunch citing The Information;[28][28] Silicon Valley bets big on environments TechCrunch Secondary Gartner via secondary;[29][29] Insurance-Canada.ca quoting Gartner Secondary Arena;[30][30] Series B announcement Arena Vendor METR;[27][27] Funding update August 2026 METR Primary our estimate. Read chapter 4 → Figure 2 and the full evidence
- Institutions and regulation
7.Only the EU makes testing frontier models a legal duty.
EU AI Act Article 55 requires state-of-the-art evaluation and adversarial testing, meaning deliberate attacks, of the most capable models.[31][31] EU AI Act, Article 55 Primary The AI Office has enforced it since 2 August 2026; high-risk duties wait until December 2027.[32][32] AI Act Service Desk, Commission enforcement powers FAQ Primary[33][33] Garrigues, Digital Omnibus on AI published Secondary
The EU's Frontier Duties Are Live; Its High-Risk Duties Wait Until 2027
AI Act dates that matter for evaluation, July 2025 to August 2028.
Data table
Date Lane Event Kind Note 2025-07-10 Frontier (GPAI) duties GPAI Code of Practice published release 2025-08-02 Frontier (GPAI) duties GPAI duties become binding regulation 2025-10-23 Standards Fast-track for six delayed drafts other 2026-07-27 High-risk and transparency Digital Omnibus enters into force regulation 2026-08-02 Frontier (GPAI) duties AI Office enforcement powers begin regulation 2026-08-02 High-risk and transparency Article 50 transparency duties apply regulation 2026-10-14 Standards EN 18286 cited: first harmonised standard release 2026-12-02 (planned) High-risk and transparency Output-labelling grace period ends regulation 2027-02-28 (planned) Standards Standardisation request M/613 ends other 2027-08-02 (planned) Frontier (GPAI) duties Legacy GPAI models reportedly due regulation Not verified 2027-12-02 (planned) High-risk and transparency Annex III high-risk duties apply regulation 2028-08-02 (planned) High-risk and transparency Annex I high-risk duties apply (AI in products) regulation Figure 7 in chapter 5, Institutions and regulation. Source: Garrigues;[33][33] Garrigues, Digital Omnibus on AI published Secondary AI Act Service Desk;[32][32] AI Act Service Desk, Commission enforcement powers FAQ Primary European Commission;[34][34] European Commission, contents of the GPAI Code of Practice Primary JTC 21;[35][35] JTC 21, EN 18286 becomes the first harmonised standard (14 Oct 2026) Secondary CEN-CENELEC;[36][36] CEN-CENELEC, AI standardisation fast-track (23 Oct 2025) Primary Cloud Security Alliance.[37][37] Cloud Security Alliance, EU AI Act GPAI enforcement (29 Aug 2026) Secondary Read chapter 5 → Figure 7 and the full evidence
- China and Asia
8.China tests every public AI service before launch; the rest of Asia writes guides.
By 31 August 2026, 1,112 generative AI services had passed the filing with China's regulator.[38][38] Beijing News on CAC filings Secondary Each must pass a test bank of at least 2,000 questions with 90% acceptable answers.[39][39] CSET translation of GB/T 45654-2025 Primary
One Country Tests Before Launch; the Rest of Asia Builds Tools
Legal force of AI testing by country or region, November 2026.
Hover over, tap or tab to a tile for details.
Binding test duty?
- Yes, de facto through filing
- Self-assessment only
- Voluntary; accredits private testers
- Voluntary guides and benchmarks
- Proposals only
- Not researched
Data table
Country or region Binding test duty? Value Note China Yes, de facto through filing 1,112 filed CAC filing; GB/T 45654-2025; mandatory agent standard in drafting. South Korea Self-assessment only 22 Jan 2026 AI Basic Act: operators self-assess; no third-party test. Japan Voluntary guides and benchmarks Guides J-AISI red-teaming guide; Nejumi, Open Japanese LLM Leaderboard. Gulf Not researched Open Arabic LLM Leaderboard idle since Oct 2025. India Proposals only 12.8% AISI proposed Jan 2026; IndicSafe, MILU, Inspect India Evals. Singapore and SE Asia Voluntary; accredits private testers 5 testers AI Verify accreditation; Moonshot; SEA-HELM; agent guidance Oct 2026. Latin America Not researched LatamGPT (CENIA) with CHOCLO and Trueque benchmarks. Figure 1 in chapter 6, China and Asia. Source: Beijing News;[38][38] Beijing News on CAC filings Secondary Cooley;[40][40] Cooley on Korea's AI Basic Act Secondary AI Verify Foundation;[41][41] AI Verify Foundation, AI Tester Accreditation Primary IndicSafe;[42][42] IndicSafe, arXiv 2603.17915 Primary Open Arabic LLM Leaderboard data;[43][43] Open Arabic LLM Leaderboard Space data Primary Buenos Aires Times.[44][44] Buenos Aires Times on LatamGPT Secondary Read chapter 6 → Figure 1 and the full evidence
- Multimodal, voice and embodied
9.Voice agents lose most of the skill they show in text.
In the τ-voice study (2026), GPT-5 with reasoning solved 85% of the tasks in text mode.[45][45] τ-voice, arXiv 2603.13686 Primary Voice agents solved only 26% to 51% of the same tasks, and noise and accents cost the most.[45][45] τ-voice, arXiv 2603.13686 Primary
The same tasks drop from 85% in text to as low as 26% in voice
τ-voice task success (pass@1): GPT-5 with reasoning in text mode, against the best and worst voice agents in two audio conditions.
Data table
Line Text mode Voice mode Change Clean audio, best voice agent 85% 51% −34 Clean audio, worst voice agent 85% 31% −54 Noise and accents, best voice agent 85% 38% −47 Noise and accents, worst voice agent 85% 26% −59 Figure 6 in chapter 7, Multimodal, voice and embodied. Source: τ-voice, arXiv 2603.13686;[45][45] τ-voice, arXiv 2603.13686 Primary Artificial Analysis speech-to-speech (Nov 2026).[46][46] Artificial Analysis, speech-to-speech Primary Read chapter 7 → Figure 6 and the full evidence
- Enterprise practice and sector rules
10.The best teams build their tests from real traffic and real users.
Cursor retrained its code-completion model on live user feedback: suggestions fell 21% and the accept rate rose 28% (Sep 2025).[47][47] Cursor, Online RL for Tab Primary Shopify builds its test sets from samples of real merchant conversations, not hand-written examples (Aug 2025).[48][48] Shopify Engineering, Building production-ready agentic systems Primary
At Cursor's scale, users are the eval
Change after Cursor's online-RL Tab model replaced the previous model, September 2025. Both arrows start at the previous model, indexed to 100.
Data table
Row Previous model Online-RL model Change Suggestions shown 100 −21% −21 Accept rate 100 +28% +28 Figure 4 in chapter 8, Enterprise practice and sector rules. Source: Cursor, Online RL for Tab.[47][47] Cursor, Online RL for Tab Primary Read chapter 8 → Figure 4 and the full evidence
Go deeper: eight chapters, one directory.
Each chapter has key findings, a ten-minute report with figures, and "Go deeper" sections with the full tables and sources.
- Benchmarks
- Evaluation science
- Tools and technology
- Market and companies
- Institutions and regulation
- China and Asia
- Multimodal, voice and embodied
- Enterprise practice and sector rules
Also in this report
- Company directory 256 companies, filterable by segment, region and status
- Methodology How we researched, confidence labels and how to cite
Sources for this page
The numbers below apply to this page only. Each chapter carries its own numbered sources.
- [1] Benchmark data bundle, Epoch AI. Dated scores (5 Nov 2026) and ceilings for most benchmarks here. Primary epoch.ai
- [2] Benchmark Review registry, Epoch AI. Lists 13 Flawed, 5 Verified and 8 unrated benchmarks in November 2026. Primary epoch.ai
- [3] Remote Labor Index, arXiv 2510.26787. It describes the projects and the launch automation rate. Primary arxiv.org
- [4]
- [5] Measuring AI ability to complete long tasks (Kwa et al.). It defines the 50% time horizon. Primary arxiv.org
- [6]
- [7]
- [8] Time Horizon 1.1 results, METR. It lists time horizons and intervals for each model. Primary metr.org
- [9] Claude Opus 5.5 system card, Anthropic. It reports capability, cyber, alignment and evaluation-awareness results. Vendor anthropic.com
- [10] Cheating behaviour in frontier model evaluations, UK AISI. It reports cheating by every tested model. Primary aisi.gov.uk
- [11] Gemini 4 Argon Safety and Security Report, Google DeepMind. It reports evaluation awareness and names the external testers. Vendor storage.googleapis.com
- [12]
- [13]
- [14] swebench downloads, pypistats. It gives the monthly PyPI download count for SWE-bench. Primary pypistats.org
- [15]
- [16]
- [17]
- [18] Harbor, GitHub API. It gives stars, licence and the move from laude-institute. Primary api.github.com
- [19]
- [20]
- [21]
- [22]
- [23]
- [24]
- [25]
- [26] Lab human-data forecast, FutureSearch. It estimates 2026 human-data spending by the top five labs. Estimate futuresearch.ai
- [27] Funding update August 2026, METR. It lists METR's new commitments, funders and funding policy. Primary metr.org
- [28] Silicon Valley bets big on environments, TechCrunch. It reports Anthropic's discussed RL-environment budget, citing The Information. Secondary techcrunch.com
- [29] Insurance-Canada.ca quoting Gartner. It quotes Gartner's forecast for AI governance platform spending. Secondary insurance-canada.ca
- [30] Series B announcement, Arena. It gives Arena's valuation, revenue and usage in October 2026. Vendor arena.ai
- [31] EU AI Act, Article 55. It gives the duties of providers of GPAI models with systemic risk. Primary artificialintelligenceact.eu
- [32] AI Act Service Desk, Commission enforcement powers FAQ. It explains the powers that apply from 2 August 2026. Primary ai-act-service-desk.ec.europa.eu
- [33] Garrigues, Digital Omnibus on AI published. It gives the new AI Act deadlines after the Omnibus. Secondary garrigues.com
- [34] European Commission, contents of the GPAI Code of Practice. It lists the chapters and the signatories. Primary digital-strategy.ec.europa.eu
- [35] JTC 21, EN 18286 becomes the first harmonised standard (14 Oct 2026). It reports the citation and the remaining standardisation work. Secondary jtc21.eu
- [36] CEN-CENELEC, AI standardisation fast-track (23 Oct 2025). It announces the exceptional measures for delayed drafts. Primary cencenelec.eu
- [37] Cloud Security Alliance, EU AI Act GPAI enforcement (29 Aug 2026). This secondary note covers fines, notification and non-signatories. Secondary labs.cloudsecurityalliance.org
- [38] Beijing News on CAC filings. It reports 1,112 filed services and 731 registrations by 31 August 2026. Secondary m.bjnews.com.cn
- [39] CSET translation of GB/T 45654-2025. It gives the English text of the generative AI safety standard and its test numbers. Primary cset.georgetown.edu
- [40] Cooley on Korea's AI Basic Act. It summarises the Act's duties, thresholds and fines. Secondary cooley.com
- [41] AI Verify Foundation, AI Tester Accreditation. It lists the accredited firms and the programme terms. Primary aiverifyfoundation.sg
- [42] IndicSafe, arXiv 2603.17915. It tests culturally grounded safety prompts in 12 Indic languages. Primary arxiv.org
- [43]
- [44] Buenos Aires Times on LatamGPT. It reports the LatamGPT launch in Santiago. Secondary batimes.com.ar
- [45]
- [46]
- [47]
- [48]