Ascending Labs Research
Ascending Labs · Research · November 2026

The State of AI Evals, 2026

Testing AI is now a billion-dollar industry, and its benchmarks break faster than anyone can fix them. Our map of who pays, what still measures anything, and who sets the rules, as of 5 November 2026.

top-five lab spend on expert data in 2026 (estimate)
$5–10B
benchmarks rated Flawed by Epoch AI
13
median judge–human agreement on hard tasks
κ 0.22
eval companies bought or in a pending deal
23
jurisdiction with a legal duty to evaluate models (EU)
1
companies in our directory
256
Read the report Explore the directory

The short answer

Ten things to know about AI evaluation in 2026.

  1. 01 Labs pay the most. Frontier labs buy expert data, test environments and pre-release testing. One estimate puts the top five at $5–10B in 2026.[1][1] Lab spending estimate FutureSearch Estimate Mercor alone passed a $2B gross run rate in June 2026.[2][2] Mercor run rate passes $2B TechCrunch Secondary Market →
  2. 02 Larger platforms keep buying the builder tools. Dynatrace paid $915M for Arize in 2026.[3][3] Dynatrace to acquire Arize Dynatrace Primary Our directory counts 23 eval firms that were bought or face a pending deal. Market →
  3. 03 Public benchmarks expire within one or two model generations. ARC-AGI-2 went from 50% to above 90% in seven months, December 2025 to July 2026.[5][5] Benchmark data bundle Epoch AI Primary Benchmarks →
  4. 04 Many famous benchmarks contain broken items. By November 2026, Epoch AI rated 13 benchmarks "Flawed".[6][6] Benchmark Review registry Epoch AI Primary In a sample of Humanity's Last Exam, 46% of items had defects.[32][32] Humanity's Last Exam review Epoch AI Primary Benchmarks →
  5. 05 Grading is now the weak point. Five LLM judges re-graded 4,000 Arena battles. Their median agreement with human votes was a Cohen's kappa of only 0.22.[8][8] Preregistered study of LLM judges on Arena battles arXiv Primary Science →
  6. 06 Models notice the test, and they cheat. Anthropic flagged 36% of Opus 5.5 audit transcripts as test-aware, against 0.4% of real use.[10][10] Claude Opus 5.5 system card Anthropic Vendor Every model that UK AISI checked tried to cheat.[9][9] Cheating behaviour in frontier model evaluations UK AISI Primary Science →
  7. 07 The harness can matter more than the model. The same model scores 62.7% or 99.9% on ARC-AGI-3, depending only on the harness.[7][7] ARC-AGI-3 leaderboard data ARC Prize Primary Inspect AI, the de facto harness, draws 2.9M downloads a month.[11][11] Inspect AI download statistics PyPI Stats Primary Tools →
  8. 08 Only the EU makes model evaluation a legal duty. Article 55 of the AI Act covers the largest general-purpose models.[13][13] EU AI Act, Article 55 Primary The Commission's enforcement powers began on 2 August 2026.[14][14] Commission's enforcement powers for the most advanced models AI Act Service Desk Primary Rules →
  9. 09 China already tests models before launch, at scale. By 31 August 2026, 1,112 generative AI services had passed the regulator's filing.[15][15] CAC generative AI filings Beijing News Secondary Its content test needs at least 90% acceptable answers.[16][16] Translation of GB/T 45654-2025 CSET Secondary China & Asia →
  10. 10 Evaluation lags observability in practice. Among agent teams, 89% trace their systems, but only 52% run offline evals (vendor survey).[18][18] State of Agent Engineering LangChain Vendor Voice agents keep only 30–45% of their text-mode task success.[17][17] τ-voice (arXiv 2603.13686) arXiv Primary Practice →Beyond text →

Read all eight chapters ↓

The map

Four layers build, run, score and govern AI tests, and labs pay for most of it

Each dot is an organisation, placed by what it does and by who mainly pays for its work. Tap or hover a dot for its numbers.

WHO MAINLY PAYS →Frontier labsApp builders and enterprisesGovernments and donorsFrontierlabs$5–10BMercorSurgeScaleHandshakeRL environmentsScale LabsEpoch AIMETRAcademicspromptfooLangfuseGalileoDeepEvalBraintrustMLflowOpenAI EvalsInspectlm-evalHarborIrregularArtificial AnalysisVals AIEpoch AIApolloLab safety frameworksMLCommonsUS CAISSIIntl. networkISO/IEC 42119NIST 800-2ArenaUK AISIMETRArizeLangSmithEU AI Office01Build the tests02Run the tests03Score and supply04Set the rules

Step 1 of 6: A map of 37 grey dots in four horizontal layers: build, run, score and supply, set the rules. Three columns show the main payer: frontier labs, app builders and enterprises, governments and donors.

Four layers make up the evals world. Some groups build the tests, others run them, score them or set the rules. Each column shows who mainly pays for the work.
Labs, non-profits and academics build the tests. Frontier labs test their own models and buy most of the supply. One estimate puts the top five at $5–10B in 2026.[1][1] Lab spending estimate FutureSearch Estimate Donor-funded Epoch AI and METR write many public tests.
Testing tools are cheap, and big platforms keep buying them. UK AISI’s Inspect draws 2.9M downloads a month.[11][11] Inspect AI download statistics PyPI Stats Primary In 2026 Dynatrace paid $915M for Arize.[3][3] Dynatrace to acquire Arize Dynatrace Primary Cisco bought Galileo, ClickHouse bought Langfuse, and OpenAI bought promptfoo.
Lab money flows to data firms and leaderboards. Mercor passed a $2B gross run rate in June 2026.[2][2] Mercor run rate passes $2B TechCrunch Secondary Surge AI earned $1.2B in 2024.[26][26] How this Google alum became a low-key AI billionaire Forbes Australia Secondary Arena reports over $100M of annualised revenue from the labs it ranks.[19][19] Arena Series B Arena Vendor
Governments set the rules, and only the EU makes testing a duty. UK AISI spends £66m a year and tests models before release.[20][20] About the AI Security Institute UK AISI Primary Article 55 of the EU AI Act obliges the largest model providers to evaluate their models.[13][13] EU AI Act, Article 55 Primary Enforcement began on 2 August 2026.[14][14] Commission's enforcement powers for the most advanced models AI Act Service Desk Primary
The same names turn up in several layers. Scale AI builds benchmarks and sells data. UK AISI writes the main harness and also sets method. These overlaps create conflicts of interest.

The main actors in AI evaluation on 5 November 2026, by layer and main payer (our classification). Dot area shows money a year: revenue, run rate, budget or spend, on the date in each tooltip. Hollow dots have no public figure; a pale outer disc marks the top of an estimated range. Ribbon width shows each supplier’s revenue, nearly all of it from labs. £ and € budgets are converted roughly, for size only.

Sources: FutureSearch (estimate), TechCrunch, Forbes Australia, Arena, Dynatrace, PyPI Stats, UK AISI, EU AI Act, TED, METR and company announcements; each tooltip links its source.[1][1] Lab spending estimate FutureSearch Estimate[2][2] Mercor run rate passes $2B TechCrunch Secondary[3][3] Dynatrace to acquire Arize Dynatrace Primary[11][11] Inspect AI download statistics PyPI Stats Primary[13][13] EU AI Act, Article 55 Primary[19][19] Arena Series B Arena Vendor[20][20] About the AI Security Institute UK AISI Primary[21][21] EU AI Office tender award notice 864574-2025 TED Primary[22][22] Funding update METR Primary[26][26] How this Google alum became a low-key AI billionaire Forbes Australia Secondary

Explore all 256 companies →

Who pays whom

$5–10B

What the top five frontier labs spend on expert data and test material in 2026, by one estimate. Labs disclose no figures, so treat this as an estimate.[1][1] Lab spending estimate FutureSearch Estimate

Source: FutureSearch estimate (2026).[1][1] Lab spending estimate FutureSearch Estimate

Lab money is about 100 times what funds independent testing

What the top five labs spend on test material, against one data supplier, one leaderboard and one independent tester

$5–10Best.Top-five labs, expert dataSpend in 2026, estimate>$2BMercorGross run rate,June 2026>$100MArenaAnnualisedrevenue, 2026~$71MMETRNew philanthropiccommitments, 2026

Square area is proportional to US dollars a year Solid = low estimate, dashed outline = high estimate

Figure 2. Labs buy data and environments worth billions. Builder teams buy tooling worth hundreds of millions (our estimate). Philanthropy and governments fund the independent layer, the smallest one. The lab figure is an estimate: labs disclose no spending. Sources: FutureSearch estimate (2026); TechCrunch (Mercor, June 2026); Arena (Series B); METR funding update (August 2026).[1][1] Lab spending estimate FutureSearch Estimate[2][2] Mercor run rate passes $2B TechCrunch Secondary[19][19] Arena Series B Arena Vendor[22][22] Funding update METR Primary

Follow the money →

256 companies sell AI evaluation, and most of them sell security

Each dot is one company or product line in our directory. Scroll to regroup them.

Step 1 of 4: 256 dots in one undivided field.

This is the evals market: 256 companies and product lines that build, run or sell AI evaluation, as of 5 November 2026.
Security and runtime control hold 113 of the 256. AI security and red-teaming is the largest segment, with 74 rows. Builder evals and observability follow with 46.
The market is mostly American. US firms hold 171 rows, or 67%. Israel holds 24, and 16 of them sell AI security. The EU holds only 26.
One in eleven was bought or faces a pending deal. 23 rows were bought or face a pending deal, mostly by security vendors and data platforms. 193 remain independent.

All 256 directory rows by segment, HQ region and status, as of 5 November 2026. Segment placement is our call; we place each company by its main buyer and product.

Source: Ascending Labs company directory (5 Nov 2026); company sites fetched on 5 Nov 2026; deal sources per row.

Explore the directory →

Chapters

Eight chapters. Skim in a minute, or go deep.

Each chapter has three layers: key findings, a ten-minute report with figures, and "Go deeper" sections with the full tables and sources.

  1. Benchmarks Static exams have hit their ceilings, audits show many ceilings were fake, and only long, expert-built tasks still spread the frontier.
  2. Evaluation science LLM judges fall near chance on hard tasks, models notice tests and cheat, and the best teams check the instrument first.
  3. Tools and technology Inspect, lm-evaluation-harness and Harbor became the standard runners; larger firms bought the platforms.
  4. Market and companies Expert data and RL environments earn billions from labs; builder tools stay small and keep getting bought.
  5. Institutions and regulation Government institutes anchor credibility, the AI Act's Article 55 is enforced from August 2026, and testing standards are thin.
  6. China and Asia China sets numeric pass marks and files every service; Japan and Korea write guides; Singapore accredits private testers.
  7. Multimodal, voice and embodied Vision tests saturated, video arenas disagree, deepfake detectors fail in the wild, and voice agents lose most of their text skill.
  8. Enterprise practice and sector rules Software teams gate releases on regressions and calibrated judges; banks validate each use case; insurers face dense testing duties.

Also in this report

How we researched

Every number keeps its date and its source.

We read the primary record wherever we could. Fourteen research tracks ran in two passes up to 5 November 2026. The first seven covered benchmarks, evaluation science, tools, the market, leaderboards and data suppliers, and institutions. The second seven checked flagged facts and added China and Asia, other modalities, enterprise practice and the company directory.

Every number carries a date and a source. The ten research documents cite about 1,400 footnotes from more than 500 websites. We prefer maintainer data files, regulator texts, filings and full system cards. We label each claim as primary, secondary, vendor, single-source or estimate.

This is a snapshot, not a lab test. We did not run the benchmarks or tools ourselves. Paywalled reports and JavaScript-only leaderboards limited what we could confirm. Each chapter lists what we could not verify.

research tracks
14
footnotes
1,423
websites cited
543
snapshot date
5 Nov 2026

Read the methodology →

This is an open snapshot of a fast-moving field. Found an error? Tell us and we will correct it: corrections policy. Cite as: Ascending Labs, State of AI Evals 2026, November 2026.

Sources for this page

Chapter pages carry their own numbered sources. The numbers below apply to this page only.

  1. [1]
    Lab spending estimate, FutureSearch. Top five labs at $5–10B in 2026. Estimate futuresearch.ai
  2. [2]
    Mercor run rate passes $2B, TechCrunch. Gross run rate above $2B, June 2026. Secondary techcrunch.com
  3. [3]
    Dynatrace to acquire Arize, Dynatrace. $915M purchase price. Primary dynatrace.com
  4. [5]
    Benchmark data bundle, Epoch AI. Scores and dates as of 5 Nov 2026. Primary epoch.ai
  5. [6]
    Benchmark Review registry, Epoch AI. 13 rated Flawed, 5 Verified, 8 unrated. Primary epoch.ai
  6. [7]
    ARC-AGI-3 leaderboard data, ARC Prize. Primary arcprize.org
  7. [8]
    Preregistered study of LLM judges on Arena battles, arXiv. Preprint, not yet peer reviewed. Primary arxiv.org
  8. [9]
    Cheating behaviour in frontier model evaluations, UK AISI. Primary aisi.gov.uk
  9. [10]
    Claude Opus 5.5 system card, Anthropic. A lab reports on its own model. Vendor anthropic.com
  10. [11]
    Inspect AI download statistics, PyPI Stats. Counts include CI and mirror traffic. Primary pypistats.org
  11. [13]
    EU AI Act, Article 55. Primary artificialintelligenceact.eu
  12. [14]
    Commission's enforcement powers for the most advanced models, AI Act Service Desk. Primary ai-act-service-desk.ec.europa.eu
  13. [15]
    CAC generative AI filings, Beijing News. 1,112 filed services by 31 Aug 2026. Secondary m.bjnews.com.cn
  14. [16]
    Translation of GB/T 45654-2025, CSET. Translation of the national standard. Secondary cset.georgetown.edu
  15. [17]
    τ-voice (arXiv 2603.13686), arXiv. Preprint. Primary arxiv.org
  16. [18]
    State of Agent Engineering, LangChain. Vendor survey of agent builders. Vendor langchain.com
  17. [19]
    Arena Series B, Arena. $3.1B valuation; revenue above $100M. Vendor arena.ai
  18. [20]
    About the AI Security Institute, UK AISI. £66m a year; 100+ technical staff. Primary aisi.gov.uk
  19. [21]
    EU AI Office tender award notice 864574-2025, TED. €7.37M across six lots. Primary ted.europa.eu
  20. [22]
    Funding update, METR. ~$71M of new commitments. Primary metr.org
  21. [24]
    LangChain is about to become a unicorn, TechCrunch. LangSmith ARR $12–16M, mid-2025. Secondary techcrunch.com
  22. [25]
    LLM Observability, Datadog. No separate fee for evaluations. Vendor datadoghq.com
  23. [26]
    How this Google alum became a low-key AI billionaire, Forbes Australia. Surge AI $1.2B revenue, 2024. Secondary forbes.com.au
  24. [32]
    Humanity's Last Exam review, Epoch AI. 46% of a 48-item sample defective. Primary epoch.ai