How we built State of AI Evals 2026

• Confirmed: the primary source matches the first-pass claim.
• Corrected: the primary source gives a different value. We used the corrected value. Example: Labelbox's October 2026 round set a $1.25B pre-money valuation, not post-money.
• New: a fact that the first pass did not have.
• Still unknown: no reliable source found. These items appear in the list of gaps below.

We read the long primary documents in full. These included three Anthropic system cards (230, 245 and 212 pages), the Gemini 4 Argon card and its 217-page safety report, the GPT-6 Astra, GPT-5.5 and GPT-5.6 hub pages, and the Grok 4.20 card.

The research documents used a few extra tags. On this site they map onto the five labels as follows.

• Aggregator (third-party sites that compile lab numbers, such as BenchLM) → secondary. Aggregator numbers often disagree with lab pages, so treat them as indicative.
• Snippet only / search summary (the page did not load, so the fact comes from a search-engine summary) → secondary, with a note.
• Paywalled (we saw only the public summary of a paid report) → secondary, with a note.
• Vendor-authored (a case study that a supplier wrote about its customer) → vendor.
• Derived / inference (our own arithmetic on sourced inputs) → estimate.

Model names follow the sources. Many 2026 model names, such as GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon or Kimi K3, appear exactly as leaderboards and model cards print them.

The company list began from a third-party market map, then checked. We fetched every company site, re-checked funding and deal facts against dated sources, and added the companies the map lacked. Each row carries one marker.

• Checked (✓): a dated source confirms the funding or deal figure, or the row rests on a cited source. 127 rows.
• Funding not verified (–): we found no dated source for funding, so the directory shows "Not verified" in place of a figure. 111 rows.
• General knowledge (†): HQ or founding year comes from general knowledge, which we did not check in this pass. 18 rows carry this as their weakest mark; 37 rows have at least one such cell.
• For rows from the market map, HQ and founding year come from that map. We did not re-check them.

• Lab launch tables and independent boards often disagree by 5 to 10 points. Harness, effort level and selective framing explain much of the gap, so we prefer maintainer data files.
• Some leaderboard leaders rest on a single source, for example Kimi K3.1 Pro at 67.2% on Terminal-Bench 4.0.
• LiveBrowseComp's claim that memory inflates BrowseComp scores by 25–40 points rests on a single source.
• FutureHouse's 2025 estimate that 29% of HLE biology and chemistry answers were wrong is disputed.
• Several key results in evaluation science are single-source preprints with no replication yet. Examples: LeakScale's contamination effect, the e-process leaderboard test and the judge-version collapse study.
• We could not verify Arena's policy changes after 2025.

• Download counts include CI and mirror traffic, so they are rough signals only. SWE-bench, for example, shows 22.1 million PyPI downloads a month.
• No primary source gives a current count of environments on the Prime Intellect hub. The last public figure is over 400, from October 2025.
• Buyers did not disclose most platform prices. Calcalist's $60–80M figure for Traceloop is unconfirmed.

• Nobody publishes a trustworthy dollar size for evals. Report-mill figures of $2–2.7B for LLM observability look generous against visible vendor revenue.
• Lab spending on expert data is an estimate ($5–10B for the top five in 2026). Claims of $15B are unverified.
• Several revenue figures are paywalled or modelled. Handshake's AI-training revenue conflicts between a paywalled report (~$300M net) and a Sacra estimate (~$450M).
• No public price exists for Galileo, Langfuse, Promptfoo, Dreadnode, TrojAI, Haize Labs, Fiddler, Enkrypt, Virtue AI or Quotient.
• Promptfoo's closing date is not public. No primary source confirms that the Traceloop deal closed.
• Surge AI's 2025 raise talks never had a confirmed close. Its 2025–26 revenue is unknown. Turing and Invisible have no 2026 primary figures.
• Sources disagree on funding for Haize Labs, Holistic AI, Alice, micro1, Kong, LiteLLM, Stacklok and Trustwise.
• Apollo Research and Artificial Analysis do not disclose their seed amounts. We found no budget or 2026 funding figure for Transluce.
• A write-up claims Labelbox profits and fivefold revenue growth in 2026; we could not verify it.
• Helicone's site says it is no longer part of Mintlify. We found no report that explains the split.
• We could not read the full Forrester Wave or the full Gartner Peer Insights list (HTTP 403).

• We found no UK AISI headcount beyond "100+ technical staff", and no dated budget beyond £66m a year.
• We found no staff or budget figures for CAISI/CAISSI, and no official line item.
• We did not read the primary text of the June 2026 US executive order.
• We found no contract durations or outputs for the six EU AI Office lots, and no record of AI Office model evaluations.
• We did not read the Haiku 5.5, Sonnet 5.5, Opus 5 or Fable 5 system cards. We read only the abstract of Meta's preparedness report.
• We found no figures on government spending for third-party evaluation contracts.
• We found no evidence that insurers already require frontier-style evaluations.
• We could not confirm the status of IEEE P3396 and P3419.
• Anthropic's embedded-evaluator role for Faculty and its withheld model access for UK AISI each rest on a single report.

• We have no current top ten for SuperCLUE, OpenCompass or FlagEval. Their sites render only in JavaScript, and our tools could not read them.
• We found no 2026 update to SuperCLUE-Safety or SuperCLUE-Agent beyond their 2023 launches.
• Our "no safety content" finding covers Hugging Face model cards only. We did not check separate technical reports, or Tencent Hunyuan and ByteDance Seed cards.
• We found no primary audit that proves contamination in a named 2026 Chinese model.
• Two CAC filing counts (February and June 2026) come from search summaries only.
• We could not confirm the status of GB 45438-2025, the labelling standard for AI-generated content, from a primary source.
• We found no public list of accredited testing bodies for the CAC filing, and no details of the May 2026 CAC action on agents.
• The scope of China's agent-security standard and Singapore's October 2026 agent extension each rest on a single news report.
• Korea's high-impact start date conflicts between sources. We found no published Korea AISI results. India's AISI status is unclear.
• We did not research Chinese security vendors' red-teaming products, the Gulf in depth, or Brazil.
• We found no evidence of Chinese participation in joint testing by the AISI network.

• We found no official, dated November 2026 leaderboard for Video-MME, LongVideoBench, MVBench or EgoSchema.
• We found no 2026 primary source for POPE scores, or for ImageReward, PickScore, GenEval or T2I-CompBench.
• Multimodal numbers for Claude Fable 5 and GPT-6.x models come only from aggregators. We treat the 94% MMMU-Pro claim for GPT-5.4 Pro as unverified.
• The Open ASR results file requires a login, so its top error rates rest on secondary blogs.
• We could not read the TTS Arena V2 or RoboArena rankings, and we did not verify current LIBERO or CALVIN top scores.
• We read the BEHAVIOR Challenge results and NAVSIM v2 scores only in search snippets. The Veo evaluator's 0.88 correlation rests on one secondary summary.
• We found no funding data for Hamming, Cekura or Bluejay, no 2026 market size for classic model-risk tools, and no enforcement data under SR 26-2.
• We did not search 2026 tests of watermark robustness or voice-clone detection.

• We could not load the OpenAI page on Morgan Stanley or the McKinsey page on ING. Their content comes from search summaries.
• No primary JPMorgan or Morgan Stanley document gives test-set sizes, judge use or thresholds.
• We found no public eval detail for Goldman Sachs, Allianz, Zurich, Lemonade, KPMG, Epic or Ambience, and no primary write-up from Duolingo, Stripe or Ramp.
• Many enterprise cases are vendor-authored: a supplier wrote about its own customer.
• We could not confirm whether US bank regulators issued their planned request for information on AI, or whether the NAIC adopted its AI evaluation tool.
• We did not read the full ECB guide's ML section, the EBA's 2023 report on ML in IRB models, or New York's Circular Letter No. 7 (2024).
• Sources disagree on the PCCP guidance date: December 2024 or August 2025.
• We could not read the NY Comptroller's report on Local Law 144; its figures come from a secondary summary.
• No public survey gives eval spend as a share of AI budget, or norms for eval headcount. We found no systematic dataset of enterprise eval failures.

• We did not re-check HQ or founding year for rows that come from the third-party market map.
• Where funding rested only on that map, the directory shows "Not verified" (111 rows).
• For 37 rows, the HQ or founding year comes from general knowledge.
• We could not verify that Protectt.ai holds a majority of Bosch AIShield.
• Calypso AI is listed in Dublin, so the EU count includes it. F5 now owns it.
• Nine rows from the two directory parts overlapped, and we merged each pair.
• The segment of each row is our call. Promptfoo, QuantPi, Advai, Irregular and Epoch AI are borderline cases.