Ascending Labs Research
← All chapters
Practice

Mature teams evaluate in one loop, but sector rules barely reach generative AI

The best companies turn every production failure into a test and calibrate their judges against experts, while most sector regulators still have no rules for testing generative AI agents.

of agent teams run offline evals
52%
of production agent teams do not evaluate at all
23%
Shopify judge agreement after calibration
κ 0.02 → 0.61
real chats in ING's daily regression set
500
sector regimes built for generative AI agents
0 of 11

5 November 2026 · 12 min read · Dotted terms are in the glossary

On this page
  1. Key findings
  2. The loop
  3. Software firms
  4. Banks and insurers
  5. Public sector and health
  6. Failures
  7. What surveys say
  8. Sector rules
  9. Playbook
  10. Sources

Key findings

  • Mature software teams run the same loop. They combine a regression suite in CI, calibrated LLM judges, internal dogfood and online business metrics (GitHub, Jan 2025; Shopify, Aug 2025).[1][1] GitHub, How we evaluate models for GitHub Copilot Primary[2][2] Shopify Engineering, Building production-ready agentic systems Primary
  • The usual release gate is "no regression against the production baseline". Few teams publish an absolute pass score.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary[3][3] Claude customer story, Intercom Vendor
  • The best golden sets come from production traffic and failures. Shopify replaced hand-written sets with samples of real merchant conversations (Aug 2025).[2][2] Shopify Engineering, Building production-ready agentic systems Primary
  • Teams now check their LLM judges against humans. Shopify raised judge agreement from a kappa of 0.02 to 0.61, close to the human level of 0.69 (Aug 2025).[2][2] Shopify Engineering, Building production-ready agentic systems Primary
  • Banks gate each use case and validate independently, but publish few numbers. ING gives the most concrete figure: daily regression tests on 500 real customer chats, plus A/B tests (snippet only).[4][4] OpenAI customer story, Morgan Stanley Vendor[5][5] The Muse, JPMorgan model risk job listing Single source[6][6] AI Expert Network, AI at ING Secondary
  • Evaluation lags observability. In a vendor survey of 1,340 respondents (Nov–Dec 2025), 89% trace their agents but only 52% run offline evals and 37% run online evals.[9][9] LangChain, State of Agent Engineering Vendor
  • Public failures share one cause: the builder controlled the metric. Klarna, Deloitte Australia and NYC's hiring-audit law all show it (2023–2025).[7][7] BNN Bloomberg / AP, Deloitte to partially refund Australian government Secondary[13][13] NY State Comptroller, Enforcement of Local Law 144 Primary[42][42] Maginative, Klarna dials back AI customer service Secondary
  • The main US bank guidance excludes generative AI. SR 26-2 (17 Apr 2026) carves out generative and agentic AI and sets no enforceable standards.[10][10] Federal Reserve, SR 26-2 attachment Primary[49][49] Federal Reserve, SR 26-2 letter Primary
  • Insurance has the densest testing duties outside AI law. EIOPA expects pre-release and ongoing tests, fairness metrics and due-diligence tests of bought AI (Aug 2025).[11][11] EIOPA, Opinion on AI governance and risk management Primary
  • Mandates without enforcement or a paying buyer fail. NYC Local Law 144's audits went largely unenforced, and CHAI's health assurance labs never launched (2025–2026).[13][13] NY State Comptroller, Enforcement of Local Law 144 Primary[14][14] Fierce Healthcare, Inside CHAI's failed assurance labs Secondary

Mature teams run one seven-step loop, and every production failure becomes a test

Public cases from GitHub, Shopify, ING, Notion, Intercom and Abridge describe the same loop. It runs from real traces, through a release gate, to online monitoring and back.

The loop starts with real traffic, not invented test cases. Teams read production conversations, group the failures and build test sets from them. They gate releases on those sets in CI, grade hard cases with calibrated judges, roll out in stages and monitor live quality.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary[2][2] Shopify Engineering, Building production-ready agentic systems Primary[6][6] AI Expert Network, AI at ING Secondary Each new failure goes back into the datasets.[17][17] Braintrust, Notion case study Vendor

The usual gate is relative, not absolute. Teams release when a candidate shows "no regression against the production baseline".[1][1] GitHub, How we evaluate models for GitHub Copilot Primary[3][3] Claude customer story, Intercom Vendor Few publish a fixed pass score. GitHub says it uses judgement when metrics conflict.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary Practitioner guides describe the method for the first four steps in more depth; this chapter adds the company evidence.[18][18] Husain and Shankar, evals FAQ Secondary

The Best Teams Turn Every Failure Into a New Test

The seven-step evaluation loop described by GitHub, Shopify, ING, Notion, Intercom and Abridge, with a real example at each step.

1/7

Step 1· Before release

Analyse errors on real traces

Read real production conversations and group the failures.

ExampleShopify samples real merchant chats; three or more product experts label each one.

Step 1 of 8: Step 1, error analysis on real traces, appears with the Shopify example.

Start with real failures. Shopify dropped hand-written golden sets. It now samples real merchant conversations, and at least three product experts label each one.[2][2] Shopify Engineering, Building production-ready agentic systems Primary
Turn them into a regression set. A regulated set needs hundreds of real cases, not thousands. ING re-ran 500 real chats daily, and Box gates launches on 330 questions.[6][6] AI Expert Network, AI at ING Secondary[15][15] Braintrust, customer stories Vendor
Gate every change in CI. GitHub runs more than 4,000 offline tests, mostly automated.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary Dropbox runs more than 10,000 tests after every merge.[15][15] Braintrust, customer stories Vendor
Calibrate the judge before trusting it. Shopify's first judge agreed with experts at a kappa of 0.02, close to random. Calibration lifted it to 0.61, near the human-to-human 0.69.[2][2] Shopify Engineering, Building production-ready agentic systems Primary
Release in stages. GitHub staff use candidate models first, like a canary.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary ING piloted with 10% of Dutch mobile-chat users and ran A/B tests.[6][6] AI Expert Network, AI at ING Secondary Shopify also replays real goals through an LLM merchant simulator before release.[2][2] Shopify Engineering, Building production-ready agentic systems Primary
Keep testing in production. GitHub runs daily tests against live models to catch silent changes.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary Intercom scores live conversations, and Abridge tracks clinician edits and star ratings.[16][16] createwith.com, Intercom Monitors Secondary[8][8] Abridge, The science of AI evaluation Vendor
Close the loop. At Notion, a data specialist turns failures into new eval sets.[17][17] Braintrust, Notion case study Vendor The loop then starts again at step 1.
Few teams run the whole loop. In LangChain's vendor survey, 45% of teams with agents in production run online evals, and 23% do not evaluate at all.[9][9] LangChain, State of Agent Engineering Vendor

Figure 1. Steps 1–4 run before release, step 5 is the release, and steps 6–7 run in production. Box, Dropbox and Notion figures come from Braintrust case studies (vendor-authored). Abridge and Intercom figures are self-reported. ING figures are snippet only.

Source: GitHub;[1][1] GitHub, How we evaluate models for GitHub Copilot Primary Shopify Engineering;[2][2] Shopify Engineering, Building production-ready agentic systems Primary AI Expert Network on ING;[6][6] AI Expert Network, AI at ING Secondary Braintrust;[15][15] Braintrust, customer stories Vendor[17][17] Braintrust, Notion case study Vendor Intercom Monitors;[16][16] createwith.com, Intercom Monitors Secondary Abridge.[8][8] Abridge, The science of AI evaluation Vendor

Metrics by stage

StageTypical metricsExample
Offline regressionUnit-test pass rate, similarity to known-good output, token useGitHub Copilot[1][1] GitHub, How we evaluate models for GitHub Copilot Primary
Judge calibrationCohen's kappa against a human-human ceilingShopify Sidekick[2][2] Shopify Engineering, Building production-ready agentic systems Primary
Expert trialClinician preference in blinded head-to-head trialsAbridge[8][8] Abridge, The science of AI evaluation Vendor
Buyer acceptanceF1 against human reviewersUK i.AI Consult[19][19] GOV.UK, Humphrey tool reviews consultation responses Primary
Online qualityResolution rate, CSAT, accept rate, clinician editsIntercom, Cursor, Abridge[3][3] Claude customer story, Intercom Vendor[20][20] Cursor, Online RL for Tab Primary[8][8] Abridge, The science of AI evaluation Vendor
Online safetyOff-topic, toxicity and prompt-hacking ratesGitHub Copilot[1][1] GitHub, How we evaluate models for GitHub Copilot Primary
Table 1. Each stage of the loop uses different metrics. Only judge calibration and buyer acceptance compare a system directly with human experts. Source: company engineering blogs and case studies cited in each row.

Software firms publish the best evidence: regression gates, calibrated judges and online metrics

Their own engineering blogs give the clearest numbers. Most other detail comes from case studies written by eval vendors about their customers.

GitHub runs a large offline suite, a judge it audits and a canary step. It runs "more than 4,000 offline tests", mostly in automated CI.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary The team breaks about 100 containerised repositories on purpose, and the model must repair them. Chat quality tests use more than 1,000 technical questions. An LLM judge grades complex answers, and the team calls alignment with human reviewers a challenge.

Shopify gave the most detailed public account of judge calibration. It samples "Ground Truth Sets" from real merchant conversations, labelled by at least three product experts.[2][2] Shopify Engineering, Building production-ready agentic systems Primary Calibration raised the judge's Cohen's kappa from 0.02 to 0.61, against 0.69 between two humans. The team trusts a judge when blind reviewers cannot tell judge labels from human labels.

Calibration took Shopify's judge from random to almost human

Agreement of Shopify Sidekick's LLM judge with expert labels (Cohen's κ), before and after calibration, against the human ceiling, August 2025.

Drag each bar: how often did Shopify's judge agree with experts, beyond chance?

slightfairmoderatesubstantialalmost perfect00.20.40.60.81Drag the bars to your guess0.50Before calibration LLM judge vs experts0.50After calibration LLM judge vs experts0.50Human vs human the ceiling

Drag a bar, or tab to it and use the arrow keys.

Data table: shown after you check your guess

Figure 2. Cohen's κ corrects agreement for chance: 0 is chance level, 1 is perfect. The human bar is the ceiling: how well two experts agree with each other. Bands follow the common Landis and Koch reading. A later, separate fix for reward hacking raised the judge's correlation from 0.66 to 0.75. Source: Shopify Engineering, Building production-ready agentic systems (Aug 2025).[2][2] Shopify Engineering, Building production-ready agentic systems Primary

Training on the evals produced reward hacking. When Shopify ran reinforcement learning against its evals, the model refused tasks, misused tags and invented IDs.[2][2] Shopify Engineering, Building production-ready agentic systems Primary Fixes raised syntax validity from about 93% to 99%. The lesson is simple: hold back a test set that training never sees.

Regression suites run in thousands of tests, golden sets in hundreds

Size of published eval suites and golden sets at named companies, 2025–2026. Square area is proportional to test cases.

10,000+Dropboxtests afterevery merge(vendor-authored)4,000+GitHubCopilotofflinetests,mostly inCI1,000+GitHubCopilottechnicalchatquestions500INGrealcustomerchats,re-rundaily(snippetonly)400+Eveexpert-gradedlegal tasks(vendor-authored)330Boxlaunch-gategolden set(vendor-authored)≈100GitHubCopilotbrokenrepositoriesto repairBox shipped its agent above80% on core evals.

Area is proportional to test cases. Indigo: regression suites; tan: golden or expert sets.

What to notice

  • A regulated golden set needs hundreds of real cases, not thousands.
Data table
ItemValueNote
Dropbox10,000+tests after every merge (vendor-authored)
GitHub Copilot4,000+offline tests, mostly in CI
GitHub Copilot1,000+technical chat questions
ING500real customer chats, re-run daily (snippet only)
Eve400+expert-graded legal tasks (vendor-authored)
Box330launch-gate golden set (vendor-authored)
GitHub Copilot≈100broken repositories to repair
Figure 3. Counts are lower bounds where the source says 'more than'. Suites differ in what one test means, so compare orders of magnitude only. Source: GitHub;[1][1] GitHub, How we evaluate models for GitHub Copilot Primary Braintrust customer stories (vendor-authored);[15][15] Braintrust, customer stories Vendor AI Expert Network on ING (snippet only).[6][6] AI Expert Network, AI at ING Secondary

Huge traffic lets online metrics replace much of the offline gate. Cursor trains its Tab model with online reinforcement learning on accept and reject signals.[20][20] Cursor, Online RL for Tab Primary Few enterprises have that volume. Vendor-reported resolution rates also need care. A competitor notes that Intercom changed the denominator of Fin's resolution rate, so rates from different vendors do not compare.[22][22] eesel.ai, Fin response quality Vendor

At Cursor's scale, users are the eval

Change after Cursor's online-RL Tab model replaced the previous model, September 2025. Both arrows start at the previous model, indexed to 100.

7085100115130Index, previous model = 100ChangeSuggestions shown100−21%−21Previous modelOnline-RL modelAccept rate100+28%+28

What to notice

  • The model shows fewer suggestions, and users accept more of them.
Data table
RowPrevious modelOnline-RL modelChange
Suggestions shown100−21%−21
Accept rate100+28%+28
Figure 4. Cursor serves more than 400 million Tab requests per day. One checkpoint rollout plus data collection takes 1.5 to 2 hours, so live accept and reject signals act as the eval. Cursor reports these figures itself. Source: Cursor, Online RL for Tab.[20][20] Cursor, Online RL for Tab Primary

  • GitHub Copilot (January 2025)

    • More than 4,000 offline tests, mostly in automated CI.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary
    • About 100 containerised repositories that pass CI are broken on purpose; the model must repair them.
    • Chat quality tests use more than 1,000 technical questions.
    • An LLM judge grades complex chat answers; the team audits the judge.
    • Staff switch to candidate models first, like a canary release.
    • Daily runs against production models flag any degradation.
    • Safety tests cover off-topic prompts, toxicity, baiting and prompt hacking; red-teaming adds more cases.
    • No formal release threshold is published; the team uses judgement when metrics conflict.

    Shopify Sidekick (August 2025)

    • Hand-curated golden sets replaced by "Ground Truth Sets" sampled from real merchant conversations.[2][2] Shopify Engineering, Building production-ready agentic systems Primary
    • At least three product experts label each conversation.
    • Judge kappa raised from 0.02 to 0.61; two humans agree at 0.69.
    • A judge is trusted when blind reviewers cannot tell its labels from human labels.
    • An LLM merchant simulator replays real goals against each candidate before release.
    • RL on the evals produced reward hacking: refusals, tag misuse and invented IDs.
    • Fixes raised syntax validity from about 93% to 99%; judge correlation rose from 0.66 to 0.75.

    Intercom Fin (vendor-authored)

    • Anthropic, Intercom's model supplier, published this case.[3][3] Claude customer story, Intercom Vendor
    • Every change is tested "against our production baseline" on answer accuracy, resolution rate, CSAT and response quality, using millions of real conversations.
    • Intercom claims a 51% resolution rate out of the box and up to 86% for some customers.[3][3] Claude customer story, Intercom Vendor[21][21] Intercom blog, Intercom vs Zendesk Vendor
    • A competitor notes the denominator changed: "Fin Constrained" conversations no longer count.[22][22] eesel.ai, Fin response quality Vendor
    • The "Monitors" feature scores live conversations continuously.[16][16] createwith.com, Intercom Monitors Secondary

    Cursor Tab (September 2025)

    • Over 400 million requests per day; online RL on accept and reject signals.[20][20] Cursor, Online RL for Tab Primary
    • One checkpoint rollout plus data collection takes 1.5 to 2 hours.
    • The new model makes 21% fewer suggestions with a 28% higher accept rate.
  • CompanyPracticeNumber
    NotionRegression and frontier evals on each new modelAbout 70 AI engineers; about 80% of AI work runs on evals and traces; new models ship in under 24 hours[17][17] Braintrust, Notion case study Vendor
    NotionA data specialist turns failures into eval setsExample: a multilingual language-adherence set with an LLM judge[17][17] Braintrust, Notion case study Vendor
    DropboxFull suite after every mergeMore than 10,000 tests[15][15] Braintrust, customer stories Vendor
    BoxLaunch gate on a golden set330 questions; Box Agent shipped above 80% on core evals[15][15] Braintrust, customer stories Vendor
    NavanJudge scores voice callsAbove 0.9 macro F1[15][15] Braintrust, customer stories Vendor
    CloudflareCI blocks regressionsNo number given[15][15] Braintrust, customer stories Vendor
    EveExpert-graded legal setMore than 400 tasks[15][15] Braintrust, customer stories Vendor
    Table 2. All figures come from case studies that Braintrust, an eval vendor, wrote about its customers.
    • The stack has converged: CI regression, then calibrated judges, then dogfood or canary, then online metrics.
    • Golden sets grow from production failures; nobody writes them fully in advance.
    • Good teams validate their judges against humans and report agreement.
    • Vendor-reported resolution rates do not compare across vendors, because definitions change.
    • No company reports eval headcount apart from AI headcount. Notion's figure of about 70 engineers covers all AI work.

Banks gate each use case and validate it independently, but publish few numbers

Public detail on bank and insurer practice is thin. Banks describe gates and validation, but not set sizes or thresholds. Insurers describe security and privacy, but not test method.

The bank pattern is a gate per use case plus independent validation. Morgan Stanley built "an evaluation (eval) framework to test every AI use case before deployment", according to its supplier OpenAI.[4][4] OpenAI customer story, Morgan Stanley Vendor JPMorgan runs an "AI Center of Excellence" inside its model risk review function. A job listing says it does "independent validation focused on AI/ML models, LLM-based applications, and generative AI systems".[5][5] The Muse, JPMorgan model risk job listing Single source Its CIO said that building generative AI is easier than "the validation and the controls".[25][25] InvestmentNews, JPMorgan joins the AI productivity war Secondary

ING gives the most concrete European bank number. It built a support chatbot with McKinsey's QuantumBlack and piloted it with 10% of Dutch mobile-chat customers.[6][6] AI Expert Network, AI at ING Secondary[27][27] CTO Magazine, ING AI chatbot Secondary[28][28] McKinsey, How ING uses generative AI Vendor The team re-ran 500 real customer chats daily and ran A/B tests. Risk staff joined from the start, and guardrails block mortgage and investment advice. The bot resolved about 20% more queries than the classic chatbot.

Insurers publish almost nothing about test method. AXA's coverage stresses privacy and security, not testing.[30][30] Intelligent Insurer, AXA develops generative AI with Microsoft Secondary[31][31] Microsoft customer story, AXA Vendor A 2025 trade article says insurers red-team both in-house and vendor AI, and warns them not to trust vendors' red-team claims alone.[32][32] InsuranceNewsNet, Red teaming for insurer AI risk Secondary We found no public eval detail for Allianz, Zurich, Lemonade or Goldman Sachs.

Banks describe their gates; only ING gives a set size

FirmKind of accountKey factWhat it describes
Morgan StanleyVendor-authored98%+ adoption among advisor teams (OpenAI's figure)Eval framework tests every AI use case before deployment; Later added translation evals and retrieval tuning; Assistant launched Sep 2023 over about 100,000 documents
JPMorgan ChaseIndependent validationMRGR AI Center of Excellence inside model risk reviewValidates LLM applications like models; Discussed GenAI pilots with US regulators in Nov 2023; No public set sizes, judge use or thresholds
INGVendor-consultant account500 real customer chats re-run dailyPilot with 10% of Dutch mobile-chat users, plus A/B tests; Risk staff involved from the start; About 20% more queries resolved than the old bot
BBVAGuardrailsBlue chatbot limited to banking operations and accountsTechnical guardrails and real-time monitoring; Simulated external cyberattacks; No published test sets
AXAInsurerSecure GPT internal assistant on Azure OpenAICoverage stresses privacy and security; No public test method; Insurers red-team in-house and vendor AI (trade article)
Table 3. What five financial firms say publicly about how they evaluate generative AI, 2023–2026. Only ING names a test-set size. The bank habit of independent validation persists even though SR 26-2 no longer covers generative AI. Most figures come from vendor case studies or search snippets. Klarna, the opposite pattern, appears in the failures section. Source: OpenAI (vendor-authored, snippet only);[4][4] OpenAI customer story, Morgan Stanley Vendor CTO Magazine;[23][23] CTO Magazine, AI in Morgan Stanley Secondary The Muse (snippet only);[5][5] The Muse, JPMorgan model risk job listing Single source Bloomberg (snippet only);[26][26] Bloomberg, JPMorgan works with US regulators on generative AI pilots Secondary AI Expert Network on ING;[6][6] AI Expert Network, AI at ING Secondary BBVA;[29][29] BBVA, Responsible AI and guardrails (Spanish) Primary Intelligent Insurer;[30][30] Intelligent Insurer, AXA develops generative AI with Microsoft Secondary InsuranceNewsNet.[32][32] InsuranceNewsNet, Red teaming for insurer AI risk Secondary

  • Bank and insurer detail

    Morgan Stanley (vendor-authored, snippet only). OpenAI reports more than 98% adoption among advisor teams in wealth management.[4][4] OpenAI customer story, Morgan Stanley Vendor The assistant launched in September 2023 and covers about 100,000 research reports and documents.[23][23] CTO Magazine, AI in Morgan Stanley Secondary The framework later added translation evals for clients in other languages, and retrieval tuning for a growing library. OpenAI's enterprise guide uses Morgan Stanley as its lead "start with evals" example.[24][24] OpenAI, AI in the Enterprise Vendor

    JPMorgan Chase. The MRGR AI Center of Excellence runs independent validation of AI/ML models, LLM-based applications and generative AI systems (job listing, snippet only).[5][5] The Muse, JPMorgan model risk job listing Single source JPMorgan discussed its generative AI pilots with US regulators in November 2023.[26][26] Bloomberg, JPMorgan works with US regulators on generative AI pilots Secondary No public JPMorgan document gives set sizes, judge use or pass thresholds.

    ING. Pilot reached 10% of Dutch mobile-chat customers; guardrails block mortgage and investment advice; risk staff joined from the start; daily regression tests on 500 real chats plus A/B tests; about 20% more queries resolved than the classic chatbot.[6][6] AI Expert Network, AI at ING Secondary[27][27] CTO Magazine, ING AI chatbot Secondary[28][28] McKinsey, How ING uses generative AI Vendor A set of 500 real chats suggests a useful scale: a regulated golden set needs hundreds of real cases, not thousands.

    BBVA. "Blue" has technical guardrails, real-time monitoring and simulated external cyberattacks. It answers only questions about banking operations and accounts.[29][29] BBVA, Responsible AI and guardrails (Spanish) Primary

    Insurers. AXA runs "AXA Secure GPT" on Azure OpenAI.[30][30] Intelligent Insurer, AXA develops generative AI with Microsoft Secondary[31][31] Microsoft customer story, AXA Vendor Insurers red-team in-house and vendor AI (2025 trade article).[32][32] InsuranceNewsNet, Red teaming for insurer AI risk Secondary No public eval detail for Allianz, Zurich, Lemonade or Goldman Sachs.

    What the finance cases show

    • The pattern "a gate per use case plus independent validation" comes from model-risk habits.
    • Banks keep this habit even where SR 26-2 no longer covers generative AI.
    • Klarna shows the opposite pattern, where cost drove the evaluation.

Public bodies and one health vendor publish the most rigorous methods

Public bodies must justify spending, so they publish more method than firms. In healthcare, one vendor's release gate sets the bar, but it grades itself.

The UK's i.AI runs a buyer-side acceptance test. Its "Consult" tool analysed a Scottish Government consultation and scored an F1 of 0.76 against human reviewers.[19][19] GOV.UK, Humphrey tool reviews consultation responses Primary The differences had a "negligible" effect on the ranking of themes. The government plans more accuracy work before a wider rollout. Its note-taking tool "Minute" reported one hour saved per one-hour meeting in 25 councils, but that is an early pilot figure.[35][35] Computer Weekly, Humphrey AI for council admin Secondary

Self-classification under-reports risk. The Netherlands Court of Audit counted 433 AI systems at 70 central-government organisations in October 2024.[36][36] Accountant.nl, Court of Audit on central government AI Secondary[37][37] Binnenlands Bestuur, Government often skips AI risk assessment Secondary[38][38] NOS, Court of Audit AI report Secondary More than half had no prior risk assessment. Only 5% appeared in the public algorithm register. The audit office saw "an incentive not to classify AI systems as high risk" (our translation). Its figures rest on the ministries' own self-assessments.

Most Dutch government AI systems skipped a risk assessment

Share of 433 AI systems at 70 Dutch central-government organisations, Netherlands Court of Audit, October 2024.

0%25%50%75%100%Share of systemsWithout prior risk assessme…all systems>50%High-risk systems without p…share of high-risk systems≈1/3In use120 of 43328%In the public algorithm reg…all systems5%↳ Only 5% are in the public register.

What to notice

  • The first bar is a lower bound ('more than half').
  • The audit office did not check the ministries' self-assessments.
Data table
ItemValue
Without prior risk assessment (all systems)>50%
High-risk systems without prior risk assessment (share of high-risk systems)≈1/3
In use (120 of 433)28%
In the public algorithm register (all systems)5%
Figure 5. Bars use different bases: the high-risk bar is a share of high-risk systems only. Source: Accountant.nl, Binnenlands Bestuur and NOS on the Court of Audit report (snippet only).[36][36] Accountant.nl, Court of Audit on central government AI Secondary[37][37] Binnenlands Bestuur, Government often skips AI risk assessment Secondary[38][38] NOS, Court of Audit AI report Secondary

Abridge publishes the most rigorous enterprise release gate we found, and wrote it itself. Its test data holds more than 10,000 hours of clinical audio with gold transcripts.[8][8] Abridge, The science of AI evaluation Vendor A release needs blinded head-to-head trials "adjudicated by licensed clinicians". The trials use anytime-valid sequential tests, so the team can stop early without false confidence. Monitoring uses clinician edits, star ratings and links from each note sentence to the transcript.

Independent health studies are rare, but they can contradict vendors. A 2025 study graded ambient notes with a modified PDQI-9 instrument and two blinded reviewers.[41][41] Frontiers in AI, Ambient AI note quality study Primary Ambient notes were more thorough than human notes, but less succinct and more prone to hallucination. Most scribes and admin tools are not medical devices, so the FDA does not review them, and vendors report their own evals.

  • Consultancies

    • McKinsey's Lilli draws on more than 40 curated internal sources and more than 100,000 documents.[33][33] OfficeChai, McKinsey Lilli Secondary[34][34] AI Expert Network, McKinsey Lilli Secondary
    • McKinsey claims 30% time savings and 20% better "content quality and accuracy". It reports these figures itself, and we found no published accuracy method.
    • Deloitte Australia agreed in October 2025 to refund part of a report fee (see Failures).[7][7] BNN Bloomberg / AP, Deloitte to partially refund Australian government Secondary

    UK government detail

    • Consult: F1 of 0.76 against human reviewers on a Scottish Government consultation; "negligible" effect on theme ranking; more accuracy evaluation planned.[19][19] GOV.UK, Humphrey tool reviews consultation responses Primary
    • Minute: pilots in 25 councils; one hour saved per one-hour meeting (early test, not a published evaluation).[35][35] Computer Weekly, Humphrey AI for council admin Secondary
  • FindingValue
    AI systems counted433, at 70 organisations
    Systems in use120 (28%)
    Systems with no prior risk assessmentMore than half
    High-risk systems with no prior risk assessmentAbout one third
    Systems in the public algorithm register5%
    Table 4. The figures rest on the ministries' own self-assessments. Source: Accountant.nl, Binnenlands Bestuur, NOS (snippet only).[36][36] Accountant.nl, Court of Audit on central government AI Secondary[37][37] Binnenlands Bestuur, Government often skips AI risk assessment Secondary[38][38] NOS, Court of Audit AI report Secondary
    • More than 10,000 hours of clinical audio with gold transcripts.[8][8] Abridge, The science of AI evaluation Vendor
    • Metrics: word error rate, recall of medical terms, precision and recall of medical concepts.
    • Blinded head-to-head trials adjudicated by licensed clinicians, with anytime-valid sequential tests.
    • Rollout starts with trained early adopters.
    • Monitoring: clinician edits, star ratings, sentence-to-transcript links.
    • Abridge's August 2025 paper says its system is "6x more likely to find and fix hallucinations" than standard models.[39][39] Abridge, Hallucinations whitepaper Vendor Secondary coverage gives 97% against 82% for GPT-4o on more than 10,000 encounters (snippet only).[40][40] HIT Consultant, Abridge hallucination results Secondary
    • i.AI's "F1 against human reviewers" is a buyer-side acceptance test that others could standardise.
    • The Dutch audit gives the clearest evidence that self-classification under-reports risk.
    • The strongest health gate combines expert trials, sequential statistics, staged rollout and edit monitoring.
    • Independent health studies remain rare and small, but the PDQI-9 study contradicts vendor claims on hallucination.

Public failures share one cause: the builder controlled the metric

The public eval failures fall into six groups. In each case, an independent test with a fixed rubric and human-checked grading would probably have exposed the problem.

Klarna let cost become the main evaluation factor. Klarna had claimed its assistant handled two-thirds of chats and did the work of 700 agents.[44][44] tech.co, Klarna reverses AI overhaul Secondary In 2025, CEO Sebastian Siemiatkowski told Bloomberg the AI-first push "went too far" and lowered quality.[42][42] Maginative, Klarna dials back AI customer service Secondary[43][43] Entrepreneur, Klarna CEO reverses course Secondary Klarna then began to rehire human agents.

[Cost was] a too predominant evaluation factor.[42][42] Maginative, Klarna dials back AI customer service Secondary
Sebastian Siemiatkowski, Klarna CEO, 2025 (as reported)

Deloitte Australia delivered fabricated references to a government client. Its AU$440,000 report for the Department of Employment and Workplace Relations contained fabricated references and a misquoted court judgment.[7][7] BNN Bloomberg / AP, Deloitte to partially refund Australian government Secondary A revised version disclosed the use of Azure OpenAI GPT-4o. Deloitte agreed to refund part of the fee, later reported as more than AU$97,000.[45][45] CFO Dive, Deloitte refunds report Secondary A citation check before delivery would have caught it.

Public eval failures and what would have caught them

CaseWhat went wrongWhat would have caught it
Klarna customer service[42][42] Maginative, Klarna dials back AI customer service SecondaryCost drove the evaluation; quality fellQuality metrics with equal weight in the release gate
Deloitte Australia report[7][7] BNN Bloomberg / AP, Deloitte to partially refund Australian government SecondaryFabricated references and a misquoted court judgmentA citation check before delivery
Shopify judge before calibration[2][2] Shopify Engineering, Building production-ready agentic systems PrimaryThe judge agreed with humans at near-random levels (κ 0.02)Judge validation against expert labels
Shopify RL training[2][2] Shopify Engineering, Building production-ready agentic systems PrimaryReward hacking: refusals, tag misuse, invented IDsHeld-out tests that training cannot see
Intercom resolution rate[22][22] eesel.ai, Fin response quality VendorThe metric's denominator changedA fixed, published metric definition
Dutch central government[36][36] Accountant.nl, Court of Audit on central government AI SecondaryAgencies under-classified risk and skipped assessmentsIndependent risk classification and testing
NYC Local Law 144[13][13] NY State Comptroller, Enforcement of Local Law 144 PrimaryMandated audits went largely unenforcedActive enforcement or buyer demand
Table 5. In every case the builder controlled the metric. We found no systematic dataset of enterprise eval failures, so this list is illustrative. Source: the cases cited in each row.

  • Six lessons from the failures

    • Put quality and cost in the same gate. Klarna shows the cost of a gate that tracks only cost.[42][42] Maginative, Klarna dials back AI customer service Secondary
    • Check every fact a reader will rely on. Deloitte had no reference check before delivery.[7][7] BNN Bloomberg / AP, Deloitte to partially refund Australian government Secondary
    • Validate the judge before you trust it. Shopify's uncalibrated judge was close to random.[2][2] Shopify Engineering, Building production-ready agentic systems Primary
    • Keep training data and test data apart. Shopify's reward hacking came from evals that fed training.[2][2] Shopify Engineering, Building production-ready agentic systems Primary
    • Freeze metric definitions. A changed denominator makes trends and comparisons meaningless.[22][22] eesel.ai, Fin response quality Vendor
    • Do not let the builder classify its own risk. The Dutch audit found a clear incentive to under-classify.[36][36] Accountant.nl, Court of Audit on central government AI Secondary

    Case detail

    • Klarna earlier claimed its assistant handled two-thirds of chats and did the work of 700 agents.[44][44] tech.co, Klarna reverses AI overhaul Secondary It then moved to rehire human agents in an "Uber type of setup".[42][42] Maginative, Klarna dials back AI customer service Secondary[43][43] Entrepreneur, Klarna CEO reverses course Secondary
    • Deloitte's report fee was AU$440,000; the refund was later put above AU$97,000.[7][7] BNN Bloomberg / AP, Deloitte to partially refund Australian government Secondary[45][45] CFO Dive, Deloitte refunds report Secondary

Evaluation lags observability: about half of agent teams run no offline evals

The best public data comes from LangChain's "State of Agent Engineering" survey. LangChain sells eval and tracing tools, so treat it as a vendor survey.

Teams trace their agents far more often than they test them. The survey ran from 18 November to 2 December 2025 and drew 1,340 responses.[9][9] LangChain, State of Agent Engineering Vendor About 89% of respondents trace or observe their systems. Only 52.4% run offline evals and 37.3% run online evals. Human review (59.8%) still leads LLM-as-judge (53.3%) as a method.

Nine in ten agent teams trace; only half run offline evals

Share of all respondents using each practice, LangChain State of Agent Engineering survey, November–December 2025 (n = 1,340).

Each square is 1 of 100 (1%).

What to notice

  • Human review (59.8%) still leads LLM-as-judge (53.3%) as a method.
Data table
PanelValue
Tracing or observability89%
Human review60%
LLM-as-judge53%
Offline evals52%
Online evals37%
Figure 6. Each grid has 100 squares; one square is one percent of respondents. Respondents could pick several practices, so the grids do not sum to 100%. Another 29.5% of all respondents run no evaluation at all. Vendor-run survey; about 49% of respondents work at firms with fewer than 100 staff. Source: LangChain, State of Agent Engineering (vendor survey).[9][9] LangChain, State of Agent Engineering Vendor

23%

of teams with agents in production do not evaluate them at all (22.8%, LangChain survey, Nov–Dec 2025).

Source: LangChain, State of Agent Engineering (vendor survey).[9][9] LangChain, State of Agent Engineering Vendor

Production teams do better, but not by much. Among teams with agents in production, 44.8% run online evals and 22.8% run no evaluation at all.[9][9] LangChain, State of Agent Engineering Vendor Quality is the top barrier to production at 32%, followed by latency at 20%. At firms with 2,000 or more staff, security ranks second at 24.9%.

Fewer than half of production agent teams test on live traffic

Teams with agents in production, by most advanced evaluation practice, LangChain survey, November–December 2025. One figure is 1% of these teams.

Production agent teams
  • Run online evals, 44.8%
  • Evaluate, but not online, 32.4% (derived)
  • No evaluation at all, 22.8%

= 1 percent of production teams

What to notice

  • The middle group is derived: 100% minus the two reported shares.
Data table
RowCategoryCount
Production agent teamsRun online evals, 44.8%44.8
Production agent teamsEvaluate, but not online, 32.4% (derived)32.4
Production agent teamsNo evaluation at all, 22.8%22.8
Figure 7. Shares of teams with agents in production; they sum to 100%. Vendor-run survey. Source: LangChain, State of Agent Engineering (vendor survey).[9][9] LangChain, State of Agent Engineering Vendor

Nobody measures eval spend, so time is the best proxy. Husain and Shankar report that 60–80% of development time went to evaluation in their projects.[18][18] Husain and Shankar, evals FAQ Secondary Notion says about 80% of AI-team work runs on evals and traces.[17][17] Braintrust, Notion case study Vendor Teams also pay for expert labels and for judge compute on every graded trace. Practitioners therefore run cheap checks on all traces and judges on samples.[18][18] Husain and Shankar, evals FAQ Secondary

  • MeasureAll respondentsTeams with agents in production
    Agents in production57.3% (51% a year earlier)100%
    Tracing or observabilityabout 89%not split out
    Offline evals52.4%not split out
    Online evals37.3%44.8%
    No evaluation at all29.5%22.8%
    Human review59.8%not split out
    LLM-as-judge53.3%not split out
    Table 6. Respondents could pick several practices, so the shares do not sum to 100%. Source: LangChain (vendor survey, 18 Nov–2 Dec 2025, 1,340 responses).[9][9] LangChain, State of Agent Engineering Vendor

  • Spend and scale

    • Menlo Ventures put enterprise generative AI spend at $37 billion in 2025, up from $1.7 billion in 2023, from about 500 US enterprise decision-makers (snippet only).[46][46] Let's Data Science, enterprise AI spend reaches $37bn Secondary[47][47] Long-term Wiki, Menlo Ventures 2025 report entry Secondary We found no 2026 edition.
    • A KPMG pulse survey reportedly found that only 11% of enterprises have scaled agents to organisation-wide value (secondary summary only).[48][48] AI2.work, enterprise AI budget scaling Single source

    Cost of evaluation

    • No reputable survey reports eval spend as a share of AI budget.
    • Husain and Shankar: 60–80% of development time on evaluation.[18][18] Husain and Shankar, evals FAQ Secondary
    • Notion: about 80% of AI-team work runs on evals and traces.[17][17] Braintrust, Notion case study Vendor
    • Expert labels: Shopify uses three experts per conversation; Abridge uses licensed clinicians.[2][2] Shopify Engineering, Building production-ready agentic systems Primary[8][8] Abridge, The science of AI evaluation Vendor
    • Judges cost compute on every graded trace, so teams run cheap checks on all traces and judges on samples.[18][18] Husain and Shankar, evals FAQ Secondary
    • About half of agent builders run no systematic offline eval.
    • Human review still leads LLM judges as a method.
    • No survey gives norms for eval headcount, and none offers a widely used maturity model.

Eleven sector regimes demand testing, but none is built for generative AI agents

Banks, insurers, device makers, employers, aviation and public agencies faced testing rules long before AI law. We rated eleven regimes on five questions.

Every regime asks for testing before use, and most for monitoring after. Many also name independence: SR 26-2, SS1/23, EIOPA, NYC Local Law 144 and the OMB memos all do.[10][10] Federal Reserve, SR 26-2 attachment Primary[54][54] Bank of England, SS1/23 model risk management principles Primary[11][11] EIOPA, Opinion on AI governance and risk management Primary[12][12] NYC DCWP, Automated employment decision tools Primary[79][79] OMB M-25-21 (3 Apr 2025) Primary Several accept black-box tests. SR 26-2 allows outcome analysis without code access, and OMB allows query-based tests of a vendor service.[10][10] Federal Reserve, SR 26-2 attachment Primary[79][79] OMB M-25-21 (3 Apr 2025) Primary

Most regimes are guidance, not law, and the newest technology often falls outside scope. Only Colorado's insurance rules, NYC Local Law 144, the EU MDR and the OMB memos bind. SR 26-2 excludes generative and agentic AI in its footnote 3.[10][10] Federal Reserve, SR 26-2 attachment Primary It also states that "non-compliance with this guidance will not result in supervisory criticism". EASA's proposal covers only offline-learning ML.[76][76] EASA, NPA 2025-07 Primary

No sector regime fully covers generative AI agents

Eleven regimes in banking, insurance, health, hiring, aviation and government, rated on five testing questions, November 2026.

RegimeCovers GenAI or agents? Pre-deployment testing Ongoing monitoring Independence required Binding?
US SR 26-2 Banking · Fed, OCC, FDIC · Apr 2026 No Excludes GenAI and agentic AI Yes Validation before first use Yes Includes vendor models Yes 'Objective experts' No Not enforceable
Covers GenAI or agents?: Excludes GenAI and agentic AIPre-deployment testing: Validation before first useOngoing monitoring: Includes vendor modelsIndependence required: 'Objective experts'Binding?: Not enforceable
UK PRA SS1/23 Banking · internal-model banks Partly or limited AI 'whatever the technology' Not covered in our sources Not covered in our sources Yes Independent validation (Principle 4) Partly or limited Internal-model banks only
Covers GenAI or agents?: AI 'whatever the technology'Independence required: Independent validation (Principle 4)Binding?: Internal-model banks only
ECB internal-models guide Banking · euro area · Jul 2025 No Capital models only Partly or limited Explainability, performance Not covered in our sources Not covered in our sources Partly or limited Supervisory guide
Covers GenAI or agents?: Capital models onlyPre-deployment testing: Explainability, performanceBinding?: Supervisory guide
EIOPA AI opinion Insurance · EU · Aug 2025 Partly or limited All AI; API links (3.37) Yes Before release (3.27) Yes Ongoing; drift (3.36) Partly or limited External audit an option (3.11) Partly or limited Supervisory opinion
Covers GenAI or agents?: All AI; API links (3.37)Pre-deployment testing: Before release (3.27)Ongoing monitoring: Ongoing; drift (3.36)Independence required: External audit an option (3.11)Binding?: Supervisory opinion
NAIC AI bulletin Insurance · 29 US jurisdictions Not covered in our sources Partly or limited Testing and validation expected Not covered in our sources Not covered in our sources No Guidance only
Pre-deployment testing: Testing and validation expectedBinding?: Guidance only
Colorado 3 CCR 702-10 Insurance · Colorado Not covered in our sources Partly or limited Bias test promised, not adopted Not covered in our sources Not covered in our sources Yes Binding governance rules
Pre-deployment testing: Bias test promised, not adoptedBinding?: Binding governance rules
FDA PCCP and lifecycle Health · US devices Partly or limited AI broadly; GenAI guidance pending Yes Pre-market gate Partly or limited Change plans; lifecycle guidance draft Partly or limited FDA review; CHAI labs stalled Partly or limited Gate binding; lifecycle draft
Covers GenAI or agents?: AI broadly; GenAI guidance pendingPre-deployment testing: Pre-market gateOngoing monitoring: Change plans; lifecycle guidance draftIndependence required: FDA review; CHAI labs stalledBinding?: Gate binding; lifecycle draft
EU MDR + AI Act Health · MDCG 2025-6 Partly or limited Device AI only Yes Notified-body assessment Yes Post-market surveillance Yes Notified body Yes High-risk duties from 2 Aug 2028 (Omnibus)
Covers GenAI or agents?: Device AI onlyPre-deployment testing: Notified-body assessmentOngoing monitoring: Post-market surveillanceIndependence required: Notified bodyBinding?: High-risk duties from 2 Aug 2028 (Omnibus)
NYC Local Law 144 Hiring tools · since Jul 2023 Partly or limited Automated hiring tools only Yes Bias audit within a year before use Partly or limited Yearly audit Yes Independent bias audit Yes Law, but weak enforcement
Covers GenAI or agents?: Automated hiring tools onlyPre-deployment testing: Bias audit within a year before useOngoing monitoring: Yearly auditIndependence required: Independent bias auditBinding?: Law, but weak enforcement
EASA NPA 2025-07 Aviation · EU · proposal No Offline ML only; GenAI later Yes Learning assurance Not covered in our sources Not covered in our sources No Proposal; next step Q2 2027
Covers GenAI or agents?: Offline ML only; GenAI laterPre-deployment testing: Learning assuranceBinding?: Proposal; next step Q2 2027
OMB M-25-21 / M-25-22 US federal agencies Partly or limited High-impact AI Yes Practice 1; black-box tests allowed Yes Practice 3 Yes In-agency reviewer Yes Binds agencies
Covers GenAI or agents?: High-impact AIPre-deployment testing: Practice 1; black-box tests allowedOngoing monitoring: Practice 3Independence required: In-agency reviewerBinding?: Binds agencies

What to notice

  • The highlighted column has no full mark: no regime is built for generative AI agents.
  • The closest are the OMB memos and the EU device rules, and both are narrow.
  • US bank guidance now excludes generative AI; insurance rules are mostly guidance.
Figure 8. The EU MDR row uses the Digital Omnibus date of 2 August 2028; MDCG 2025-6 had given 2 August 2027. CHAI, a voluntary health scheme, is covered separately. Source: Federal Reserve;[10][10] Federal Reserve, SR 26-2 attachment Primary[49][49] Federal Reserve, SR 26-2 letter Primary Bank of England;[54][54] Bank of England, SS1/23 model risk management principles Primary ECB;[55][55] ECB, Revised guide to internal models Primary EIOPA;[11][11] EIOPA, Opinion on AI governance and risk management Primary NAIC;[57][57] NAIC, AI model bulletin adoption map Primary Debevoise;[60][60] Debevoise, Colorado extends AI regulation Secondary RAPS;[63][63] RAPS, Final FDA guidance on PCCP Secondary[65][65] RAPS, FDA device centre guidance agenda for FY2027 Secondary MDCG summaries;[71][71] OpenRegulatory, MDCG 2025-6 Secondary[72][72] Reuschlaw, Interplay between MDR, IVDR and the AI Act Secondary Garrigues;[73][73] Garrigues, Digital Omnibus on AI published Secondary NYC DCWP;[12][12] NYC DCWP, Automated employment decision tools Primary EASA;[76][76] EASA, NPA 2025-07 Primary OMB.[79][79] OMB M-25-21 (3 Apr 2025) Primary[80][80] OMB M-25-22 (3 Apr 2025) Primary

Only four of the eleven regimes bind

Legal force of the eleven sector regimes that demand AI testing, November 2026.

What to notice

  • Even binding NYC LL144 is weakly enforced.
Data table
SegmentValue
Binding4
Partly binding4
Not binding3
Figure 9. Binding: Colorado 3 CCR 702-10, NYC Local Law 144, EU MDR with the AI Act, and the OMB memos. Partly binding or supervisory: PRA SS1/23, the ECB guide, the EIOPA opinion and the FDA regime. Not binding: SR 26-2, the NAIC bulletin and the EASA proposal. Classification follows the "Binding?" column of the matrix above. Source: as for the matrix above.

Insurance has the densest testing duties outside AI law. EIOPA's opinion of 6 August 2025 expects tests "before release as well as on an ongoing basis", fairness metrics and drift monitoring.[11][11] EIOPA, Opinion on AI governance and risk management Primary Paragraph 3.11 names "external audits, or performing due diligence testing" for bought AI. Paragraph 3.37 asks for tests of connections to other AI systems through APIs, which fits agents directly. In the US, the NAIC bulletin reached 29 jurisdictions by 3 November 2026, but it stays guidance.[57][57] NAIC, AI model bulletin adoption map Primary Colorado binds, but its quantitative test rule has stalled since September 2023.[60][60] Debevoise, Colorado extends AI regulation Secondary

Mandates without enforcement or a paying buyer fail. NYC Local Law 144 requires an independent bias audit within a year before an employer uses a hiring tool.[12][12] NYC DCWP, Automated employment decision tools Primary The State Comptroller audited enforcement in December 2025.[13][13] NY State Comptroller, Enforcement of Local Law 144 Primary A secondary summary says the audit found 17 potential violations where the city found one.[74][74] GLACIS, Guide to New York AI rules Single source In health, CHAI planned certified assurance labs in 2024. By February 2026 the network had "never materialized".[14][14] Fierce Healthcare, Inside CHAI's failed assurance labs Secondary[66][66] Fierce Healthcare, CHAI draft model card and lab certification Secondary

Sector rules arrived piecemeal, and generative AI keeps slipping out of scope

Key dates for sector regimes that demand AI testing, 2023 to 2028.

202320242025202620272028NowSR 26-2 is the newest major rule,and it explicitly drops generativeAI.BankingSR 26-2 replaces SR11-7; excludes GenAI ECB revisedinternal-modelsguide PRA SS1/23 published InsuranceNAIC evaluation-toolpilot, 12 states Colorado extendsrules to auto andhealth EIOPA opinion on AI NAIC AI bulletinadopted HealthFDA revises CDSguidance MDCG 2025-6 on MDRand AI Act STAT: CHAI labeffort 'dead' FDA PCCP guidancefinal A June 2026 CRSreport dates it to CHAI draftsassurance-labcertification MDR/AI Act dutiesfor regulatedproducts Moved from 2 Aug 2027by the Digital Hiring, aviation, governmentEASA nextconsultation (Q22027) Planned; monthapproximate NY Comptrolleraudits LL144enforcement EASA NPA 2025-07 OMB M-25-21 andM-25-22 NYC LL144enforcement starts

What to notice

  • Dates after November 2026 are planned.
Data table
DateLaneEventKindNote
2023-05-17BankingPRA SS1/23 publishedregulation
2023-07-05Hiring, aviation, governmentNYC LL144 enforcement startsregulation
2023-12-04InsuranceNAIC AI bulletin adoptedregulation
2024-10HealthCHAI drafts assurance-lab certificationother
2024-12HealthFDA PCCP guidance finalregulationA June 2026 CRS report dates it to August 2025
2025-04-03Hiring, aviation, governmentOMB M-25-21 and M-25-22regulation
2025-05HealthSTAT: CHAI lab effort 'dead'shutdown
2025-06HealthMDCG 2025-6 on MDR and AI Actregulation
2025-07-28BankingECB revised internal-models guideregulation
2025-08-06InsuranceEIOPA opinion on AIregulation
2025-10-15InsuranceColorado extends rules to auto and healthregulation
2025-11-10Hiring, aviation, governmentEASA NPA 2025-07regulation
2025-12-02Hiring, aviation, governmentNY Comptroller audits LL144 enforcementother
2026-01-06HealthFDA revises CDS guidanceregulation
2026-03InsuranceNAIC evaluation-tool pilot, 12 statesother
2026-04-17BankingSR 26-2 replaces SR 11-7; excludes GenAIregulation
2027-04 (planned)Hiring, aviation, governmentEASA next consultation (Q2 2027)regulationPlanned; month approximate
2028-08-02 (planned)HealthMDR/AI Act duties for regulated productsregulationMoved from 2 Aug 2027 by the Digital Omnibus
Figure 10. Selected dates only. Full regime detail sits in the 'Go deeper' block below. Source: the regulator and secondary sources cited on each event.

The rules leave room for outside testers, but only with a buyer. SR 26-2 lets banks validate vendor products "either by internal or outside parties".[10][10] Federal Reserve, SR 26-2 attachment Primary EIOPA names external audits, and OMB M-25-22 requires vendors to give agencies the "access and time necessary" for independent evaluation.[11][11] EIOPA, Opinion on AI governance and risk management Primary[80][80] OMB M-25-22 (3 Apr 2025) Primary Black-box access is enough for many of these duties. Yet LL144 and CHAI show that a mandate alone does not create demand.

  • RegimeSector and placeLegal forceWhat it demandsCovers generative AI or agents?
    SR 26-2 / OCC 2026-13US banksGuidance; "non-compliance ... will not result in supervisory criticism"Effective challenge by "objective experts", validation before first use, monitoring that includes vendor modelsNo; footnote 3 excludes them
    PRA SS1/23UK banks with internal-model approvalSupervisory statement; internal-model banks onlyIndependent model validation (Principle 4)Partly; it covers AI "whatever the technology"
    ECB guide to internal modelsEuro-area banksSupervisory guideExplainable ML; performance must justify complexityNo; capital models only
    EIOPA AI opinionEU insurersSupervisory opinionTests before release (3.27), ongoing drift metrics (3.36), external audit as an option (3.11)Partly; all AI, including API links to other AI (3.37)
    NAIC model bulletinInsurers in 29 US jurisdictionsGuidance onlyTesting and validation within an AI programmeNot stated in our sources
    Colorado 3 CCR 702-10Colorado insurersBinding governance rulesRemediation of bias found by quantitative tests; test rule promised, not adoptedNot stated in our sources
    FDA PCCP and lifecycle guidanceUS medical devicesPre-market gate binds; PCCP guidance final; lifecycle guidance draftPre-market review; pre-agreed change plans with validationPartly; AI broadly, generative AI guidance pending
    EU MDR/IVDR + AI ActEU medical devicesBinding lawNotified-body assessment; post-market surveillancePartly; device AI only
    NYC Local Law 144NYC employersBinding law, weakly enforcedIndependent bias audit within a year before use; public summaryPartly; automated hiring tools only
    EASA NPA 2025-07EU aviationProposal; next step Q2 2027Learning assurance for offline-learning MLNo; generative AI comes later
    OMB M-25-21 / M-25-22US federal agenciesBinding on agenciesPre-deployment tests (Practice 1), black-box tests allowed, monitoring (Practice 3), in-agency reviewerPartly; high-impact AI only
    Table 7. Legal force, demands and scope of eleven sector regimes, November 2026.

  • US banking: SR 26-2

    • Issued by the Federal Reserve, the OCC and the FDIC on 17 April 2026; replaces SR 11-7 (2011) and SR 21-8 on anti-money-laundering models.[49][49] Federal Reserve, SR 26-2 letter Primary The OCC published it as Bulletin 2026-13.[50][50] Baker Tilly, Updated interagency guidance on model risk management Secondary[51][51] Openlayer, SR 26-2 guide for AI teams Vendor
    • "Most relevant to banking organizations with over $30 billion in total assets".[49][49] Federal Reserve, SR 26-2 letter Primary
    • Footnote 3: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance."[10][10] Federal Reserve, SR 26-2 attachment Primary Principles apply to "traditional statistical and quantitative models and non-generative, non-agentic AI models".
    • "This guidance does not set forth enforceable standards or prescriptive requirements; accordingly, non-compliance with this guidance will not result in supervisory criticism."[10][10] Federal Reserve, SR 26-2 attachment Primary
    • Within scope: effective challenge by "objective experts" with "sufficient independence" and the "organizational standing and influence to effect any change"; validation "generally occurs prior to a model's first use"; quality "depends on the rigor and effectiveness of the review rather than on organizational structure"; vendor products validated "either by internal or outside parties"; "ongoing monitoring and outcome analysis" of vendor models even without code or data.[10][10] Federal Reserve, SR 26-2 attachment Primary
    • Vendor commentators call the carve-out a placeholder; one reports a planned request for information on AI in banking, which we could not confirm.[52][52] CIMCON, SR 26-2 regulates your models, not your AI agents Vendor[53][53] Monitaur, Governing generative and agentic AI under SR 26-2 Vendor
    • Large banks still validate LLM applications (JPMorgan MRGR). The habit persists without the mandate.[5][5] The Muse, JPMorgan model risk job listing Single source

    UK banking: PRA SS1/23

    • Published 17 May 2023; in effect 17 May 2024; updated version in effect 23 April 2026 after minor amendments.[54][54] Bank of England, SS1/23 model risk management principles Primary
    • Applies to UK banks, building societies and PRA-designated investment firms with internal-model approval (IRB, IMA, IMM).
    • Covers in-house and vendor models "whatever the technology", with a sub-principle on "managing the risks associated with the use of artificial intelligence".
    • Principle 4 demands independent model validation. The PRA held a roundtable on AI and ML model risk in November 2025.

    Euro-area banking: ECB guide to internal models

    • Revised guide published 28 July 2025, with a new section on machine learning: models must be explainable, and performance must justify complexity.[55][55] ECB, Revised guide to internal models Primary[56][56] A&O Shearman, ECB publishes revised guide to internal models Secondary
    • It targets capital models. Customer-facing generative AI in EU banks falls under DORA, EBA outsourcing rules and the AI Act instead.

  • EU insurance: EIOPA opinion (EIOPA-BoS-25-360, 6 Aug 2025)

    Legal bases include the IDD, Solvency II and DORA.[11][11] EIOPA, Opinion on AI governance and risk management Primary

    ParagraphWhat it expects
    3.11Firms stay "ultimately responsible" for third-party AI; where IP limits insight, use contract clauses, "external audits, or performing due diligence testing and monitoring"
    3.15Monitor and, where appropriate, audit outcomes, "including with the use of fairness and non-discrimination metrics"; Annex I lists example metrics
    3.27Secure and test high-impact uses "before release as well as on an ongoing basis"
    3.31Firms may outsource oversight functions, but not responsibility, "while ensuring independence"
    3.36Use performance metrics such as accuracy and recall to detect drift
    3.37Testing should cover connections to other AI systems through APIs
    3.38Test resilience against data poisoning and adversarial attacks
    Table 8. The paragraphs of EIOPA's opinion that set testing expectations for insurers.

    US insurance: NAIC bulletin and Colorado

    • NAIC adopted its AI model bulletin on 4 December 2023; 29 jurisdictions had adopted it by 3 November 2026, most recently Puerto Rico, Mississippi, South Carolina and North Dakota in 2026.[57][57] NAIC, AI model bulletin adoption map Primary California, Colorado, New York and Texas have their own insurance AI rules.
    • Twelve states piloted an "AI Risk Evaluation Supplement" in March 2026; adoption at the 2026 Fall National Meeting is unconfirmed.[58][58] Openlayer, NAIC model bulletin and AI governance Vendor[59][59] actuary.info, NAIC AI bulletin and model law transition Secondary Secondary sources report no formal enforcement action under the bulletin by early 2026.
    • Colorado is the only US state with binding insurance AI governance rules.[60][60] Debevoise, Colorado extends AI regulation Secondary It extended its life-insurer rule to private auto and health benefit plans in August 2025; the amendments took effect on 15 October 2025, with governance due by 1 July 2026.
    • Insurers must remediate unfair discrimination "detected through quantitative testing". The quantitative testing rule was drafted in September 2023 and not updated since. The final text dropped a required bias and data-quality evaluation of external data.
    • Bulletin B-10.004 waives the description of quantitative tests in the 2024 and 2025 annual reports.[61][61] Colorado Division of Insurance, Bulletin B-10.004 Primary

  • US healthcare: FDA

    • The PCCP guidance for AI-enabled device software is final; most sources date it to December 2024, a June 2026 CRS report to August 2025.[62][62] Akin Gump, FDA finalises PCCP guidance Secondary[63][63] RAPS, Final FDA guidance on PCCP Secondary[64][64] CRS IF13245, FDA and AI-enabled devices Primary It covers AI broadly; makers agree planned changes and validation methods in advance.
    • The lifecycle guidance is still a January 2025 draft; the device centre's FY2027 agenda (1 Oct 2026) makes its final version an A-list priority.[65][65] RAPS, FDA device centre guidance agenda for FY2027 Secondary The agenda adds a top-priority draft on generative AI conversational devices for mental health and a low-priority item on risk assessment for generative AI devices.
    • The FDA revised its Clinical Decision Support software guidance on 6 January 2026.[64][64] CRS IF13245, FDA and AI-enabled devices Primary

    US healthcare: CHAI

    • An October 2024 draft would have certified independent labs against ISO 17025 and conflict-of-interest tests; final designs were promised for April 2025.[66][66] Fierce Healthcare, CHAI draft model card and lab certification Secondary
    • STAT reported in May 2025 that "CHAI's 'assurance lab' effort is dead".[67][67] Ground News, STAT on CHAI's assurance labs Secondary Becker's reported that the federal government sought to sideline CHAI.[68][68] Becker's, Administration seeks to sideline CHAI Secondary
    • In February 2026 Fierce Healthcare reported that the network "never materialized" (snippet only).[14][14] Fierce Healthcare, Inside CHAI's failed assurance labs Secondary
    • URAC, the Consumer Technology Association and the American Heart Association now run their own health AI programmes.[69][69] Distilinfo, CHAI's promise meets reality Secondary CHAI's site lists an "Applied Model Card", a public registry and v1.0 testing frameworks, but no certified labs.[70][70] CHAI, home page Primary

    EU healthcare: MDR, IVDR and the AI Act

    • MDCG 2025-6 (June 2025) is a joint FAQ of the Medical Device Coordination Group and the AI Board.[71][71] OpenRegulatory, MDCG 2025-6 Secondary[72][72] Reuschlaw, Interplay between MDR, IVDR and the AI Act Secondary
    • Device AI is high-risk under AI Act Article 6(1) when a notified body must assess the device. Makers may embed AI Act testing, monitoring and documentation into MDR or IVDR quality systems, and post-market surveillance can carry the monitoring duties.
    • MDCG 2025-6 gave 2 August 2027 as the start date; the Digital Omnibus moved duties for AI inside regulated products to 2 August 2028.[73][73] Garrigues, Digital Omnibus on AI published Secondary The Rules chapter covers the Omnibus.

  • Employment: NYC Local Law 144

    • An employer may use an automated employment decision tool only after "a bias audit within one year of the use of the tool", must post a summary, and must give candidates notice 10 business days before use.[12][12] NYC DCWP, Automated employment decision tools Primary DCWP began enforcement on 5 July 2023.
    • The State Comptroller audited enforcement on 2 December 2025.[13][13] NY State Comptroller, Enforcement of Local Law 144 Primary A secondary summary says it rated enforcement "ineffective": DCWP found one issue where auditors found 17 potential violations, and 75% of calls about these tools went to the wrong place.[74][74] GLACIS, Guide to New York AI rules Single source We could not check these figures against the report. No confirmed 2026 amendment.[75][75] Beancount.io, NYC Local Law 144 guide for 2026 Secondary

    Aviation: EASA NPA 2025-07

    • Published 10 November 2025; comments closed 10 February 2026. Step 1 of rulemaking task RMT.0742.[76][76] EASA, NPA 2025-07 Primary[77][77] EASA news, First regulatory proposal on AI in aviation Primary
    • Proposes a generic "DS.AI" framework for AI assurance, human factors and ethics, covering only offline-learning ML that does not contribute to critical failures. Generative, reinforcement-learning and hybrid AI come later.
    • The next consultation, expected in Q2 2027, embeds the framework into airworthiness, air traffic and drone rules.[78][78] ICAO DGCA61 paper, EASA approach to AI in aviation Primary "Learning assurance" is the most rigorous sector test doctrine we found, but it does not yet reach LLM agents.

    US federal agencies: OMB M-25-21 and M-25-22

    • Agencies must test high-impact AI before deployment; without code, model or data access, they query the service or send evaluation data to the vendor.[79][79] OMB M-25-21 (3 Apr 2025) Primary
    • An in-agency reviewer "who has not been involved in the development" reviews each impact assessment.
    • Agencies test and review AI periodically after deployment; non-compliant AI must stop.
    • Vendors must give agencies the "access and time necessary" for independent evaluation.[80][80] OMB M-25-22 (3 Apr 2025) Primary
    • The EU's model contractual clauses for AI procurement (MCC-AI) exist in High-Risk and Light versions; they are optional.[81][81] Covington Inside Privacy, Updated AI model contractual clauses Secondary

  • What the regimes share

    • Most regimes are guidance, not law; only Colorado, NYC LL144, the EU MDR and the OMB memos bind.
    • Every regime asks for testing before use and monitoring after use.
    • Many name independence: SR 26-2, SS1/23, EIOPA, LL144 and OMB M-25-21.
    • Several accept outside parties: SR 26-2 for vendor models, EIOPA for external audits.
    • Black-box testing is acceptable under SR 26-2 (outcome analysis) and OMB (query-based tests).
    • The newest technology often falls outside scope: SR 26-2 and EASA both exclude generative AI.
    • No regime targets generative AI agents directly.

    Other sectors not checked

    • No US or EU education rule outside the AI Act that demands AI testing was found; we did not search this area in depth.
    • Automotive rules (UNECE, ISO PAS 8800) and EU employment rules (Platform Work Directive) were not checked in this pass.

A mature programme climbs six levels, from vibes to independent assurance

This playbook combines the public cases and the sector rules above. It is our synthesis, not a published standard.

Most teams sit on the lower rungs. About 23% of production agent teams run no evals at all, which is level 0.[9][9] LangChain, State of Agent Engineering Vendor A regression gate built from real traffic, as at Box, Dropbox and Cloudflare, marks level 2.[15][15] Braintrust, customer stories Vendor Calibrated judges with published agreement, as at Shopify, mark level 3.[2][2] Shopify Engineering, Building production-ready agentic systems Primary Level 5 adds a party outside the build team that validates and challenges results. JPMorgan's model risk review and the OMB memos show this level.[5][5] The Muse, JPMorgan model risk job listing Single source[79][79] OMB M-25-21 (3 Apr 2025) Primary

Six levels of evaluation maturity, with a real example at each

A maturity ladder synthesised from public company cases and sector rules, November 2026.

  1. 5. Independent assurance

    Rare outside banks and government A party outside the build team validates and challenges results.

    • JPMorgan MRGR
    • OMB independent review

  2. 4. Closed loop

    Leading teams Staged rollout, online metrics and A/B tests; failures flow back into the suite.

    • GitHub
    • ING
    • Abridge

  3. 3. Calibrated grading

    Leading teams Judges carry published agreement with experts; simulators test multi-turn flows.

    • Shopify

  4. 2. Regression gate

    Established A suite built from real traffic runs in CI; no regression against production.

    • Box
    • Dropbox
    • Cloudflare

  5. 1. Smoke tests

    Common starting point A small fixed set runs before releases.

    • Many early teams

  6. 0. Vibes

    ≈23% of production agent teams Developers try prompts by hand; no stored tests.

    • LangChain survey, Nov–Dec 2025

What to notice

  • No survey offers a widely used maturity model; this ladder is our synthesis.
  • Each level keeps the practices of the levels below it.
Figure 11. Examples show where a named organisation publicly describes the practice, not a full audit of its programme. Source: LangChain;[9][9] LangChain, State of Agent Engineering Vendor Braintrust;[15][15] Braintrust, customer stories Vendor Shopify;[2][2] Shopify Engineering, Building production-ready agentic systems Primary GitHub;[1][1] GitHub, How we evaluate models for GitHub Copilot Primary ING;[6][6] AI Expert Network, AI at ING Secondary Abridge;[8][8] Abridge, The science of AI evaluation Vendor JPMorgan job listing;[5][5] The Muse, JPMorgan model risk job listing Single source OMB M-25-21.[79][79] OMB M-25-21 (3 Apr 2025) Primary

Build golden sets from real cases, and keep a held-out set. Start with hundreds of real production cases: ING used 500 chats and Box 330 questions.[6][6] AI Expert Network, AI at ING Secondary[15][15] Braintrust, customer stories Vendor Add each new production failure as a test. Keep a separate frontier set for new capabilities, as Notion does.[17][17] Braintrust, Notion case study Vendor Hold back a test set that never feeds training, which prevents reward hacking.[2][2] Shopify Engineering, Building production-ready agentic systems Primary

Calibrate every judge and publish the agreement. Write one narrow judge per failure mode.[18][18] Husain and Shankar, evals FAQ Secondary Measure each against expert labels with kappa or F1, and compare it with the agreement between two humans.[2][2] Shopify Engineering, Building production-ready agentic systems Primary Audit judges on a schedule, because models and traffic change.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary

Gate on regression, with hard floors and one named decision owner. Set the gate as "no regression on the core suite against production". Add hard floors for safety and compliance tests that cannot trade against quality gains. Weigh quality and cost in the same decision, as Klarna's reversal shows.[42][42] Maginative, Klarna dials back AI customer service Secondary Write down who decides when metrics conflict, and use sequential tests for expert trials.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary[8][8] Abridge, The science of AI evaluation Vendor

Map the programme to the rules

Programme elementRules that expect it
Test before first useSR 26-2, SS1/23, EIOPA 3.27, FDA, OMB M-25-21
Independent reviewSR 26-2, SS1/23 Principle 4, LL144, OMB M-25-21
Vendor model testingSR 26-2, EIOPA 3.11, OMB M-25-22
Ongoing monitoring and drift metricsSR 26-2, EIOPA 3.36, FDA lifecycle draft, OMB M-25-21
Fairness metricsEIOPA 3.15, Colorado, LL144
Tests of API connections to other AIEIOPA 3.37
Adversarial and poisoning testsEIOPA 3.38
Pre-agreed change validationFDA PCCP
Table 9. One programme can satisfy several regimes at once. Only EIOPA asks explicitly for tests of AI-to-AI connections, the closest any rule comes to agents. Source: SR 26-2;[10][10] Federal Reserve, SR 26-2 attachment Primary PRA SS1/23;[54][54] Bank of England, SS1/23 model risk management principles Primary EIOPA;[11][11] EIOPA, Opinion on AI governance and risk management Primary FDA;[62][62] Akin Gump, FDA finalises PCCP guidance Secondary[65][65] RAPS, FDA device centre guidance agenda for FY2027 Secondary NYC DCWP;[12][12] NYC DCWP, Automated employment decision tools Primary Colorado;[60][60] Debevoise, Colorado extends AI regulation Secondary OMB.[79][79] OMB M-25-21 (3 Apr 2025) Primary[80][80] OMB M-25-22 (3 Apr 2025) Primary

  • Team set-up

    • Give one named owner responsibility for each eval suite.
    • Put domain experts on labelling: Shopify uses product experts, Abridge licensed clinicians.[2][2] Shopify Engineering, Building production-ready agentic systems Primary[8][8] Abridge, The science of AI evaluation Vendor
    • Involve risk staff from the start, as ING did.[6][6] AI Expert Network, AI at ING Secondary
    • Keep an independent reviewer apart from the build team for high-impact uses; SR 26-2 and OMB M-25-21 both describe this role.[10][10] Federal Reserve, SR 26-2 attachment Primary[79][79] OMB M-25-21 (3 Apr 2025) Primary
    • Plan for a large share of engineering time: practitioners report 60–80% of development time on evaluation.[18][18] Husain and Shankar, evals FAQ Secondary

    Golden sets

    • Sample real production cases, not invented ones.
    • Start with hundreds of cases (ING 500 chats; Box 330 questions).[6][6] AI Expert Network, AI at ING Secondary[15][15] Braintrust, customer stories Vendor
    • Add each new production failure as a test.
    • Keep a separate frontier set for new capabilities, as Notion does.[17][17] Braintrust, Notion case study Vendor
    • Hold back a test set that never feeds training.[2][2] Shopify Engineering, Building production-ready agentic systems Primary

    Judges and their calibration

    • Write one narrow judge per failure mode.[18][18] Husain and Shankar, evals FAQ Secondary
    • Measure each judge against expert labels with kappa or F1.
    • Compare the judge with human-human agreement; Shopify used 0.69 as its ceiling.[2][2] Shopify Engineering, Building production-ready agentic systems Primary
    • Audit judges on a schedule.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary
    • Publish agreement statistics with every result.

    User simulators

    • Use simulators for multi-turn products and agents.
    • Seed the simulator with real user goals, as Shopify does.[2][2] Shopify Engineering, Building production-ready agentic systems Primary
    • Treat simulator results as a pre-release screen; they do not replace real-user tests.

  • Release gates

    • "No regression on the core suite against production".
    • Hard floors for safety and compliance tests that do not trade against quality gains.
    • Quality and cost in the same decision.[42][42] Maginative, Klarna dials back AI customer service Secondary
    • A written decision owner for conflicting metrics.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary
    • Sequential tests for expert trials, so a team can stop early with valid statistics.[8][8] Abridge, The science of AI evaluation Vendor

    Online monitoring and A/B tests

    • Roll out to staff first, then to a small share of users.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary[6][6] AI Expert Network, AI at ING Secondary
    • Track behaviour signals such as accept rate, edits, resolution and escalation.[20][20] Cursor, Online RL for Tab Primary[8][8] Abridge, The science of AI evaluation Vendor
    • Run daily regression against the live model to catch silent provider changes.[1][1] GitHub, How we evaluate models for GitHub Copilot Primary
    • Run A/B tests for major changes, as ING did.[6][6] AI Expert Network, AI at ING Secondary
    • Fix metric definitions, and version them when they must change.[22][22] eesel.ai, Fin response quality Vendor
    • Monitor fairness metrics where decisions affect customers; EIOPA expects this for insurers.[11][11] EIOPA, Opinion on AI governance and risk management Primary

    Costs

    • Budget for expert labels, judge compute and engineering time; no survey gives a share of AI budget.
    • Run cheap deterministic checks on all traffic and expensive judges on samples.[18][18] Husain and Shankar, evals FAQ Secondary
    • Re-run the full suite on each new model release; Notion ships new frontier models within 24 hours because its suite runs fast.[17][17] Braintrust, Notion case study Vendor
    • The OpenAI Morgan Stanley page and the McKinsey ING page did not load; their content comes from search summaries.
    • No primary JPMorgan or Morgan Stanley document gives set sizes, judge use or thresholds.
    • No public eval detail for Goldman Sachs, Allianz, Zurich, Lemonade, KPMG, Epic or Ambience; Epic's open-source "Seismometer" was not checked.
    • No primary eval write-up from Duolingo, Stripe or Ramp.
    • Whether US bank regulators issued the planned request for information on AI.
    • The full ECB ML section, the EBA's 2023 report on ML in IRB models and New York's Circular Letter No. 7 (2024) were not read.
    • Whether the NAIC adopted its AI evaluation tool at the 2026 Fall National Meeting.
    • The PCCP guidance date (December 2024 or August 2025).
    • The Fierce Healthcare article of February 2026, the official MDCG 2025-6 text and the NY Comptroller's LL144 report could not be read.
    • Whether any 2026 OMB memo replaced M-25-21 or M-25-22.
    • Eval spend as a share of AI budget, eval headcount norms, and a systematic dataset of enterprise eval failures.

Sources

Every number keeps its date and source. Badges mark source quality: primary, secondary, vendor (a claim about the publisher's own product), single source, or estimate.

  1. [1]
    GitHub, How we evaluate models for GitHub Copilot. Primary github.blog
  2. [2]
    Shopify Engineering, Building production-ready agentic systems. Primary shopify.engineering
  3. [3]
    Claude customer story, Intercom. Vendor-authored. Vendor claude.com
  4. [4]
    OpenAI customer story, Morgan Stanley. Vendor-authored; snippet only. Vendor openai.com
  5. [5]
    The Muse, JPMorgan model risk job listing. Snippet only. Single source themuse.com
  6. [6]
    AI Expert Network, AI at ING. Snippet only. Secondary aiexpert.network
  7. [7]
    BNN Bloomberg / AP, Deloitte to partially refund Australian government. Snippet only. Secondary bnnbloomberg.ca
  8. [8]
    Abridge, The science of AI evaluation. Self-report. Vendor abridge.com
  9. [9]
    LangChain, State of Agent Engineering. Vendor-run survey, n = 1,340. Vendor langchain.com
  10. [10]
    Federal Reserve, SR 26-2 attachment. Primary federalreserve.gov
  11. [11]
    EIOPA, Opinion on AI governance and risk management. Primary eiopa.europa.eu
  12. [12]
    NYC DCWP, Automated employment decision tools. Primary nyc.gov
  13. [13]
    NY State Comptroller, Enforcement of Local Law 144. Not read in full. Primary osc.ny.gov
  14. [14]
    Fierce Healthcare, Inside CHAI's failed assurance labs. Snippet only. Secondary fiercehealthcare.com
  15. [15]
    Braintrust, customer stories. Vendor-authored. Vendor braintrust.dev
  16. [16]
    createwith.com, Intercom Monitors. Snippet only. Secondary createwith.com
  17. [17]
    Braintrust, Notion case study. Vendor-authored. Vendor braintrust.dev
  18. [18]
    Husain and Shankar, evals FAQ. Practitioner guide. Secondary hamel.dev
  19. [19]
    GOV.UK, Humphrey tool reviews consultation responses. Snippet only. Primary gov.uk
  20. [20]
    Cursor, Online RL for Tab. Company self-report. Primary cursor.com
  21. [21]
    Intercom blog, Intercom vs Zendesk. Snippet only. Vendor intercom.com
  22. [22]
    eesel.ai, Fin response quality. Competitor blog; snippet only. Vendor eesel.ai
  23. [23]
    CTO Magazine, AI in Morgan Stanley. Snippet only. Secondary ctomagazine.com
  24. [24]
    OpenAI, AI in the Enterprise. Not fetched. Vendor cdn.openai.com
  25. [25]
    InvestmentNews, JPMorgan joins the AI productivity war. Snippet only. Secondary investmentnews.com
  26. [26]
    Bloomberg, JPMorgan works with US regulators on generative AI pilots. Snippet only. Secondary bloomberg.com
  27. [27]
    CTO Magazine, ING AI chatbot. Snippet only. Secondary ctomagazine.com
  28. [28]
    McKinsey, How ING uses generative AI. Vendor-consultant; timed out. Vendor mckinsey.com
  29. [29]
    BBVA, Responsible AI and guardrails (Spanish). Snippet only. Primary bbva.com
  30. [30]
    Intelligent Insurer, AXA develops generative AI with Microsoft. Snippet only. Secondary intelligentinsurer.com
  31. [31]
    Microsoft customer story, AXA. Snippet only. Vendor microsoft.com
  32. [32]
    InsuranceNewsNet, Red teaming for insurer AI risk. Snippet only. Secondary insurancenewsnet.com
  33. [33]
    OfficeChai, McKinsey Lilli. Snippet only. Secondary officechai.com
  34. [34]
    AI Expert Network, McKinsey Lilli. Snippet only. Secondary aiexpert.network
  35. [35]
    Computer Weekly, Humphrey AI for council admin. Snippet only. Secondary computerweekly.com
  36. [36]
    Accountant.nl, Court of Audit on central government AI. Snippet only. Secondary accountant.nl
  37. [37]
    Binnenlands Bestuur, Government often skips AI risk assessment. Snippet only. Secondary binnenlandsbestuur.nl
  38. [38]
    NOS, Court of Audit AI report. Snippet only. Secondary nos.nl
  39. [39]
    Abridge, Hallucinations whitepaper. Vendor abridge.com
  40. [40]
    HIT Consultant, Abridge hallucination results. Snippet only. Secondary hitconsultant.net
  41. [41]
    Frontiers in AI, Ambient AI note quality study. Snippet only. Primary public-pages-files-2025.frontiersin.org
  42. [42]
    Maginative, Klarna dials back AI customer service. Snippet only. Secondary maginative.com
  43. [43]
    Entrepreneur, Klarna CEO reverses course. Snippet only. Secondary entrepreneur.com
  44. [44]
    tech.co, Klarna reverses AI overhaul. Snippet only. Secondary tech.co
  45. [45]
    CFO Dive, Deloitte refunds report. Snippet only. Secondary cfodive.com
  46. [46]
    Let's Data Science, enterprise AI spend reaches $37bn. Snippet only. Secondary letsdatascience.com
  47. [47]
    Long-term Wiki, Menlo Ventures 2025 report entry. Snippet only. Secondary longtermwiki.com
  48. [48]
    AI2.work, enterprise AI budget scaling. Secondary; snippet only. Single source ai2.work
  49. [49]
    Federal Reserve, SR 26-2 letter. Primary federalreserve.gov
  50. [50]
    Baker Tilly, Updated interagency guidance on model risk management. Snippet only. Secondary bakertilly.com
  51. [51]
    Openlayer, SR 26-2 guide for AI teams. Snippet only. Vendor openlayer.com
  52. [52]
    CIMCON, SR 26-2 regulates your models, not your AI agents. Snippet only. Vendor cimcon.com
  53. [53]
    Monitaur, Governing generative and agentic AI under SR 26-2. Snippet only. Vendor monitaur.ai
  54. [54]
    Bank of England, SS1/23 model risk management principles. Primary bankofengland.co.uk
  55. [55]
    ECB, Revised guide to internal models. Snippet only. Primary bankingsupervision.europa.eu
  56. [56]
    A&O Shearman, ECB publishes revised guide to internal models. Snippet only. Secondary finreg.aoshearman.com
  57. [57]
    NAIC, AI model bulletin adoption map. Primary content.naic.org
  58. [58]
    Openlayer, NAIC model bulletin and AI governance. Snippet only. Vendor openlayer.com
  59. [59]
    actuary.info, NAIC AI bulletin and model law transition. Snippet only. Secondary actuary.info
  60. [60]
    Debevoise, Colorado extends AI regulation. Secondary debevoise.com
  61. [61]
    Colorado Division of Insurance, Bulletin B-10.004. Snippet only. Primary doi.colorado.gov
  62. [62]
    Akin Gump, FDA finalises PCCP guidance. Snippet only. Secondary akingump.com
  63. [63]
    RAPS, Final FDA guidance on PCCP. Snippet only. Secondary raps.org
  64. [64]
    CRS IF13245, FDA and AI-enabled devices. Snippet only. Primary everycrsreport.com
  65. [65]
    RAPS, FDA device centre guidance agenda for FY2027. Secondary raps.org
  66. [66]
    Fierce Healthcare, CHAI draft model card and lab certification. Snippet only. Secondary fiercehealthcare.com
  67. [67]
    Ground News, STAT on CHAI's assurance labs. Secondary ground.news
  68. [68]
    Becker's, Administration seeks to sideline CHAI. Snippet only. Secondary beckershospitalreview.com
  69. [69]
    Distilinfo, CHAI's promise meets reality. Snippet only. Secondary distilinfo.com
  70. [70]
    CHAI, home page. Primary chai.org
  71. [71]
    OpenRegulatory, MDCG 2025-6. Snippet only. Secondary openregulatory.com
  72. [72]
    Reuschlaw, Interplay between MDR, IVDR and the AI Act. Snippet only. Secondary reuschlaw.de
  73. [73]
    Garrigues, Digital Omnibus on AI published. Secondary garrigues.com
  74. [74]
    GLACIS, Guide to New York AI rules. Secondary, not checked against the report. Single source glacis-landing.pages.dev
  75. [75]
    Beancount.io, NYC Local Law 144 guide for 2026. Snippet only. Secondary beancount.io
  76. [76]
    EASA, NPA 2025-07. Snippet only. Primary easa.europa.eu
  77. [77]
    EASA news, First regulatory proposal on AI in aviation. Snippet only. Primary easa.europa.eu
  78. [78]
    ICAO DGCA61 paper, EASA approach to AI in aviation. Snippet only. Primary icao.int
  79. [79]
    OMB M-25-21 (3 Apr 2025). Primary whitehouse.gov
  80. [80]
    OMB M-25-22 (3 Apr 2025). Primary whitehouse.gov
  81. [81]
    Covington Inside Privacy, Updated AI model contractual clauses. Secondary insideprivacy.com