Open source runs the eval stack, but only its lower layers have settled

Only the bottom of the eval stack has settled

Every evaluation passes through the same seven layers, whatever the tool. The sandbox and the harness have settled. Grading, statistics and reporting have not.

Every evaluation follows the same path. A team writes tasks and runs them in a sandbox through a harness. Graders score each answer, statistics add error bars, traces keep the evidence, and reports show the result. Figure 1 shows each layer with its leading names and a status tag.

The lower layers have settled. Docker containers, OpenAI-compatible APIs and the two main harnesses work almost everywhere. Cloud sandbox vendors such as E2B, Modal and Daytona now compete as interchangeable back ends.^{[155][155] Inspect sandboxing docs UK AISI Primary}

The upper layers still change fast. Each tool grades, computes statistics, stores and reports in its own way. A different judge can change a ranking. The OpenTelemetry GenAI conventions still carry "Development" status.^{[142][142] GenAI conventions overview OpenTelemetry Primary}

Two harnesses carry model benchmarking, and Harbor carries agent tasks

A model-benchmark harness runs a fixed set of tasks against a model and reports scores. Labs, safety institutes and researchers use them to compare models.

UK AISI's Inspect AI leads for agentic, sandboxed and safety evals. It draws 2.9 million PyPI downloads a month, almost three times the count for lm-evaluation-harness.^{[3][3] inspect-ai downloads pypistats Primary[7][7] lm-eval downloads pypistats Primary} Its README lists over 200 ready-made evaluations.^{[21][21] Inspect AI README UK AISI Primary} Each evaluation combines a dataset, solvers and scorers, and MCP tools plug in directly.^{[22][22] Inspect docs folder UK AISI Primary}

Inspect has become a de facto verification standard. Epoch AI's Benchmarking Hub runs on Inspect and shows each question in the Inspect viewer.^{[27][27] Benchmarking Hub update Epoch AI Primary} Hugging Face marks a score as verified only after an Inspect run in HF Jobs.^{[28][28] Eval Results Hugging Face Hub docs Primary} Groq's openbench runs over 95 benchmarks on it.^{[25][25] openbench README Groq Primary}

EleutherAI's lm-evaluation-harness stays the standard for open weights. It defines each task as a YAML file and supports vLLM, SGLang, llama.cpp and OpenAI-compatible APIs.^{[30][30] lm-evaluation-harness README EleutherAI Primary} It also powered the Hugging Face Open LLM Leaderboard.

Harbor became the shared unit for agent benchmarks. A Harbor task is a small folder with an instruction, a Dockerfile and a test script.^{[32][32] create-task skill Harbor Primary} Its 86 adapters include SWE-bench, OSWorld, GAIA and HLE.^{[33][33] Adapters folder Harbor Primary} Inspect, verifiers, OpenEnv and NeMo Evaluator all import Harbor tasks. Harbor is the official harness for Terminal-Bench 2.0.^{[31][31] Harbor README Primary}

Inspect also ships the strongest statistics of any harness. It offers standard errors, bootstrap errors, Wilson intervals, clustered errors and Krippendorff's alpha for judge agreement.^{[29][29] Inspect metrics docs UK AISI Primary}

Academic leaderboard harnesses age fast. Six well-known projects froze or retired since mid-2024. Stanford's HELM entered maintenance mode on 1 June 2026.^{[39][39] HELM README Stanford CRFM Primary} OpenAI stopped updating simple-evals in July 2025.^{[41][41] simple-evals README OpenAI Primary} Hugging Face retired the Open LLM Leaderboard on 13 March 2025.^{[44][44] End of the Open LLM Leaderboard Hugging Face Primary}

The app eval library became a commodity; platforms now compete on traces and simulation

These tools test a team's own application or agent, not a bare model. Code-first libraries run tests in a script or in CI. Platforms add trace storage, dashboards, datasets and human review.

Every tool now ships the same core. All of them offer LLM judges, datasets, experiments and a pass-or-fail exit code for CI.^{[92][92] CI/CD docs promptfoo Primary[93][93] Experiments in CI/CD Langfuse Primary} Most gates compare a pass rate with a fixed threshold. Few tools say with statistical confidence whether a change made the agent worse.

Only MLflow ships a paired test. Version 3.18.0, from early November 2026, compares two runs with McNemar or Wilcoxon tests and a bootstrap interval.^{[86][86] Comparing evaluation runs MLflow Primary} No other platform ships a paired test today.

Download counts favour platforms with production SDKs. Langfuse draws 23.6 million PyPI downloads a month, and MLflow 19.9 million.^{[59][59] langfuse downloads pypistats Primary[61][61] mlflow downloads pypistats Primary} Both counts include SDK installs in production apps, so they overstate pure eval use. Among code-first libraries, autoevals and DeepEval lead.^{[55][55] autoevals downloads pypistats Primary[49][49] deepeval downloads pypistats Primary}

Platforms now compete on three things. They compete on trace storage at scale, on simulated users and on scoring agent trajectories. DeepEval ships a conversation simulator, and LangWatch Scenario tests agents against a simulated user and a judge.^{[83][83] Scenario README LangWatch Primary} LangChain's agentevals scores trajectories with strict, unordered, subset and superset match modes.^{[79][79] agentevals README LangChain Primary}

Three gaps stay common. Few tools test a pure black-box agent without code. Few ship stateful copies of business tools. Only MLflow runs a paired test between a candidate and a baseline.

Some popular libraries have slowed. Ragas made RAG metrics such as faithfulness popular, but its last commit dates from 24 February 2026.^{[76][76] Ragas commits GitHub Primary} Giskard rewrote its library as v3 for multi-turn agents and no longer maintains v2.^{[77][77] OSS README Giskard Primary} LangChain archived its langsmith-sdk monorepo on 16 October 2026; the platform stays closed.^{[69][69] langsmith-sdk GitHub Primary}

Eval environments and RL training environments merged into one layer

An environment gives an agent a world to act in. It holds tools, data and rules, and it scores what the agent does. In 2026 the same environment often serves for evaluation and for RL training.

OpenEnv offers one step API for any environment. It exposes reset(), step() and state() over a WebSocket, and each environment is a Docker image.^{[106][106] OpenEnv README Primary} A twelve-member technical committee now governs it, including Meta-PyTorch, Hugging Face, Nvidia, Microsoft and Modal. Version 0.12.0 shipped on 5 November 2026.^{[94][94] OpenEnv GitHub API Primary}

verifiers links environments to training. Prime Intellect's library defines tasksets, harnesses and agents, and each rollout produces a trace.^{[108][108] verifiers overview Prime Intellect Primary} A HarborTaskset loads Harbor tasks such as Terminal-Bench 2.^{[109][109] verifiers Harbor docs Prime Intellect Primary} The library draws 762,000 PyPI downloads a month.^{[97][97] verifiers downloads pypistats Primary}

The pieces mostly cooperate. Harbor defines tasks and sandboxes, and OpenEnv defines the step API and hosting. verifiers adds rubrics and training links, and Inspect adds logging and scoring. The story below follows one task through both uses.

Simulated users are maturing in two places. Sierra's tau2-bench v2.0 tests customer-service agents in six languages and full-duplex voice. Its text and voice sets hold 900 tasks each.^{[111][111] tau2-bench README Sierra Primary} Meridian Labs' Petri runs an auditing agent against a target model and scores transcripts with a rubric.^{[117][117] Petri README Meridian Labs Primary}

Four open red-team tools lead, and their owners now matter as much as their features

Red-team tools attack a model or an agent on purpose. They look for jailbreaks, prompt injection, data leaks and unsafe tool use.

promptfoo has the widest plugin set and the most users. It holds 72 red-team plugin docs, including excessive agency, MCP attacks and memory poisoning.^{[124][124] Red-team plugins folder promptfoo Primary} Domain packs cover insurance, finance and medicine. By default it generates attacks through a remote service, and its docs rate local generation quality as generally low.^{[125][125] Red-team configuration promptfoo Primary}

garak and PyRIT suit code-driven testing. NVIDIA's garak combines probes, detectors and harnesses, and a REST generator connects it to any endpoint.^{[126][126] garak README NVIDIA Primary} Microsoft's PyRIT composes single-turn, multi-turn and compound attacks in Python, over HTTP, WebSockets or Playwright.^{[128][128] PyRIT README Microsoft Primary} DeepTeam lists over 50 vulnerabilities and scores them locally with LLM judges.^{[129][129] DeepTeam README Confident AI Primary}

Owners now shape the choice. OpenAI owns promptfoo, which may worry buyers who test competing models. Microsoft and NVIDIA back PyRIT and garak, so these two make the safest long-term bets for neutrality.

OpenTelemetry won the transport, but the vocabulary and result formats remain open

OpenTelemetry carries traces in almost every eval platform. On top of it, attribute vocabularies, result formats and tool protocols are at different stages.

OTLP ingestion became normal. Langfuse, MLflow, promptfoo, TruLens and DeepEval all accept or export OpenTelemetry traces. Two attribute vocabularies compete on top of the transport.

OpenInference leads in practice. Arize's vocabulary draws 8.6 million monthly downloads.^{[138][138] OpenInference conventions downloads pypistats Primary} It instruments the OpenAI Agents SDK, the Claude Agent SDK, LangChain and AutoGen, and ships a bridge for OpenLLMetry.^{[147][147] OpenInference README Primary} The vocabularies therefore seem to converge.

The official GenAI conventions are still a draft. They moved out of the core OpenTelemetry repo in 2026 and cover inference, retrieval, agent spans and MCP.^{[141][141] Core GenAI README OpenTelemetry Primary[142][142] GenAI conventions overview OpenTelemetry Primary} Their status stays "Development", and the schema URL section still reads "TODO".

A neutral slot for eval scores now exists. The gen_ai.evaluation.result event carries a name, a score, a label and an explanation on the evaluated span.^{[145][145] GenAI events OpenTelemetry Primary} It is the first neutral way to put eval scores on production traces. Judge model, rubric version and confidence intervals still need custom attributes.

No result format has won. Inspect writes binary .eval logs, about one eighth the size of JSON, with access to single samples.^{[149][149] Inspect eval logs docs UK AISI Primary} Many projects read or export this format, so it acts as the richest de facto format. The neutral alternative barely registers.

ClickHouse became the default trace store. Langfuse, Opik and self-managed Weave all run on it.^{[84][84] Self-hosting docs Langfuse Primary[87][87] Self-host architecture Opik Primary[89][89] Weave self-managed docs CoreWeave Primary} ClickHouse also bought Langfuse in January 2026. Datasets mostly live on Hugging Face, and Langfuse versions its datasets on every change.^{[148][148] Datasets docs Langfuse Primary}

MCP became the common way for eval agents to reach tools. Inspect, verifiers, OpenEnv, Harbor and promptfoo all support MCP tools. The spec has five dated versions, from November 2024 to July 2026.^{[152][152] Schema folder MCP Primary} The project is moving from MIT to Apache-2.0 for code and spec.^{[153][153] LICENSE MCP Primary}

Docker is the sandbox contract, but judges and statistics still lack standards

Three supporting layers decide how far readers can trust a result. They cover where the agent runs, who grades it and how sure the number is.

Docker became the universal sandbox contract. Inspect ships Docker and local sandboxes, and external packages add Kubernetes, Daytona, Modal, EC2 and Proxmox.^{[155][155] Inspect sandboxing docs UK AISI Primary} Harbor runs on nine cloud back ends, and OpenEnv runs each environment as a Docker image. Cloud vendors now compete as interchangeable back ends.

Check Daytona's licence before you build on it. Daytona has 71,377 GitHub stars, but the GitHub API reports no licence, and our research did not confirm one.^{[159][159] Daytona GitHub Primary}

Most harnesses call models through an OpenAI-compatible API. vLLM serves open models behind that API, with over 96,000 stars under Apache-2.0.^{[160][160] vLLM GitHub Primary} Teams can serve open judge models through vLLM to cut grading costs.

LLM judges now grade most open-ended answers. Every framework ships judge templates, and open guard models give a cheaper option.^{[161][161] Granite Guardian 4.1 8B model card IBM Primary[162][162] Qwen3Guard models Hugging Face API Primary[163][163] Llama Guard 4 model card Meta Primary} No open judge publishes human agreement on agent transcripts in a regulated domain. Most published numbers compare judges with GPT-4-class labels, not with human experts.

Averaging judges is a cheap gain. Composo found that averaging eight judge scores added 9.8 points on RewardBench 2. Three scores gave most of that gain.^{[164][164] Judge ensembling paper Composo Primary}

Statistics remain the weakest layer. Most CI gates still compare a pass rate with a fixed threshold. Inspect offers the best single-run intervals and supports early stopping.^{[29][29] Inspect metrics docs UK AISI Primary[165][165] Inspect early stopping docs UK AISI Primary} MLflow alone ships a paired test.^{[86][86] Comparing evaluation runs MLflow Primary} No tool combines paired tests, non-inferiority margins and monitoring over time.

Larger companies bought most eval platforms in 2026, and no open licence changed

2026 was the consolidation year for eval and observability tools. Data platforms, observability vendors, security vendors and model labs bought most independent players.

Seven deals touched open eval projects between January and October 2026. ClickHouse bought Langfuse on 16 January.^{[168][168] Langfuse acquisition ClickHouse Vendor} OpenAI bought promptfoo, ServiceNow bought Traceloop, and Cisco closed Galileo in May.^{[73][73] Joining OpenAI promptfoo Vendor[170][170] Joining ServiceNow Traceloop Vendor[171][171] Galileo acquisition Cisco Vendor} Dynatrace closed its purchase of Arize, the owner of Phoenix, on 1 October.^{[175][175] Dynatrace completes Arize deal engineering.com Secondary}

Buyers fall into three groups. ClickHouse, Dynatrace, Cisco with Splunk, and ServiceNow wanted traces and evals for their platforms. Palo Alto Networks, Check Point, Zscaler, A10 Networks and Snyk wanted AI security tools. OpenAI bought promptfoo, and Anthropic took the Humanloop team.^{[179][179] Humanloop team joins Anthropic Data Phoenix Secondary}

Most prices stayed private. Dynatrace paid a reported $915M for Arize, about $815M of it in cash.^{[173][173] Arize acquisition Dynatrace Vendor[174][174] Dynatrace and Arize ITPro Secondary} Calcalist reported $60–80M for Traceloop, but nobody confirmed it.^{[183][183] Traceloop price Calcalist Single source} Press estimates put the Deepchecks price at up to $20M.^{[184][184] Deepchecks price Dealroom citing Calcalist Estimate}

Ownership changed, but licences did not. No eval project in scope moved to a BSL or AGPL licence in 2025 or 2026. No acquirer changed an open licence after a deal. promptfoo stays MIT under OpenAI, and Langfuse's core stays MIT under ClickHouse.^{[74][74] promptfoo README Primary[168][168] Langfuse acquisition ClickHouse Vendor}

Evals now sit inside bigger suites. Buyers should expect roadmaps to bend toward the parent platform. Several projects also moved home without a sale: Harbor, OpenEnv, Petri and Ragas changed GitHub organisations.

Standardise on the plumbing; results, judges and statistics are still open ground

The table sums up what a team can treat as settled today and what still moves.

The lower layers are safe to standardise on. Docker, OpenAI-compatible APIs, OpenTelemetry and MCP will likely stay.

The harness choice depends on the job. Inspect suits agentic and safety work, lm-evaluation-harness suits open weights, and Harbor suits container tasks.

The upper layers still lack standards. Result formats, judge validation and contract-grade statistics remain open. Lab-owned red-team tools also raise a neutrality question.

Sources

Every number keeps its date and source. Badges mark source quality: primary, secondary, vendor (a claim about the publisher's own product), single source, or estimate.

1. [1]
   swebench downloads, pypistats. It gives the monthly PyPI download count for SWE-bench. Primary pypistats.org
2. [2]
   Inspect AI, GitHub. It gives stars, licence and release data. Primary github.com
3. [3]
   inspect-ai downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
4. [4]
   inspect_evals, GitHub. It gives stars, licence and release data. Primary github.com
5. [5]
   inspect-evals downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
6. [6]
   lm-evaluation-harness, GitHub. It gives stars, licence and release data. Primary github.com
7. [7]
   lm-eval downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
8. [8]
   Harbor, GitHub API. It gives stars, licence and the move from laude-institute. Primary api.github.com
9. [9]
   harbor downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
10. [10]
    SWE-bench, GitHub. It gives stars, licence and push data. Primary github.com
11. [11]
    OpenCompass, GitHub. It gives stars, licence and release data. Primary github.com
12. [12]
    lighteval, GitHub. It gives stars, licence and release data. Primary github.com
13. [13]
    lighteval downloads, pypistats. It gives a monthly count that looks too low. Primary pypistats.org
14. [14]
    NeMo Evaluator, GitHub. It gives stars, licence and release data. Primary github.com
15. [15]
    openbench, GitHub. It gives stars, licence and release data. Primary github.com
16. [16]
    HELM, GitHub. It gives stars, licence and last push. Primary github.com
17. [17]
    OpenAI Evals, GitHub. It gives stars and push data. Primary github.com
18. [18]
    simple-evals, GitHub. It gives stars and push data. Primary github.com
19. [19]
    BIG-bench, GitHub. It shows the archived repo and its last push. Primary github.com
20. [20]
    HAL harness, GitHub. It gives stars and push data. Primary github.com
21. [21]
    Inspect AI README, UK AISI. It describes the framework and its ready-made evals. Primary github.com
22. [22]
    Inspect docs folder, UK AISI. It covers tasks, scorers, agents, the agent bridge and MCP tools. Primary github.com
23. [23]
    Meridian Labs repos, GitHub search. It lists the Inspect add-ons and Petri projects. Primary github.com
24. [24]
    inspect_evals README. It names Generality Labs as maintainer and sets contributor rules. Primary github.com
25. [25]
    openbench README, Groq. It describes benchmarks built on Inspect. Primary github.com
26. [26]
    NeMo Evaluator README, NVIDIA. It lists interceptors, confidence intervals and Inspect export. Primary github.com
27. [27]
    Benchmarking Hub update, Epoch AI. It explains how the hub uses Inspect. Primary epoch.ai
28. [28]
    Eval Results, Hugging Face Hub docs. It defines a verified result as an Inspect run in HF Jobs. Primary huggingface.co
29. [29]
    Inspect metrics docs, UK AISI. It lists the error, interval and agreement metrics. Primary github.com
30. [30]
    lm-evaluation-harness README, EleutherAI. It covers YAML tasks, plugins, backends and named users. Primary github.com
31. [31]
    Harbor README. It describes agents, cloud providers and the Terminal-Bench 2.0 role. Primary github.com
32. [32]
    create-task skill, Harbor. It defines the task folder layout. Primary github.com
33. [33]
    Adapters folder, Harbor. It holds the 86 benchmark adapters. Primary github.com
34. [34]
    Terminal-Bench 1 README. It sends new users to Harbor. Primary github.com
35. [35]
    SWE-bench README. It covers the Docker harness, Modal runs and Multimodal v2. Primary github.com
36. [36]
    lighteval README, Hugging Face. It claims over 1,000 tasks. Primary github.com
37. [37]
    OpenCompass README. It lists the 2026 news, including AgentCompass. Primary github.com
38. [38]
    Capabilities Index package, Epoch AI. It describes the Bayesian item-response fit. Primary github.com
39. [39]
    HELM README, Stanford CRFM. It states the start of maintenance mode. Primary github.com
40. [40]
    Maintenance mode policy, HELM. It explains what the team still maintains. Primary crfm-helm.readthedocs.io
41. [41]
    simple-evals README, OpenAI. It announces the end of updates in July 2025. Primary github.com
42. [42]
    Evals README, OpenAI. It points users to the OpenAI Dashboard. Primary github.com
43. [43]
    HAL harness README, Princeton PLI. It states that the repo stays archived for reference. Primary github.com
44. [44]
    End of the Open LLM Leaderboard, Hugging Face. It announces the retirement on 13 March 2025. Primary huggingface.co
45. [45]
    Leaderboards docs, Hugging Face. It describes the Open LLM Leaderboard in the past tense. Primary huggingface.co
46. [46]
    promptfoo, GitHub. It gives stars, licence and release data. Primary github.com
47. [47]
    promptfoo downloads, npm. It gives the monthly npm download count. Primary api.npmjs.org
48. [48]
    DeepEval, GitHub. It gives stars, licence and push data. Primary github.com
49. [49]
    deepeval downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
50. [50]
    Ragas, GitHub. It gives stars, licence and the last push. Primary github.com
51. [51]
    Giskard, GitHub. It gives stars, licence and release data. Primary github.com
52. [52]
    openevals, GitHub. It gives stars, licence and release data. Primary github.com
53. [53]
    agentevals, GitHub. It gives stars, licence and push data. Primary github.com
54. [54]
    autoevals, GitHub. It gives stars, licence and release data. Primary github.com
55. [55]
    autoevals downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
56. [56]
    TruLens, GitHub. It gives stars, licence and release data. Primary github.com
57. [57]
    LangWatch Scenario, GitHub. It gives stars, licence and release data. Primary github.com
58. [58]
    Langfuse, GitHub. It gives stars, licence and release data. Primary github.com
59. [59]
    langfuse downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
60. [60]
    MLflow, GitHub. It gives stars, licence and push data. Primary github.com
61. [61]
    mlflow downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
62. [62]
    Opik, GitHub. It gives stars, licence and release data. Primary github.com
63. [63]
    opik downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
64. [64]
    Arize Phoenix, GitHub. It gives stars, licence and release data. Primary github.com
65. [65]
    W&B Weave, GitHub. It gives stars, licence and release data. Primary github.com
66. [66]
    Evidently, GitHub. It gives stars, licence and the last push. Primary github.com
67. [67]
    Agenta, GitHub. It gives stars and release data. Primary github.com
68. [68]
    Kiln, GitHub. It gives stars and push data. Primary github.com
69. [69]
    langsmith-sdk, GitHub. It records the archive date and the two new SDK repos. Primary github.com
70. [70]
    HTTP provider docs, promptfoo. It describes calls to any HTTP endpoint. Primary github.com
71. [71]
    Tracing docs, promptfoo. It describes the OTLP receiver and trajectory checks. Primary github.com
72. [72]
    GitHub Action docs, promptfoo. It describes the before-and-after comparison. Primary github.com
73. [73]
    Joining OpenAI, promptfoo. It announces the deal and the user claims. Vendor promptfoo.dev
74. [74]
    promptfoo README. It states that Promptfoo now belongs to OpenAI and stays MIT-licensed. Primary github.com
75. [75]
    CI/CD docs, DeepEval. It describes unit tests in CI. Primary github.com
76. [76]
    Ragas commits, GitHub. It shows the last commit on 24 February 2026. Primary github.com
77. [77]
    OSS README, Giskard. It describes the v3 rewrite and the end of v2. Primary github.com
78. [78]
    openevals README, LangChain. It describes the ready-made judges. Primary github.com
79. [79]
    agentevals README, LangChain. It describes trajectory judges and match modes. Primary github.com
80. [80]
    autoevals repo, Braintrust. It holds the scorer library. Primary github.com
81. [81]
    Braintrust Series B, SiliconANGLE. It reports the $80M round, as a secondary source. Secondary siliconangle.com
82. [82]
    TruLens README. It describes OpenTelemetry tracing and feedback functions. Primary github.com
83. [83]
    Scenario README, LangWatch. It describes scenarios, simulated users and judges. Primary github.com
84. [84]
    Self-hosting docs, Langfuse. It lists PostgreSQL, Redis, ClickHouse and S3. Primary langfuse.com
85. [85]
    llms.txt, Langfuse. It lists the OpenTelemetry endpoint and the docs. Primary langfuse.com
86. [86]
    Comparing evaluation runs, MLflow. It describes the paired tests between two runs. Primary github.com
87. [87]
    Self-host architecture, Opik. It lists ClickHouse, MySQL, Redis and MinIO. Primary github.com
88. [88]
    LICENSE, Arize Phoenix. It sets the Elastic License 2.0 terms. Primary github.com
89. [89]
    Weave self-managed docs, CoreWeave. It describes Weave on ClickHouse under CoreWeave. Primary docs.coreweave.com
90. [90]
    LICENSE, Agenta. It puts the ee/ folder under separate terms. Primary github.com
91. [91]
    Kiln README. It describes the MIT library and the fair-code app. Primary github.com
92. [92]
    CI/CD docs, promptfoo. It describes exit-code gates. Primary github.com
93. [93]
    Experiments in CI/CD, Langfuse. It describes regression errors in CI. Primary langfuse.com
94. [94]
    OpenEnv, GitHub API. It gives stars, licence and the move from meta-pytorch. Primary api.github.com
95. [95]
    openenv downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
96. [96]
    verifiers, GitHub. It gives stars, licence and release data. Primary github.com
97. [97]
    verifiers downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
98. [98]
    community-environments, GitHub. It gives stars, licence and push data. Primary github.com
99. [99]
    tau2-bench, GitHub. It gives stars, licence and release data. Primary github.com
100. [100]
     AppWorld, GitHub. It gives stars, licence and release data. Primary github.com
101. [101]
     appworld downloads, pypistats. It gives a monthly count that likely includes CI pulls. Primary pypistats.org
102. [102]
     BrowserGym, GitHub. It gives stars and release data. Primary github.com
103. [103]
     AgentLab, GitHub. It gives stars and release data. Primary github.com
104. [104]
     OSWorld, GitHub. It gives stars, licence and the last release. Primary github.com
105. [105]
     inspect_petri, GitHub API. It gives stars, licence and the move from safety-research. Primary api.github.com
106. [106]
     OpenEnv README. It describes the API, integrations, MCP use and governance. Primary github.com
107. [107]
     verifiers README, Prime Intellect. It describes the library and its links to the Hub. Primary github.com
108. [108]
     verifiers overview, Prime Intellect. It defines tasksets, harnesses, agents and toolsets. Primary github.com
109. [109]
     verifiers Harbor docs, Prime Intellect. It describes loading Harbor tasks. Primary github.com
110. [110]
     Post on X, Prime Intellect. It claims over 400 crowdsourced environments in October 2025. Single source threadreaderapp.com
111. [111]
     tau2-bench README, Sierra. It describes version 2.0, languages, voice and task counts. Primary github.com
112. [112]
     AppWorld README, Stony Brook NLP. It describes the app world and the award. Primary github.com
113. [113]
     BrowserGym README, ServiceNow. It lists the bundled web benchmarks. Primary github.com
114. [114]
     AgentLab README, ServiceNow. It describes parallel experiments with Ray. Primary github.com
115. [115]
     OSWorld README, XLANG Lab. It describes OSWorld-Verified and the VM options. Primary github.com
116. [116]
     ControlArena repository. It describes the AI control library from UK AISI and Redwood. Primary github.com
117. [117]
     Petri README, Meridian Labs. It describes the auditing agent and the v3 API. Primary github.com
118. [118]
     garak, GitHub. It gives stars, licence and release data. Primary github.com
119. [119]
     garak downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
120. [120]
     PyRIT, GitHub. It gives stars, licence and release data. Primary github.com
121. [121]
     pyrit downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
122. [122]
     DeepTeam, GitHub. It gives stars, licence and release data. Primary github.com
123. [123]
     deepteam downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
124. [124]
     Red-team plugins folder, promptfoo. It holds the 72 plugin docs. Primary github.com
125. [125]
     Red-team configuration, promptfoo. It describes remote and local attack generation. Primary github.com
126. [126]
     garak README, NVIDIA. It describes probes, detectors, harnesses and the REST generator. Primary github.com
127. [127]
     garak probes folder, NVIDIA. It holds the 44 probe modules. Primary github.com
128. [128]
     PyRIT README, Microsoft. It describes the scanner, the GUI and the targets. Primary github.com
129. [129]
     DeepTeam README, Confident AI. It lists the vulnerabilities and local scoring. Primary github.com
130. [130]
     AgentDojo, GitHub. It gives stars, licence and push data. Primary github.com
131. [131]
     agent-scan README, Snyk. It describes the scan and the token requirement. Primary github.com
132. [132]
     PINT benchmark, GitHub. It shows the archived repo. Primary github.com
133. [133]
     Project Moonshot repository. It shows the beta status, version and licence. Primary github.com
134. [134]
     OWASP AI Testing Guide. It gives a method for testing the trustworthiness of AI systems. Primary owasp.org
135. [135]
     Top 10 for Agentic Applications for 2026, OWASP. It lists the main security risks of agentic applications. Primary genai.owasp.org
136. [136]
     OTel GenAI conventions, GitHub API. It gives the creation date, stars and licence. Primary api.github.com
137. [137]
     OpenInference, GitHub. It gives stars and licence. Primary github.com
138. [138]
     OpenInference conventions downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
139. [139]
     OpenLLMetry, GitHub. It gives stars, licence and release data. Primary github.com
140. [140]
     traceloop-sdk downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
141. [141]
     Core GenAI README, OpenTelemetry. It states that the GenAI conventions moved to a new repo. Primary github.com
142. [142]
     GenAI conventions overview, OpenTelemetry. It lists the scope and the status. Primary github.com
143. [143]
     Semantic conventions releases, OpenTelemetry. It lists v1.45.0. Primary github.com
144. [144]
     GenAI agent spans, OpenTelemetry. It defines the agent and workflow spans. Primary github.com
145. [145]
     GenAI events, OpenTelemetry. It defines the evaluation result event. Primary github.com
146. [146]
     Semantic conventions, OpenInference. It lists the span kinds. Primary github.com
147. [147]
     OpenInference README. It lists the instrumentations and the OpenLLMetry bridge. Primary github.com
148. [148]
     Datasets docs, Langfuse. It describes versioning on every change. Primary langfuse.com
149. [149]
     Inspect eval logs docs, UK AISI. It describes the .eval format and storage options. Primary github.com
150. [150]
     Every Eval Ever README, EvalEval. It describes the schema, the database and the converters. Primary github.com
151. [151]
     every-eval-ever downloads, pypistats. It gives the monthly PyPI download count. Primary pypistats.org
152. [152]
     Schema folder, MCP. It lists the dated spec versions. Primary github.com
153. [153]
     LICENSE, MCP. It describes the move from MIT to Apache-2.0. Primary github.com
154. [154]
     MCP conventions, OpenTelemetry. It defines spans for MCP calls. Primary github.com
155. [155]
     Inspect sandboxing docs, UK AISI. It lists the built-in and external sandboxes. Primary github.com
156. [156]
     E2B, GitHub. It gives stars, licence and push data. Primary github.com
157. [157]
     gVisor, GitHub. It gives stars and licence. Primary github.com
158. [158]
     Modal client, GitHub. It gives stars and licence for the client SDK. Primary github.com
159. [159]
     Daytona, GitHub. It gives stars and the last push but no licence. Primary github.com
160. [160]
     vLLM, GitHub. It gives stars, licence and push data. Primary github.com
161. [161]
     Granite Guardian 4.1 8B model card, IBM. It gives criteria, scores and the licence. Primary huggingface.co
162. [162]
     Qwen3Guard models, Hugging Face API. It gives sizes and download counts. Primary huggingface.co
163. [163]
     Llama Guard 4 model card, Meta. It describes the taxonomy and the scores. Primary github.com
164. [164]
     Judge ensembling paper, Composo. It measures the gain from criteria and ensembles. Primary arxiv.org
165. [165]
     Inspect early stopping docs, UK AISI. It describes early stopping and optional stopping. Primary github.com
166. [166]
     LICENSE, Langfuse. It lists the ee/ folders outside the MIT licence. Primary github.com
167. [167]
     Enterprise LICENSE, LangWatch. It covers the closed enterprise features. Primary github.com
168. [168]
     Langfuse acquisition, ClickHouse. It announces the deal and the licence pledge. Vendor clickhouse.com
169. [169]
     Helicone acquisition, Mintlify. It announces the deal and the maintenance mode. Vendor mintlify.com
170. [170]
     Joining ServiceNow, Traceloop. It announces the deal and the open-source pledge. Vendor traceloop.com
171. [171]
     Galileo acquisition, Cisco. It confirms the close. Vendor cisco.com
172. [172]
     Portkey acquisition, Palo Alto Networks. It confirms the close. Vendor paloaltonetworks.com
173. [173]
     Arize acquisition, Dynatrace. It announces the deal and the Phoenix pledge. Vendor dynatrace.com
174. [174]
     Dynatrace and Arize, ITPro. It reports the $915M price. Secondary itpro.com
175. [175]
     Dynatrace completes Arize deal, engineering.com. It reports the close on 1 October 2026. Secondary engineering.com
176. [176]
     Invariant Labs acquisition, Snyk. It announces the deal. Vendor snyk.io
177. [177]
     Lakera acquisition, Check Point. It announces the deal. Vendor checkpoint.com
178. [178]
     Zscaler and SPLX, Help Net Security. It reports the deal. Secondary helpnetsecurity.com
179. [179]
     Humanloop team joins Anthropic, Data Phoenix. It reports the team move. Secondary dataphoenix.info
180. [180]
     Check Point and Deepchecks, Security Boulevard. It reports the purchase of the Deepchecks team and IP. Secondary securityboulevard.com
181. [181]
     TrojAI acquisition, A10 Networks. It announces the deal of 15 June 2026. Vendor businesswire.com
182. [182]
     Beacon Software and Haize Labs, Osler. It describes the deal and the new research group. Secondary osler.com
183. [183]
     Traceloop price, Calcalist. It reports an unconfirmed price range. Single source calcalistech.com
184. [184]
     Deepchecks price, Dealroom citing Calcalist. It estimates the price at up to $20M. Estimate dealroom.co