# Agent Reliability — full corpus

> Testing, benchmarking and auditing autonomous AI agents — methods, harnesses, evidence

License: CC-BY-4.0. Every claim lists its sources; confidence is declared per claim.

## Observability standards for agents (OpenTelemetry GenAI)

- id: agent-observability-standards · type: entity · evidence_tier: primary · confidence: 0.8 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/agent-observability-standards · api: https://agentreliability.dev/api/k/agent-observability-standards.json

The emerging standard layer for agent telemetry: OpenTelemetry's GenAI semantic conventions define spans, metrics and events for model calls, tool execution, MCP interactions and agent operations, so traces from different stacks become comparable. Still in active development — adopt it for vocabulary and interoperability, pin versions, and expect churn until stabilization.

Claims:
- OpenTelemetry maintains dedicated semantic conventions for generative AI — spans, metrics and events for GenAI clients, MCP and provider-specific operations — covering agent spans, tool execution and token usage. [confidence 0.9; sources: OpenTelemetry semantic conventions for generative AI]
- The GenAI conventions live in their own repository under Apache-2.0 and are in active development, with the schema URL still marked TODO — the vocabulary is adoptable, the stability guarantees are not yet. [confidence 0.85; sources: OpenTelemetry semantic conventions for generative AI]
- Framework backing for instrumenting deployed AI: NIST AI RMF makes Measure a core function across the lifecycle, which post-deployment telemetry operationalizes. [confidence 0.85; sources: NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1)]

## Release gates for agent systems

- id: agent-release-gates · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/agent-release-gates · api: https://agentreliability.dev/api/k/agent-release-gates.json

The ship/no-ship decision expressed as declared, machine-checkable thresholds instead of judgment calls: minimum external quality score, minimum factuality, minimum interface success rate, zero critical security findings. Declaring the gate in configuration makes it auditable and Goodhart-visible; a gate that lives in someone's head renegotiates itself under deadline pressure.

Claims:
- A production example of declared gates: this instance's blueprint specifies minimum external quality (CAS-E 85), minimum factuality (0.98), minimum MCP interface success (0.99) and zero critical security findings as release conditions, in version-controlled configuration approved by a human gate. [confidence 0.9; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]
- Framework backing for threshold governance: NIST AI RMF's Manage function requires risks to be prioritized and acted on with defined processes across the lifecycle, which declared release thresholds operationalize. [confidence 0.85; sources: NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1)]
- Gate metrics must be the panel kind: consistency across repeated trials (pass^k) rather than single-run success, or the gate certifies demos instead of deployments. [confidence 0.85; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]

## Agent reliability FAQ

- id: agent-reliability-faq · type: faq · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/agent-reliability-faq · api: https://agentreliability.dev/api/k/agent-reliability-faq.json

The questions teams actually ask before shipping an autonomous agent: how to stop hallucinated actions against critical APIs, what a benchmark should measure, whether one successful demo means anything, and where to start when there is no eval at all. Answers are bound to sources, not opinion.

Claims:
- Hallucinated actions are caught by grading state, not prose: comparing the final state of the acted-on system against an annotated goal state detects wrong actions that a plausible transcript hides. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- Until verified, an agent belongs in a contained environment; execution-graded evaluation presumes an execution environment the agent cannot escape or corrupt. [confidence 0.85; sources: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?]
- An agent benchmark should measure consistency across repeated trials, not just single-attempt success: pass^k exposes agents that pass once but fail the user who retries. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- Capability trend and reliability are different measurements: the length of tasks agents can complete has grown exponentially, which says nothing about consistency on any one task. [confidence 0.85; sources: METR — Model Evaluation & Threat Research]
- Before building a benchmark, survey the shelf: a public catalog tracks 69 agentic benchmarks and the field's move from measuring capability to measuring agency. [confidence 0.85; sources: EUREKA — applied AI, agents and robotics knowledge platform]
- Model swaps run on the provider's calendar: deprecated models get as little as 60 days between notice and retirement, after which requests fail — so a frozen regression suite for model upgrades is standing infrastructure, not a one-off project. [confidence 0.9; sources: Anthropic model deprecations (Claude API documentation)]
- A score difference smaller than the eval's noise floor is not a finding: eval questions behave as samples from a larger population and comparisons need standard errors — reported noise from formatting choices alone has reached about 5 points. [confidence 0.85; sources: Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations; Challenges in evaluating AI systems]
- Run each task as many times as it runs unreviewed in production, and read the right metric: all-of-k reliability collapses with repetition (pass^8 under 25% where single attempts clear 50%), while any-of-k capability climbs with it (28.8% to 70.2% from 1 to 100 samples) — so the same suite supports opposite conclusions depending on which one you quote. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains; Evaluating Large Language Models Trained on Code]

## Agent reliability glossary

- id: agent-reliability-glossary · type: glossary · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/agent-reliability-glossary · api: https://agentreliability.dev/api/k/agent-reliability-glossary.json

Working definitions for the terms this instance uses everywhere: eval, harness, canary, red-teaming, regression gate, pass^k, fault injection, sandbox. Each definition is bound to a source that uses the term operationally, so the glossary states how the field works rather than how a dictionary reads.

Claims:
- An eval is a structured, repeatable test that measures an LLM or LLM-based system against a defined dimension; frameworks package evals as registries of reusable templates. [confidence 0.9; sources: openai/evals — framework for evaluating LLMs and LLM systems]
- An evaluation harness is the fixed machinery that presents tasks, captures behavior and computes verdicts; it is part of the measurement — formatting details alone have shifted benchmark scores by about 5%. [confidence 0.85; sources: Challenges in evaluating AI systems]
- A canary is a continuously-run probe with a known-good baseline whose deviation signals trouble early — the operational descendant of the chaos-engineering steady-state hypothesis. [confidence 0.85; sources: Principles of Chaos Engineering]
- Red-teaming is adversarial evaluation: probing a system for harmful or dangerous capability — persuasion, cyber offense, self-proliferation — under hostile framing instead of benign-path testing. [confidence 0.85; sources: Evaluating Frontier Models for Dangerous Capabilities]
- A regression gate is an automated check that blocks a change when previously-verified behavior stops holding; eval frameworks exist largely to detect how model or system versions change application behavior. [confidence 0.8; sources: openai/evals — framework for evaluating LLMs and LLM systems]
- pass^k is the probability that a system succeeds on all k repeated trials of the same task — a reliability metric that separates consistency from single-attempt capability, introduced with tau-bench. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- Fault injection is deliberately introducing realistic disruptions — server failures, network outages, degraded dependencies — to verify a system's behavior against a steady-state baseline, per the chaos engineering method. [confidence 0.85; sources: Principles of Chaos Engineering]
- A sandbox is a contained execution environment for untrusted code and agent actions; evaluation frameworks implement it with backends such as Docker and Kubernetes so tested behavior cannot reach production systems. [confidence 0.85; sources: Inspect — evaluation framework for large language models]

## Agent telemetry and actor classification

- id: agent-telemetry-actor-classification · type: entity · evidence_tier: secondary · confidence: 0.8 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/agent-telemetry-actor-classification · api: https://agentreliability.dev/api/k/agent-telemetry-actor-classification.json

Instrumenting a deployed agent so every action is attributable: which actor (human, autonomous agent, crawler, operator) initiated the request, which tools were called, what state changed. Actor classification is the prerequisite for everything downstream — rate limits, audits, incident forensics, and knowing whether a regression hit humans or machines.

Claims:
- NIST AI RMF's Measure and Manage functions require ongoing measurement and management of AI risk across the lifecycle, including deployment and evaluation — post-deployment telemetry is framework-mandated, not optional polish. [confidence 0.85; sources: NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1)]
- Classifying each request by actor type (human, agent, crawler) and binding actions to actors is part of the production methodology behind this instance, whose consumers explicitly include AI agents and LLM crawlers alongside humans. [confidence 0.8; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]
- Monitoring effectiveness against autonomous AI systems is an active research area: METR publishes behavioral datasets and studies on it. [confidence 0.8; sources: METR — Model Evaluation & Threat Research]

## AgentBench

- id: agentbench · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/agentbench · api: https://agentreliability.dev/api/k/agentbench.json

A multi-environment benchmark evaluating LLMs as agents across eight distinct settings (operating system, database, knowledge graph, games, web tasks and more). Its durable findings: commercial frontier models act competently as agents while sub-70B open models lag sharply, and the failures concentrate in long-horizon reasoning, decision-making and instruction following — not in single-turn knowledge.

Claims:
- AgentBench evaluates LLMs as autonomous agents across eight distinct interactive environments rather than a single task family. [confidence 0.95; sources: AgentBench: Evaluating LLMs as Agents]
- AgentBench found top commercial LLMs show strong agent ability in complex environments while open-source models up to 70B trail by a significant margin, with key obstacles in long-term reasoning, decision-making and instruction following. [confidence 0.9; sources: AgentBench: Evaluating LLMs as Agents]

## Behavioral canaries

- id: behavioral-canaries · type: entity · evidence_tier: secondary · confidence: 0.8 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/behavioral-canaries · api: https://agentreliability.dev/api/k/behavioral-canaries.json

Small, fixed probe tasks run continuously against a deployed agent. Each canary has a known-good outcome; a change in that outcome is treated as an early regression signal — model drift, prompt rot, tool API changes — before users feel it. The chaos-engineering steady-state hypothesis applied to agent behavior instead of infrastructure metrics.

Claims:
- A behavioral canary is a probe task with a pinned expected outcome, executed on a schedule against the production agent; outcome drift is an alarm, and the practice is part of the production methodology behind this instance. [confidence 0.85; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]
- Chaos engineering grounds the pattern: define steady state as measurable output that indicates normal behavior, then treat deviations from that baseline as the signal worth alarming on. [confidence 0.9; sources: Principles of Chaos Engineering]

## Calibrating your LLM judge before it gates anything

- id: calibrating-your-llm-judge · type: guide · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/calibrating-your-llm-judge · api: https://agentreliability.dev/api/k/calibrating-your-llm-judge.json

The protocol that turns an LLM judge from vibes into an instrument: measure its agreement against a human-audited sample, probe its known biases directly (position, verbosity, self-preference), put error bars on every comparison it feeds, and define the triggers that force recalibration. A judge is promoted to gatekeeper only after this protocol — and demoted the day its judge model changes.

Claims:
- Strong LLM judges can reach over 80% agreement with human preferences — the same level humans reach with each other — which makes human-agreement measurement both the calibration method and the ceiling. [confidence 0.9; sources: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena]
- The failure modes to probe are documented: position bias, verbosity bias, self-enhancement bias, and limited reasoning ability on hard grading tasks. [confidence 0.95; sources: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena]
- Judge-fed comparisons need uncertainty accounting like any eval: treat graded items as samples from a larger population and report standard errors, or noise reads as verdicts. [confidence 0.85; sources: Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations]
- Model-graded evaluation is a first-class template in open eval frameworks, so the calibration protocol wraps standard tooling rather than bespoke infrastructure. [confidence 0.9; sources: openai/evals — framework for evaluating LLMs and LLM systems]

Takeaways:
- A judge's verdicts mean nothing until its agreement with humans is measured on your domain — published numbers do not transfer.
- Probe each documented bias directly - swap positions, control length, test self-preference - instead of hoping they average out.
- Human-level agreement (~80%) is the ceiling, not a defect; design gates that tolerate it.
- Recalibrate on every judge-model change and every prompt change to the judge — the instrument changed.

## Why calibrate at all

A judge model will always produce a score. The question a gate has to
answer is whether that score tracks anything. Published agreement
numbers (claim c1) were measured on other domains, other prompts,
other judge models — none of it transfers to your setup automatically.
Calibration is the transfer measurement.

## The protocol

1. **Build the audit sample.** Pull 100-200 real graded items from
   your own pipeline, stratified across the score range — not the easy
   tail. Have humans grade them blind, with the same rubric the judge
   gets. This human layer is the reference; budget for disagreement
   review between annotators.
2. **Measure agreement.** Compare judge verdicts against the human
   reference. Expect the ceiling around human-human agreement (claim
   c1): a judge at 80% agreement with a human panel that agrees 81%
   with itself is calibrated, not broken.
3. **Probe each documented bias (claim c2).**
   - *Position*: re-grade pairwise items with the order swapped; the
     flip rate is your position bias.
   - *Verbosity*: grade length-controlled variants of the same
     content; score drift with length is the verbosity bias.
   - *Self-preference*: if the judge model also generates candidates,
     measure the delta it awards its own outputs.
   - *Reasoning limit*: check agreement separately on the hardest
     stratum; judges degrade there first.
4. **Bound the noise (claim c3).** Re-run the judge on the same items;
   the run-to-run variance plus the sampling error defines the minimum
   score difference that means anything. Publish that number next to
   every judge-fed comparison.
5. **Write the promotion decision.** The judge may gate only the
   decisions whose required precision its measured agreement supports.
   Everything else it reports as telemetry.

## Recalibration triggers

Recalibrate — not spot-check — when any of these change: the judge
model (including provider-forced upgrades), the judge prompt or
rubric, the task distribution being graded, or the candidate model
family. The instrument changed; its old calibration is a memory. Wire
the trigger into the same pipeline that catches model deprecations.

## What this does not fix

Calibration bounds a judge; it does not make it deterministic. Ship
gates that need reproducible verdicts stay on execution-based grading
(see the harness-vs-judge comparison). The calibrated judge earns the
open-ended slice — style, helpfulness, judgment — and the standard
tooling makes running it cheap (claim c4). The expensive part was
never the API call; it is the human reference sample, and that cost
is the honest price of trusting a model's opinion.

## Check-counter regression alarm

- id: check-counter-alarm · type: entity · evidence_tier: primary · confidence: 0.9 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/check-counter-alarm · api: https://agentreliability.dev/api/k/check-counter-alarm.json

A meta-guardian that watches the guardians: the validator exports the number of distinct check families it runs, and CI tracks that counter. If a refactor silently drops a check, the count falls and the alarm fires — catching the failure mode where protection disappears while every remaining light stays green.

Claims:
- The Citarium validator exports its distinct check families as an explicit constant described in source as "the counter CI tracks", so a drop in the number of checks is itself a detectable regression. [confidence 0.95; sources: Citarium content validator source (@citarium/core v0.1.0, compiler/validate.ts)]
- The check-counter regression alarm is listed among the Citarium methodology's living case studies with real incident history. [confidence 0.9; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]

## Choosing k and the threshold for pass^k

- id: choosing-k-for-pass-hat-k · type: guide · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-09
- canonical: https://agentreliability.dev/k/choosing-k-for-pass-hat-k · api: https://agentreliability.dev/api/k/choosing-k-for-pass-hat-k.json

The corpus recommends pass^k everywhere and never says which k or what number to demand. This is the arithmetic and the decision rule: k comes from how many times the task runs between human reviews, the threshold comes from the cost of one failure, and both must be set before you see the score. Includes why pass@k and pass^k move in opposite directions and why measured pass^k is almost never p^k.

Claims:
- pass^k is the probability that a system succeeds on all k repeated trials of the same task, introduced with tau-bench as a reliability metric distinct from single-attempt capability. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- The gap between capability and reliability is large in practice: state-of-the-art function-calling agents succeeded on under 50% of tau-bench tasks, and pass^8 fell under 25% in the retail domain. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- Repeated sampling moves the opposite metric in the opposite direction: on HumanEval, Codex solved 28.8% of problems with one sample and 70.2% with 100 samples per problem, because any-of-k credits a single success. [confidence 0.95; sources: Evaluating Large Language Models Trained on Code]
- A pass^k estimate is a sample statistic and needs uncertainty accounting like any eval score: treat graded items as draws from a larger population and report standard errors alongside the point estimate. [confidence 0.85; sources: Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations]
- Incidental setup details can move eval scores by around 5 percentage points on their own, so a threshold quoted without a stated harness and prompt configuration is not a reproducible gate. [confidence 0.85; sources: Challenges in evaluating AI systems]

Takeaways:
- k is not a compute-budget choice - it is how many times the task runs between human reviews.
- pass@k rises with k and pass^k falls with k; they answer opposite questions, so never quote one as the other.
- Set the threshold from the cost of a single failure before you see the score, or you will be negotiating with your own results.
- Measured pass^k is almost never the p^k the independence assumption predicts - correlated failures make it worse, and only measurement tells you which.

## Two metrics that look alike and point opposite ways

`pass@k` asks whether **any** of k attempts succeeded. `pass^k` asks
whether **all** k succeeded (claim c1). The names differ by one
character and the numbers move in opposite directions as k grows:
Codex went from 28.8% to 70.2% on HumanEval by sampling 100 times
instead of once (claim c3), while tau-bench agents that clear under 50%
of tasks on a single attempt drop below 25% at pass^8 in retail (claim
c2).

The distinction is what you are buying with the extra attempts. `pass@k`
is a capability measure and assumes something downstream can pick the
winning attempt — a test suite, a compiler, a human reviewer. If no such
filter exists in production, `pass@k` is a number about a system you are
not running. `pass^k` is the reliability measure: it describes what the
user experiences when the same task runs k times and nobody is checking.

## Choosing k

k is not a budget decision, and it is not "as many as we can afford".
**k is the number of times the task runs between human reviews.**

Work it out from your own traffic:

- A weekly report the agent generates and an analyst reads before it
  circulates: a human filter exists every run, so k = 1. Use pass@1 and
  spend your budget on the review, not on repeated trials.
- A support workflow that fires 200 times a day and is spot-checked
  weekly: roughly 1,400 unreviewed runs between reviews. You cannot
  measure pass^1400, but you now know your gate is about the tail, not
  the average — pick the largest k you can afford to measure and treat
  it as a lower bound on the exposure, not a description of it.
- A nightly migration step in a pipeline of 12 sequential agent calls,
  none reviewed until the pipeline finishes: k = 12, and the relevant
  question is whether the whole chain survives, not whether each step
  usually works.

Two corrections to the naive reading. First, the k that matters is
per-task-instance, not global volume: 200 runs of *different* tasks is a
sample of size 200 at k=1, not pass^200. Second, if a retry is automatic
and cheap and the failure is detectable, that retry belongs in the
system under test — measure the agent *with* its retry loop at the k
your reviews actually imply, rather than crediting a retry the harness
performs and production does not.

## Choosing the threshold

Set the number from the cost of one failure, and set it **before** you
see the score. The order matters more than the value: a threshold chosen
after the measurement is a description of what you got, not a gate.

The rule that survives contact with a release meeting: state the failure
budget in incidents, then convert. If the workflow runs N unreviewed
times per review cycle and you can tolerate at most one bad outcome per
cycle, you need a per-run failure rate below 1/N — and it is the
per-run rate, not the pass^k figure, that you should be arguing about.
pass^k is how you *observe* that rate under repetition; the failure
budget is what you are actually deciding.

Then write down, next to the number: the harness version, the prompt
configuration, the model version, and the task set. Incidental
configuration differences alone are worth several points of score
(claim c5), so an undocumented threshold cannot be re-checked next
quarter — and a gate you cannot re-run is a gate that quietly stops
gating.

## The independence trap

The arithmetic everyone does first: if each trial succeeds with
probability p, then pass^k = p^k. At p = 0.95, pass^10 ≈ 0.60. At
p = 0.99, pass^10 ≈ 0.90. This is worth doing once, because it explains
why a "95% reliable" agent feels broken over a working week and why
buying reliability at the third nine is the expensive part.

It is also wrong in a specific direction. Agent failures are not
independent draws: the same task hits the same ambiguous instruction,
the same brittle selector, the same tool that rate-limits at the same
point. Correlated failures cluster, so measured pass^k is typically
**below** p^k — and the clustering is itself the diagnosis, because a
task that fails 4 times out of 8 in one cluster and never in another
has a deterministic bug wearing a probabilistic costume. Use p^k to size
the sample and to argue for the budget; use the measured value to gate,
and look at the failure clustering before you look at the mean.

## The sample budget

A pass^k estimate over n tasks costs n × k runs, and the estimate has
error bars like any other (claim c4). Two consequences worth planning
for:

- **Rare events need runs you may not want to pay for.** Distinguishing
  a 1% failure rate from a 3% failure rate takes hundreds of trials per
  task. If the budget does not stretch, say so in the release note —
  "pass^8 = 0.94 ± 0.05, cannot distinguish from 0.90" is an honest
  gate; "pass^8 = 0.94" alone invites a comparison the data does not
  support.
- **Report the interval, gate on the bound.** Gate on the lower end of
  the confidence interval, not the point estimate. It is the only
  version of the number that stays true when the same suite runs again
  next week.

## Where this lands

Both selections — k and the threshold — belong in the release gate
configuration, declared and version-controlled next to the other
thresholds, not chosen per release by whoever is shipping. That is the
whole point of writing them down before the run: the gate holds when
the number comes back inconvenient.

## Designing an agent sandbox that actually contains failures

- id: designing-an-agent-sandbox · type: guide · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/designing-an-agent-sandbox · api: https://agentreliability.dev/api/k/designing-an-agent-sandbox.json

Requirements and checks for an execution environment where an agent can fail safely: no default egress, ephemeral seeded state, fake credentials that look real, a bounded blast radius, and an observation channel the agent cannot write to. Includes a containment checklist you can test the same way you test the agent.

Claims:
- Chaos engineering requires that the fallout from experiments be minimized and contained; blast radius is a design input, not an afterthought. [confidence 0.9; sources: Principles of Chaos Engineering]
- Execution-graded benchmarks presume a working environment: SWE-bench tasks require editing real codebases and interacting with execution environments to verify resolution. [confidence 0.85; sources: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?]
- Large-scale agent evaluation is infrastructure work — METR's Hawk platform, built on Inspect, exists specifically to run agent evaluations at scale. [confidence 0.85; sources: METR — Model Evaluation & Threat Research]
- Dangerous-capability evaluations cover cybersecurity and self-proliferation — behavior whose escape from the test environment is itself the incident being prevented. [confidence 0.85; sources: Evaluating Frontier Models for Dangerous Capabilities]

Takeaways:
- A sandbox with egress, shared state or real credentials is production with extra steps.
- Every containment property must be testable — and falsified once before it is trusted.
- Keep the grader and telemetry outside the boundary, read-only; an agent that can edit its transcript passes any transcript-graded eval.
- Adopt existing evaluation infrastructure; build only the fixtures and graders that encode your domain.

## What the sandbox is for

An agent under test will do the wrong thing — that is the point of
testing it. The sandbox's job is to make the wrong thing observable and
cheap. If a test failure can reach a production API, mutate shared
state, or exfiltrate a real credential, you do not have a sandbox; you
have production with extra steps.

## Containment properties (each one testable)

1. **No default egress.** The environment reaches only an explicit
   allowlist of endpoints, all of them fakes or fixtures. Test it: run
   an agent whose task is to call an unlisted endpoint; the call must
   fail visibly.
2. **Ephemeral, seeded state.** Every run starts from a fixture and
   throws its world away afterwards. Test it: run twice, diff the
   starting states; any difference is a leak.
3. **Fake credentials that look real.** The agent must not learn it is
   being tested from the shape of its secrets, and a leaked test secret
   must be worthless. Test it: plant the credential in output and verify
   the alarm fires and nothing real was exposed.
4. **Bounded blast radius (claim c1).** Enumerate, in writing, the worst
   thing a fully adversarial agent could do inside the environment. If
   the answer includes any production noun, redesign.
5. **An observation channel the agent cannot write to.** Telemetry and
   grading read from outside the environment. An agent that can edit its
   own transcript can pass any transcript-graded eval.

## Grading inside the sandbox

Execution-based grading is the reason the environment exists (claim
c2): the sandbox hosts the codebase, the database, or the API fixture
whose final state the harness compares against a goal state. Keep the
grader outside the sandbox boundary and read-only.

## Build vs adopt

Platform work at evaluation scale is real engineering (claim c3).
Adopt existing open infrastructure (Inspect-based platforms such as
METR's Hawk) when your tasks fit; build only the fixtures and graders
that encode your domain. The capability ceiling of your sandbox should
match the capability you are probing — evaluations that probe
cybersecurity or self-proliferation behavior (claim c4) treat
containment as the primary deliverable, and so should any team whose
agent holds credentials to critical APIs.

## Checklist

- [ ] Unlisted egress fails, visibly, with an alarm.
- [ ] Two consecutive runs start from identical state.
- [ ] Test credentials are worthless outside the sandbox.
- [ ] Written worst-case contains no production noun.
- [ ] Grader and telemetry are read-only from outside.
- [ ] The sandbox itself has been falsified: you have watched each
      property fail once.

## Deterministic harnesses vs LLM-as-judge: when each wins

- id: deterministic-harnesses-vs-llm-as-judge · type: comparison · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/deterministic-harnesses-vs-llm-as-judge · api: https://agentreliability.dev/api/k/deterministic-harnesses-vs-llm-as-judge.json

A decision comparison between mechanical grading (state diffs, executable tests) and model-based grading. Deterministic harnesses win wherever an outcome can be computed; judges win on open-ended quality — at the cost of documented biases that must be measured before the judge's verdicts mean anything.

Claims:
- Deterministic grading scales to real agent tasks: tau-bench compares the final database state against an annotated goal state, so the verdict is reproducible and independent of transcript plausibility. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- LLM judges reach over 80% agreement with human preferences on open-ended chat, while carrying position, verbosity and self-enhancement biases and limited reasoning on hard grading. [confidence 0.9; sources: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena]
- Mature eval frameworks treat both as templates: OpenAI's evals ships basic (mechanical) evals and model-graded evals side by side, selected per task. [confidence 0.9; sources: openai/evals — framework for evaluating LLMs and LLM systems]
- Mechanical grading is not automatically objective: formatting inconsistencies alone shifted MMLU scores by about 5%, so harness details are part of the measurement. [confidence 0.85; sources: Challenges in evaluating AI systems]

Takeaways:
- Computable outcome means harness; judgment call means calibrated judge; never let a judge grade what a diff can.
- Judge agreement with humans tops out around human-to-human agreement — that is the ceiling, not a defect.
- Deterministic gates decide ship/no-ship; judge scores ride along as telemetry until calibrated on your domain.
- Harness details are part of the measurement — formatting alone has moved scores about 5 points.

## The routing rule

If the outcome can be computed — a test passes, a database state matches
a goal, a file compiles — grade it deterministically. Reach for an LLM
judge only when the property you care about is a judgment call (tone,
helpfulness, reasoning quality) that no diff can express. Never let a
judge grade something a diff could have graded: you would be adding
noise and bias to a solved measurement.

## Head to head

| Dimension | Deterministic harness | LLM-as-judge |
|---|---|---|
| Reproducibility | Same input, same verdict (claim c1) | Verdicts vary with prompt, position, judge model (claim c2) |
| Coverage | Only computable outcomes | Open-ended quality, style, safety judgment |
| Known biases | Harness bugs, format sensitivity (claim c4) | Position, verbosity, self-enhancement (claim c2) |
| Cost per verdict | Near zero after fixtures exist | One strong-model call per item |
| Gaming surface | Overfit to fixtures | Please-the-judge outputs, verbosity inflation |
| Failure visibility | Loud (test crashes, diff mismatch) | Silent (plausible score, wrong reason) |

## Where each wins

**The harness wins** for tool-using agents acting on systems of record:
grade the final state, not the conversation (claim c1). It also wins for
anything regression-gated in CI, where verdict stability is the point.

**The judge wins** for open-ended output where human preference is the
ground truth and you can afford calibration: measured agreement with
humans above 80% is achievable (claim c2), which is human-level — and
also the ceiling.

**Both lose** when treated as free. Harness details are part of the
measurement (claim c4); a judge is an instrument that needs a
calibration sheet before its verdicts count.

## The production pattern

Layer them, as mature frameworks already do (claim c3): deterministic
gates decide ship/no-ship; judge scores ride along as trend telemetry
until their agreement with a human-audited sample has been measured on
your own domain. Promote a judge to gatekeeper only after that
calibration — and re-falsify it when the judge model changes.

## Deterministic vs probabilistic evaluation

- id: deterministic-vs-probabilistic-evaluation · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/deterministic-vs-probabilistic-evaluation · api: https://agentreliability.dev/api/k/deterministic-vs-probabilistic-evaluation.json

Two grading regimes for agent behavior. Deterministic evaluation checks outcomes mechanically — exact state comparison, executable tests — and reproduces run to run. Probabilistic evaluation delegates judgment to a model or human grader, trading reproducibility for coverage of open-ended outputs. Production reliability work starts deterministic and adds probabilistic grading only where mechanical checks cannot reach.

Claims:
- tau-bench grades agents deterministically by comparing the final database state of each episode against an annotated goal state, which catches wrong actions independently of how plausible the transcript looks. [confidence 0.95; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- Probabilistic grading with an LLM judge reaches over 80% agreement with human preferences on MT-Bench and Chatbot Arena, but carries documented position, verbosity and self-enhancement biases. [confidence 0.9; sources: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena]
- Even nominally objective benchmarks are fragile measurements: Anthropic reports that formatting inconsistencies alone moved MMLU scores by about 5%. [confidence 0.9; sources: Challenges in evaluating AI systems]

## Evaluating retrieval (RAG) inside agents

- id: evaluating-rag-in-agents · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/evaluating-rag-in-agents · api: https://agentreliability.dev/api/k/evaluating-rag-in-agents.json

The retrieval layer of an agent fails in three distinct ways — wrong passages retrieved, right passages used unfaithfully, fluent answer ungrounded in either — and each needs its own metric. Reference-free RAG evaluation makes this measurable without annotated ground truth, at the price of inheriting LLM-judge biases. In an agent, retrieved content is also an attack surface, so RAG evaluation and injection testing share fixtures.

Claims:
- Reference-free RAG evaluation exists and is dimensional: Ragas measures the retrieval system's capacity to find relevant, focused context passages, the model's faithful use of those passages, and generation quality — all without ground-truth human annotations. [confidence 0.9; sources: Ragas: Automated Evaluation of Retrieval Augmented Generation]
- Reference-free metrics are model-graded, so they inherit the documented failure modes of LLM judges — position, verbosity and self-enhancement biases with limited reasoning on hard cases — and need the same calibration against human-audited samples. [confidence 0.85; sources: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena]
- In a tool-using agent, retrieved content is untrusted input: prompt injection through third-party content is the top-ranked LLM application risk, so the retrieval fixtures used for quality evaluation double as the seeding channel for injection tests. [confidence 0.85; sources: OWASP Top 10 for LLM Applications and Generative AI (2025)]

## How to falsify your first guardian

- id: falsify-your-first-guardian · type: guide · evidence_tier: primary · confidence: 0.9 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/falsify-your-first-guardian · api: https://agentreliability.dev/api/k/falsify-your-first-guardian.json

A step-by-step procedure for proving that one automated check in your agent pipeline can actually fail: pick the guardian, plant a defect it must catch, confirm red, restore, confirm green, and record the episode. Thirty minutes of work that converts a green light from an assumption into evidence.

Claims:
- The falsification cycle is plant a defect, observe the guardian go red, restore the correct state, observe it go green; a guardian is only trusted after it has been seen in both states. [confidence 0.95; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]
- In the Citarium framework the content validator is the first guardian falsified: it runs from CI on day one and each of its check families exists because an incident paid for it. [confidence 0.95; sources: Citarium content validator source (@citarium/core v0.1.0, compiler/validate.ts)]
- Robust evaluations are extremely difficult to develop and implement, which is why an untested check deserves no trust by default. [confidence 0.85; sources: Challenges in evaluating AI systems]

Takeaways:
- A guardian that has never been observed red is an assumption, not a control.
- The cycle is plant a defect, expect red, restore, expect green — then record the episode.
- Red for the wrong reason falsifies a different check; the target stays unproven.
- Falsify every new guardian in the pull request that lands it, and re-falsify after pipeline refactors.

## Why falsify

A guardian — any automated check standing between your agent and
production — earns trust the same way a scientific hypothesis does: by
surviving an attempt to break it. A check that has always been green
tells you nothing; maybe it works, maybe it silently stopped running,
maybe it never could fail. Robust evaluations are hard to build (claim
c3), so the default assumption about any untested check is that it is
broken.

## The cycle (claim c1)

1. **Pick one guardian.** Start with the check whose failure would hurt
   most: schema validation, a claim-to-source binding check, a
   destructive-action gate in your agent harness.
2. **Plant a defect.** Introduce the exact class of error the guardian
   exists to catch. Make it minimal and realistic: an unregistered
   source id, a future date in a freshness label, a tool call outside
   the allowlist. Commit it on a branch, never on main.
3. **Expect red.** Run the pipeline. The guardian must fail, and fail
   for the planted reason — read the error message and verify it points
   at your defect, not at an accidental one.
4. **Restore.** Revert the planted defect completely.
5. **Expect green.** Run the pipeline again. It must pass. If it stays
   red, your restore was incomplete or the guardian has a state leak —
   both are findings.
6. **Record the episode.** Note the guardian, the defect class, the
   observed red, and the date. This record is what "our CI catches X"
   now cites.

## Worked example

The Citarium content validator ships as each instance's first guardian
(claim c2). Falsifying it on this very repository looks like: add a
claim citing a source id that is not in `sources.yaml`, run the test
command, observe the cross-reference check fail with the exact claim id,
revert, observe green. The validator's own source code documents that
every one of its check families traces to a real incident — falsification
keeps that inventory honest.

## After the first one

- Falsify each new guardian **when it lands**, in the same pull request.
- Re-falsify after any refactor of the pipeline that runs the guardians.
- Track the number of active checks so a silently dropped guardian is
  itself an alarm — see the check-counter regression alarm.

## Failure modes to expect

- **The guardian passes with the defect planted.** The check does not
  cover what you thought. Fix the check before trusting anything else.
- **Red, but for the wrong reason.** Your defect tripped a different
  check. You have falsified that one instead; the target is still
  unproven.
- **Green after restore fails.** Cached state or ordering dependence in
  the pipeline. Treat as a real defect in the harness.

## Fault injection for agents

- id: fault-injection-for-agents · type: entity · evidence_tier: secondary · confidence: 0.8 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/fault-injection-for-agents · api: https://agentreliability.dev/api/k/fault-injection-for-agents.json

Deliberately breaking the world around an agent to observe how it fails: tool errors, malformed API responses, timeouts, permission denials, adversarial content in retrieved data. The agent equivalent of chaos engineering's real-world events — the goal is verified recovery behavior, not the absence of faults.

Claims:
- Chaos engineering prescribes a four-step method — define steady state, form a hypothesis, introduce variables that reflect real events, and look for differences against a control group — with blast radius minimized and contained. [confidence 0.9; sources: Principles of Chaos Engineering]
- The events worth injecting are real-world disruptions (server failures, network outages, traffic spikes) rather than theoretical scenarios; for agents, the equivalent surface is the tool layer and the data the agent retrieves. [confidence 0.8; sources: Principles of Chaos Engineering]
- Frontier-model evaluation already stress-tests systems under hostile conditions: DeepMind's dangerous-capabilities framework probes cybersecurity, persuasion and self-proliferation behavior rather than benign-path performance. [confidence 0.85; sources: Evaluating Frontier Models for Dangerous Capabilities]

## From incident to check: making failures pay rent

- id: from-incident-to-check · type: guide · evidence_tier: primary · confidence: 0.9 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/from-incident-to-check · api: https://agentreliability.dev/api/k/from-incident-to-check.json

The loop that turns every production incident into a permanent guardian: write the incident down, derive the smallest check that would have caught it, falsify that check, and add it to a counted inventory so it can never silently vanish. After enough cycles, your validator is a compressed history of everything that has actually gone wrong.

Claims:
- The Citarium validator is built this way in production: its source states that every check exists because an incident paid for it, and individual checks carry their incident of origin in code comments (the locale-contract check cites the pt-BR incident directly). [confidence 0.95; sources: Citarium content validator source (@citarium/core v0.1.0, compiler/validate.ts)]
- New checks enter service through falsification — plant the incident's defect, observe red, restore, observe green — so the check demonstrably catches the failure it was born from. [confidence 0.9; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]
- The check inventory is itself guarded: the validator exports its count of check families for CI to track, so a check silently dropped in a refactor raises an alarm. [confidence 0.95; sources: Citarium content validator source (@citarium/core v0.1.0, compiler/validate.ts)]

Takeaways:
- Every incident produces exactly one durable artifact — a falsified, counted check.
- Smallest check that rejects the defect wins; broad checks cry wolf and get deleted.
- Write the incident of origin into the check so the next maintainer knows why it exists.
- Only the counter notices a check's absence — uncounted protection disappears silently.

## The rule

Every incident produces exactly one durable artifact: a check that
would have caught it, proven by falsification, added to a counted
inventory. Everything else in the postmortem — narrative, blame,
slideware — decays. The check compounds.

## The loop, step by step

1. **Capture the defect while it is fresh.** Not the outage story — the
   minimal wrong state. An agent cited an unregistered source; a label
   promised a locale the prose broke; a date claimed the future. One
   sentence, one reproducible state.
2. **Derive the smallest check that rejects that state.** Smallest
   matters: a check that rejects too much gets deleted the first time
   it cries wolf, and then the incident can return. Precision over
   recall is the survival trait for guardians (the Citarium
   locale-contract check is deliberately high-precision for exactly
   this reason — claim c1).
3. **Falsify it before trusting it.** Plant the original incident's
   defect, watch the new check go red, restore, watch green (claim c2).
   The incident becomes the check's own test case — the one defect this
   check must provably catch forever.
4. **Count it.** Add the check to the tracked inventory so its
   disappearance is an alarm, not a mystery discovered during the next
   incident (claim c3).
5. **Write the origin into the check.** A comment naming the incident
   turns the validator into documentation: the next maintainer learns
   why the check exists before deciding to weaken it.

## What this produces over time

A validator built by this loop is a compressed incident history (claim
c1): reading its checks is reading everything that has gone wrong,
with each entry executable and self-verifying. Teams that skip the
loop keep relearning failures; teams that run it convert failure into
infrastructure at a fixed exchange rate — one incident, one guardian.

## Anti-patterns

- **The broad check.** Born from an incident but rejecting a whole
  category "to be safe". It fires on legitimate work, gets deleted,
  and takes the incident's protection with it.
- **The uncounted check.** Added in a hurry, never inventoried,
  silently lost in a refactor. Falsified once is not falsified
  forever — only the counter notices absence.
- **The narrative postmortem.** Action items assigned, none of them a
  check. Six months later the document exists and the protection does
  not.

## GAIA benchmark

- id: gaia-benchmark · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/gaia-benchmark · api: https://agentreliability.dev/api/k/gaia-benchmark.json

A benchmark of 466 real-world questions for general AI assistants, designed so answers are unambiguous to grade but require reasoning, multi-modality, web browsing and tool use to reach. Its signature result is the human-AI gap: 92% for human respondents against 15% for GPT-4 with plugins at publication — questions conceptually simple for people, hard for tool-using models.

Claims:
- GAIA comprises 466 real-world questions that jointly test reasoning, multi-modality handling, web browsing and general tool-use proficiency. [confidence 0.95; sources: GAIA: a benchmark for General AI Assistants]
- At publication, human respondents scored 92% on GAIA against 15% for GPT-4 equipped with plugins — the reverse of benchmarks where models beat humans on professional-exam material. [confidence 0.9; sources: GAIA: a benchmark for General AI Assistants]

## Goodhart resistance in agent metrics

- id: goodhart-resistance · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/goodhart-resistance · api: https://agentreliability.dev/api/k/goodhart-resistance.json

Designing agent evaluation so that optimizing the reported number does not destroy what the number means. Known countermeasures: separate the metric you optimize from the metric you report, measure consistency across repeated trials rather than best-of-N, and audit with a panel of honesty figures instead of a single score.

Claims:
- Goodhart's Law is not one failure but at least four distinct mechanisms — regressional, extremal, causal and adversarial — each requiring different defenses. [confidence 0.9; sources: Categorizing Variants of Goodhart's Law]
- Single-run success overstates reliability: on tau-bench, state-of-the-art function-calling agents pass under 50% of tasks once, and pass^8 falls under 25% in the retail domain, which is why pass^k exists as a consistency metric. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- The Citarium methodology separates CAS-I (internal score) from CAS-E (external score) precisely so the optimized metric and the reported metric are not the same object. [confidence 0.85; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]
- The Citarium audit command reports a panel of honesty figures — evidence-tier mix vs quota, unused sources, moat count, staleness — rather than one aggregatable score, making the audit itself harder to Goodhart. [confidence 0.9; sources: Citarium audit command source (@citarium/cli v0.1.0, commands.ts)]

## Guardian falsification

- id: guardian-falsification · type: entity · evidence_tier: primary · confidence: 0.9 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/guardian-falsification · api: https://agentreliability.dev/api/k/guardian-falsification.json

A discipline for proving that an automated check (a "guardian") can actually fail: deliberately plant a defect, watch the guardian go red, restore the correct state, watch it go green. A guardian that has never been observed red is an assumption, not a control. Practiced in production in the Citarium framework, where every validator check family traces back to a real incident.

Claims:
- Guardian falsification validates a check by planting a known defect, confirming the guardian turns red, restoring the correct state, and confirming it turns green again; only a guardian that has been observed failing counts as evidence rather than decoration. [confidence 0.95; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]
- The Citarium content validator is deployed as each instance's first guardian and runs from CI on day one; its source states that every check exists because an incident paid for it. [confidence 0.95; sources: Citarium content validator source (@citarium/core v0.1.0, compiler/validate.ts)]
- The falsified-guardian methodology is a living case study with real incident history in the Citarium framework, not a thought experiment. [confidence 0.9; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]

## Human approval gates and graduated autonomy

- id: human-approval-gates · type: entity · evidence_tier: primary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/human-approval-gates · api: https://agentreliability.dev/api/k/human-approval-gates.json

The control that keeps an agent's blast radius human-sized: specific action classes (destructive, irreversible, outward-facing) require a human yes before execution, and autonomy expands only as the agent earns it with evidence. The complement to sandboxing — the sandbox contains what the agent can do, gates contain what it may do.

Claims:
- Anthropic's agent-engineering guidance is explicit: agents should pause for human feedback at checkpoints or when encountering blockers, because autonomous operation compounds errors — which is why it pairs the advice with extensive sandboxed testing and guardrails. [confidence 0.9; sources: Building effective agents]
- OWASP ranks Excessive Agency (LLM06:2025) as a top-10 risk: an agent granted more autonomy or permissions than its task requires turns any upstream failure — injection, hallucination — into unauthorized actions. [confidence 0.9; sources: OWASP Top 10 for LLM Applications and Generative AI (2025)]
- Autonomy should be graduated, not granted: add agent complexity only when it demonstrably improves outcomes, starting from the simplest solution that works. [confidence 0.85; sources: Building effective agents]
- A production example of gates as configuration: this instance's own blueprint declares an autonomy level and named human gates (blueprint approval, ship approval) as machine-readable contract, with exceptions that expire after a fixed period. [confidence 0.9; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]

## Independent third-party evaluation

- id: independent-evaluation · type: entity · evidence_tier: secondary · confidence: 0.8 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/independent-evaluation · api: https://agentreliability.dev/api/k/independent-evaluation.json

Evaluation performed by organizations that did not build the system: external red teams, research nonprofits running autonomy evaluations, and government institutes publishing open harnesses. Self-evaluation has a structural conflict of interest — the same incentive gradient Goodhart describes — and the emerging ecosystem of independent evaluators is the field's answer. For buyers, a vendor's willingness to be independently evaluated is itself a reliability signal.

Claims:
- Frontier-lab practice already includes external capacity: Anthropic's red-teaming account recommends funding standards development, supporting independent testing organizations, professionalizing red teaming with certification, and giving vetted third parties access to systems. [confidence 0.9; sources: Challenges in red teaming AI systems]
- Independent evaluators exist and publish: METR runs autonomy evaluations of frontier models from multiple labs, partnering with developers while also conducting independent assessments of publicly released models. [confidence 0.85; sources: METR — Model Evaluation & Threat Research]
- Public evaluation infrastructure lowers the barrier: the UK AI Security Institute maintains Inspect as open source with 200+ prebuilt evaluations, so third-party evaluation does not require third-party tooling from scratch. [confidence 0.85; sources: Inspect — evaluation framework for large language models]

## Inspect (UK AI Security Institute evaluation framework)

- id: inspect-eval-framework · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/inspect-eval-framework · api: https://agentreliability.dev/api/k/inspect-eval-framework.json

The open-source evaluation framework from the UK AI Security Institute: composable solvers and scorers, sandboxed execution of untrusted model code, agent primitives, and a registry of 200+ prebuilt evals. The closest thing agent evaluation has to shared infrastructure — METR's Hawk platform builds on it — and the default answer to "should we write our own harness runner?" (usually no).

Claims:
- Inspect is an open-source LLM evaluation framework by the UK AI Security Institute (GitHub UKGovernmentBEIS/inspect_ai) providing solvers, scorers with both text-based and model-graded options, and over 200 prebuilt evaluations. [confidence 0.9; sources: Inspect — evaluation framework for large language models]
- Inspect runs untrusted model code in sandboxes via Docker, Kubernetes, Modal, Proxmox and other backends through an extension API, and ships agent primitives — ReAct, multi-agent composition, and driving external agents such as Claude Code, Codex CLI and Gemini CLI. [confidence 0.9; sources: Inspect — evaluation framework for large language models]
- METR's Hawk large-scale agent-evaluation platform is built on Inspect infrastructure — third-party adoption at frontier-evaluation scale. [confidence 0.85; sources: METR — Model Evaluation & Threat Research]

## LLM-as-judge and its failure modes

- id: llm-as-judge · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/llm-as-judge · api: https://agentreliability.dev/api/k/llm-as-judge.json

Using a strong language model to grade another model's output. Cheap, scalable, and surprisingly aligned with human preference on open-ended chat — but a biased instrument: judges favor certain answer positions, longer answers, and their own outputs, and their reasoning limits cap what they can grade. Usable in production only with its biases measured and bounded.

Claims:
- Strong LLM judges like GPT-4 reach over 80% agreement with human preferences on MT-Bench and Chatbot Arena — the same level as agreement between humans. [confidence 0.9; sources: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena]
- LLM judges exhibit position bias, verbosity bias and self-enhancement bias, plus limited reasoning ability on hard grading tasks; these are documented, measurable failure modes. [confidence 0.95; sources: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena]
- Model-graded evaluation is a first-class template in OpenAI's evals framework, alongside basic (deterministic) eval templates. [confidence 0.9; sources: openai/evals — framework for evaluating LLMs and LLM systems]

## Measuring agent reliability: bounded metrics and decision tables

- id: measuring-agent-reliability · type: guide · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/measuring-agent-reliability · api: https://agentreliability.dev/api/k/measuring-agent-reliability.json

Which numbers actually predict whether an agent can ship: consistency across repeated trials (pass^k), execution-verified success rates, task time horizon, and a panel of honesty figures instead of one score. Includes a decision table mapping each question a team asks to the metric that answers it.

Claims:
- pass^k measures the probability that an agent succeeds on all k repeated trials of the same task; on tau-bench, agents under 50% single-attempt success drop under 25% at pass^8 in retail, so consistency and capability are different axes. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- Execution-verified resolution rate on real tasks is a bounded, reproducible measure: SWE-bench grades 2,294 real GitHub issues by running the result, and its initial best baseline resolved 1.96%. [confidence 0.9; sources: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?]
- METR's time-horizon metric — the length of tasks agents can complete — has risen exponentially across six years of frontier models, making it a capability trend measure rather than a reliability measure. [confidence 0.85; sources: METR — Model Evaluation & Threat Research]
- NIST AI RMF makes Measure one of its four core functions: metric selection and ongoing measurement are governance obligations across the lifecycle, not an engineering nicety. [confidence 0.85; sources: NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1)]
- The Citarium audit command reports honesty figures — tier mix against quota, unused sources, moat count, oldest verification — as a panel, because a single aggregated score invites gaming. [confidence 0.9; sources: Citarium audit command source (@citarium/cli v0.1.0, commands.ts)]
- The EUREKA platform maintains a public catalog of 69 agentic benchmarks and tracks the field's shift from measuring capability to measuring agency. [confidence 0.85; sources: EUREKA — applied AI, agents and robotics knowledge platform]

Takeaways:
- Users experience pass^k, not pass@1 — gate on consistency, report the panel, never one aggregate score.
- Success is defined by execution (state matches goal), never by reading the transcript.
- Time horizon tells you what to delegate; it says nothing about reliability at a fixed point.
- Check the public benchmark shelf before inventing your own suite.

## The question behind every metric

"Is the agent good?" is not measurable. "Does the agent complete this
task, verified by execution, k times in a row, inside budget?" is. The
metrics below are bounded — each has a defined range, a defined
denominator, and a defined way to fail — because unbounded or aggregated
scores are where gaming starts.

## The four metrics that earn their place

1. **pass^k — consistency (claim c1).** The probability of succeeding
   all k times on the same task. This is the production metric: a user
   who hits the agent eight times experiences pass^8, not pass@1. The
   observed gap (under 50% at one attempt, under 25% at eight in
   tau-bench retail) is the difference between a demo and a deploy.
2. **Execution-verified success rate (claim c2).** Success defined by
   running the result — tests pass, final state matches goal state —
   never by reading the transcript. Real-task suites (SWE-bench's 2,294
   GitHub issues) set the standard here.
3. **Time horizon (claim c3).** How long a task the agent can carry
   autonomously. Track it to know what class of work to trust the agent
   with; do not confuse its exponential trend with reliability at any
   fixed point.
4. **Honesty-figure panels (claim c5).** For the evaluation system
   itself: a set of small, separately-alarmed numbers (coverage vs
   quota, unused inventory, staleness) that resist collapse into one
   optimizable score.

## Decision table

| Question the team is asking | Metric that answers it | Ship gate example |
|---|---|---|
| Will it work for a user who retries? | pass^k on your task suite | pass^4 above target on critical paths |
| Did it really do the task? | execution-verified success | grader outside sandbox, state-diff based |
| What work can we delegate? | time horizon | tasks under proven horizon only |
| Is our eval still honest? | honesty-figure panel | zero unused fixtures, staleness bounded |
| Are we governance-complete? | Measure-function mapping (claim c4) | each risk has a metric and an owner |

## Choosing benchmarks

Do not invent a benchmark before checking the shelf: the field already
has dozens of agentic suites — a public catalog tracks 69 of them
(claim c6). Adopt the closest real-task suite for comparability, then
add a private, rotating extension for your domain (see opaque rotating
test sets) so your number cannot be trained against.

## Anti-patterns

- **One aggregate quality score.** Invites Goodhart dynamics; keep the
  panel disaggregated (claim c5).
- **Transcript-graded success.** Plausible narration is what these
  systems do best; grade state, not prose.
- **pass@1 in the deck, pass^k in production.** Report the metric your
  users will experience.

## Opaque rotating test sets

- id: opaque-rotating-test-sets · type: entity · evidence_tier: secondary · confidence: 0.8 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/opaque-rotating-test-sets · api: https://agentreliability.dev/api/k/opaque-rotating-test-sets.json

Held-out evaluation tasks that the evaluated system (and its training pipeline) never sees, refreshed on a schedule. Opacity prevents direct optimization against the benchmark; rotation limits contamination and slow overfitting as public data leaks into training corpora. The price is comparability: rotated scores are trends, not fixed leaderboards.

Claims:
- OpenAI's evals framework explicitly supports private evals built on your own data without public exposure, so teams can grade models without publishing the test set. [confidence 0.9; sources: openai/evals — framework for evaluating LLMs and LLM systems]
- Optimizing against a known, static metric degrades what the metric measures — Goodhart dynamics with at least four distinct mechanisms — which is the failure opaque, rotating sets are designed to resist. [confidence 0.85; sources: Categorizing Variants of Goodhart's Law]
- METR publishes datasets and research on evaluation integrity threats for autonomous AI systems, treating benchmark integrity as a first-class research problem. [confidence 0.8; sources: METR — Model Evaluation & Threat Research]

## OSWorld

- id: osworld · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/osworld · api: https://agentreliability.dev/api/k/osworld.json

A benchmark of 369 open-ended tasks executed in real operating systems (Ubuntu, Windows, macOS) spanning web and desktop apps, file I/O and multi-application workflows. Every task ships its own initial-state setup and an execution-based evaluation script, making it a working template for reproducible computer-use agent evaluation. Headline gap at publication: humans 72.36%, best model 12.24%.

Claims:
- OSWorld comprises 369 real computer tasks — web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications — running on real operating systems including Ubuntu, Windows and macOS. [confidence 0.95; sources: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments]
- Each OSWorld task defines a detailed initial-state setup plus a custom execution-based evaluation script, so grading is reproducible and independent of the agent's narration. [confidence 0.9; sources: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments]
- At publication humans accomplished over 72.36% of OSWorld tasks against 12.24% for the best model, with difficulties concentrated in GUI grounding and operational knowledge. [confidence 0.9; sources: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments]

## Process vs outcome evaluation of agent trajectories

- id: process-vs-outcome-evaluation · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/process-vs-outcome-evaluation · api: https://agentreliability.dev/api/k/process-vs-outcome-evaluation.json

Two places to point the grader: the final state (outcome) or every intermediate step (process). Outcome grading is cheap, objective and gameable by lucky trajectories; step-level grading catches reasoning that arrives at the right answer wrong, at the cost of expensive labels. The research result: process supervision beat outcome supervision decisively on hard math. For agents, outcome grading gates releases while process signals explain failures and catch right-for-the-wrong-reason passes.

Claims:
- Process supervision — feedback on each intermediate reasoning step — significantly outperforms outcome supervision on the MATH dataset, with the process-supervised reward model reaching 78% on a representative test subset. [confidence 0.9; sources: Let's Verify Step by Step]
- Step-level evaluation has a public data foundation: PRM800K releases 800,000 step-level human feedback labels used to train the best process reward model. [confidence 0.9; sources: Let's Verify Step by Step]
- Outcome grading remains the production gate for tool-using agents: comparing final system state against an annotated goal is reproducible and narration-independent, which is what release decisions need. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]

## Prompt-injection testing for agents

- id: prompt-injection-testing · type: entity · evidence_tier: primary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/prompt-injection-testing · api: https://agentreliability.dev/api/k/prompt-injection-testing.json

Systematically probing whether an agent acts on instructions smuggled into content it reads — tool results, retrieved documents, web pages, emails. For tool-using agents this is the top-ranked risk turned into a test surface: every untrusted input channel gets seeded with adversarial instructions, and the assertion is that the agent's actions never follow them. Runs in the sandbox, gates the release.

Claims:
- Prompt injection is the top-ranked risk in the OWASP Top 10 for LLM Applications (LLM01 in the 2025 edition): user or third-party content altering the system's intended behavior. [confidence 0.95; sources: OWASP Top 10 for LLM Applications and Generative AI (2025)]
- OWASP separately ranks Excessive Agency (LLM06:2025) — an agent granted more autonomy or permissions than its task needs — which is the amplifier that turns a successful injection into destructive actions against connected systems. [confidence 0.9; sources: OWASP Top 10 for LLM Applications and Generative AI (2025)]
- OWASP frames these risks with mitigations spanning development, deployment and management phases, so injection resistance is a lifecycle property to keep testing, not a one-time filter to install. [confidence 0.85; sources: OWASP Top 10 for LLM Applications and Generative AI (2025)]

## Public benchmarks vs private task suites: what each buys you

- id: public-benchmarks-vs-private-task-suites · type: comparison · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/public-benchmarks-vs-private-task-suites · api: https://agentreliability.dev/api/k/public-benchmarks-vs-private-task-suites.json

Public benchmarks buy comparability, transparency and community scrutiny; private suites buy contamination resistance, domain fit and Goodhart resistance. Production teams need both, in a fixed shape: public suites to place yourself in the field, a private rotating suite to gate your releases. Using either for the other's job is how teams fool themselves.

Claims:
- The public side has a reference implementation: HELM publishes transparent, reproducible benchmark suites and leaderboards across many scenarios and metrics, explicitly to standardize open evaluation practice. [confidence 0.9; sources: HELM — Holistic Evaluation of Language Models]
- Eval tooling supports the private side natively: OpenAI's evals framework lets teams build private evals on their own data without public exposure. [confidence 0.9; sources: openai/evals — framework for evaluating LLMs and LLM systems]
- A public, static target invites Goodhart dynamics — optimizing the known metric degrades what it measures through at least four distinct mechanisms. [confidence 0.85; sources: Categorizing Variants of Goodhart's Law]
- Benchmark integrity is an active threat surface: METR publishes research and datasets on evaluation integrity for autonomous systems. [confidence 0.8; sources: METR — Model Evaluation & Threat Research]
- Public agent benchmarks still discriminate today: humans beat the best models by tens of points on GAIA-style real-world assistant tasks (92% vs 15% at publication), so saturation is not yet the dominant failure. [confidence 0.85; sources: GAIA: a benchmark for General AI Assistants]

Takeaways:
- Public benchmarks answer "where do we stand"; private suites answer "may this ship" — one suite doing both jobs is how teams fool themselves.
- A public static target invites Goodhart optimization by construction.
- Gate releases only on the private, rotating suite; track public placement as telemetry.
- Publish your public scores — community scrutiny is the cheapest auditing available.

## The jobs are different

A public benchmark answers "where does my system stand in the field?"
A private suite answers "may this release ship?" The failure pattern
behind most "great scores, broken agent" stories is one suite doing
both jobs.

## Head to head

| Dimension | Public benchmark | Private task suite |
|---|---|---|
| Comparability | Across labs and models (claim c1) | Only against your own history |
| Contamination | Training data absorbs it over time | Resistant while it stays private (claim c2) |
| Goodhart exposure | High — the target is known (claim c3) | Low, if rotated and access-controlled |
| Domain fit | Generic by design | Encodes your APIs, policies, edge cases |
| Scrutiny | Community-audited methodology | Your bugs are your own |
| Cost | Free to adopt | You build and maintain the fixtures |

## Where public wins

Placement and honesty about the field. Open, reproducible,
multi-metric suites exist precisely so claims are checkable (claim
c1), and current agent benchmarks still leave a wide human-model gap
(claim c5), so a public score carries real signal. Publish yours:
scrutiny you did not pay for is the cheapest auditing available.

## Where private wins

Release gating. Your agent's critical paths — your APIs, your
policies, your failure modes — appear in no public suite. A private
suite built from them cannot be trained against (claim c2), and
rotation keeps slow leakage from mattering. This is also the only
suite where a regression means, unambiguously, that your product got
worse.

## The operating pattern

- Adopt the closest public suites once, rerun on each major model
  change, and track your placement as telemetry.
- Gate releases only on the private suite, rotated on a schedule,
  with integrity treated as a live threat (claim c4).
- Never promote a public score into a ship decision: the incentive to
  optimize the visible number is a Goodhart mechanism working as
  described (claim c3).

## Running your first agent red-team exercise

- id: red-teaming-your-agent · type: guide · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/red-teaming-your-agent · api: https://agentreliability.dev/api/k/red-teaming-your-agent.json

A bounded, repeatable adversarial exercise for a tool-using agent: scope the rules of engagement inside the sandbox, enumerate the attack surfaces (injection channels, tool permissions, data paths), run manual and automated passes, triage by blast radius, and convert every confirmed finding into a falsified check so the exercise compounds instead of evaporating.

Claims:
- Red teaming is a family of methods, not one activity: practiced forms include domain-expert teaming, frontier-threats testing (cybersecurity and autonomous-AI risks among them), automated model-vs-model attack generation, and crowdsourced and community exercises. [confidence 0.9; sources: Challenges in red teaming AI systems]
- Adversarial capability evaluation has a published reference shape: DeepMind's dangerous-capabilities framework probes persuasion, cybersecurity, self-proliferation and self-reasoning rather than benign-path behavior. [confidence 0.85; sources: Evaluating Frontier Models for Dangerous Capabilities]
- For tool-using agents the standard attack surface is named by OWASP: prompt injection (LLM01) as the entry vector and excessive agency (LLM06) as the amplifier that turns it into unauthorized actions. [confidence 0.9; sources: OWASP Top 10 for LLM Applications and Generative AI (2025)]
- Findings only compound if they become guardians: the incident-pays-for-check pattern — each confirmed failure produces one falsified, counted check — is how the Citarium validator was built in production. [confidence 0.9; sources: Citarium content validator source (@citarium/core v0.1.0, compiler/validate.ts)]

Takeaways:
- A confirmed finding is an unauthorized action executed, not a scary transcript.
- The surface is injection channels, dangerous tool permissions and data paths — one seeded probe per channel.
- Triage by blast radius of the action executed, not by ease of fix.
- The deliverable is checks, not a report — every confirmed finding becomes a falsified, counted guardian.

## Before you start: rules of engagement

Run the exercise against the sandboxed agent, never production. Fix
the scope in writing: which agent build, which tools enabled, which
attack classes are in bounds, what counts as a "confirmed" finding
(an unauthorized action executed, not a scary transcript). One page,
agreed before the first probe.

## Enumerate the attack surface

For a tool-using agent the map is short and standard (claim c3):

1. **Injection channels** — every untrusted input the agent reads:
   retrieved documents, web pages, tool results, user uploads, email.
   One seeded probe per channel.
2. **Tool permissions** — every tool whose misuse is irreversible or
   outward-facing: writes, deletes, sends, payments. The question is
   whether upstream text can reach them without a gate.
3. **Data paths** — what secrets or personal data the agent can see,
   and every channel through which output leaves (messages, commits,
   URLs, logs).

## Run the passes

- **Manual pass.** A person plays adversary on each surface: planted
  instructions in retrieved content, confused-deputy requests through
  legitimate-looking tasks, escalation chains across tools. Expert
  attention goes where blast radius is highest.
- **Automated pass.** Use a model to generate attack variants at
  volume against the same surfaces — the automated red/blue pattern
  from published practice (claim c1). Volume finds the phrasing your
  manual pass did not think of.
- **Capability pass (if your agent is powerful).** For agents with
  broad system access, borrow the shape of dangerous-capability
  evaluation — probe what the agent could do under hostile prompting,
  not just what it does on the happy path (claim c2).

## Triage by blast radius

Score each confirmed finding by the worst action actually executed:
irreversible external action > internal state corruption > policy
bypass without action > refusal-wording weakness. Fix order follows
blast radius, not ease of fix.

## The output contract

Every confirmed finding produces exactly one artifact: a check that
would have caught it — falsified (plant the attack, watch red,
restore, watch green) and added to the counted inventory (claim c4).
The exercise report is allowed to exist, but the checks are the
deliverable; findings that stay findings are incidents on a delay.

## Cadence

Repeat on every new tool grant, every model swap, and on a fixed
calendar even when nothing changed — automated variants drift cheap,
so the re-run costs little and the surface moves under you either
way.

## Regression-gating a model upgrade

- id: regression-gating-model-upgrades · type: guide · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/regression-gating-model-upgrades · api: https://agentreliability.dev/api/k/regression-gating-model-upgrades.json

The playbook for swapping the model behind a production agent without discovering the differences in production: freeze the harness, baseline the incumbent with pass^k on critical paths, run the candidate on the identical suite, diff the panel, roll out behind canaries. Model swaps are not optional — providers retire models on notice — so this gate runs on the provider's calendar, not yours.

Claims:
- Model retirement is a scheduled certainty, not an optional refactor: Anthropic deprecates models with at least 60 days' notice for publicly released models, and requests to retired models fail outright. [confidence 0.95; sources: Anthropic model deprecations (Claude API documentation)]
- The vendor's own guidance is to test applications with the replacement model well before the retirement date and to audit usage by API key and model to locate everything still pointing at the deprecated version. [confidence 0.9; sources: Anthropic model deprecations (Claude API documentation)]
- Consistency is the metric that catches upgrade regressions: pass^k measures success across k repeated trials, and agents with similar single-attempt scores can differ widely in it. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- Detecting how model versions change application behavior is a stated purpose of eval frameworks — the upgrade gate is the canonical use case for a frozen eval suite. [confidence 0.85; sources: openai/evals — framework for evaluating LLMs and LLM systems]
- Harness details are part of the measurement — formatting inconsistencies alone have shifted benchmark scores by about 5% — so the harness must stay frozen while the model changes, or the diff measures the harness. [confidence 0.85; sources: Challenges in evaluating AI systems]

Takeaways:
- Model swaps run on the provider's calendar — 60-day deprecation windows make the gate standing infrastructure.
- Freeze the harness completely; a moving harness makes the model diff meaningless.
- Diff per critical path with pass^k, not the average — pass@1 up with pass^8 down is the dangerous pattern.
- Roll out behind behavioral canaries and recalibrate any LLM judge whose model also changed.

## Why this gate exists

You will swap models whether you plan to or not: providers retire
models on a published calendar, with as little as 60 days between
notice and shutdown (claim c1). Teams that treat the swap as an
emergency prompt-patching session in week 8 ship regressions; teams
with a standing gate treat it as a routine run.

## The gate, step by step

1. **Inventory what actually runs.** Audit usage by key and by model
   (claim c2). Agents accrete model references — fallback paths,
   subagents, background jobs — and the one you forget is the one that
   breaks at retirement.
2. **Freeze the harness.** Same task fixtures, same graders, same
   parsing, same temperature of everything around the model. Any
   harness change between baseline and candidate invalidates the diff
   (claim c5).
3. **Baseline the incumbent.** Run the frozen suite against the current
   model with repeated trials per task — pass^4 or pass^8 on critical
   paths, not pass@1 (claim c3). Store per-task results, not the
   aggregate.
4. **Run the candidate on the identical suite.** Same trials, same
   order-independence. This is the frozen-suite use case eval
   frameworks exist for (claim c4).
5. **Diff the panel, not the average.** A candidate that is +3 points
   on average and -40 on one critical path fails the gate. Ship gates
   are per-path thresholds; the average is telemetry.
6. **Roll out behind canaries.** Route a slice of traffic to the
   candidate with behavioral canaries pinned to known-good outcomes;
   drift on a canary rolls back automatically. Keep the incumbent
   reachable until the canaries have survived your longest business
   cycle.
7. **Re-falsify judge-graded checks.** If any gate uses an LLM judge,
   the judge model may have changed too — recalibrate it against a
   human-audited sample before trusting its verdicts on the new
   candidate.

## What the diff will show (and what to do)

- **Format drift.** New models answer in different shapes; parsers
  break before reasoning does. Fix parsers, not prompts, first.
- **Instruction-following shifts.** Old prompt hacks (emphatic
  repetition, workaround phrasing) may now misfire. Delete them and
  re-test rather than stacking new hacks on top.
- **Consistency changes.** pass@1 up, pass^8 down is a real and
  dangerous pattern (claim c3): the demo improves while the user
  experience degrades. The gate must weight pass^k.

## Standing infrastructure

Run this gate on every provider deprecation notice and every scheduled
model refresh. The suite you freeze for it is the same suite that
gates prompt changes and tool changes — one investment, three gates.

## Sandboxed execution environments

- id: sandboxed-execution · type: entity · evidence_tier: secondary · confidence: 0.8 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/sandboxed-execution · api: https://agentreliability.dev/api/k/sandboxed-execution.json

Contained environments where an agent can act — run code, call tools, mutate state — without its failures reaching production systems. Execution-graded benchmarks already presume one; for dangerous-capability testing the sandbox is the safety case. A sandbox that leaks (network egress, shared state, real credentials) converts a test failure into an incident.

Claims:
- SWE-bench grades by execution: resolving real GitHub issues requires the model to edit codebases and interact with execution environments, so the environment is part of the benchmark's design. [confidence 0.9; sources: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?]
- METR built Hawk, an open-source platform for large-scale AI agent evaluations on Inspect infrastructure — evidence that serious agent evaluation is an infrastructure problem, not a prompt problem. [confidence 0.85; sources: METR — Model Evaluation & Threat Research]
- Dangerous-capability evaluation domains include cybersecurity and self-proliferation, where the evaluated behavior is exactly the kind that must not escape the test environment. [confidence 0.85; sources: Evaluating Frontier Models for Dangerous Capabilities]

## Statistical rigor in agent evals

- id: statistical-rigor-in-evals · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/statistical-rigor-in-evals · api: https://agentreliability.dev/api/k/statistical-rigor-in-evals.json

Treating an eval score as a measurement with uncertainty instead of a fact: report error bars, model your task set as a sample from a larger population of possible tasks, and never call a two-point difference a regression (or an improvement) without knowing the noise floor. The cheapest upgrade available to most eval pipelines is a standard error next to every number.

Claims:
- Eval questions should be conceptualized as draws from an unseen super-population, with results analyzed and reported using statistical formulas that minimize noise and maximize the informativeness of model comparisons. [confidence 0.9; sources: Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations]
- Measurement noise in evals is empirically material: formatting inconsistencies alone have moved MMLU scores by about 5% — larger than many claimed model-to-model differences. [confidence 0.9; sources: Challenges in evaluating AI systems]
- Repeated trials are the raw material of agent statistics: pass^k exists because single-run success and consistency across k runs diverge sharply in practice. [confidence 0.85; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]

## SWE-bench Verified

- id: swe-bench-verified · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/swe-bench-verified · api: https://agentreliability.dev/api/k/swe-bench-verified.json

The human-validated subset of SWE-bench: 500 tasks filtered from the original test set for quality, graded by unit-test verification against post-PR behavior. Its existence is the interesting fact — a flagship benchmark needed a human audit of its own tasks, which makes "who benchmarks the benchmark?" a standing question for anyone gating decisions on eval numbers.

Claims:
- SWE-bench Verified is a subset of 500 samples from the SWE-bench test set that have been human-validated for quality. [confidence 0.9; sources: SWE-bench Verified dataset card (princeton-nlp/SWE-bench_Verified)]
- Verified keeps execution-based grading: task resolution is checked by unit-test verification using post-PR behavior as the reference solution. [confidence 0.9; sources: SWE-bench Verified dataset card (princeton-nlp/SWE-bench_Verified)]
- The parent benchmark it filters is large and real — 2,294 GitHub issues from Python repositories requiring multi-file edits and execution-environment interaction — which is exactly the scale at which unaudited task quality becomes a measurement risk. [confidence 0.9; sources: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?]

## Testing MCP servers and integrations

- id: testing-mcp-servers · type: entity · evidence_tier: primary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/testing-mcp-servers · api: https://agentreliability.dev/api/k/testing-mcp-servers.json

What reliability testing means for Model Context Protocol integrations: conformance to the JSON-RPC contract, capability-negotiation edge cases, and — because the spec itself declares tool descriptions untrusted — injection probes through every server-supplied string. An MCP interface is an API whose consumer is a model, so its test suite gates on success rate like any other API, plus adversarial content like no other API.

Claims:
- MCP standardizes stateful JSON-RPC 2.0 connections between hosts, clients and servers with capability negotiation; servers expose tools, resources and prompts, and clients may offer sampling, roots and elicitation — each a distinct surface a conformance suite must cover. [confidence 0.95; sources: Model Context Protocol specification (2025-06-18)]
- The specification itself instructs implementors to treat tool behavior descriptions and annotations as untrusted unless they come from a trusted server, to treat tools as arbitrary code execution, and to require explicit user consent before any tool invocation — making description-borne injection a spec-named test case, not a hypothetical. [confidence 0.95; sources: Model Context Protocol specification (2025-06-18)]
- MCP telemetry is standardizable today: OpenTelemetry's GenAI semantic conventions include MCP alongside model and agent spans, so integration health can be measured with shared vocabulary. [confidence 0.85; sources: OpenTelemetry semantic conventions for generative AI]
- A production example of gating on it: this instance's blueprint ships an MCP interface and declares a minimum MCP success rate of 0.99 among its release thresholds. [confidence 0.9; sources: agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)]

## WebArena

- id: webarena · type: entity · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/webarena · api: https://agentreliability.dev/api/k/webarena.json

A self-hosted, realistic web environment for evaluating autonomous agents: functional sites for e-commerce, forum discussion, collaborative software development and content management, plus maps and knowledge-base tools. Tasks are long-horizon and graded on functional correctness of the end state. Headline result at publication: best GPT-4 agent 14.41% against human 78.24%.

Claims:
- WebArena provides realistic self-hosted environments across four domains — e-commerce, social forums, collaborative software development and content management — with tool and knowledge-base access, so agent evaluation runs against functional sites rather than static snapshots. [confidence 0.95; sources: WebArena: A Realistic Web Environment for Building Autonomous Agents]
- WebArena grades functional correctness of task completion on long-horizon tasks; at publication the best GPT-4-based agent reached 14.41% end-to-end success against 78.24% for humans. [confidence 0.9; sources: WebArena: A Realistic Web Environment for Building Autonomous Agents]

## Your first agent evals: from zero to a release gate in two weeks

- id: your-first-agent-evals · type: guide · evidence_tier: secondary · confidence: 0.85 · last_verified: 2026-08-08
- canonical: https://agentreliability.dev/k/your-first-agent-evals · api: https://agentreliability.dev/api/k/your-first-agent-evals.json

The bootstrap path for a team whose agent has no evals at all: harvest real tasks from production logs, write execution-based graders for them, adopt an existing eval framework instead of building one, baseline with repeated trials and error bars, and wire the result into CI as a gate. Two weeks of part-time work; the output is the suite every other reliability practice hangs from.

Claims:
- Start simple is vendor-stated best practice: add agent complexity only when it demonstrably improves outcomes, with extensive sandboxed testing and guardrails before autonomy grows. [confidence 0.85; sources: Building effective agents]
- Framework adoption beats framework building on day one: open eval frameworks ship reusable templates for basic and model-graded evals plus private-eval support, and Inspect adds sandboxed execution, agent primitives and 200+ prebuilt evaluations. [confidence 0.9; sources: openai/evals — framework for evaluating LLMs and LLM systems; Inspect — evaluation framework for large language models]
- Grade end state, not narration: comparing the final state of the acted-on system against a goal state is how agent benchmarks catch wrong actions hidden behind plausible transcripts. [confidence 0.9; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains]
- Baselines need repeated trials and uncertainty: consistency across k runs diverges sharply from single-run success, and comparisons without standard errors mistake noise for findings. [confidence 0.85; sources: tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains; Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations]

Takeaways:
- Twenty log-harvested tasks, one grader style, one adopted framework, one CI job — resist enlargement until the gate exists.
- Define done as end state; a task whose outcome cannot be asserted gets replaced, not judge-graded.
- Baseline with 4-8 trials per task and a standard error, or noise will read as regressions.
- Invented tasks stay green forever and predict nothing — logs or it does not count.

## The shape of the problem

Teams without evals do not lack conviction — they lack a first suite
small enough to finish. The answer is twenty tasks, one grader style,
one adopted framework, one CI job. Resist every enlargement until the
gate exists.

## Week 1: tasks and graders

1. **Harvest 20 real tasks from logs.** Not invented scenarios: pull
   the last 20 things users actually asked the agent to do, keep the
   inputs verbatim, strip secrets. Ten routine, five hard, five that
   previously failed (those five are your future regression tests).
2. **Define done as end state.** For each task, write down what must
   be true of the world afterwards — records changed, files present,
   API responses matching — never what the transcript should say
   (claim c3). If a task's outcome cannot be stated as a checkable
   condition, replace the task; do not reach for a judge yet.
3. **Write the graders as plain assertions.** State diffs, exact
   checks, executable verification. Boring by design: a grader you
   can falsify in one minute is worth ten clever ones.

## Week 2: framework, baseline, gate

4. **Adopt, do not build** (claim c2). Pick an existing framework —
   the choice matters less than not writing your own runner. Fixtures
   and graders encode your domain; everything else is commodity.
5. **Baseline with repeated trials.** Run each task 4-8 times against
   the current agent; record per-task pass rates and a standard error
   for the suite (claim c4). This number panel — not any single score
   — is what "the agent got worse" will mean from now on.
6. **Wire the CI gate.** The suite runs on every prompt change, tool
   change and model change; per-task thresholds on the critical five,
   suite-level trend for the rest. Keep the agent's autonomy matched
   to what the gate proves (claim c1).

## After the gate exists

- Falsify it: plant a known regression, watch red, restore (see
  falsify-your-first-guardian).
- Feed it: every new incident adds one task with a grader (see
  from-incident-to-check).
- Grow deliberately: more tasks on the paths users actually hit, a
  calibrated judge only where no assertion can reach.

## Two traps

- **The invented-task suite.** Twenty scenarios nobody asked for,
  green forever, predictive of nothing. Logs or it does not count.
- **The framework project.** Three sprints into a bespoke runner, no
  gate yet. The runner was never the hard part; your fixtures are.