https://agentreliability.dev Testing, benchmarking and auditing autonomous AI agents — methods, harnesses, evidence Sun, 09 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/choosing-k-for-pass-hat-k https://agentreliability.dev/k/choosing-k-for-pass-hat-k The corpus recommends pass^k everywhere and never says which k or what number to demand. This is the arithmetic and the decision rule: k comes from how many times the task runs between human reviews, the threshold comes from the cost of one failure, and both must be set before you see the score. Includes why pass@k and pass^k move in opposite directions and why measured pass^k is almost never p^k. Sun, 09 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/agent-observability-standards https://agentreliability.dev/k/agent-observability-standards The emerging standard layer for agent telemetry: OpenTelemetry's GenAI semantic conventions define spans, metrics and events for model calls, tool execution, MCP interactions and agent operations, so traces from different stacks become comparable. Still in active development — adopt it for vocabulary and interoperability, pin versions, and expect churn until stabilization. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/agent-release-gates https://agentreliability.dev/k/agent-release-gates The ship/no-ship decision expressed as declared, machine-checkable thresholds instead of judgment calls: minimum external quality score, minimum factuality, minimum interface success rate, zero critical security findings. Declaring the gate in configuration makes it auditable and Goodhart-visible; a gate that lives in someone's head renegotiates itself under deadline pressure. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/agent-reliability-faq https://agentreliability.dev/k/agent-reliability-faq The questions teams actually ask before shipping an autonomous agent: how to stop hallucinated actions against critical APIs, what a benchmark should measure, whether one successful demo means anything, and where to start when there is no eval at all. Answers are bound to sources, not opinion. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/agent-reliability-glossary https://agentreliability.dev/k/agent-reliability-glossary Working definitions for the terms this instance uses everywhere: eval, harness, canary, red-teaming, regression gate, pass^k, fault injection, sandbox. Each definition is bound to a source that uses the term operationally, so the glossary states how the field works rather than how a dictionary reads. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/agent-telemetry-actor-classification https://agentreliability.dev/k/agent-telemetry-actor-classification Instrumenting a deployed agent so every action is attributable: which actor (human, autonomous agent, crawler, operator) initiated the request, which tools were called, what state changed. Actor classification is the prerequisite for everything downstream — rate limits, audits, incident forensics, and knowing whether a regression hit humans or machines. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/agentbench https://agentreliability.dev/k/agentbench A multi-environment benchmark evaluating LLMs as agents across eight distinct settings (operating system, database, knowledge graph, games, web tasks and more). Its durable findings: commercial frontier models act competently as agents while sub-70B open models lag sharply, and the failures concentrate in long-horizon reasoning, decision-making and instruction following — not in single-turn knowledge. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/behavioral-canaries https://agentreliability.dev/k/behavioral-canaries Small, fixed probe tasks run continuously against a deployed agent. Each canary has a known-good outcome; a change in that outcome is treated as an early regression signal — model drift, prompt rot, tool API changes — before users feel it. The chaos-engineering steady-state hypothesis applied to agent behavior instead of infrastructure metrics. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/calibrating-your-llm-judge https://agentreliability.dev/k/calibrating-your-llm-judge The protocol that turns an LLM judge from vibes into an instrument: measure its agreement against a human-audited sample, probe its known biases directly (position, verbosity, self-preference), put error bars on every comparison it feeds, and define the triggers that force recalibration. A judge is promoted to gatekeeper only after this protocol — and demoted the day its judge model changes. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/check-counter-alarm https://agentreliability.dev/k/check-counter-alarm A meta-guardian that watches the guardians: the validator exports the number of distinct check families it runs, and CI tracks that counter. If a refactor silently drops a check, the count falls and the alarm fires — catching the failure mode where protection disappears while every remaining light stays green. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/designing-an-agent-sandbox https://agentreliability.dev/k/designing-an-agent-sandbox Requirements and checks for an execution environment where an agent can fail safely: no default egress, ephemeral seeded state, fake credentials that look real, a bounded blast radius, and an observation channel the agent cannot write to. Includes a containment checklist you can test the same way you test the agent. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/deterministic-harnesses-vs-llm-as-judge https://agentreliability.dev/k/deterministic-harnesses-vs-llm-as-judge A decision comparison between mechanical grading (state diffs, executable tests) and model-based grading. Deterministic harnesses win wherever an outcome can be computed; judges win on open-ended quality — at the cost of documented biases that must be measured before the judge's verdicts mean anything. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/deterministic-vs-probabilistic-evaluation https://agentreliability.dev/k/deterministic-vs-probabilistic-evaluation Two grading regimes for agent behavior. Deterministic evaluation checks outcomes mechanically — exact state comparison, executable tests — and reproduces run to run. Probabilistic evaluation delegates judgment to a model or human grader, trading reproducibility for coverage of open-ended outputs. Production reliability work starts deterministic and adds probabilistic grading only where mechanical checks cannot reach. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/evaluating-rag-in-agents https://agentreliability.dev/k/evaluating-rag-in-agents The retrieval layer of an agent fails in three distinct ways — wrong passages retrieved, right passages used unfaithfully, fluent answer ungrounded in either — and each needs its own metric. Reference-free RAG evaluation makes this measurable without annotated ground truth, at the price of inheriting LLM-judge biases. In an agent, retrieved content is also an attack surface, so RAG evaluation and injection testing share fixtures. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/falsify-your-first-guardian https://agentreliability.dev/k/falsify-your-first-guardian A step-by-step procedure for proving that one automated check in your agent pipeline can actually fail: pick the guardian, plant a defect it must catch, confirm red, restore, confirm green, and record the episode. Thirty minutes of work that converts a green light from an assumption into evidence. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/fault-injection-for-agents https://agentreliability.dev/k/fault-injection-for-agents Deliberately breaking the world around an agent to observe how it fails: tool errors, malformed API responses, timeouts, permission denials, adversarial content in retrieved data. The agent equivalent of chaos engineering's real-world events — the goal is verified recovery behavior, not the absence of faults. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/from-incident-to-check https://agentreliability.dev/k/from-incident-to-check The loop that turns every production incident into a permanent guardian: write the incident down, derive the smallest check that would have caught it, falsify that check, and add it to a counted inventory so it can never silently vanish. After enough cycles, your validator is a compressed history of everything that has actually gone wrong. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/gaia-benchmark https://agentreliability.dev/k/gaia-benchmark A benchmark of 466 real-world questions for general AI assistants, designed so answers are unambiguous to grade but require reasoning, multi-modality, web browsing and tool use to reach. Its signature result is the human-AI gap: 92% for human respondents against 15% for GPT-4 with plugins at publication — questions conceptually simple for people, hard for tool-using models. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/goodhart-resistance https://agentreliability.dev/k/goodhart-resistance Designing agent evaluation so that optimizing the reported number does not destroy what the number means. Known countermeasures: separate the metric you optimize from the metric you report, measure consistency across repeated trials rather than best-of-N, and audit with a panel of honesty figures instead of a single score. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/guardian-falsification https://agentreliability.dev/k/guardian-falsification A discipline for proving that an automated check (a "guardian") can actually fail: deliberately plant a defect, watch the guardian go red, restore the correct state, watch it go green. A guardian that has never been observed red is an assumption, not a control. Practiced in production in the Citarium framework, where every validator check family traces back to a real incident. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/human-approval-gates https://agentreliability.dev/k/human-approval-gates The control that keeps an agent's blast radius human-sized: specific action classes (destructive, irreversible, outward-facing) require a human yes before execution, and autonomy expands only as the agent earns it with evidence. The complement to sandboxing — the sandbox contains what the agent can do, gates contain what it may do. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/independent-evaluation https://agentreliability.dev/k/independent-evaluation Evaluation performed by organizations that did not build the system: external red teams, research nonprofits running autonomy evaluations, and government institutes publishing open harnesses. Self-evaluation has a structural conflict of interest — the same incentive gradient Goodhart describes — and the emerging ecosystem of independent evaluators is the field's answer. For buyers, a vendor's willingness to be independently evaluated is itself a reliability signal. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/inspect-eval-framework https://agentreliability.dev/k/inspect-eval-framework The open-source evaluation framework from the UK AI Security Institute: composable solvers and scorers, sandboxed execution of untrusted model code, agent primitives, and a registry of 200+ prebuilt evals. The closest thing agent evaluation has to shared infrastructure — METR's Hawk platform builds on it — and the default answer to "should we write our own harness runner?" (usually no). Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/llm-as-judge https://agentreliability.dev/k/llm-as-judge Using a strong language model to grade another model's output. Cheap, scalable, and surprisingly aligned with human preference on open-ended chat — but a biased instrument: judges favor certain answer positions, longer answers, and their own outputs, and their reasoning limits cap what they can grade. Usable in production only with its biases measured and bounded. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/measuring-agent-reliability https://agentreliability.dev/k/measuring-agent-reliability Which numbers actually predict whether an agent can ship: consistency across repeated trials (pass^k), execution-verified success rates, task time horizon, and a panel of honesty figures instead of one score. Includes a decision table mapping each question a team asks to the metric that answers it. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/opaque-rotating-test-sets https://agentreliability.dev/k/opaque-rotating-test-sets Held-out evaluation tasks that the evaluated system (and its training pipeline) never sees, refreshed on a schedule. Opacity prevents direct optimization against the benchmark; rotation limits contamination and slow overfitting as public data leaks into training corpora. The price is comparability: rotated scores are trends, not fixed leaderboards. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/osworld https://agentreliability.dev/k/osworld A benchmark of 369 open-ended tasks executed in real operating systems (Ubuntu, Windows, macOS) spanning web and desktop apps, file I/O and multi-application workflows. Every task ships its own initial-state setup and an execution-based evaluation script, making it a working template for reproducible computer-use agent evaluation. Headline gap at publication: humans 72.36%, best model 12.24%. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/process-vs-outcome-evaluation https://agentreliability.dev/k/process-vs-outcome-evaluation Two places to point the grader: the final state (outcome) or every intermediate step (process). Outcome grading is cheap, objective and gameable by lucky trajectories; step-level grading catches reasoning that arrives at the right answer wrong, at the cost of expensive labels. The research result: process supervision beat outcome supervision decisively on hard math. For agents, outcome grading gates releases while process signals explain failures and catch right-for-the-wrong-reason passes. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/prompt-injection-testing https://agentreliability.dev/k/prompt-injection-testing Systematically probing whether an agent acts on instructions smuggled into content it reads — tool results, retrieved documents, web pages, emails. For tool-using agents this is the top-ranked risk turned into a test surface: every untrusted input channel gets seeded with adversarial instructions, and the assertion is that the agent's actions never follow them. Runs in the sandbox, gates the release. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/public-benchmarks-vs-private-task-suites https://agentreliability.dev/k/public-benchmarks-vs-private-task-suites Public benchmarks buy comparability, transparency and community scrutiny; private suites buy contamination resistance, domain fit and Goodhart resistance. Production teams need both, in a fixed shape: public suites to place yourself in the field, a private rotating suite to gate your releases. Using either for the other's job is how teams fool themselves. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/red-teaming-your-agent https://agentreliability.dev/k/red-teaming-your-agent A bounded, repeatable adversarial exercise for a tool-using agent: scope the rules of engagement inside the sandbox, enumerate the attack surfaces (injection channels, tool permissions, data paths), run manual and automated passes, triage by blast radius, and convert every confirmed finding into a falsified check so the exercise compounds instead of evaporating. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/regression-gating-model-upgrades https://agentreliability.dev/k/regression-gating-model-upgrades The playbook for swapping the model behind a production agent without discovering the differences in production: freeze the harness, baseline the incumbent with pass^k on critical paths, run the candidate on the identical suite, diff the panel, roll out behind canaries. Model swaps are not optional — providers retire models on notice — so this gate runs on the provider's calendar, not yours. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/sandboxed-execution https://agentreliability.dev/k/sandboxed-execution Contained environments where an agent can act — run code, call tools, mutate state — without its failures reaching production systems. Execution-graded benchmarks already presume one; for dangerous-capability testing the sandbox is the safety case. A sandbox that leaks (network egress, shared state, real credentials) converts a test failure into an incident. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/statistical-rigor-in-evals https://agentreliability.dev/k/statistical-rigor-in-evals Treating an eval score as a measurement with uncertainty instead of a fact: report error bars, model your task set as a sample from a larger population of possible tasks, and never call a two-point difference a regression (or an improvement) without knowing the noise floor. The cheapest upgrade available to most eval pipelines is a standard error next to every number. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/swe-bench-verified https://agentreliability.dev/k/swe-bench-verified The human-validated subset of SWE-bench: 500 tasks filtered from the original test set for quality, graded by unit-test verification against post-PR behavior. Its existence is the interesting fact — a flagship benchmark needed a human audit of its own tasks, which makes "who benchmarks the benchmark?" a standing question for anyone gating decisions on eval numbers. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/testing-mcp-servers https://agentreliability.dev/k/testing-mcp-servers What reliability testing means for Model Context Protocol integrations: conformance to the JSON-RPC contract, capability-negotiation edge cases, and — because the spec itself declares tool descriptions untrusted — injection probes through every server-supplied string. An MCP interface is an API whose consumer is a model, so its test suite gates on success rate like any other API, plus adversarial content like no other API. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/webarena https://agentreliability.dev/k/webarena A self-hosted, realistic web environment for evaluating autonomous agents: functional sites for e-commerce, forum discussion, collaborative software development and content management, plus maps and knowledge-base tools. Tasks are long-horizon and graded on functional correctness of the end state. Headline result at publication: best GPT-4 agent 14.41% against human 78.24%. Sat, 08 Aug 2026 00:00:00 GMT https://agentreliability.dev/k/your-first-agent-evals https://agentreliability.dev/k/your-first-agent-evals The bootstrap path for a team whose agent has no evals at all: harvest real tasks from production logs, write execution-based graders for them, adopt an existing eval framework instead of building one, baseline with repeated trials and error bars, and wire the result into CI as a gate. Two weeks of part-time work; the output is the suite every other reliability practice hangs from. Sat, 08 Aug 2026 00:00:00 GMT