{ "id": "measuring-agent-reliability", "type": "guide", "name": "Measuring agent reliability: bounded metrics and decision tables", "summary": "Which numbers actually predict whether an agent can ship: consistency across repeated trials (pass^k), execution-verified success rates, task time horizon, and a panel of honesty figures instead of one score. Includes a decision table mapping each question a team asks to the metric that answers it.", "locale": "en", "tags": [ "guide", "metrics", "reliability", "benchmarks", "evaluation" ], "relations": [ { "rel": "applies", "target": "goodhart-resistance" }, { "rel": "related", "target": "deterministic-vs-probabilistic-evaluation" }, { "rel": "refined_by", "target": "statistical-rigor-in-evals" }, { "rel": "related", "target": "swe-bench-verified" }, { "rel": "preceded_by", "target": "your-first-agent-evals" } ], "questions": [ "What metrics should I track before shipping an agent?", "What should an agent benchmark measure?", "Is pass@1 enough to decide an agent is production-ready?" ], "claims": [ { "id": "c1", "text": "pass^k measures the probability that an agent succeeds on all k repeated trials of the same task; on tau-bench, agents under 50% single-attempt success drop under 25% at pass^8 in retail, so consistency and capability are different axes.", "sources": [ "tau-bench" ], "confidence": 0.9 }, { "id": "c2", "text": "Execution-verified resolution rate on real tasks is a bounded, reproducible measure: SWE-bench grades 2,294 real GitHub issues by running the result, and its initial best baseline resolved 1.96%.", "sources": [ "swe-bench" ], "confidence": 0.9 }, { "id": "c3", "text": "METR's time-horizon metric — the length of tasks agents can complete — has risen exponentially across six years of frontier models, making it a capability trend measure rather than a reliability measure.", "sources": [ "metr" ], "confidence": 0.85 }, { "id": "c4", "text": "NIST AI RMF makes Measure one of its four core functions: metric selection and ongoing measurement are governance obligations across the lifecycle, not an engineering nicety.", "sources": [ "nist-ai-rmf" ], "confidence": 0.85 }, { "id": "c5", "text": "The Citarium audit command reports honesty figures — tier mix against quota, unused sources, moat count, oldest verification — as a panel, because a single aggregated score invites gaming.", "sources": [ "citarium-cli-audit" ], "confidence": 0.9 }, { "id": "c6", "text": "The EUREKA platform maintains a public catalog of 69 agentic benchmarks and tracks the field's shift from measuring capability to measuring agency.", "sources": [ "santismm-eureka" ], "confidence": 0.85 } ], "takeaways": [ "Users experience pass^k, not pass@1 — gate on consistency, report the panel, never one aggregate score.", "Success is defined by execution (state matches goal), never by reading the transcript.", "Time horizon tells you what to delegate; it says nothing about reliability at a fixed point.", "Check the public benchmark shelf before inventing your own suite." ], "faqs": [], "evidence_tier": "secondary", "evidence": { "level": "production", "source_types": [ "production_system", "personal_experience", "benchmark", "paper" ] }, "moat_flag": true, "winning_edge": "Signed by a practitioner who operates both a benchmark catalog (EUREKA) and a production audit pipeline; the decision table encodes choices a team actually faces, with every threshold sourced instead of invented.", "confidence": 0.85, "last_verified": "2026-08-08", "body": "## The question behind every metric\n\n\"Is the agent good?\" is not measurable. \"Does the agent complete this\ntask, verified by execution, k times in a row, inside budget?\" is. The\nmetrics below are bounded — each has a defined range, a defined\ndenominator, and a defined way to fail — because unbounded or aggregated\nscores are where gaming starts.\n\n## The four metrics that earn their place\n\n1. **pass^k — consistency (claim c1).** The probability of succeeding\n all k times on the same task. This is the production metric: a user\n who hits the agent eight times experiences pass^8, not pass@1. The\n observed gap (under 50% at one attempt, under 25% at eight in\n tau-bench retail) is the difference between a demo and a deploy.\n2. **Execution-verified success rate (claim c2).** Success defined by\n running the result — tests pass, final state matches goal state —\n never by reading the transcript. Real-task suites (SWE-bench's 2,294\n GitHub issues) set the standard here.\n3. **Time horizon (claim c3).** How long a task the agent can carry\n autonomously. Track it to know what class of work to trust the agent\n with; do not confuse its exponential trend with reliability at any\n fixed point.\n4. **Honesty-figure panels (claim c5).** For the evaluation system\n itself: a set of small, separately-alarmed numbers (coverage vs\n quota, unused inventory, staleness) that resist collapse into one\n optimizable score.\n\n## Decision table\n\n| Question the team is asking | Metric that answers it | Ship gate example |\n|---|---|---|\n| Will it work for a user who retries? | pass^k on your task suite | pass^4 above target on critical paths |\n| Did it really do the task? | execution-verified success | grader outside sandbox, state-diff based |\n| What work can we delegate? | time horizon | tasks under proven horizon only |\n| Is our eval still honest? | honesty-figure panel | zero unused fixtures, staleness bounded |\n| Are we governance-complete? | Measure-function mapping (claim c4) | each risk has a metric and an owner |\n\n## Choosing benchmarks\n\nDo not invent a benchmark before checking the shelf: the field already\nhas dozens of agentic suites — a public catalog tracks 69 of them\n(claim c6). Adopt the closest real-task suite for comparability, then\nadd a private, rotating extension for your domain (see opaque rotating\ntest sets) so your number cannot be trained against.\n\n## Anti-patterns\n\n- **One aggregate quality score.** Invites Goodhart dynamics; keep the\n panel disaggregated (claim c5).\n- **Transcript-graded success.** Plausible narration is what these\n systems do best; grade state, not prose.\n- **pass@1 in the deck, pass^k in production.** Report the metric your\n users will experience.", "canonical_url": "https://agentreliability.dev/k/measuring-agent-reliability", "api_url": "https://agentreliability.dev/api/k/measuring-agent-reliability.json", "jsonld": { "@context": "https://schema.org", "name": "Measuring agent reliability: bounded metrics and decision tables", "description": "Which numbers actually predict whether an agent can ship: consistency across repeated trials (pass^k), execution-verified success rates, task time horizon, and a panel of honesty figures instead of one score. Includes a decision table mapping each question a team asks to the metric that answers it.", "url": "https://agentreliability.dev/k/measuring-agent-reliability", "license": "https://spdx.org/licenses/CC-BY-4.0.html", "dateModified": "2026-08-08", "citation": [ { "@type": "CreativeWork", "name": "Citarium audit command source (@citarium/cli v0.1.0, commands.ts)" }, { "@type": "CreativeWork", "name": "EUREKA — applied AI, agents and robotics knowledge platform", "url": "https://santismm.com" }, { "@type": "CreativeWork", "name": "NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1)", "url": "https://www.nist.gov/itl/ai-risk-management-framework" }, { "@type": "CreativeWork", "name": "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", "url": "https://arxiv.org/abs/2310.06770" }, { "@type": "CreativeWork", "name": "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains", "url": "https://arxiv.org/abs/2406.12045" }, { "@type": "CreativeWork", "name": "METR — Model Evaluation & Threat Research", "url": "https://metr.org/" } ], "author": { "@type": "Person", "name": "Santiago Santa María Morales", "jobTitle": "practitioner — harness engineering and agent evaluation in production" }, "@type": "Article", "headline": "Measuring agent reliability: bounded metrics and decision tables" } }