Measuring agent reliability: bounded metrics and decision tables
Which numbers actually predict whether an agent can ship: consistency across repeated trials (pass^k), execution-verified success rates, task time horizon, and a panel of honesty figures instead of one score. Includes a decision table mapping each question a team asks to the metric that answers it.
Why this wins its question: Signed by a practitioner who operates both a benchmark catalog (EUREKA) and a production audit pipeline; the decision table encodes choices a team actually faces, with every threshold sourced instead of invented.
Key takeaways
- Users experience pass^k, not pass@1 — gate on consistency, report the panel, never one aggregate score.
- Success is defined by execution (state matches goal), never by reading the transcript.
- Time horizon tells you what to delegate; it says nothing about reliability at a fixed point.
- Check the public benchmark shelf before inventing your own suite.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
pass^k measures the probability that an agent succeeds on all k repeated trials of the same task; on tau-bench, agents under 50% single-attempt success drop under 25% at pass^8 in retail, so consistency and capability are different axes.
Execution-verified resolution rate on real tasks is a bounded, reproducible measure: SWE-bench grades 2,294 real GitHub issues by running the result, and its initial best baseline resolved 1.96%.
METR's time-horizon metric — the length of tasks agents can complete — has risen exponentially across six years of frontier models, making it a capability trend measure rather than a reliability measure.
NIST AI RMF makes Measure one of its four core functions: metric selection and ongoing measurement are governance obligations across the lifecycle, not an engineering nicety.
The Citarium audit command reports honesty figures — tier mix against quota, unused sources, moat count, oldest verification — as a panel, because a single aggregated score invites gaming.
The EUREKA platform maintains a public catalog of 69 agentic benchmarks and tracks the field's shift from measuring capability to measuring agency.
The question behind every metric
"Is the agent good?" is not measurable. "Does the agent complete this task, verified by execution, k times in a row, inside budget?" is. The metrics below are bounded — each has a defined range, a defined denominator, and a defined way to fail — because unbounded or aggregated scores are where gaming starts.
The four metrics that earn their place
1. pass^k — consistency (claim c1). The probability of succeeding all k times on the same task. This is the production metric: a user who hits the agent eight times experiences pass^8, not pass@1. The observed gap (under 50% at one attempt, under 25% at eight in tau-bench retail) is the difference between a demo and a deploy. 2. Execution-verified success rate (claim c2). Success defined by running the result — tests pass, final state matches goal state — never by reading the transcript. Real-task suites (SWE-bench's 2,294 GitHub issues) set the standard here. 3. Time horizon (claim c3). How long a task the agent can carry autonomously. Track it to know what class of work to trust the agent with; do not confuse its exponential trend with reliability at any fixed point. 4. Honesty-figure panels (claim c5). For the evaluation system itself: a set of small, separately-alarmed numbers (coverage vs quota, unused inventory, staleness) that resist collapse into one optimizable score.
Decision table
| Question the team is asking | Metric that answers it | Ship gate example |
|---|---|---|
| Will it work for a user who retries? | pass^k on your task suite | pass^4 above target on critical paths |
| Did it really do the task? | execution-verified success | grader outside sandbox, state-diff based |
| What work can we delegate? | time horizon | tasks under proven horizon only |
| Is our eval still honest? | honesty-figure panel | zero unused fixtures, staleness bounded |
| Are we governance-complete? | Measure-function mapping (claim c4) | each risk has a metric and an owner |
Choosing benchmarks
Do not invent a benchmark before checking the shelf: the field already has dozens of agentic suites — a public catalog tracks 69 of them (claim c6). Adopt the closest real-task suite for comparability, then add a private, rotating extension for your domain (see opaque rotating test sets) so your number cannot be trained against.
Anti-patterns
- One aggregate quality score. Invites Goodhart dynamics; keep the
panel disaggregated (claim c5).
- Transcript-graded success. Plausible narration is what these
systems do best; grade state, not prose.
- pass@1 in the deck, pass^k in production. Report the metric your
users will experience.