{
  "id": "choosing-k-for-pass-hat-k",
  "type": "guide",
  "name": "Choosing k and the threshold for pass^k",
  "summary": "The corpus recommends pass^k everywhere and never says which k or what number to demand. This is the arithmetic and the decision rule: k comes from how many times the task runs between human reviews, the threshold comes from the cost of one failure, and both must be set before you see the score. Includes why pass@k and pass^k move in opposite directions and why measured pass^k is almost never p^k.",
  "locale": "en",
  "tags": [
    "guide",
    "evals",
    "reliability",
    "thresholds",
    "metrics",
    "statistics",
    "gating"
  ],
  "relations": [
    {
      "rel": "applies",
      "target": "agent-release-gates"
    },
    {
      "rel": "related",
      "target": "statistical-rigor-in-evals"
    },
    {
      "rel": "related",
      "target": "measuring-agent-reliability"
    },
    {
      "rel": "related",
      "target": "deterministic-vs-probabilistic-evaluation"
    },
    {
      "rel": "explains",
      "target": "agent-reliability-glossary"
    }
  ],
  "questions": [
    "How do I choose k for pass^k?",
    "What pass^k threshold should I require before shipping an agent?",
    "Why is pass^k so much lower than my success rate?"
  ],
  "claims": [
    {
      "id": "c1",
      "text": "pass^k is the probability that a system succeeds on all k repeated trials of the same task, introduced with tau-bench as a reliability metric distinct from single-attempt capability.",
      "sources": [
        "tau-bench"
      ],
      "confidence": 0.9
    },
    {
      "id": "c2",
      "text": "The gap between capability and reliability is large in practice: state-of-the-art function-calling agents succeeded on under 50% of tau-bench tasks, and pass^8 fell under 25% in the retail domain.",
      "sources": [
        "tau-bench"
      ],
      "confidence": 0.9
    },
    {
      "id": "c3",
      "text": "Repeated sampling moves the opposite metric in the opposite direction: on HumanEval, Codex solved 28.8% of problems with one sample and 70.2% with 100 samples per problem, because any-of-k credits a single success.",
      "sources": [
        "codex-humaneval"
      ],
      "confidence": 0.95
    },
    {
      "id": "c4",
      "text": "A pass^k estimate is a sample statistic and needs uncertainty accounting like any eval score: treat graded items as draws from a larger population and report standard errors alongside the point estimate.",
      "sources": [
        "error-bars-evals"
      ],
      "confidence": 0.85
    },
    {
      "id": "c5",
      "text": "Incidental setup details can move eval scores by around 5 percentage points on their own, so a threshold quoted without a stated harness and prompt configuration is not a reproducible gate.",
      "sources": [
        "anthropic-eval-challenges"
      ],
      "confidence": 0.85
    }
  ],
  "takeaways": [
    "k is not a compute-budget choice - it is how many times the task runs between human reviews.",
    "pass@k rises with k and pass^k falls with k; they answer opposite questions, so never quote one as the other.",
    "Set the threshold from the cost of a single failure before you see the score, or you will be negotiating with your own results.",
    "Measured pass^k is almost never the p^k the independence assumption predicts - correlated failures make it worse, and only measurement tells you which."
  ],
  "faqs": [],
  "evidence_tier": "secondary",
  "evidence": {
    "level": "benchmark",
    "source_types": [
      "benchmark",
      "paper"
    ]
  },
  "moat_flag": false,
  "winning_edge": "Every source that recommends pass^k defines it and stops. The published material answers \"what is pass^k\"; a team about to gate a release needs \"which k, what number, and why is mine lower than the multiplication says\" — the two selection rules, the independence trap, and the sample budget, in one place and with the arithmetic worked.",
  "confidence": 0.85,
  "last_verified": "2026-08-09",
  "body": "## Two metrics that look alike and point opposite ways\n\n`pass@k` asks whether **any** of k attempts succeeded. `pass^k` asks\nwhether **all** k succeeded (claim c1). The names differ by one\ncharacter and the numbers move in opposite directions as k grows:\nCodex went from 28.8% to 70.2% on HumanEval by sampling 100 times\ninstead of once (claim c3), while tau-bench agents that clear under 50%\nof tasks on a single attempt drop below 25% at pass^8 in retail (claim\nc2).\n\nThe distinction is what you are buying with the extra attempts. `pass@k`\nis a capability measure and assumes something downstream can pick the\nwinning attempt — a test suite, a compiler, a human reviewer. If no such\nfilter exists in production, `pass@k` is a number about a system you are\nnot running. `pass^k` is the reliability measure: it describes what the\nuser experiences when the same task runs k times and nobody is checking.\n\n## Choosing k\n\nk is not a budget decision, and it is not \"as many as we can afford\".\n**k is the number of times the task runs between human reviews.**\n\nWork it out from your own traffic:\n\n- A weekly report the agent generates and an analyst reads before it\n  circulates: a human filter exists every run, so k = 1. Use pass@1 and\n  spend your budget on the review, not on repeated trials.\n- A support workflow that fires 200 times a day and is spot-checked\n  weekly: roughly 1,400 unreviewed runs between reviews. You cannot\n  measure pass^1400, but you now know your gate is about the tail, not\n  the average — pick the largest k you can afford to measure and treat\n  it as a lower bound on the exposure, not a description of it.\n- A nightly migration step in a pipeline of 12 sequential agent calls,\n  none reviewed until the pipeline finishes: k = 12, and the relevant\n  question is whether the whole chain survives, not whether each step\n  usually works.\n\nTwo corrections to the naive reading. First, the k that matters is\nper-task-instance, not global volume: 200 runs of *different* tasks is a\nsample of size 200 at k=1, not pass^200. Second, if a retry is automatic\nand cheap and the failure is detectable, that retry belongs in the\nsystem under test — measure the agent *with* its retry loop at the k\nyour reviews actually imply, rather than crediting a retry the harness\nperforms and production does not.\n\n## Choosing the threshold\n\nSet the number from the cost of one failure, and set it **before** you\nsee the score. The order matters more than the value: a threshold chosen\nafter the measurement is a description of what you got, not a gate.\n\nThe rule that survives contact with a release meeting: state the failure\nbudget in incidents, then convert. If the workflow runs N unreviewed\ntimes per review cycle and you can tolerate at most one bad outcome per\ncycle, you need a per-run failure rate below 1/N — and it is the\nper-run rate, not the pass^k figure, that you should be arguing about.\npass^k is how you *observe* that rate under repetition; the failure\nbudget is what you are actually deciding.\n\nThen write down, next to the number: the harness version, the prompt\nconfiguration, the model version, and the task set. Incidental\nconfiguration differences alone are worth several points of score\n(claim c5), so an undocumented threshold cannot be re-checked next\nquarter — and a gate you cannot re-run is a gate that quietly stops\ngating.\n\n## The independence trap\n\nThe arithmetic everyone does first: if each trial succeeds with\nprobability p, then pass^k = p^k. At p = 0.95, pass^10 ≈ 0.60. At\np = 0.99, pass^10 ≈ 0.90. This is worth doing once, because it explains\nwhy a \"95% reliable\" agent feels broken over a working week and why\nbuying reliability at the third nine is the expensive part.\n\nIt is also wrong in a specific direction. Agent failures are not\nindependent draws: the same task hits the same ambiguous instruction,\nthe same brittle selector, the same tool that rate-limits at the same\npoint. Correlated failures cluster, so measured pass^k is typically\n**below** p^k — and the clustering is itself the diagnosis, because a\ntask that fails 4 times out of 8 in one cluster and never in another\nhas a deterministic bug wearing a probabilistic costume. Use p^k to size\nthe sample and to argue for the budget; use the measured value to gate,\nand look at the failure clustering before you look at the mean.\n\n## The sample budget\n\nA pass^k estimate over n tasks costs n × k runs, and the estimate has\nerror bars like any other (claim c4). Two consequences worth planning\nfor:\n\n- **Rare events need runs you may not want to pay for.** Distinguishing\n  a 1% failure rate from a 3% failure rate takes hundreds of trials per\n  task. If the budget does not stretch, say so in the release note —\n  \"pass^8 = 0.94 ± 0.05, cannot distinguish from 0.90\" is an honest\n  gate; \"pass^8 = 0.94\" alone invites a comparison the data does not\n  support.\n- **Report the interval, gate on the bound.** Gate on the lower end of\n  the confidence interval, not the point estimate. It is the only\n  version of the number that stays true when the same suite runs again\n  next week.\n\n## Where this lands\n\nBoth selections — k and the threshold — belong in the release gate\nconfiguration, declared and version-controlled next to the other\nthresholds, not chosen per release by whoever is shipping. That is the\nwhole point of writing them down before the run: the gate holds when\nthe number comes back inconvenient.",
  "canonical_url": "https://agentreliability.dev/k/choosing-k-for-pass-hat-k",
  "api_url": "https://agentreliability.dev/api/k/choosing-k-for-pass-hat-k.json",
  "jsonld": {
    "@context": "https://schema.org",
    "name": "Choosing k and the threshold for pass^k",
    "description": "The corpus recommends pass^k everywhere and never says which k or what number to demand. This is the arithmetic and the decision rule: k comes from how many times the task runs between human reviews, the threshold comes from the cost of one failure, and both must be set before you see the score. Includes why pass@k and pass^k move in opposite directions and why measured pass^k is almost never p^k.",
    "url": "https://agentreliability.dev/k/choosing-k-for-pass-hat-k",
    "license": "https://spdx.org/licenses/CC-BY-4.0.html",
    "dateModified": "2026-08-09",
    "citation": [
      {
        "@type": "CreativeWork",
        "name": "Challenges in evaluating AI systems",
        "url": "https://www.anthropic.com/news/evaluating-ai-systems"
      },
      {
        "@type": "CreativeWork",
        "name": "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains",
        "url": "https://arxiv.org/abs/2406.12045"
      },
      {
        "@type": "CreativeWork",
        "name": "Evaluating Large Language Models Trained on Code",
        "url": "https://arxiv.org/abs/2107.03374"
      },
      {
        "@type": "CreativeWork",
        "name": "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations",
        "url": "https://arxiv.org/abs/2411.00640"
      }
    ],
    "author": {
      "@type": "Person",
      "name": "Santiago Santa María Morales",
      "jobTitle": "practitioner — harness engineering and agent evaluation in production"
    },
    "@type": "Article",
    "headline": "Choosing k and the threshold for pass^k"
  }
}