{
  "id": "calibrating-your-llm-judge",
  "type": "guide",
  "name": "Calibrating your LLM judge before it gates anything",
  "summary": "The protocol that turns an LLM judge from vibes into an instrument: measure its agreement against a human-audited sample, probe its known biases directly (position, verbosity, self-preference), put error bars on every comparison it feeds, and define the triggers that force recalibration. A judge is promoted to gatekeeper only after this protocol — and demoted the day its judge model changes.",
  "locale": "en",
  "tags": [
    "guide",
    "llm-as-judge",
    "calibration",
    "grading",
    "evaluation"
  ],
  "relations": [
    {
      "rel": "applies",
      "target": "llm-as-judge"
    },
    {
      "rel": "related",
      "target": "deterministic-harnesses-vs-llm-as-judge"
    },
    {
      "rel": "related",
      "target": "statistical-rigor-in-evals"
    },
    {
      "rel": "related",
      "target": "regression-gating-model-upgrades"
    }
  ],
  "questions": [
    "How do I know if I can trust my LLM judge?",
    "How many samples do I need to calibrate a judge?",
    "When does a judge need recalibration?"
  ],
  "claims": [
    {
      "id": "c1",
      "text": "Strong LLM judges can reach over 80% agreement with human preferences — the same level humans reach with each other — which makes human-agreement measurement both the calibration method and the ceiling.",
      "sources": [
        "mt-bench-llm-judge"
      ],
      "confidence": 0.9
    },
    {
      "id": "c2",
      "text": "The failure modes to probe are documented: position bias, verbosity bias, self-enhancement bias, and limited reasoning ability on hard grading tasks.",
      "sources": [
        "mt-bench-llm-judge"
      ],
      "confidence": 0.95
    },
    {
      "id": "c3",
      "text": "Judge-fed comparisons need uncertainty accounting like any eval: treat graded items as samples from a larger population and report standard errors, or noise reads as verdicts.",
      "sources": [
        "error-bars-evals"
      ],
      "confidence": 0.85
    },
    {
      "id": "c4",
      "text": "Model-graded evaluation is a first-class template in open eval frameworks, so the calibration protocol wraps standard tooling rather than bespoke infrastructure.",
      "sources": [
        "openai-evals"
      ],
      "confidence": 0.9
    }
  ],
  "takeaways": [
    "A judge's verdicts mean nothing until its agreement with humans is measured on your domain — published numbers do not transfer.",
    "Probe each documented bias directly - swap positions, control length, test self-preference - instead of hoping they average out.",
    "Human-level agreement (~80%) is the ceiling, not a defect; design gates that tolerate it.",
    "Recalibrate on every judge-model change and every prompt change to the judge — the instrument changed."
  ],
  "faqs": [],
  "evidence_tier": "secondary",
  "evidence": {
    "level": "benchmark",
    "source_types": [
      "benchmark",
      "paper"
    ]
  },
  "moat_flag": false,
  "winning_edge": "The corpus-completing piece: existing coverage documents judge biases or reports agreement numbers; this guide is the operational protocol — sample, audit, probe, bound, retrigger — that turns those findings into a promotion decision for the judge.",
  "confidence": 0.85,
  "last_verified": "2026-08-08",
  "body": "## Why calibrate at all\n\nA judge model will always produce a score. The question a gate has to\nanswer is whether that score tracks anything. Published agreement\nnumbers (claim c1) were measured on other domains, other prompts,\nother judge models — none of it transfers to your setup automatically.\nCalibration is the transfer measurement.\n\n## The protocol\n\n1. **Build the audit sample.** Pull 100-200 real graded items from\n   your own pipeline, stratified across the score range — not the easy\n   tail. Have humans grade them blind, with the same rubric the judge\n   gets. This human layer is the reference; budget for disagreement\n   review between annotators.\n2. **Measure agreement.** Compare judge verdicts against the human\n   reference. Expect the ceiling around human-human agreement (claim\n   c1): a judge at 80% agreement with a human panel that agrees 81%\n   with itself is calibrated, not broken.\n3. **Probe each documented bias (claim c2).**\n   - *Position*: re-grade pairwise items with the order swapped; the\n     flip rate is your position bias.\n   - *Verbosity*: grade length-controlled variants of the same\n     content; score drift with length is the verbosity bias.\n   - *Self-preference*: if the judge model also generates candidates,\n     measure the delta it awards its own outputs.\n   - *Reasoning limit*: check agreement separately on the hardest\n     stratum; judges degrade there first.\n4. **Bound the noise (claim c3).** Re-run the judge on the same items;\n   the run-to-run variance plus the sampling error defines the minimum\n   score difference that means anything. Publish that number next to\n   every judge-fed comparison.\n5. **Write the promotion decision.** The judge may gate only the\n   decisions whose required precision its measured agreement supports.\n   Everything else it reports as telemetry.\n\n## Recalibration triggers\n\nRecalibrate — not spot-check — when any of these change: the judge\nmodel (including provider-forced upgrades), the judge prompt or\nrubric, the task distribution being graded, or the candidate model\nfamily. The instrument changed; its old calibration is a memory. Wire\nthe trigger into the same pipeline that catches model deprecations.\n\n## What this does not fix\n\nCalibration bounds a judge; it does not make it deterministic. Ship\ngates that need reproducible verdicts stay on execution-based grading\n(see the harness-vs-judge comparison). The calibrated judge earns the\nopen-ended slice — style, helpfulness, judgment — and the standard\ntooling makes running it cheap (claim c4). The expensive part was\nnever the API call; it is the human reference sample, and that cost\nis the honest price of trusting a model's opinion.",
  "canonical_url": "https://agentreliability.dev/k/calibrating-your-llm-judge",
  "api_url": "https://agentreliability.dev/api/k/calibrating-your-llm-judge.json",
  "jsonld": {
    "@context": "https://schema.org",
    "name": "Calibrating your LLM judge before it gates anything",
    "description": "The protocol that turns an LLM judge from vibes into an instrument: measure its agreement against a human-audited sample, probe its known biases directly (position, verbosity, self-preference), put error bars on every comparison it feeds, and define the triggers that force recalibration. A judge is promoted to gatekeeper only after this protocol — and demoted the day its judge model changes.",
    "url": "https://agentreliability.dev/k/calibrating-your-llm-judge",
    "license": "https://spdx.org/licenses/CC-BY-4.0.html",
    "dateModified": "2026-08-08",
    "citation": [
      {
        "@type": "CreativeWork",
        "name": "openai/evals — framework for evaluating LLMs and LLM systems",
        "url": "https://github.com/openai/evals"
      },
      {
        "@type": "CreativeWork",
        "name": "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena",
        "url": "https://arxiv.org/abs/2306.05685"
      },
      {
        "@type": "CreativeWork",
        "name": "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations",
        "url": "https://arxiv.org/abs/2411.00640"
      }
    ],
    "author": {
      "@type": "Person",
      "name": "Santiago Santa María Morales",
      "jobTitle": "practitioner — harness engineering and agent evaluation in production"
    },
    "@type": "Article",
    "headline": "Calibrating your LLM judge before it gates anything"
  }
}