{
  "id": "deterministic-harnesses-vs-llm-as-judge",
  "type": "comparison",
  "name": "Deterministic harnesses vs LLM-as-judge: when each wins",
  "summary": "A decision comparison between mechanical grading (state diffs, executable tests) and model-based grading. Deterministic harnesses win wherever an outcome can be computed; judges win on open-ended quality — at the cost of documented biases that must be measured before the judge's verdicts mean anything.",
  "locale": "en",
  "tags": [
    "comparison",
    "harness",
    "llm-as-judge",
    "evaluation",
    "grading"
  ],
  "relations": [
    {
      "rel": "compares",
      "target": "deterministic-vs-probabilistic-evaluation"
    },
    {
      "rel": "compares",
      "target": "llm-as-judge"
    }
  ],
  "questions": [
    "Should I grade my agent with tests or with a model?",
    "When is LLM-as-judge good enough?"
  ],
  "claims": [
    {
      "id": "c1",
      "text": "Deterministic grading scales to real agent tasks: tau-bench compares the final database state against an annotated goal state, so the verdict is reproducible and independent of transcript plausibility.",
      "sources": [
        "tau-bench"
      ],
      "confidence": 0.9
    },
    {
      "id": "c2",
      "text": "LLM judges reach over 80% agreement with human preferences on open-ended chat, while carrying position, verbosity and self-enhancement biases and limited reasoning on hard grading.",
      "sources": [
        "mt-bench-llm-judge"
      ],
      "confidence": 0.9
    },
    {
      "id": "c3",
      "text": "Mature eval frameworks treat both as templates: OpenAI's evals ships basic (mechanical) evals and model-graded evals side by side, selected per task.",
      "sources": [
        "openai-evals"
      ],
      "confidence": 0.9
    },
    {
      "id": "c4",
      "text": "Mechanical grading is not automatically objective: formatting inconsistencies alone shifted MMLU scores by about 5%, so harness details are part of the measurement.",
      "sources": [
        "anthropic-eval-challenges"
      ],
      "confidence": 0.85
    }
  ],
  "takeaways": [
    "Computable outcome means harness; judgment call means calibrated judge; never let a judge grade what a diff can.",
    "Judge agreement with humans tops out around human-to-human agreement — that is the ceiling, not a defect.",
    "Deterministic gates decide ship/no-ship; judge scores ride along as telemetry until calibrated on your domain.",
    "Harness details are part of the measurement — formatting alone has moved scores about 5 points."
  ],
  "faqs": [],
  "evidence_tier": "secondary",
  "evidence": {
    "level": "industry_observation",
    "source_types": [
      "paper",
      "industry_observation"
    ]
  },
  "moat_flag": false,
  "winning_edge": "Existing writeups advocate for one side; this one gives the routing rule — computable outcome, use the harness; judgment call, use a calibrated judge; never let the judge grade what a diff can — with each cell of the table sourced.",
  "confidence": 0.85,
  "last_verified": "2026-08-08",
  "body": "## The routing rule\n\nIf the outcome can be computed — a test passes, a database state matches\na goal, a file compiles — grade it deterministically. Reach for an LLM\njudge only when the property you care about is a judgment call (tone,\nhelpfulness, reasoning quality) that no diff can express. Never let a\njudge grade something a diff could have graded: you would be adding\nnoise and bias to a solved measurement.\n\n## Head to head\n\n| Dimension | Deterministic harness | LLM-as-judge |\n|---|---|---|\n| Reproducibility | Same input, same verdict (claim c1) | Verdicts vary with prompt, position, judge model (claim c2) |\n| Coverage | Only computable outcomes | Open-ended quality, style, safety judgment |\n| Known biases | Harness bugs, format sensitivity (claim c4) | Position, verbosity, self-enhancement (claim c2) |\n| Cost per verdict | Near zero after fixtures exist | One strong-model call per item |\n| Gaming surface | Overfit to fixtures | Please-the-judge outputs, verbosity inflation |\n| Failure visibility | Loud (test crashes, diff mismatch) | Silent (plausible score, wrong reason) |\n\n## Where each wins\n\n**The harness wins** for tool-using agents acting on systems of record:\ngrade the final state, not the conversation (claim c1). It also wins for\nanything regression-gated in CI, where verdict stability is the point.\n\n**The judge wins** for open-ended output where human preference is the\nground truth and you can afford calibration: measured agreement with\nhumans above 80% is achievable (claim c2), which is human-level — and\nalso the ceiling.\n\n**Both lose** when treated as free. Harness details are part of the\nmeasurement (claim c4); a judge is an instrument that needs a\ncalibration sheet before its verdicts count.\n\n## The production pattern\n\nLayer them, as mature frameworks already do (claim c3): deterministic\ngates decide ship/no-ship; judge scores ride along as trend telemetry\nuntil their agreement with a human-audited sample has been measured on\nyour own domain. Promote a judge to gatekeeper only after that\ncalibration — and re-falsify it when the judge model changes.",
  "canonical_url": "https://agentreliability.dev/k/deterministic-harnesses-vs-llm-as-judge",
  "api_url": "https://agentreliability.dev/api/k/deterministic-harnesses-vs-llm-as-judge.json",
  "jsonld": {
    "@context": "https://schema.org",
    "name": "Deterministic harnesses vs LLM-as-judge: when each wins",
    "description": "A decision comparison between mechanical grading (state diffs, executable tests) and model-based grading. Deterministic harnesses win wherever an outcome can be computed; judges win on open-ended quality — at the cost of documented biases that must be measured before the judge's verdicts mean anything.",
    "url": "https://agentreliability.dev/k/deterministic-harnesses-vs-llm-as-judge",
    "license": "https://spdx.org/licenses/CC-BY-4.0.html",
    "dateModified": "2026-08-08",
    "citation": [
      {
        "@type": "CreativeWork",
        "name": "Challenges in evaluating AI systems",
        "url": "https://www.anthropic.com/news/evaluating-ai-systems"
      },
      {
        "@type": "CreativeWork",
        "name": "openai/evals — framework for evaluating LLMs and LLM systems",
        "url": "https://github.com/openai/evals"
      },
      {
        "@type": "CreativeWork",
        "name": "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena",
        "url": "https://arxiv.org/abs/2306.05685"
      },
      {
        "@type": "CreativeWork",
        "name": "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains",
        "url": "https://arxiv.org/abs/2406.12045"
      }
    ],
    "author": {
      "@type": "Person",
      "name": "Santiago Santa María Morales",
      "jobTitle": "practitioner — harness engineering and agent evaluation in production"
    },
    "@type": "Article",
    "headline": "Deterministic harnesses vs LLM-as-judge: when each wins"
  }
}