{
  "id": "public-benchmarks-vs-private-task-suites",
  "type": "comparison",
  "name": "Public benchmarks vs private task suites: what each buys you",
  "summary": "Public benchmarks buy comparability, transparency and community scrutiny; private suites buy contamination resistance, domain fit and Goodhart resistance. Production teams need both, in a fixed shape: public suites to place yourself in the field, a private rotating suite to gate your releases. Using either for the other's job is how teams fool themselves.",
  "locale": "en",
  "tags": [
    "comparison",
    "benchmarks",
    "evaluation",
    "contamination",
    "goodhart"
  ],
  "relations": [
    {
      "rel": "compares",
      "target": "opaque-rotating-test-sets"
    },
    {
      "rel": "compares",
      "target": "gaia-benchmark"
    },
    {
      "rel": "related",
      "target": "measuring-agent-reliability"
    },
    {
      "rel": "related",
      "target": "goodhart-resistance"
    }
  ],
  "questions": [
    "Should I evaluate my agent on public benchmarks or build my own suite?",
    "Why do teams with great benchmark scores still ship broken agents?"
  ],
  "claims": [
    {
      "id": "c1",
      "text": "The public side has a reference implementation: HELM publishes transparent, reproducible benchmark suites and leaderboards across many scenarios and metrics, explicitly to standardize open evaluation practice.",
      "sources": [
        "helm-stanford"
      ],
      "confidence": 0.9
    },
    {
      "id": "c2",
      "text": "Eval tooling supports the private side natively: OpenAI's evals framework lets teams build private evals on their own data without public exposure.",
      "sources": [
        "openai-evals"
      ],
      "confidence": 0.9
    },
    {
      "id": "c3",
      "text": "A public, static target invites Goodhart dynamics — optimizing the known metric degrades what it measures through at least four distinct mechanisms.",
      "sources": [
        "goodhart-variants"
      ],
      "confidence": 0.85
    },
    {
      "id": "c4",
      "text": "Benchmark integrity is an active threat surface: METR publishes research and datasets on evaluation integrity for autonomous systems.",
      "sources": [
        "metr"
      ],
      "confidence": 0.8
    },
    {
      "id": "c5",
      "text": "Public agent benchmarks still discriminate today: humans beat the best models by tens of points on GAIA-style real-world assistant tasks (92% vs 15% at publication), so saturation is not yet the dominant failure.",
      "sources": [
        "gaia-benchmark-paper"
      ],
      "confidence": 0.85
    }
  ],
  "takeaways": [
    "Public benchmarks answer \"where do we stand\"; private suites answer \"may this ship\" — one suite doing both jobs is how teams fool themselves.",
    "A public static target invites Goodhart optimization by construction.",
    "Gate releases only on the private, rotating suite; track public placement as telemetry.",
    "Publish your public scores — community scrutiny is the cheapest auditing available."
  ],
  "faqs": [],
  "evidence_tier": "secondary",
  "evidence": {
    "level": "industry_observation",
    "source_types": [
      "industry_observation",
      "paper"
    ]
  },
  "moat_flag": false,
  "winning_edge": "Assigns each benchmark type a job description and a failure mode instead of debating which is \"better\" — and gives the combined operating pattern (public to locate, private to gate) as the deliverable.",
  "confidence": 0.85,
  "last_verified": "2026-08-08",
  "body": "## The jobs are different\n\nA public benchmark answers \"where does my system stand in the field?\"\nA private suite answers \"may this release ship?\" The failure pattern\nbehind most \"great scores, broken agent\" stories is one suite doing\nboth jobs.\n\n## Head to head\n\n| Dimension | Public benchmark | Private task suite |\n|---|---|---|\n| Comparability | Across labs and models (claim c1) | Only against your own history |\n| Contamination | Training data absorbs it over time | Resistant while it stays private (claim c2) |\n| Goodhart exposure | High — the target is known (claim c3) | Low, if rotated and access-controlled |\n| Domain fit | Generic by design | Encodes your APIs, policies, edge cases |\n| Scrutiny | Community-audited methodology | Your bugs are your own |\n| Cost | Free to adopt | You build and maintain the fixtures |\n\n## Where public wins\n\nPlacement and honesty about the field. Open, reproducible,\nmulti-metric suites exist precisely so claims are checkable (claim\nc1), and current agent benchmarks still leave a wide human-model gap\n(claim c5), so a public score carries real signal. Publish yours:\nscrutiny you did not pay for is the cheapest auditing available.\n\n## Where private wins\n\nRelease gating. Your agent's critical paths — your APIs, your\npolicies, your failure modes — appear in no public suite. A private\nsuite built from them cannot be trained against (claim c2), and\nrotation keeps slow leakage from mattering. This is also the only\nsuite where a regression means, unambiguously, that your product got\nworse.\n\n## The operating pattern\n\n- Adopt the closest public suites once, rerun on each major model\n  change, and track your placement as telemetry.\n- Gate releases only on the private suite, rotated on a schedule,\n  with integrity treated as a live threat (claim c4).\n- Never promote a public score into a ship decision: the incentive to\n  optimize the visible number is a Goodhart mechanism working as\n  described (claim c3).",
  "canonical_url": "https://agentreliability.dev/k/public-benchmarks-vs-private-task-suites",
  "api_url": "https://agentreliability.dev/api/k/public-benchmarks-vs-private-task-suites.json",
  "jsonld": {
    "@context": "https://schema.org",
    "name": "Public benchmarks vs private task suites: what each buys you",
    "description": "Public benchmarks buy comparability, transparency and community scrutiny; private suites buy contamination resistance, domain fit and Goodhart resistance. Production teams need both, in a fixed shape: public suites to place yourself in the field, a private rotating suite to gate your releases. Using either for the other's job is how teams fool themselves.",
    "url": "https://agentreliability.dev/k/public-benchmarks-vs-private-task-suites",
    "license": "https://spdx.org/licenses/CC-BY-4.0.html",
    "dateModified": "2026-08-08",
    "citation": [
      {
        "@type": "CreativeWork",
        "name": "openai/evals — framework for evaluating LLMs and LLM systems",
        "url": "https://github.com/openai/evals"
      },
      {
        "@type": "CreativeWork",
        "name": "Categorizing Variants of Goodhart's Law",
        "url": "https://arxiv.org/abs/1803.04585"
      },
      {
        "@type": "CreativeWork",
        "name": "METR — Model Evaluation & Threat Research",
        "url": "https://metr.org/"
      },
      {
        "@type": "CreativeWork",
        "name": "GAIA: a benchmark for General AI Assistants",
        "url": "https://arxiv.org/abs/2311.12983"
      },
      {
        "@type": "CreativeWork",
        "name": "HELM — Holistic Evaluation of Language Models",
        "url": "https://crfm.stanford.edu/helm/"
      }
    ],
    "author": {
      "@type": "Person",
      "name": "Santiago Santa María Morales",
      "jobTitle": "practitioner — harness engineering and agent evaluation in production"
    },
    "@type": "Article",
    "headline": "Public benchmarks vs private task suites: what each buys you"
  }
}