{
  "id": "regression-gating-model-upgrades",
  "type": "guide",
  "name": "Regression-gating a model upgrade",
  "summary": "The playbook for swapping the model behind a production agent without discovering the differences in production: freeze the harness, baseline the incumbent with pass^k on critical paths, run the candidate on the identical suite, diff the panel, roll out behind canaries. Model swaps are not optional — providers retire models on notice — so this gate runs on the provider's calendar, not yours.",
  "locale": "en",
  "tags": [
    "guide",
    "model-upgrades",
    "regression",
    "deprecations",
    "gating"
  ],
  "relations": [
    {
      "rel": "applies",
      "target": "behavioral-canaries"
    },
    {
      "rel": "applies",
      "target": "measuring-agent-reliability"
    },
    {
      "rel": "related",
      "target": "deterministic-vs-probabilistic-evaluation"
    }
  ],
  "questions": [
    "How do I upgrade the model behind my agent without breaking it?",
    "My model is being deprecated — what do I test before switching?",
    "Can I trust a new model version with my existing prompts?"
  ],
  "claims": [
    {
      "id": "c1",
      "text": "Model retirement is a scheduled certainty, not an optional refactor: Anthropic deprecates models with at least 60 days' notice for publicly released models, and requests to retired models fail outright.",
      "sources": [
        "anthropic-model-deprecations"
      ],
      "confidence": 0.95
    },
    {
      "id": "c2",
      "text": "The vendor's own guidance is to test applications with the replacement model well before the retirement date and to audit usage by API key and model to locate everything still pointing at the deprecated version.",
      "sources": [
        "anthropic-model-deprecations"
      ],
      "confidence": 0.9
    },
    {
      "id": "c3",
      "text": "Consistency is the metric that catches upgrade regressions: pass^k measures success across k repeated trials, and agents with similar single-attempt scores can differ widely in it.",
      "sources": [
        "tau-bench"
      ],
      "confidence": 0.9
    },
    {
      "id": "c4",
      "text": "Detecting how model versions change application behavior is a stated purpose of eval frameworks — the upgrade gate is the canonical use case for a frozen eval suite.",
      "sources": [
        "openai-evals"
      ],
      "confidence": 0.85
    },
    {
      "id": "c5",
      "text": "Harness details are part of the measurement — formatting inconsistencies alone have shifted benchmark scores by about 5% — so the harness must stay frozen while the model changes, or the diff measures the harness.",
      "sources": [
        "anthropic-eval-challenges"
      ],
      "confidence": 0.85
    }
  ],
  "takeaways": [
    "Model swaps run on the provider's calendar — 60-day deprecation windows make the gate standing infrastructure.",
    "Freeze the harness completely; a moving harness makes the model diff meaningless.",
    "Diff per critical path with pass^k, not the average — pass@1 up with pass^8 down is the dangerous pattern.",
    "Roll out behind behavioral canaries and recalibrate any LLM judge whose model also changed."
  ],
  "faqs": [],
  "evidence_tier": "secondary",
  "evidence": {
    "level": "industry_observation",
    "source_types": [
      "industry_observation",
      "paper"
    ]
  },
  "moat_flag": false,
  "winning_edge": "Ties the abstract advice (\"test before you switch\") to the concrete clock (60-day deprecation windows), the concrete metric (pass^k diff per critical path) and the concrete rollout mechanism (behavioral canaries) — the pieces exist elsewhere, the assembled gate does not.",
  "confidence": 0.85,
  "last_verified": "2026-08-08",
  "body": "## Why this gate exists\n\nYou will swap models whether you plan to or not: providers retire\nmodels on a published calendar, with as little as 60 days between\nnotice and shutdown (claim c1). Teams that treat the swap as an\nemergency prompt-patching session in week 8 ship regressions; teams\nwith a standing gate treat it as a routine run.\n\n## The gate, step by step\n\n1. **Inventory what actually runs.** Audit usage by key and by model\n   (claim c2). Agents accrete model references — fallback paths,\n   subagents, background jobs — and the one you forget is the one that\n   breaks at retirement.\n2. **Freeze the harness.** Same task fixtures, same graders, same\n   parsing, same temperature of everything around the model. Any\n   harness change between baseline and candidate invalidates the diff\n   (claim c5).\n3. **Baseline the incumbent.** Run the frozen suite against the current\n   model with repeated trials per task — pass^4 or pass^8 on critical\n   paths, not pass@1 (claim c3). Store per-task results, not the\n   aggregate.\n4. **Run the candidate on the identical suite.** Same trials, same\n   order-independence. This is the frozen-suite use case eval\n   frameworks exist for (claim c4).\n5. **Diff the panel, not the average.** A candidate that is +3 points\n   on average and -40 on one critical path fails the gate. Ship gates\n   are per-path thresholds; the average is telemetry.\n6. **Roll out behind canaries.** Route a slice of traffic to the\n   candidate with behavioral canaries pinned to known-good outcomes;\n   drift on a canary rolls back automatically. Keep the incumbent\n   reachable until the canaries have survived your longest business\n   cycle.\n7. **Re-falsify judge-graded checks.** If any gate uses an LLM judge,\n   the judge model may have changed too — recalibrate it against a\n   human-audited sample before trusting its verdicts on the new\n   candidate.\n\n## What the diff will show (and what to do)\n\n- **Format drift.** New models answer in different shapes; parsers\n  break before reasoning does. Fix parsers, not prompts, first.\n- **Instruction-following shifts.** Old prompt hacks (emphatic\n  repetition, workaround phrasing) may now misfire. Delete them and\n  re-test rather than stacking new hacks on top.\n- **Consistency changes.** pass@1 up, pass^8 down is a real and\n  dangerous pattern (claim c3): the demo improves while the user\n  experience degrades. The gate must weight pass^k.\n\n## Standing infrastructure\n\nRun this gate on every provider deprecation notice and every scheduled\nmodel refresh. The suite you freeze for it is the same suite that\ngates prompt changes and tool changes — one investment, three gates.",
  "canonical_url": "https://agentreliability.dev/k/regression-gating-model-upgrades",
  "api_url": "https://agentreliability.dev/api/k/regression-gating-model-upgrades.json",
  "jsonld": {
    "@context": "https://schema.org",
    "name": "Regression-gating a model upgrade",
    "description": "The playbook for swapping the model behind a production agent without discovering the differences in production: freeze the harness, baseline the incumbent with pass^k on critical paths, run the candidate on the identical suite, diff the panel, roll out behind canaries. Model swaps are not optional — providers retire models on notice — so this gate runs on the provider's calendar, not yours.",
    "url": "https://agentreliability.dev/k/regression-gating-model-upgrades",
    "license": "https://spdx.org/licenses/CC-BY-4.0.html",
    "dateModified": "2026-08-08",
    "citation": [
      {
        "@type": "CreativeWork",
        "name": "Challenges in evaluating AI systems",
        "url": "https://www.anthropic.com/news/evaluating-ai-systems"
      },
      {
        "@type": "CreativeWork",
        "name": "openai/evals — framework for evaluating LLMs and LLM systems",
        "url": "https://github.com/openai/evals"
      },
      {
        "@type": "CreativeWork",
        "name": "Anthropic model deprecations (Claude API documentation)",
        "url": "https://platform.claude.com/docs/en/docs/about-claude/model-deprecations"
      },
      {
        "@type": "CreativeWork",
        "name": "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains",
        "url": "https://arxiv.org/abs/2406.12045"
      }
    ],
    "author": {
      "@type": "Person",
      "name": "Santiago Santa María Morales",
      "jobTitle": "practitioner — harness engineering and agent evaluation in production"
    },
    "@type": "Article",
    "headline": "Regression-gating a model upgrade"
  }
}