{
  "id": "falsify-your-first-guardian",
  "type": "guide",
  "name": "How to falsify your first guardian",
  "summary": "A step-by-step procedure for proving that one automated check in your agent pipeline can actually fail: pick the guardian, plant a defect it must catch, confirm red, restore, confirm green, and record the episode. Thirty minutes of work that converts a green light from an assumption into evidence.",
  "locale": "en",
  "tags": [
    "guide",
    "guardians",
    "falsification",
    "testing",
    "ci"
  ],
  "relations": [
    {
      "rel": "explains",
      "target": "guardian-falsification"
    },
    {
      "rel": "related",
      "target": "check-counter-alarm"
    }
  ],
  "questions": [
    "How do I falsify a guardian?",
    "How do I prove my CI would catch a real failure?"
  ],
  "claims": [
    {
      "id": "c1",
      "text": "The falsification cycle is plant a defect, observe the guardian go red, restore the correct state, observe it go green; a guardian is only trusted after it has been seen in both states.",
      "sources": [
        "citarium-methodology-brief"
      ],
      "confidence": 0.95
    },
    {
      "id": "c2",
      "text": "In the Citarium framework the content validator is the first guardian falsified: it runs from CI on day one and each of its check families exists because an incident paid for it.",
      "sources": [
        "citarium-core-validator"
      ],
      "confidence": 0.95
    },
    {
      "id": "c3",
      "text": "Robust evaluations are extremely difficult to develop and implement, which is why an untested check deserves no trust by default.",
      "sources": [
        "anthropic-eval-challenges"
      ],
      "confidence": 0.85
    }
  ],
  "takeaways": [
    "A guardian that has never been observed red is an assumption, not a control.",
    "The cycle is plant a defect, expect red, restore, expect green — then record the episode.",
    "Red for the wrong reason falsifies a different check; the target stays unproven.",
    "Falsify every new guardian in the pull request that lands it, and re-falsify after pipeline refactors."
  ],
  "faqs": [],
  "evidence_tier": "primary",
  "evidence": {
    "level": "production",
    "source_types": [
      "production_system",
      "personal_experience",
      "industry_observation"
    ]
  },
  "moat_flag": true,
  "winning_edge": "A reproducible procedure with a live production example (the Citarium validator) behind every step — not testing philosophy. The reader can run the whole cycle against their own CI in one sitting.",
  "confidence": 0.9,
  "last_verified": "2026-08-08",
  "body": "## Why falsify\n\nA guardian — any automated check standing between your agent and\nproduction — earns trust the same way a scientific hypothesis does: by\nsurviving an attempt to break it. A check that has always been green\ntells you nothing; maybe it works, maybe it silently stopped running,\nmaybe it never could fail. Robust evaluations are hard to build (claim\nc3), so the default assumption about any untested check is that it is\nbroken.\n\n## The cycle (claim c1)\n\n1. **Pick one guardian.** Start with the check whose failure would hurt\n   most: schema validation, a claim-to-source binding check, a\n   destructive-action gate in your agent harness.\n2. **Plant a defect.** Introduce the exact class of error the guardian\n   exists to catch. Make it minimal and realistic: an unregistered\n   source id, a future date in a freshness label, a tool call outside\n   the allowlist. Commit it on a branch, never on main.\n3. **Expect red.** Run the pipeline. The guardian must fail, and fail\n   for the planted reason — read the error message and verify it points\n   at your defect, not at an accidental one.\n4. **Restore.** Revert the planted defect completely.\n5. **Expect green.** Run the pipeline again. It must pass. If it stays\n   red, your restore was incomplete or the guardian has a state leak —\n   both are findings.\n6. **Record the episode.** Note the guardian, the defect class, the\n   observed red, and the date. This record is what \"our CI catches X\"\n   now cites.\n\n## Worked example\n\nThe Citarium content validator ships as each instance's first guardian\n(claim c2). Falsifying it on this very repository looks like: add a\nclaim citing a source id that is not in `sources.yaml`, run the test\ncommand, observe the cross-reference check fail with the exact claim id,\nrevert, observe green. The validator's own source code documents that\nevery one of its check families traces to a real incident — falsification\nkeeps that inventory honest.\n\n## After the first one\n\n- Falsify each new guardian **when it lands**, in the same pull request.\n- Re-falsify after any refactor of the pipeline that runs the guardians.\n- Track the number of active checks so a silently dropped guardian is\n  itself an alarm — see the check-counter regression alarm.\n\n## Failure modes to expect\n\n- **The guardian passes with the defect planted.** The check does not\n  cover what you thought. Fix the check before trusting anything else.\n- **Red, but for the wrong reason.** Your defect tripped a different\n  check. You have falsified that one instead; the target is still\n  unproven.\n- **Green after restore fails.** Cached state or ordering dependence in\n  the pipeline. Treat as a real defect in the harness.",
  "canonical_url": "https://agentreliability.dev/k/falsify-your-first-guardian",
  "api_url": "https://agentreliability.dev/api/k/falsify-your-first-guardian.json",
  "jsonld": {
    "@context": "https://schema.org",
    "name": "How to falsify your first guardian",
    "description": "A step-by-step procedure for proving that one automated check in your agent pipeline can actually fail: pick the guardian, plant a defect it must catch, confirm red, restore, confirm green, and record the episode. Thirty minutes of work that converts a green light from an assumption into evidence.",
    "url": "https://agentreliability.dev/k/falsify-your-first-guardian",
    "license": "https://spdx.org/licenses/CC-BY-4.0.html",
    "dateModified": "2026-08-08",
    "citation": [
      {
        "@type": "CreativeWork",
        "name": "Citarium content validator source (@citarium/core v0.1.0, compiler/validate.ts)"
      },
      {
        "@type": "CreativeWork",
        "name": "agent-reliability editorial brief and blueprint (Gate 1 approved, 2026-08-08)"
      },
      {
        "@type": "CreativeWork",
        "name": "Challenges in evaluating AI systems",
        "url": "https://www.anthropic.com/news/evaluating-ai-systems"
      }
    ],
    "author": {
      "@type": "Person",
      "name": "Santiago Santa María Morales",
      "jobTitle": "practitioner — harness engineering and agent evaluation in production"
    },
    "@type": "Article",
    "headline": "How to falsify your first guardian"
  }
}