From incident to check: making failures pay rent
The loop that turns every production incident into a permanent guardian: write the incident down, derive the smallest check that would have caught it, falsify that check, and add it to a counted inventory so it can never silently vanish. After enough cycles, your validator is a compressed history of everything that has actually gone wrong.
Why this wins its question: The loop is documented from a validator that was actually built this way — check by check, incident by incident — with the code as evidence. Postmortem literature stops at "write action items"; this guide's action item is always the same executable artifact.
Key takeaways
- Every incident produces exactly one durable artifact — a falsified, counted check.
- Smallest check that rejects the defect wins; broad checks cry wolf and get deleted.
- Write the incident of origin into the check so the next maintainer knows why it exists.
- Only the counter notices a check's absence — uncounted protection disappears silently.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
The Citarium validator is built this way in production: its source states that every check exists because an incident paid for it, and individual checks carry their incident of origin in code comments (the locale-contract check cites the pt-BR incident directly).
New checks enter service through falsification — plant the incident's defect, observe red, restore, observe green — so the check demonstrably catches the failure it was born from.
The check inventory is itself guarded: the validator exports its count of check families for CI to track, so a check silently dropped in a refactor raises an alarm.
The rule
Every incident produces exactly one durable artifact: a check that would have caught it, proven by falsification, added to a counted inventory. Everything else in the postmortem — narrative, blame, slideware — decays. The check compounds.
The loop, step by step
1. Capture the defect while it is fresh. Not the outage story — the minimal wrong state. An agent cited an unregistered source; a label promised a locale the prose broke; a date claimed the future. One sentence, one reproducible state. 2. Derive the smallest check that rejects that state. Smallest matters: a check that rejects too much gets deleted the first time it cries wolf, and then the incident can return. Precision over recall is the survival trait for guardians (the Citarium locale-contract check is deliberately high-precision for exactly this reason — claim c1). 3. Falsify it before trusting it. Plant the original incident's defect, watch the new check go red, restore, watch green (claim c2). The incident becomes the check's own test case — the one defect this check must provably catch forever. 4. Count it. Add the check to the tracked inventory so its disappearance is an alarm, not a mystery discovered during the next incident (claim c3). 5. Write the origin into the check. A comment naming the incident turns the validator into documentation: the next maintainer learns why the check exists before deciding to weaken it.
What this produces over time
A validator built by this loop is a compressed incident history (claim c1): reading its checks is reading everything that has gone wrong, with each entry executable and self-verifying. Teams that skip the loop keep relearning failures; teams that run it convert failure into infrastructure at a fixed exchange rate — one incident, one guardian.
Anti-patterns
- The broad check. Born from an incident but rejecting a whole
category "to be safe". It fires on legitimate work, gets deleted, and takes the incident's protection with it.
- The uncounted check. Added in a hurry, never inventoried,
silently lost in a refactor. Falsified once is not falsified forever — only the counter notices absence.
- The narrative postmortem. Action items assigned, none of them a
check. Six months later the document exists and the protection does not.