{
  "id": "your-first-agent-evals",
  "type": "guide",
  "name": "Your first agent evals: from zero to a release gate in two weeks",
  "summary": "The bootstrap path for a team whose agent has no evals at all: harvest real tasks from production logs, write execution-based graders for them, adopt an existing eval framework instead of building one, baseline with repeated trials and error bars, and wire the result into CI as a gate. Two weeks of part-time work; the output is the suite every other reliability practice hangs from.",
  "locale": "en",
  "tags": [
    "guide",
    "evals",
    "bootstrap",
    "ci",
    "getting-started"
  ],
  "relations": [
    {
      "rel": "leads_to",
      "target": "measuring-agent-reliability"
    },
    {
      "rel": "leads_to",
      "target": "falsify-your-first-guardian"
    },
    {
      "rel": "related",
      "target": "from-incident-to-check"
    },
    {
      "rel": "related",
      "target": "inspect-eval-framework"
    }
  ],
  "questions": [
    "We have no evals at all — where do we start?",
    "How long does it take to stand up agent evals?",
    "Should I write my own eval framework?"
  ],
  "claims": [
    {
      "id": "c1",
      "text": "Start simple is vendor-stated best practice: add agent complexity only when it demonstrably improves outcomes, with extensive sandboxed testing and guardrails before autonomy grows.",
      "sources": [
        "anthropic-building-agents"
      ],
      "confidence": 0.85
    },
    {
      "id": "c2",
      "text": "Framework adoption beats framework building on day one: open eval frameworks ship reusable templates for basic and model-graded evals plus private-eval support, and Inspect adds sandboxed execution, agent primitives and 200+ prebuilt evaluations.",
      "sources": [
        "openai-evals",
        "inspect-ai"
      ],
      "confidence": 0.9
    },
    {
      "id": "c3",
      "text": "Grade end state, not narration: comparing the final state of the acted-on system against a goal state is how agent benchmarks catch wrong actions hidden behind plausible transcripts.",
      "sources": [
        "tau-bench"
      ],
      "confidence": 0.9
    },
    {
      "id": "c4",
      "text": "Baselines need repeated trials and uncertainty: consistency across k runs diverges sharply from single-run success, and comparisons without standard errors mistake noise for findings.",
      "sources": [
        "tau-bench",
        "error-bars-evals"
      ],
      "confidence": 0.85
    }
  ],
  "takeaways": [
    "Twenty log-harvested tasks, one grader style, one adopted framework, one CI job — resist enlargement until the gate exists.",
    "Define done as end state; a task whose outcome cannot be asserted gets replaced, not judge-graded.",
    "Baseline with 4-8 trials per task and a standard error, or noise will read as regressions.",
    "Invented tasks stay green forever and predict nothing — logs or it does not count."
  ],
  "faqs": [],
  "evidence_tier": "secondary",
  "evidence": {
    "level": "industry_observation",
    "source_types": [
      "industry_observation",
      "paper"
    ]
  },
  "moat_flag": false,
  "winning_edge": "A dated, ordered bootstrap plan with a stop condition (a working CI gate) — the existing literature explains why evals matter or how one framework works, not the shortest path from nothing to a gate.",
  "confidence": 0.85,
  "last_verified": "2026-08-08",
  "body": "## The shape of the problem\n\nTeams without evals do not lack conviction — they lack a first suite\nsmall enough to finish. The answer is twenty tasks, one grader style,\none adopted framework, one CI job. Resist every enlargement until the\ngate exists.\n\n## Week 1: tasks and graders\n\n1. **Harvest 20 real tasks from logs.** Not invented scenarios: pull\n   the last 20 things users actually asked the agent to do, keep the\n   inputs verbatim, strip secrets. Ten routine, five hard, five that\n   previously failed (those five are your future regression tests).\n2. **Define done as end state.** For each task, write down what must\n   be true of the world afterwards — records changed, files present,\n   API responses matching — never what the transcript should say\n   (claim c3). If a task's outcome cannot be stated as a checkable\n   condition, replace the task; do not reach for a judge yet.\n3. **Write the graders as plain assertions.** State diffs, exact\n   checks, executable verification. Boring by design: a grader you\n   can falsify in one minute is worth ten clever ones.\n\n## Week 2: framework, baseline, gate\n\n4. **Adopt, do not build** (claim c2). Pick an existing framework —\n   the choice matters less than not writing your own runner. Fixtures\n   and graders encode your domain; everything else is commodity.\n5. **Baseline with repeated trials.** Run each task 4-8 times against\n   the current agent; record per-task pass rates and a standard error\n   for the suite (claim c4). This number panel — not any single score\n   — is what \"the agent got worse\" will mean from now on.\n6. **Wire the CI gate.** The suite runs on every prompt change, tool\n   change and model change; per-task thresholds on the critical five,\n   suite-level trend for the rest. Keep the agent's autonomy matched\n   to what the gate proves (claim c1).\n\n## After the gate exists\n\n- Falsify it: plant a known regression, watch red, restore (see\n  falsify-your-first-guardian).\n- Feed it: every new incident adds one task with a grader (see\n  from-incident-to-check).\n- Grow deliberately: more tasks on the paths users actually hit, a\n  calibrated judge only where no assertion can reach.\n\n## Two traps\n\n- **The invented-task suite.** Twenty scenarios nobody asked for,\n  green forever, predictive of nothing. Logs or it does not count.\n- **The framework project.** Three sprints into a bespoke runner, no\n  gate yet. The runner was never the hard part; your fixtures are.",
  "canonical_url": "https://agentreliability.dev/k/your-first-agent-evals",
  "api_url": "https://agentreliability.dev/api/k/your-first-agent-evals.json",
  "jsonld": {
    "@context": "https://schema.org",
    "name": "Your first agent evals: from zero to a release gate in two weeks",
    "description": "The bootstrap path for a team whose agent has no evals at all: harvest real tasks from production logs, write execution-based graders for them, adopt an existing eval framework instead of building one, baseline with repeated trials and error bars, and wire the result into CI as a gate. Two weeks of part-time work; the output is the suite every other reliability practice hangs from.",
    "url": "https://agentreliability.dev/k/your-first-agent-evals",
    "license": "https://spdx.org/licenses/CC-BY-4.0.html",
    "dateModified": "2026-08-08",
    "citation": [
      {
        "@type": "CreativeWork",
        "name": "openai/evals — framework for evaluating LLMs and LLM systems",
        "url": "https://github.com/openai/evals"
      },
      {
        "@type": "CreativeWork",
        "name": "Inspect — evaluation framework for large language models",
        "url": "https://inspect.aisi.org.uk/"
      },
      {
        "@type": "CreativeWork",
        "name": "Building effective agents",
        "url": "https://www.anthropic.com/research/building-effective-agents"
      },
      {
        "@type": "CreativeWork",
        "name": "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains",
        "url": "https://arxiv.org/abs/2406.12045"
      },
      {
        "@type": "CreativeWork",
        "name": "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations",
        "url": "https://arxiv.org/abs/2411.00640"
      }
    ],
    "author": {
      "@type": "Person",
      "name": "Santiago Santa María Morales",
      "jobTitle": "practitioner — harness engineering and agent evaluation in production"
    },
    "@type": "Article",
    "headline": "Your first agent evals: from zero to a release gate in two weeks"
  }
}