{
  "id": "designing-an-agent-sandbox",
  "type": "guide",
  "name": "Designing an agent sandbox that actually contains failures",
  "summary": "Requirements and checks for an execution environment where an agent can fail safely: no default egress, ephemeral seeded state, fake credentials that look real, a bounded blast radius, and an observation channel the agent cannot write to. Includes a containment checklist you can test the same way you test the agent.",
  "locale": "en",
  "tags": [
    "guide",
    "sandbox",
    "isolation",
    "safety",
    "execution"
  ],
  "relations": [
    {
      "rel": "explains",
      "target": "sandboxed-execution"
    },
    {
      "rel": "enables",
      "target": "fault-injection-for-agents"
    }
  ],
  "questions": [
    "How do I build a sandbox for agent testing?",
    "How do I keep an agent test failure from becoming an incident?"
  ],
  "claims": [
    {
      "id": "c1",
      "text": "Chaos engineering requires that the fallout from experiments be minimized and contained; blast radius is a design input, not an afterthought.",
      "sources": [
        "principles-chaos"
      ],
      "confidence": 0.9
    },
    {
      "id": "c2",
      "text": "Execution-graded benchmarks presume a working environment: SWE-bench tasks require editing real codebases and interacting with execution environments to verify resolution.",
      "sources": [
        "swe-bench"
      ],
      "confidence": 0.85
    },
    {
      "id": "c3",
      "text": "Large-scale agent evaluation is infrastructure work — METR's Hawk platform, built on Inspect, exists specifically to run agent evaluations at scale.",
      "sources": [
        "metr"
      ],
      "confidence": 0.85
    },
    {
      "id": "c4",
      "text": "Dangerous-capability evaluations cover cybersecurity and self-proliferation — behavior whose escape from the test environment is itself the incident being prevented.",
      "sources": [
        "deepmind-dangerous-capabilities"
      ],
      "confidence": 0.85
    }
  ],
  "takeaways": [
    "A sandbox with egress, shared state or real credentials is production with extra steps.",
    "Every containment property must be testable — and falsified once before it is trusted.",
    "Keep the grader and telemetry outside the boundary, read-only; an agent that can edit its transcript passes any transcript-graded eval.",
    "Adopt existing evaluation infrastructure; build only the fixtures and graders that encode your domain."
  ],
  "faqs": [],
  "evidence_tier": "secondary",
  "evidence": {
    "level": "industry_observation",
    "source_types": [
      "industry_observation",
      "paper"
    ]
  },
  "moat_flag": false,
  "winning_edge": "Turns \"run it in a sandbox\" into a falsifiable checklist — each containment property is stated as a test you can run against the sandbox itself, in the same spirit as guardian falsification.",
  "confidence": 0.85,
  "last_verified": "2026-08-08",
  "body": "## What the sandbox is for\n\nAn agent under test will do the wrong thing — that is the point of\ntesting it. The sandbox's job is to make the wrong thing observable and\ncheap. If a test failure can reach a production API, mutate shared\nstate, or exfiltrate a real credential, you do not have a sandbox; you\nhave production with extra steps.\n\n## Containment properties (each one testable)\n\n1. **No default egress.** The environment reaches only an explicit\n   allowlist of endpoints, all of them fakes or fixtures. Test it: run\n   an agent whose task is to call an unlisted endpoint; the call must\n   fail visibly.\n2. **Ephemeral, seeded state.** Every run starts from a fixture and\n   throws its world away afterwards. Test it: run twice, diff the\n   starting states; any difference is a leak.\n3. **Fake credentials that look real.** The agent must not learn it is\n   being tested from the shape of its secrets, and a leaked test secret\n   must be worthless. Test it: plant the credential in output and verify\n   the alarm fires and nothing real was exposed.\n4. **Bounded blast radius (claim c1).** Enumerate, in writing, the worst\n   thing a fully adversarial agent could do inside the environment. If\n   the answer includes any production noun, redesign.\n5. **An observation channel the agent cannot write to.** Telemetry and\n   grading read from outside the environment. An agent that can edit its\n   own transcript can pass any transcript-graded eval.\n\n## Grading inside the sandbox\n\nExecution-based grading is the reason the environment exists (claim\nc2): the sandbox hosts the codebase, the database, or the API fixture\nwhose final state the harness compares against a goal state. Keep the\ngrader outside the sandbox boundary and read-only.\n\n## Build vs adopt\n\nPlatform work at evaluation scale is real engineering (claim c3).\nAdopt existing open infrastructure (Inspect-based platforms such as\nMETR's Hawk) when your tasks fit; build only the fixtures and graders\nthat encode your domain. The capability ceiling of your sandbox should\nmatch the capability you are probing — evaluations that probe\ncybersecurity or self-proliferation behavior (claim c4) treat\ncontainment as the primary deliverable, and so should any team whose\nagent holds credentials to critical APIs.\n\n## Checklist\n\n- [ ] Unlisted egress fails, visibly, with an alarm.\n- [ ] Two consecutive runs start from identical state.\n- [ ] Test credentials are worthless outside the sandbox.\n- [ ] Written worst-case contains no production noun.\n- [ ] Grader and telemetry are read-only from outside.\n- [ ] The sandbox itself has been falsified: you have watched each\n      property fail once.",
  "canonical_url": "https://agentreliability.dev/k/designing-an-agent-sandbox",
  "api_url": "https://agentreliability.dev/api/k/designing-an-agent-sandbox.json",
  "jsonld": {
    "@context": "https://schema.org",
    "name": "Designing an agent sandbox that actually contains failures",
    "description": "Requirements and checks for an execution environment where an agent can fail safely: no default egress, ephemeral seeded state, fake credentials that look real, a bounded blast radius, and an observation channel the agent cannot write to. Includes a containment checklist you can test the same way you test the agent.",
    "url": "https://agentreliability.dev/k/designing-an-agent-sandbox",
    "license": "https://spdx.org/licenses/CC-BY-4.0.html",
    "dateModified": "2026-08-08",
    "citation": [
      {
        "@type": "CreativeWork",
        "name": "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?",
        "url": "https://arxiv.org/abs/2310.06770"
      },
      {
        "@type": "CreativeWork",
        "name": "Evaluating Frontier Models for Dangerous Capabilities",
        "url": "https://arxiv.org/abs/2403.13793"
      },
      {
        "@type": "CreativeWork",
        "name": "METR — Model Evaluation & Threat Research",
        "url": "https://metr.org/"
      },
      {
        "@type": "CreativeWork",
        "name": "Principles of Chaos Engineering",
        "url": "https://principlesofchaos.org/"
      }
    ],
    "author": {
      "@type": "Person",
      "name": "Santiago Santa María Morales",
      "jobTitle": "practitioner — harness engineering and agent evaluation in production"
    },
    "@type": "Article",
    "headline": "Designing an agent sandbox that actually contains failures"
  }
}