SWE-bench Verified
The human-validated subset of SWE-bench: 500 tasks filtered from the original test set for quality, graded by unit-test verification against post-PR behavior. Its existence is the interesting fact — a flagship benchmark needed a human audit of its own tasks, which makes "who benchmarks the benchmark?" a standing question for anyone gating decisions on eval numbers.
Why this wins its question: Reads Verified as evidence about benchmarks in general — task quality is a measured variable, not an assumption — rather than as one more leaderboard entry, and connects that lesson to private-suite hygiene.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
SWE-bench Verified is a subset of 500 samples from the SWE-bench test set that have been human-validated for quality.
Verified keeps execution-based grading: task resolution is checked by unit-test verification using post-PR behavior as the reference solution.
The parent benchmark it filters is large and real — 2,294 GitHub issues from Python repositories requiring multi-file edits and execution-environment interaction — which is exactly the scale at which unaudited task quality becomes a measurement risk.