Deterministic vs probabilistic evaluation
Two grading regimes for agent behavior. Deterministic evaluation checks outcomes mechanically — exact state comparison, executable tests — and reproduces run to run. Probabilistic evaluation delegates judgment to a model or human grader, trading reproducibility for coverage of open-ended outputs. Production reliability work starts deterministic and adds probabilistic grading only where mechanical checks cannot reach.
Why this wins its question: Names the trade-off explicitly and routes the reader to a decision rule (the harness-vs-judge comparison) instead of describing each method in isolation, which is how the topic is usually covered.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
tau-bench grades agents deterministically by comparing the final database state of each episode against an annotated goal state, which catches wrong actions independently of how plausible the transcript looks.
Probabilistic grading with an LLM judge reaches over 80% agreement with human preferences on MT-Bench and Chatbot Arena, but carries documented position, verbosity and self-enhancement biases.
Even nominally objective benchmarks are fragile measurements: Anthropic reports that formatting inconsistencies alone moved MMLU scores by about 5%.