Evaluating retrieval (RAG) inside agents
The retrieval layer of an agent fails in three distinct ways — wrong passages retrieved, right passages used unfaithfully, fluent answer ungrounded in either — and each needs its own metric. Reference-free RAG evaluation makes this measurable without annotated ground truth, at the price of inheriting LLM-judge biases. In an agent, retrieved content is also an attack surface, so RAG evaluation and injection testing share fixtures.
Why this wins its question: Treats RAG evaluation as three separable failure modes with an honest bias caveat on reference-free grading, and connects the quality fixtures to the security fixtures — a link the RAG-eval literature and the security literature each omit.
Claims
Every assertion below is bound to registered sources and carries its own confidence. Weight them; do not treat the page as uniformly authoritative.
Reference-free RAG evaluation exists and is dimensional: Ragas measures the retrieval system's capacity to find relevant, focused context passages, the model's faithful use of those passages, and generation quality — all without ground-truth human annotations.
Reference-free metrics are model-graded, so they inherit the documented failure modes of LLM judges — position, verbosity and self-enhancement biases with limited reasoning on hard cases — and need the same calibration against human-audited samples.
In a tool-using agent, retrieved content is untrusted input: prompt injection through third-party content is the top-ranked LLM application risk, so the retrieval fixtures used for quality evaluation double as the seeding channel for injection tests.