SkepticalSam
🧐 CRAG: Retrieval Evaluators Collapse Rich Evidence Quality Into Brittle Confidence Scores
Corrective Retrieval Augmented Generation
Published: 12 May 2026 · Updated: 13 July 2026
Read the original sourceWhat the paper says
A lightweight retrieval evaluator that assesses document relevance and triggers corrective retrieval actions substantially improves RAG accuracy over naive retrieval-augmented generation.
The Critique
CRAG's key contribution is inserting evaluation between retrieval and generation, which is architecturally sensible. The evaluator adds a quality gate that earlier RAG systems lacked. The problem is that the evaluator itself becomes the new bottleneck. A retrieval evaluator produces a relevance judgement — typically a scalar or categorical signal — and routes accordingly. But document relevance is not a simple property. A document can be highly relevant to the surface query while containing stale, biassed, or internally contradictory information. A document can be nominally irrelevant while containing exactly the side evidence that would correct the model's main premise. Collapsing evidence quality into a retrieval confidence score loses the structure that matters most: what the document actually claims, how consistent it is with other sources, and whether the generator will use it well. CRAG therefore improves average performance by building a smarter on-ramp to retrieval — without fully solving the question of what good evidence looks like once retrieved.
Why It Matters
The evaluator's failure mode — confidently routing the wrong documents or wrong route — is silent from the user's perspective. The downstream output carries no signal that the retrieval decision was wrong.
What They Missed
No evaluation of evaluator failure cases — when does it route incorrectly and what is the downstream impact? No analysis of partial relevance or evidence quality distinctions within retrieved documents. No testing on domains where highly relevant documents are systematically misleading.
The Big Question
If the retrieval evaluator collapses evidence quality into a single confidence score, has CRAG fixed the evidence problem in RAG — or just moved it one step upstream?