Placing a Small Relevance Critic in Front of RAG — Not Passing Along Everything Retrieved

Introduction

I run a personal knowledge base (RAG) that stores my day-to-day records and retrieves them via semantic search. I covered raising retrieval accuracy itself with numbers in Measurement-Driven Tuning of RAG Retrieval.

But even once search gets good, a problem remains. Vector search pulls in things that are “semantically close,” but some of what’s close isn’t actually relevant to the question. This is the flip side of embeddings’ honesty, which I also wrote about in a different article (The Trap Where Vector Search Mixes In a Different Category).

If you pass everything retrieved straight to the LLM as-is, the irrelevant memories become noise and muddy the answer. This article is about placing a small critic between retrieval and the LLM that judges “is this actually relevant?”

The problem: retrieval isn’t the same as correctness

A naive RAG stuffs everything that ranks highly in search straight into the LLM’s prompt. There’s no quality gate.

Question
  │
  ▼
Search (pull the semantically closest items from the top)
  │  ← things that are merely close but "irrelevant" also come up alongside
  ▼
Pass everything retrieved to the LLM as-is  ← irrelevant memories muddy the answer

“Semantically close” doesn’t necessarily mean “relevant.” When records that merely share similar wording, but have nothing to do with the question, get mixed in, the LLM gets dragged along by them.

The idea: put a critic between retrieval and the LLM (CRAG)

The core countermeasure is simple. After retrieval, before passing to the LLM, judge “is this relevant?” one item at a time. Let the relevant ones through, drop the irrelevant ones. Because this corrects quality after retrieval, this pattern is called CRAG (Corrective RAG).

Question
  │
  ▼
Search (pull semantically close items)
  │
  ▼
┌────────────────────────────────────────────────┐
│ Critic: is this memory relevant to this question? (binary judgment) │
└────────────────────────────────────────────────┘
  │
  ├─ Relevant   ─▶ Pass to the LLM
  └─ Irrelevant ─▶ Drop (don't pollute the answer)

Why a “small critic” instead of a “large LLM”

The judgment itself could be done by a large LLM too. But querying a large model per retrieved item, every single search, is slow and heavy. And if you throw it at an external API, you’re sending your own memories to the cloud each time.

So I prepared a small model dedicated purely to judgment. And rather than a generic judge, I fine-tuned it with training data matched to my own RAG’s domain. Since the critic’s job is only a binary “use it / don’t use it,” a small model is enough for the task.

[Have a large LLM judge every time]
   Search results ─▶ Large model ─▶ Judgment   … slow, heavy, (if external) sends data to the cloud

[Small critic fine-tuned on my own domain]
   Search results ─▶ Small model (local) ─▶ Use / don't use   … fast, light, completes locally

The benefit of completing this locally isn’t just speed. It can judge without ever sending my memories outside (privacy). For a RAG that handles personal records, this matters.

What I built (measured results)

The base is a small instruction-tuned model (Qwen2.5-3B). I fine-tuned this for binary relevance judgment with QLoRA (4-bit quantization + LoRA). Training ran on my laptop GPU (RTX 2070 Max-Q) and took about 1 hour 53 minutes. Final loss was about 1.45, and binary judgment accuracy came out to roughly 76%.

76% isn’t perfect. But the critic’s job is to “drop things that are clearly irrelevant,” not to be a perfect selector. As a first-pass filter, it works well enough. In fact, this critic is now running every time a search fires, quietly dropping low-relevance chunks.

Rather than inflate the numbers, I’ll be honest. Small, fast, self-contained, and it drops the obvious noise — that was the goal, and it’s met.

Keeping retrieval from being treated as correctness

Retrieval isn’t correctness. Put a critic in front of the pass-through. Don’t hand everything that ranked highly in search straight to the LLM as-is. “Semantically close” and “relevant” are different things. Insert one gate in between that confirms relevance.

The critic doesn’t need to rely on a large model — you can build it small and dedicated. For a limited task like binary judgment, fine-tuning a small model on your own domain’s training data is enough. It’s fast, light, and runs locally.

Keep it self-contained locally, and don’t send memory outside. If you’re handling personal records, choose an architecture where you don’t have to send the contents to the cloud just to make a judgment.

Making search fast is only half of tuning. Filtering what comes back by whether it’s actually relevant — only with this critic in place does a RAG start returning things that are “useful,” not just “close.”

Raising retrieval accuracy itself with numbers is covered in Measurement-Driven Tuning of RAG Retrieval.
The same underlying trap — “semantically close” doesn’t mean “safe to surface” — is summarized in The Trap Where Vector Search Mixes In a Different Category.
The story of implementing this semantic search without adding a dedicated vector DB in the first place is in “Meaning” Instead of “Surface” — Implementing Semantic Search Without Adding a Dedicated Vector DB.
The record of the small-model fine-tuning (QLoRA) on my local GPU that this critic is built on is in Fine-Tuning a Personal RAG with Nothing but an RTX 2070 Laptop.