Designing a learning plan that won't drown in the noise — narrowing RAG/LLM tuning to "three, deep" and making 'measure it' the unit of learning
Introduction
There are simply too many tools and techniques to learn in this field. Evaluation, retrieval, reranking, fine-tuning, observability — each has several standard libraries, and new names flow by almost every week. Chase all of them and every single one ends up half-finished. So before learning anything, I first designed what to drop and what to go deep on. This article is the design record of that learning roadmap itself.
The subject is a personal RAG I run myself — a mechanism like an external long-term memory that retrieves past records by meaning. The premise of RAG itself I leave to this article.
This article is also the story of a “measurable RAG”, paired with the “RAG that doesn’t break” and the “RAG that doesn’t get contaminated.” On top of a foundation that doesn’t break and doesn’t get contaminated, how do I measure and grow my own skill — that’s the design here. Let me put the key points up front.
- Don’t touch tools broadly and shallowly. Narrow down to three, deep (evaluation, retrieval, fine-tuning), and leave the rest at “just know it”
- Don’t stop at “I read it.” Make connecting to my own real data and measuring it the smallest unit of learning
- Set up evaluation (the means of measuring) first. Without it, improvements to retrieval or fine-tuning end at “feels like it got better”
Why I narrowed it down to “three, deep”
What I wanted to avoid most was ending up with a shallow touch on twenty tools and the illusion of “knowing a bit of everything.” It’s an easy trap to fall into — the count of tools touched goes up, but the arm that can actually measure something doesn’t grow by even one.
So I laid out the standard tools layer by layer and took stock of where I already had strength and where I was thin. What emerged was that what needed filling in wasn’t “the number of tools” but three thin legs: measuring, comparing, and building data. Here’s how the narrowing turned out.
Learning scope for RAG/LLM tuning
│
┌─────────┬───────────┼───────────┬─────────────┐
▼ ▼ ▼ ▼ ▼
Eval(E) Retrieval(R) FT(F) Orchestra- …many others
│ │ │ tion(O) (stay shallow)
═══╪══════════╪══════════╪═══ │
Three pillars to go deep on (E→R→F) Just know it
│ (don't overinvest)
└──▶ Each pillar = 1 unit: "measure with real data" (next section)
The three I go deep on are evaluation, retrieval, and fine-tuning. Orchestration (the framework that wires multiple parts together) I decided was fine at the level of “can read it and talk about it.” The reason comes later, but in short: I already have the concept in hand from building it myself, so it’s enough if I can “translate” it into the standard framework’s vocabulary.
Stock-take of what I have and what’s missing (a map of where I stand)
Here is the stock-take itself, the basis for the narrowing, laid out as-is. It’s not self-scoring — it’s a plain design map of “what exists at which layer, and what’s thin.”
| Layer | Standard tools | My current state | Thickness |
|---|---|---|---|
| Evaluation | Ragas / DeepEval | Built my own relevance critic (detailed below) | Thin (standardization and automated regression not yet done) |
| Observability | LangSmith / Langfuse | Logging is self-built, no tracing infrastructure yet | Thin |
| Retrieval / reranking | pgvector / various rerankers | Hybrid search + reranking in real operation | Core is in hand (side-by-side comparison not yet done) |
| Wiring (O) | LangChain / LlamaIndex | Implemented independently | Concept is in hand (standard vocabulary not yet done) |
| Foundation / FT | QLoRA / LoRA | Trained and deployed for my personal RAG | The real thing exists (data design and evaluation still thin) |
Three places are thin: (A) standardizing/automating measurement, (B) side-by-side comparison of retrieval methods, (C) data design for fine-tuning. I ordered the learning tracks to fill these three, in that order.
The learning pattern — making “measure it” one unit
I fixed a single common procedure for every track, up front, as a template. This is the crux of the design.
① Read (official docs, concepts)
│
▼
② Run it on my own real data ◀─────────┐
│ (don't stop at toy data/tutorials)
▼ │ Can't measure = not yet running
③ Measure (produce numbers with ground-truth data) ─┘
│ Only here may I call it "understood"
▼
④ Write it up (lock in the learning + portfolio)
I don’t call “① read it” “understood.” A unit only counts as complete once it reaches ③ measure with my own real data. Getting something to run on a tutorial’s sample data isn’t “made it work” — it’s merely “traced it.” If you measure and can’t produce a number, you go back to ②; that return edge is built into the template. It’s the same thinking as the discipline of recording a personal RAG’s successes and failures as one set, “through to a successful resolution.”
Why I put evaluation first, before anything else
I put an order on the three pillars: Evaluation (E) → Retrieval (R) → Fine-tuning (F). There’s a clear reason evaluation comes first.
Change the retrieval or redo the fine-tuning without a measuring stick, and the best you can say about whether it’s good or bad is “feels like it got somewhat better.” In fact, the retrieval improvement could only be described in numbers because I set up “measuring” first. I brought in hybrid search and moved recall — at the moment it matters most — from 0.2 to 1.0. This “0.2 → 1.0” is a number that can only be stated once a measuring stick exists. How I set up that measuring stick is written in this article.
E: Evaluation foundation (the measuring stick)
│ Without this, ↓ can't state good/bad in numbers
┌────┴────┐
▼ ▼
R: Retrieval F: Fine-tuning
(compare) (train)
R or F without a measuring stick is no different from gambling. That’s why I put evaluation first. It’s the thinnest place right now, and the one place where filling it in turns every remaining judgment into a number — the highest return on investment there is.
Inside the three pillars, and O as “just know it”
Here I lay out each track’s “concepts to learn / deliverable to build / completion condition.” Every completion condition is defined by a number or a working deliverable.
| Pillar | Concepts to learn (breaking down first exposure) | Deliverable to build | Completion condition |
|---|---|---|---|
| E Evaluation | Faithfulness (whether the answer stays true to its grounding — i.e., isn’t making things up), context recall/precision, LLM-as-judge (having an LLM itself do the scoring), regression testing | A foundation for evaluation connecting a golden set (ground-truth data) to Ragas (an evaluation library), plus a regression suite that automatically compares before/after a change. Visualize traces with Langfuse (a self-hosted foundation for visualizing processing) | When I change the hybrid-search coefficients or the reranker, I can judge good/bad with a number |
| R Retrieval | Dense vectors (retrieve by semantic closeness) / sparse vectors (retrieve by term match) / hybrid, reranker (a device that re-sorts retrieved results by relevance), chunking (how documents are cut up), query transformation | Implement a pgvector (a Postgres extension for vector search) version of retrieval and compare it side-by-side with the existing version on identical data (recall, precision, speed) | Can explain, with measured numbers, “which method and why” |
| F Fine-tuning | Dataset design (diversity, leak prevention), evaluation splits, LoRA/QLoRA (memory-efficient fine-tuning techniques) | Re-design the data intentionally, retrain, and evaluate/compare using data not used for training (hold-out) | Can explain, with numbers, “why this data composition, and how accuracy moved” |
The relevance critic I built myself for E, and the fine-tuned model I trained for F, are already real things I have in hand. I leave the details to each implementation article.
Placing a Small Relevance Critic in Front of RAG — Not Passing Along Everything Retrieved
Fine-Tuning a Personal RAG with Nothing but an RTX 2070 Laptop
E’s aim is to connect and replace this self-built critic with the standard measuring stick that is Ragas — the work of transplanting a hunch I invented myself onto a common scale that can be compared against others.
The fourth item, O (orchestration), on the other hand, I don’t chase deeply. Thanks to the activation layer and wiring layer I built independently, the concept itself is already in hand.
Injecting ‘What to Recall Right Now’ into AI Every Turn — An Activation Layer for Personal RAG
So for O, I decided it’s enough to reach the point of being able to “read” a standard framework like LangChain and “explain my own mechanism using its vocabulary.” I’ll trace through just one small piece and invest no further.
Anti-patterns crushed in advance
Flip this design around, and it’s really a list of “failures I’m prone to making,” laid out in advance with a countermeasure tied to each one.
- Touching tools broadly and shallowly: touch twenty tools and can’t measure a single one. → That’s why I narrowed to three, deep, and left O at just-know-it.
- Improving without measuring: stops at “feels like it got better.” → That’s why I set up evaluation (E) first.
- Being satisfied with toy data: wraps up with a tutorial’s sample. → That’s why “connect to my own real data and measure it” is the unit.
- Overinvesting in the framework: trying to master the wiring library. → I already hold the concept from building it myself, so it’s enough if I can translate it into the standard vocabulary.
All four are the kind of failure that gets crushed by design, not by knowledge. Crush them on the map before you start learning, and there’s less time to burn once you’re already running.
Learning, too, only counts as “progress” once it’s measured
This roadmap has one backbone. Don’t conflate “read it” with “can do it” — make it one unit only once you measure with your own real data. And set up the measuring stick (evaluation) before anything else. Once the measuring stick is in place, both retrieval and fine-tuning turn from “a bet” into “a measurement,” and I can proceed while confirming progress in numbers.
This is the third installment of the same philosophy as keeping memory from breaking and correcting it so it doesn’t get contaminated. Not broken, not contaminated, and measurable — only once all three are in place does external memory turn from something that “just grows” into something that “gets trained.” The not-broken side and the not-contaminated side are written in the two companion articles.
And the weekly checkup that keeps all of this healthy is in this piece.