Fine-Tuning a Personal RAG with Nothing but an RTX 2070 Laptop

Why I Wanted to Keep It Local

my-rag-brain holds everything: personal thought logs, work decision records, observations. I didn’t want to send any of that to a cloud API.

I wanted it to run locally.

What Tiny-Critic Does

Chunks retrieved from ChromaDB by a RAG system aren’t necessarily relevant to the query. Even if the distance is close, the content can be off.

Tiny-Critic is a small model that runs a second binary judgment — relevant or not — on chunks that have already been retrieved. It removes the irrelevant ones, narrowing the context before it reaches the LLM.

Building the Training Data

I generated labeled data from RAG logs on hand — pairs of queries and hit chunks.

# Balance 2,114 pairs down to 1,036 (yes/no = 50:50)
positives = [(q, chunk, 1) for q, chunk in relevant_pairs]
negatives = [(q, chunk, 0) for q, chunk in irrelevant_pairs]
balanced = positives[:518] + negatives[:518]

Skewed training data converges toward one side. Balancing was essential.

4-bit Quantized Training with QLoRA

I used a 4-bit quantized Qwen2.5-3B-Instruct with LoRA adapters. This setup runs on the RTX 2070 Max-Q’s 8GB of memory.

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)
model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen/Qwen2.5-3B-Instruct",
    quantization_config=bnb_config,
    num_labels=2,
)
model = get_peft_model(model, LoraConfig(
    r=8, lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
))

Results

Item	Value
Training data	2,114 pairs → balanced to 1,036
Training time	~1 hour 53 min (RTX 2070 Max-Q)
Final loss	1.454
Token accuracy	~76% (peak ~81%)

76% isn’t perfect. But as a zero-cost filter against the problem of irrelevant chunks slipping through, it works well enough.

Why No Cloud

Both training and inference run on the GPU sitting in front of me. Personal logs don’t go anywhere.

This is less about security and more about the feeling of owning your own data. Decisions about what goes into the RAG system aren’t constrained by any third-party terms of service.

Noticing that Gemini’s behavior matched my own RAG’s design philosophy is covered in The AI Arrived at the Same Design Philosophy I Had.
Implementing a design that lets you get back to the source (source_origin) is in What Gemini Had That I Didn’t — Noticed Only After Using It.
The measurement-driven record of raising retrieval quality itself is in A RAG with only vector search “forgets when it matters most” — taking recall from 0.2 to 1.0 with hybrid search + measurement.
The design philosophy behind this whole personal RAG is in The night when AI corrected me 5 times — Design philosophy to make personal RAG your companion for 5 years.