I built a way to retract a wrong lesson already stored in my RAG — automatically clearing the contamination that skews my companion every turn
Introduction
I’ve given the AI agents that work across multiple systems for me a shared memory (RAG — a mechanism that retrieves past decisions, failures, and lessons via vector search). With this, the agent behaves not as “a stranger meeting me for the first time every time” but as “a companion who knows me.” I’ll leave the premise to these two earlier articles.
Injecting ‘What to Recall Right Now’ into AI Every Turn — An Activation Layer for Personal RAG
I once found and closed one pitfall in this shared memory: a decision from one project leaking, via the shared memory, into another project’s design document.
This time I go one step further. Beyond just “detecting and separating” contamination, I built a mechanism to retract, after the fact, a wrong lesson that had already been written. Let me state the roles clearly up front: the RAG itself (the DB that accumulates records) is a passive vault — it doesn’t retract anything on its own. What actually executes the retraction is the operational mechanism I built around it (a daily batch job). Please read this not as a story of “the RAG cleverly purifying itself,” but as a story of “retrofitting an operational retraction process onto the vault after the fact.”
A wrong lesson skews the companion every turn
This memory has a mechanism for marking important items “priority: high.” Marked records get automatically injected at the start of every conversation as “recall this right now” (this injection is the activation layer — see the link above for details). It’s powerful, but that’s also its danger. If I write a lesson in a definitive tone as “priority: high” before the root cause is even confirmed, that error keeps getting injected every single turn.
I actually did this. For a bug in some tool, I wrote “diagnosis confirmed: this is the cause” before I had fully pinned down the cause. Later, it turned out to be wrong. But the whole time, the marked, wrong lesson kept being injected in front of my companion, quietly skewing the foundation of its judgment. If backups are the preparation for memory “not breaking,” this is the problem of memory “not getting contaminated.”
Why adding a correction note doesn’t work
The naive idea is “just append a correction.” But this barely works. In an append-only memory, even if you add a correction later, the original wrong record isn’t erased — it just stays there. Since search retrieves things that are semantically close, both the error and the correction get pulled up on the same topic, and the marked error keeps getting injected every turn regardless. A correction ends up as “juxtaposition,” not “overwrite.”
Companion recalls memory on some topic
│
├─▶ Wrong lesson (priority: high) ──▶ injected every turn ──┐
│ │ keeps skewing judgment
└─▶ Correction note added later ──▶ just retrieved together ┘
↑
Only "juxtaposed" — the error was never erased
With human memory, old errors fade through forgetting or being overwritten. An append-only log has no such “fading” mechanism. A RAG isn’t trustworthy just because you can “write to it” — it only becomes trustworthy once it also lets you “retract an error” — that’s the spine of this article.
The mechanism I built — one line at the start, then the daily batch deletes it
So I retrofitted a mechanism that lets me explicitly declare “retract this.” When I notice an error, I write this at the start of the line of the correction entry, together with a phrase that uniquely identifies the wrong lesson I want removed:
RETRACTS: <a phrase that uniquely identifies the wrong lesson to remove>
I added a step to the back end of the daily batch job that scans the entire record set for this marker and automatically deletes matching records from the DB (I put this in the same daily batch as the backup automation — I wrote about the full batch design in the companion article A). It’s this operational script that executes the deletion — the DB doesn’t decide anything on its own. The flow looks like this.
Notice the error
│
▼
Write RETRACTS: <unique phrase> at the start of the correction entry
│
▼
Daily batch (every day, on the local PC)
│
├─▶ Scan all records to pick up the marker
│ │
│ ├─ Match found ──▶ delete that record from both stores (raw notes and curated wisdom)
│ └─ No match ──▶ do nothing (idempotent)
▼
The wrong lesson disappears, injection into the activation layer and all
“Write one line the moment you notice, and the rest is deleted automatically.” Not juxtaposing a correction — cutting off the source.
Safety design to prevent misfires
A mechanism that automatically deletes DB records is dangerous if built carelessly. If a retraction sweeps up unrelated records along with it, that’s its own form of memory destruction. So I put the most effort into the safety side.
- Delete nothing by default (dry-run — instead of actually deleting, it just displays the “candidates for deletion”). Real deletion is limited to the batch job’s production run only. Manual checks never delete anything.
- Ignore phrases that are too short (anything under a certain character length is excluded). A common short word used for a retraction would match and sweep up unrelated records too.
- Never delete the retraction declaration itself (a record containing
RETRACTS:— i.e., the retraction entry itself — is excluded from the target set). Otherwise the retraction text would delete itself. Only aRETRACTS:at the start of the line counts as a marker; a mere mention of the word in the middle of some prose is not treated as one. - A backup exists before deletion. The daily batch takes a DB snapshot earlier in its run, so the state before the retraction is always preserved.
- Running it any number of times gives the same result (idempotent — if there are zero matches, nothing happens).
Before putting this into production, I confirmed it against not just the happy path (a marker is present) but also boundary and adversarial inputs (a phrase that’s too short, a mention of RETRACTS: inside a retraction sentence, the same word buried in ordinary prose) — verifying it doesn’t misfire on an unintended match.
In normal operation (zero markers), this is what actually prints. By default it deletes nothing and only lists the “candidates”; --apply is added by the daily batch alone (internal path and script name redacted).
$ retraction --dry-run
[retraction] scanning data/ ...
[retraction] RETRACTS markers found: 0
[retraction] would delete: 0 records
(no changes; --apply is added only by the daily batch)
Automation as insurance — two layers alongside the discipline of fixing it on the spot
The other thing I valued was not treating automation as a cure-all. The retraction marker is “insurance against what slips through,” not the front line. The moment I notice a wrong lesson, I delete it from the DB on the spot, right there, and fix the original note too. I made this the first-line discipline, and the marker-driven daily auto-retraction the second net.
Front line: delete on the spot the moment I notice + fix the source (immediate, certain)
│ if something slips through
▼
Second net: RETRACTS: marker → daily batch auto-retracts (insurance)
Not betting on a single approach is the same thinking as keeping backups in three layers. Immediate action is certain, but it only works when a human happens to notice. Automatic retraction is exhaustive, but only runs once a day. Only with both together does the gap actually shrink.
What I’m deliberately leaving simple for now
This mechanism is still deliberately kept simple. What can be retracted is the record on the DB side; I haven’t automated the physical erasure from the original Markdown (a tombstone — the process of planting a marker and invalidating it) as well. Automatically editing files is fragile and its side effects are hard to read. Since the actual damage — being injected every turn and skewing judgment — lives on the DB side, I prioritized cutting that off first, with certainty. The matched original files are recorded in a log, so I can delete them by hand if needed.
There’s also a smarter-sounding direction: “have the AI automatically find and delete lessons that look contradictory.” But that’s probabilistic, and it could mistakenly delete correct memories too. I chose to first lay a foundation of a certain, explicit marker — a judgment call that puts certainty ahead of cleverness.
Retrofitting “retraction” onto an append-only memory
Boiled down, what I did is one thing. I gave a memory that could previously only append, the operation of “treat this as if it never happened,” after the fact. A knowledge base that only accumulates lets errors settle like sediment as time passes, and a marked error gets dredged back up every single day. A memory that can’t forget doesn’t get smarter — it gets warped. Only by building retraction into the operations did memory become something that gets “corrected,” not just “grows.”
This shouldn’t be limited to personal RAGs. Whether it’s logs or design documents, any knowledge base that grows by appending alone needs, somewhere, a mechanism that treats “retraction and replacement” as prime real estate. I wrote about making memory “unbreakable” in the companion article below.
And the weekly checkup that keeps all of this healthy is in this piece.