Checking a personal RAG's health every week by mechanism, not willpower — a multi-angle health check and a time-gate that forces the check to run
Introduction
I run a personal setup that works like an external long-term memory. Feed it daily work logs and decisions, and it can pull back whatever is closest in meaning — a RAG (a mechanism that recalls past records via vector search). I’ll leave the background to this article.
Everyone builds something and gets it running at least once. The hard part is continuing to properly check, every single week, whether it is still working correctly right now. This article covers two things. One is the substance of checking, from multiple angles, whether the memory is healthy. The other is the story of how I made that check happen every week by mechanism — instead of leaving it to “whenever I feel like it” or “whenever I notice.”
Deciding what counts as “healthy,” first
“It seems to be working somehow” isn’t a check. I defined up front what counts as healthy and what counts as abnormal, and in what state. I split it into five axes.
| Axis | What to look at | What anomaly it catches |
|---|---|---|
| Uptime / corruption | Whether querying the DB for a count, a fetch, or a search causes an immediate crash | Corruption of the index (the internal table of contents that speeds up search) of a DB format that opens its file directly |
| Consistency | Whether the difference in record counts between the two systems that keep the same records stays constant | Inconsistency where only one side breaks or is missing records |
| Ingestion | Whether the daily automatic ingestion process’s logs are free of errors | A silent clog where records quietly stop coming in |
| Search | Whether relevant queries return reasonable results, and irrelevant input doesn’t misfire | A false high-confidence hit, or a crash |
| Backup | Whether all the layered backups are at the latest date | A backup that can’t be restored (covered in a separate article, below) |
"Is my companion's memory healthy?"
│
┌──────────┬─────────┼──────────┬────────────┐
▼ ▼ ▼ ▼ ▼
Uptime/ Consistency Ingestion Search Backup
corruption (count (0 errors) (normal + (all layers
│ diff) abnormal) latest)
│ │
└─ Crash the instant └─ Don't return a high-
it's touched = confidence false hit on
sign of corruption nonsense/empty input
Axis 1, “index corruption,” is the failure mode that has twice nearly wiped out this memory entirely in the past. That story and its root-cause fix are in a separate article.
A check of only the happy path doesn’t prove “it won’t break”
The check that’s easiest to shortcut is #4, search. “Query with a related term, get a plausible-looking result. OK” — that only looks at the happy path (input that works). Passing the happy path only shows that it “works correctly”; it doesn’t show that it “won’t break.”
So I deliberately mix in the unhappy path (boundary, invalid, adversarial input) into the check.
Normal query "how the retraction marker works" → close records rank top (relevance: high) … as expected
Nonsense word "qwzxlkjhgfd" → no crash, relevance clearly low … no misfire
Empty query " " → no crash, returns low relevance … no misfire
What I want to see is whether, given irrelevant input, it avoids returning a plausible-looking wrong hit with high confidence. The quality of a search shows up not in how well it hits when it should, but in how correctly it misses when it should miss. A check that’s satisfied with the happy path alone doesn’t deserve to be called a “multi-angle check.”
So far this has been about the content. The problem was that the check “wasn’t run”
Both the perspectives and the procedure were in place. And yet, this check was not being run on a regular basis. The times I actually ran it were when someone asked from outside, “is it really working?” As a rule, I believed I would “check every week.” But that was nothing more than a resolution in my head — there was no gate guaranteeing it would actually run.
This is a failure with a sense of déjà vu. I had once decided that I would “take a backup before any destructive operation,” yet I proceeded without ever actually creating one, and broke my memory. The structure is exactly the same — between “deciding to do it” and “having done it,” there was no enforcement at all.
I decided to “take a backup” but never actually made one — and broke my memory vector DB twice
Entrust discipline to “willpower” or “memory,” and it slips. That’s true whether it’s a human or an AI assistant doing the hands-on work. So this time, I stopped relying on willpower.
The fix — put it on a time-gate, not willpower
What I did was simple. Record the date the previous check was completed in a single file, and once a fixed period (7 days) has passed since then, automatically inject “the check is due” the moment a session opens. The side that receives the injection runs the check right there, and once done, updates the completion date to today. No further nudge comes until the next 7 days.
Session starts
│
▼
Read the completion-marker date ──(missing/broken/empty)──┐ ← safe side: treat it as "due" regardless
│ valid date │
▼ │
today − last ≥ 7 days ? ──no──▶ Do nothing (pass silently)
│ yes │
▼ ◀───────────────────────────────────────────────────┘
Inject "check is due" at the start of the session
│
▼
Run the check → report the result → update the completion marker to today
│(if forgotten)
└──────────────▶ Nudged again next time (resilient to being missed)
There are two key points. One: if the completion marker is broken, empty, or missing, it falls to the safe side and is treated as “due.” It falls toward nagging you extra, not toward letting the check be forgotten. The other: even if you forget to update the completion marker, the next session will nudge you again. Even if it’s missed once, it will eventually be caught.
If the deadline has come, the moment a session opens, a nudge like this gets injected straight into the opening (actual output, with paths and internal names withheld).
[Weekly Gate] The personal RAG's weekly health check is due. Last=2026-07-01 / 27 days elapsed.
Run the check in this local session (a cloud environment cannot reach the local DB):
(1)Uptime/corruption (2)Consistency (3)Ingestion (4)Search(normal+abnormal) (5)Backup freshness
After running it, update the completion marker to today's date.
If it’s still within the deadline, this nudge doesn’t appear, and it passes through silently. It’s only the difference between appearing or not, but that alone changes the check from “do it whenever I feel like it” to “it will always land in front of me once the deadline arrives.”
This idea itself — of raising “advice” up to “enforcement” — is something I already built out once in a different setting. I’ll leave the discussion of the underlying principle to that article.
Not mistaking where the mechanism runs
When building this gate, there was a trap I nearly stepped into. What the check targets is a DB running on my local PC, and logs that exist only locally. If I tried to run this check from an environment away from that machine — say, running it in the cloud — it couldn’t reach either the local DB or the local logs from there.
What’s troublesome is the way it breaks when it can’t reach. Even when it fails to reach and can’t run the check, it can look as if it “finished with no errors” — a failure quietly turning into a success (a silent false negative). For a health check, this is the worst case: it produces a state of “thinking I checked, when in fact I saw nothing at all.”
So I wrote “this check must always be done in a local session” explicitly, both in the wording of the nudge and in the mechanism’s assumptions. Running it automatically on a fixed schedule is not, in itself, the goal. The goal is the fact that “I genuinely looked.” Rather than spinning uselessly in a place that can’t reach, it’s better to genuinely look once in a place that can.
Testing the gate itself from multiple angles
Once I built a gate that forces the check to run, I also needed to verify that the gate itself works correctly — not just on the happy path. Here too, I imposed the same discipline on myself.
- Happy path: the nudge appears if enough time has passed since last time; it passes silently if not yet.
- Boundary: how it behaves on exactly the deadline day.
- Unhappy path: the completion marker is broken, empty, or simply missing — does it fall to the safe side (nudging) in this case?
Only once the unhappy path has passed too can I say “this gate can’t be bypassed.” Building a gate and testing only the happy path is the same as installing a lock, confirming only that it “closes,” and never trying whether it “can be pried open.” (The exact internal procedure and configuration values are withheld here. It’s enough if the idea behind the mechanism comes across.)
Turning the discipline I decided on into a form that “continues”
What I built isn’t some grand monitoring platform. I only defined the check’s perspectives from multiple angles, and tied its execution to a deadline instead of to willpower. Even so, the effect is clear. A resolution of “do it every week” will always erode if left alone. Even after it erodes, next week I’ll get caught in the same place again — and on that single point, the resolution turned into a discipline that “continues.”
Recurring maintenance doesn’t continue through sheer diligence. It only continues once it’s been cast down into a structure built to continue. The stories of keeping memory from breaking, correcting it so it doesn’t get contaminated, and making it measurable, I wrote across a paired set of three articles. This article is about the mechanism for checking that keeps all of that healthy, going forward.