Tech Blog

I decided to "take a backup" but never actually made one — and broke my memory vector DB twice

RAG ChromaDB Operations Backup Vector DB

Introduction

I run a personal vector DB. I drop the day’s work logs and judgment calls into it, and an AI assistant pulls them back out via semantic search — an externalized long-term memory of sorts. I wrote about the design philosophy behind it in Designing a RAG That Stays With You for 5 Years. This article isn’t about that design. It’s about an accident that happened while operating it.

Let me lay out two premises first. Without them, neither the meaning of the accident nor the panic in the moment will come across.

Premise 1: the contents are kept in my own hands, never entrusted to a third party. The contents are personal records, so the policy is to not send them outside as-is. The code (the mechanism) is under version control, but the records themselves — the data — are never synced externally. This policy is correct. But at the time, I had implemented it as “everything stays local, period,” so there was no safety net where a cloud service or Git history could “automatically roll back to a prior state” for the data. Whether I could roll back depended entirely on whether I myself had produced a backup artifact by hand. It may sound dramatic, but this data is my own externalized memory, and it can’t be replaced.

Premise 2: this DB is operated together with an AI coding assistant. Day-to-day migrations and cleanup are, more often than not, done by having the assistant move its hands. Which means I need to design and tune “how the assistant behaves.” This accident happened because that tuning wasn’t yet sufficient.

I broke that DB. Twice. Both times the same way, and both times in a way that could have been prevented.

What happened

The task was a migration: moving data to a different location. It’s a textbook destructive operation — if it fails partway through, the original data can end up broken.

As a process, I had already decided that “you take a backup before a destructive operation.” That judgment was correct as far as it goes. But at this point it was only written down as an operating rule (advice) — there was no mechanical gate to confirm it had actually been followed. The assistant said “I’ll take a backup” before starting the migration, but proceeded with the migration without ever producing the artifact. The rule existed, but nothing enforced it.

Right after running the migration, search stopped returning anything. Every query came back empty, and eventually the DB itself stopped responding with an error. That’s the moment the blood drained from my face. Per Premise 1, this memory isn’t in the cloud. And I hadn’t created a recoverable artifact either. For a few minutes, the thought “did months of records just vanish, entirely, right now?” crossed my mind — and honestly, I was quite shaken.

Laying the process out as a diagram makes it clear exactly where the accident happened.

Right before the destructive operation (data migration)
        │
        ├─▶ Create the backup artifact ─▶ Verify its location ─▶ Run the migration
        │                                              │
        │                                              ├─ Success ─▶ Done
        │                                              └─ Failure ─▶ Restore from artifact ─▶ Recovery (short)
        │
        └─▶ "Intend to" back up, but never create the artifact ─▶ Run the migration ─▶ Search wiped out
                                                        │
                                                        └─▶ No recoverable artifact ─▶ Manual recovery (hours)
                                                                    ▲
                                          (the second time also went down this same lower path)┘

How it broke, and how I recovered it

Once I calmed down enough to check the contents, I breathed a little easier. Only the search index had broken — the records themselves were intact.

This DB has an HNSW index to make semantic search fast — a graph structure you traverse via nearby vectors to reach your target. Because processes running in parallel grabbed the same DB independently, this graph ended up inconsistent on disk, and search collapsed entirely. Meanwhile, the records themselves (the documents with their metadata) sat untouched somewhere else. What broke was “the retrieval mechanism,” not “the memory itself.”

Once I understood that, recovery could proceed calmly. Here’s the procedure:

Search completely dead (only the index is corrupted / the body of data is intact)
        │
        ├─▶ ① First, back up the entire DB directory (to avoid losing any more)
        ├─▶ ② Export all documents and metadata via raw SQL ─▶ Verify the count (confirm zero loss)
        ├─▶ ③ Delete and rebuild the corrupted index (collection)
        └─▶ ④ Re-embed everything and re-ingest it ─▶ Counts match — search is back

The first time, I recovered every single record (3,707 at the time) without losing a single one. But rebuilding the embeddings took about 2 hours and 19 minutes. The data survived, but that time never came back.

Then, a few days later, it happened again. A different destructive migration, and the same failure mode. I recovered everything with the same procedure (now grown to 4,732 records), but the feeling of “not this again” and the fatigue of burning several more hours weighed heavier than the first time. Twice is not a coincidence. I had to admit it was structural.

Why the second time happened

I split the cause into two layers: the surface-level trigger, and the real cause underneath it.

Surface cause: multiple processes were touching the DB directly, at the same time

This vector DB originally ran with a “program opens the DB file directly” design. Convenient, but with a pitfall: if multiple processes each open and write to the same DB file independently, the internal index can end up in a contradictory state. The migration process and a background auto-ingestion process were both grabbing the same artifact separately. That contention is what triggered the index corruption.

The first time, I stopped right there. Once the index was rebuilt and search was working again, I treated it as “fixed” and never pushed to ask “what actually corrupted the index in the first place.” So the same structure remained, and a few days later it tripped me up in the exact same spot.

Underlying cause: there was no gate to enforce the decision

Here’s the other, deeper cause. The judgment “I will take a backup” was never distinguished from the fact “I did take a backup” — and there was no gate enforcing that distinction. Writing a rule down as advice, without a gate at the point of execution to confirm it was followed, means the rule can simply slip through.

This is especially true in an operation where an AI assistant does the hands-on work: the gap between “what was stated” and “what was done” widens easily. The declaration “I’ll take a backup” and the fact that the artifact actually exists are two different things. When you do the work yourself, those two things stay loosely connected in your head without you noticing; but the moment the party doing the work changes, that connection turns out not to have existed at all. This time, that’s exactly where I got caught.

What I rebuilt

1. Consolidated the DB’s entry point into one

I dropped the “program opens the file directly” approach and moved to a design where a single resident process holds exclusive access to the DB, and everything else connects to it to read and write. With the write path physically reduced to one entry point, there’s no longer room for multiple processes to grab the same artifact separately and leave the index in a contradictory state.

【Before】Each process opens the DB file directly (contention can occur)
   Migration process     ─┐
                          ├─▶ Same DB file ─▶ Index can become inconsistent
   Auto-ingestion process ─┘        ▲
                                    └─ A separate process writes at the same time

【After】A resident process holds the DB; everything else connects (entry point unified)
   Migration process     ─┐
                          ├─▶ Resident process (holds the DB) ─▶ DB
   Auto-ingestion process ─┘        │
                     Layered mutual exclusion
                     serializes concurrent writes

2. Guarded concurrent writes with multiple layers

On top of unifying the entry point, I stacked several layers of mutual exclusion so that processes I don’t want running simultaneously never overlap. If one layer is slipped through, another layer catches it. A safety design works better when caught across multiple layers than when it relies on one perfect wall.

3. Promoted the “decided rule” from advice to enforcement

This was the one that mattered most. The rule “confirm the artifact exists and where it lives, before proceeding with a destructive operation” had already been decided. But this incident’s real lesson was: “writing the rule down isn’t enough.” Without a gate to confirm it was followed, advice will eventually slip through.

So I built a gate into the assistant’s operating procedure: the instant a destructive operation is about to run, mechanically confirm a backup artifact exists, and if it doesn’t, stop right there. In other words, I promoted advice into enforcement.

Request for a destructive operation
      │
      ├─[Back when it was only advice]─▶ The rule "existed," but nothing checked it ─▶ Slips through with no artifact ─▶ Corruption
      │
      └─[After adding the gate]─▶ Mechanically confirm the backup artifact exists
                                    │
                                    ├─ Artifact exists ─▶ Allow the destructive operation
                                    └─ No artifact ─▶ Stop here (do not proceed)

This is exactly the kind of tuning that keeps an AI assistant working safely. Whether it’s a human or an AI, any design that relies on “being careful” will eventually leak somewhere in a busy workflow. A line you must never cross should be guarded by a gate, not by attentiveness.

After the fix, I didn’t just confirm normal operation worked — I checked, from both the happy path and the failure path, whether it holds up under an operation designed to break it. Working correctly under normal conditions only shows “it works right,” not “it won’t break.”

The hole that’s still open: a local backup can vanish along with the local machine

The three fixes stopped the recurrence of that particular failure mode. But there’s a hole I haven’t closed yet. Even if you produce a backup artifact locally, if the local machine — the disk, the device — is ever lost, the backup goes down with it. Per Premise 1, this memory isn’t in the cloud. Being local-only means a single hardware failure, theft, or disaster could wipe out the backup along with everything else.

Which brings back the original dilemma: “I don’t want to entrust the contents to a third party. But local alone isn’t durable enough.” — These two can coexist. Encrypt it locally first, then send only the encrypted blob offsite. The destination can’t read the contents (the key stays local only), and yet, if the local disk dies, you can still restore from offsite.

Local (yourself only)                     Remote (cloud storage)
   Plaintext DB ─▶ Encrypt locally ─▶ Send only the encrypted blob ─▶ Store
                  │                                       │
             Key stays local only          The destination can't read the contents
                  │
                  ├─ Goal 1: restore remotely even if the local disk dies (durability)
                  └─ Goal 2: never hand contents to a third party (preserves Premise 1)

Here’s the design I landed on: restic, a well-established tool that encrypts client-side, for incremental, encrypted backups, with my own AWS S3 account as the destination. Three points matter:

  • Encryption completes locally. restic encrypts locally before sending, so only the encrypted blob ever lands at the destination. I keep the key (a strong passphrase) both locally and in an offline backup copy, and never hand either plaintext or the key to the destination. Premise 1 (never send contents outside) is preserved this way.
  • The destination is isolated with least privilege. A dedicated set of backup credentials, permitted only to read and write that one storage location, with public access blocked entirely. This confines the damage from a leaked key or leaked permission to one single spot.
  • Deduplication tames the bloat. My previous naive approach copied everything wholesale on every backup, so for a few hundred MB of actual data, backups alone had ballooned to several GB. restic reuses the parts that haven’t changed, so no matter how many generations of history you keep, the total size caps out near the scale of the real data.

I haven’t implemented this yet — that’s homework for next time. But I’ve already decided what to use and how to guard it. Local, remote, encryption, least privilege — combine these four and you can have both “never hand over the contents” and “durable offsite,” at the same time.

What I changed to prevent a third time

“Taking” a backup isn’t a completed state. Define “done” not by a decision but by the existence of an artifact. For a backup, that means: something recoverable actually sits there. Whether or not you eventually set up an offsite copy, producing a locally recoverable artifact right before a destructive operation is the first line of defense.

When something breaks, first separate “the body” from “the index.” In this case, nothing was truly lost — what broke was only the index, which can be rebuilt. Even in the middle of a panic, separating what’s actually gone from what can be regenerated lets recovery proceed calmly. So I keep the body and the index stored separately, as a last-resort insurance policy.

A line you want to protect should be protected by enforcement, not advice. “Being careful” only holds up under normal conditions. As long as an AI assistant is doing the actual work, the gap between “stated” and “done” needs to be closed by a gate, not by attentiveness. That’s what it means to tune an assistant to work safely.

The hours I burned twice don’t come back. And yet the irreplaceable memory itself was never lost even once, precisely because the body stayed separate from the index. A gate that confirms, before a destructive operation, “is something recoverable really sitting there” needs to exist as a mechanism, not as willpower. That’s the decision I made so I don’t get tripped up in the same spot again.

Feel free to send a message

Job offers, project referrals, feedback, questions — anything is welcome. I sincerely hope to connect with people who share high ambitions. I will keep taking on the challenges I have staked my life on. Thank you very much.