I touched a "DB that opens its file directly" from multiple processes, and broke it twice — until I narrowed the writer down to one, for good

Introduction

I run a personal knowledge base (RAG) that stores my daily records and past conversations and lets me pull them back by meaning. Its contents live in a vector DB — a database that turns text into arrays of numbers (vectors) and stores them so they can be searched by semantic closeness. Past judgments and lessons accumulate in here; to me it’s close to memory itself.

That vector DB broke, index and all, twice. Search and even counting records stopped working — my memory became unreadable. The cause wasn’t some flashy bug — it was a structural error in how it was used. This article is a record of tracking down why it broke, and rebuilding it into a form that never breaks again.

After breaking it, I made another mistake too — “deciding to take a backup and then never actually doing it” — the preventive side of that story is written up in The story of deciding “I’ll take a backup” and not doing it, and breaking a memory vector DB twice. This article is the root side of that — why it broke in the first place.

What broke

The symptom was clear. Basic operations against the vector DB — counting records, searching — suddenly started crashing with an Access Violation (illegal memory access). The file holding the data itself was still there, but the HNSW index used to pull it out — the index that speeds up nearest-neighbor search over vectors; if this breaks, search can’t work at all — was corrupted.

The first time, I recovered it. But a few days later, it broke the exact same way. If it breaks in the same place twice, that’s not chance — that’s structure. I stopped doing symptomatic fixes and switched to root-causing it.

Why it broke: everyone was opening the file at the same time

What I was using was the DB’s standard embedded mode (embedded DB = a mode where, inside the app’s own process, the DB opens its file directly to read and write, without standing up a separate server — a handy approach). For single-person use, this is fine as-is.

The problem was that actual operation wasn’t “single-person.” This knowledge base had multiple processes touching the same DB file.

        the same DB file (chroma_db)
                 ▲  ▲  ▲  ▲
   ┌─────────────┘  │  │  └─────────────┐
   │        ┌───────┘  └───────┐        │
 MCP server  another client  input hook   throwaway script
 (resident)  (per conversation) (every prompt) (batch/repair etc.)
   │            │               │            │
   └──── everyone opens and writes the same file at once ────┘
                     │
              the HNSW index breaks (violates the single-process assumption)

An embedded DB updates its index file assuming one process has it open. When multiple processes write to it at the same time, the index updates collide and it breaks. A tool built for one person was, unknowingly, being used by many — that was the true identity of the two breakages.

Root fix one: narrow the writer down to one

The core of the fix is simple. Make only one process open the DB file.

The way to do it: switch the DB to server mode (a mode where the DB is started as its own independent server process, and everyone else connects to that server to read and write). This way, only the one server process opens the file directly, and every other process merely asks over HTTP. With a single writer, the collision itself disappears.

【Before】each process opens the file directly      → multiple writers → it breaks
【After】each process ──HTTP──▶ Chroma server ──▶ file
                                (only this one opens the file)

In the code, I routed all access to the DB through a single entry point (get_client()), and banned use of the old direct-file-open method (PersistentClient). Rather than leaving the ban as “just be careful,” I pinned it in place with a comment on the entry-point function, closing off the path by which it could recur.

def get_client():
    """Get a Chroma client (auto-starts the server if not running, then connects).
    Using PersistentClient directly is banned (a recurrence path for multi-process corruption).
    All access to chroma_db must go through this function."""
    # HttpClient = via the server. Never opens the file directly
    client = chromadb.HttpClient(host="127.0.0.1", port=8800)
    client.heartbeat()
    return client

There’s a small but effective point. The port number itself happens to prevent starting the server twice. If some process tries to start the server while someone else is already using that same port, the latecomer fails to claim the port and exits immediately. The plain fact that “a port can only be held by one process” works, as-is, as the exclusive lock that keeps “only one server.” Rather than adding a mechanism, I leaned on a constraint that already existed.

Root fix two: even with the file intact, the contents duplicate

Going server-side made the physical corruption — “the index file breaks” — disappear. But another, separate concurrency problem remained. Logical duplication.

This knowledge base checks, before writing, whether the same content already exists, and rejects it if so (deduplication). This check has two steps: “read (check if it exists) → write if not.” Here, when multiple processes come in to write nearly the same content, nearly at the same time, both read “not there yet,” and both write. The dedup check gets slipped past by the pair of them together.

Process A: checks if it exists → no ┐
Process B: checks if it exists → no ┤ ← both checked "before writing," so both got "no"
Process A: writes              ┘
Process B: writes              → the same content ends up written twice

Server mode made “the writer to the file” a single one, but it didn’t bundle together the “one continuous operation from check to write.” So I routed the entire write operation through a single gate (a mutex). A mutex is a checkpoint that lets only one party through at a time. Do the whole “check → write” inside this gate, and B has to wait until A gets all the way through, so B correctly reads “it’s already there.”

# Cross-process write mutex.
# Serializes the "dedup check (read) → write" sequence as one unit, preventing double registration from concurrent writes.
# Reentrant (taking it in a nested call doesn't deadlock against itself).
# A timeout is never swallowed — it surfaces as an exception (never fail silently).
_WRITE_MUTEX = FileLock(str(DATA_DIR / ".write_mutex.lock"), timeout=180)

There’s one thing I deliberately chose. When the gate times out, I made it surface as an exception rather than being silently ignored. An abnormally long wait is a sign something’s stuck, and swallowing it just seeds the next failure. A defense only means something if you can notice when it fails.

Guarantee “won’t break” with structure, not with being careful

Looking back, both breakages and the duplication shared the same root: using a tool built on the assumption of a single process, in the reality of multiple processes. And the fix was consistent too — bundle the concurrent-access paths into one.

The physical corruption disappeared by narrowing the file-opening process down to one (via the server).
The logical duplication disappeared by routing check-to-write through a single gate (the mutex).

What mattered was that neither ended with “be careful from now on.” The old direct-file-open method was made unusable at its entry-point function, the double server start was prevented by an already-existing constraint (the port), and a timeout was surfaced as an exception. Being careful breaks the moment you’re tired. Only once you make it a shape that can’t break does “never breaks again” become a guarantee.

The more valuable the data, the less you should entrust how to protect it to your own attentiveness. Rebuild it as a structure that doesn’t break even when touched at the same time — it took breaking it twice for that to finally sink in.

The failure of “deciding to take a backup and not actually doing it” after breaking it, and how I turned the prevention into a mechanism, is in The story of deciding “I’ll take a backup” and not doing it, and breaking a memory vector DB twice. This article is the root side (why it broke).
The story of raising an incident’s lesson from “advice” to “enforcement” is collected in Building a mechanism that makes an AI coding assistant “not do it”.