Detecting and separating another project's decision that had leaked in — the pitfall of agents and RAG that span multiple systems

Introduction

On my own, I develop several systems in parallel: an all-in-one streaming platform, a self-made bank used for its payments, and a knowledge base that stockpiles cross-cutting memos and lessons. I gave the AI agent that works across them a shared memory (RAG). RAG is a mechanism that recalls past judgments, failures, and lessons by vector search (a search that pulls “what is semantically close”); with it, the agent acts not as “a stranger meeting you for the first time each time” but as “a partner who knows me.”

This is powerful. But span it across parallel development, and there was a quiet trap. A decision specific to one project leaked, by way of the shared memory, into the other project’s design doc — and as if that project itself had decided it. This is a record of how I detected that leak and separated the memory to plug it. The whole picture of deploying agents as a “fleet” is in Building a streaming platform and a bank in parallel.

Why shared memory is strong, and why it’s dangerous

The benefit of giving a spanning agent shared memory is that it can pull learning across projects. But the same mechanism carries project-specific decisions across the boundary too. A knowledge base pulls “what is semantically close.” Similar topics exist in other projects, so something you did not decide here gets pulled just for being “close.”

spanning agent ──▶ searches shared RAG: "MVP policy for the customer-facing front"
                          │  pulls semantically-close memory
        ┌─────────────────┴─────────────────┐
   platform's own decision            working on the bank side
   "don't build the screen out,           │
    fixed layout, responsive later" ─(leak)─▶ written into the bank's design doc
                                           as if it were the bank's decision ×

What the search returns is a “semantically close” memory; “which project’s decision it is” does not come with it. So the side that pulled it will mistake the origin unless it looks.

# when the spanning agent pulled shared memory (dangerous without checking the origin)
query : "MVP policy for the customer-facing front"
 └─ hit : "don't build the screen out, fixed layout, responsive later"  [origin: platform]
          ↑ semantically close. But the bank made no such decision

(MVP = the first version that delivers value with the minimum; a policy of trimming the build-out and shipping something that works first.)

It actually leaked

On the platform side there was a project-specific MVP policy: “for the MVP, don’t build the screen out; mobile is fixed-portrait, Web is fixed at the same size, responsive support is pushed to a later phase.” The narrow fixed-width layout is a consequence of that policy.

But while building the bank system’s customer-facing screen, the same narrow fixed width was implemented — and it was written into the bank system’s own design doc as an “MVP build-out policy.” The bank side made no such decision.

[leak]   the platform's "build-out policy" gets written into the bank's design doc as the bank's decision ×
[proper] that decision is platform-specific → remove it from the bank's design doc, return to the bank's own policy ✓

What made me notice was a comment that came up in review: “isn’t this another project’s decision?” The agent had pulled the memory on the similar topic of “MVP for the customer-facing front” and, without distinguishing the origin project, settled it into this project’s design doc.

Why it happens

What the knowledge base’s search measures is only “semantic closeness.” Whether the memory is which project’s decision, or a decision that actually exists in this design doc — that belonging information is not in the closeness. In other words, “semantically close” is not “a decision of this project.”

(I stepped on the same “semantically close ≠ OK to mix” trap on the product side of vector search too — where works of another category slip into recommendations. That’s in Vector search quietly mixes in whatever is “semantically close”.)

How to plug it

The countermeasures are a role split of what goes into memory, and verification before adoption. First, put only cross-cutting “wisdom” into shared memory, and keep project-specific “facts” on the code / design-doc side.

# splitting what goes into memory (operating discipline)
shared RAG    … transferable wisdom (design principles, failure lessons, technical know-how)  ← may cross
code/design doc … project-specific facts (config values, IDs, that project's decisions)       ← must not cross

On top of that, verify any decision the memory presents against this project’s design doc before adopting it. Pass this step through the pre-flight gate.

# pre-flight gate (excerpt) — put the design doc above memory
・Even if memory says "this is decided," adopt only after confirming it actually exists in this design doc
・If a memo/memory conflicts with the design doc, the design doc wins
・For a pulled memory, look at its origin project (the more similar the topic, the more you suspect a cross-project leak)

Line the diagram and the discipline up, and the point of defense collapses to one — wisdom may cross, decisions must not.

”Semantically close” is not “a decision of this project”

Giving a spanning agent shared memory, what landed is that memory’s strength and danger are two sides of one thing. Let wisdom cross, don’t let decisions cross. Put only transferable wisdom into shared memory, and keep specific decisions in the design doc and code. Then verify a decision the memory presents against whether it actually exists in this design doc before adopting it, and look at the origin project of a pulled memory. The more similar the topic, the more you suspect a cross-project leak.

A spanning agent + shared memory is a powerful way to build “a partner who knows me.” But the same mechanism carries in the semantically-close decision of another project. This separation and verification was the key to making cross-project operation safe. The fleet itself is in Building a streaming platform and a bank in parallel; folding failures into gates in the instructions is in Lifting “technically correct” to “a finished product”.

The whole picture — deploying agents as a “fleet” and building two systems in parallel — is in Building a streaming platform and a bank in parallel.
Folding agent failures into pre-flight gates in the instructions is in Lifting “technically correct” to “a finished product”.
The same “semantically close ≠ OK to mix,” stepped on in product-side vector search (another category slipping into recommendations), is in Vector search quietly mixes in whatever is “semantically close” — the trap of different catalog types bleeding into recommendations.
The table of contents for this whole AI assistant operations series is in AI Assistant Operation Notes — Practical record for raising Copilot/Claude Code as a companion (series table of contents).