Injecting 'What to Recall Right Now' into AI Every Turn — An Activation Layer for Personal RAG

Introduction

In my personal AI assistant setup, I store past decisions, lessons, and information about myself in a knowledge base (RAG). But storing alone isn’t enough. If the AI doesn’t go retrieve that memory on its own, the lessons I’ve stored just sit there, dead, never used even once.

So I built a mechanism that captures the conversation’s input every time and automatically injects memories relevant to the current topic into the AI. I call this the “Activation Layer.” The “storing” side of memory (raising retrieval accuracy with numbers) is covered in Measurement-Driven Tuning of RAG Retrieval. This article is the flip side — how to make the AI recall stored memory at the exact moment it’s needed.

Storing alone doesn’t let you recall

No matter how much memory you store, if it doesn’t surface “when you need it,” it’s not much different from not having it at all. Even for humans, if you can’t recall something you supposedly remember at the crucial moment, the outcome is the same as not knowing it.

The same goes for an AI assistant. Even if you store past lessons in a RAG, if the AI doesn’t search on its own asking “is this relevant right now?”, that lesson doesn’t do its job. But asking it every single time to “search the past first” doesn’t stick, and if you forget to ask, you repeat the same mistake.

What’s needed isn’t search itself but recall — a state where memories relevant to the current context are already pulled in by the AI on its own, before anyone asks. I built that as a mechanism.

[Storage only]  Accumulate memory ─▶ (AI never retrieves it on its own) ─▶ Dead storage, repeated failures

[With recall]   Accumulate memory ─▶ Auto-inject relevant memory every conversation ─▶ Respond already having recalled it

Hooking the conversation input to inject memory

What I used is a hook in my AI tool (Claude Code) — a mechanism that lets you insert your own script at a specific point in time. Here I use the hook that fires “the moment the user submits input” (UserPromptSubmit).

Here’s the flow: when the user types something, the hook fires, uses that input as a clue to gather related memories from the knowledge base, folds them into a short digest, and injects it at the top of the AI’s prompt. The AI starts its response already “having recalled the relevant memories.”

User submits input
     │
     ▼
 Input-submit hook (UserPromptSubmit) fires
     │
     ├─ Gather "memories relevant to this input" from the knowledge base
     ├─ Fold them into a short digest
     ▼
 Inject at the top of the AI's prompt ─▶ AI responds having already recalled them

The important part is that this runs automatically every single turn. Neither the user nor the AI has to consciously think “remember this.” Recall is always working quietly underneath the conversation.

Splitting the two kinds of recall instead of mixing them

This is the heart of the design. There are two kinds of “recall” with different natures, and they must not be lumped together.

One is content-based recall. Convert the immediately preceding input into a semantic vector, run a semantic search over the knowledge base, and pull memories close to the current topic. With this, even an old lesson stored long ago comes back to life if the topic matches. Without it, the AI can only “remember recent things” — the companion ends up living on nothing but fresh memory, a kind of amnesia.

The other is recency-based recall (forced injection by recency). Regardless of content, it unconditionally injects the latest N items. Recent decisions and work in progress must not be dropped, even if they share no words with the current input.

These two are separate systems even in human memory. “Chain-associating your way to something related” and “remembering what you just decided” work differently. So I keep them as separate sections. On top of that, absolute prohibitions that must never be violated (priority=high lessons) and basic information about myself (profile) go into a fixed slot every time, regardless of content or recency.

Injected digest (assembled every turn)
  ├─ ① Profile (about myself)                          … fixed slot
  ├─ ② Absolute prohibitions (priority=high lessons)    … fixed slot
  ├─ ③ Recent lessons/conclusions (forced by recency)   … recency
  └─ ④ Records relevant to the current context (semantic search) … content-based

In code, ③ and ④ are pulled separately. ④ runs a semantic search on the input text, ③ takes items in descending date order — the retrieval methods themselves are different.

# (4) Content-based recall: semantic search on the immediately preceding input, pull memories relevant to the current context
res = vault.query(query_texts=[prompt], n_results=TOP_N_QUERY * 2,
                  where={"source_type": {"$in": ["lesson", "conclusion", "knowledge"]}})

# (3) Recency: regardless of content, unconditionally inject the latest N items
items.sort(key=lambda x: x.date, reverse=True)   # descending by date
recency = items[:TOP_N_RECENCY]

And so the same memory doesn’t appear twice across different slots, whatever text was already adopted in an earlier slot is excluded from later slots. The recall budget is limited, so it shouldn’t be filled with duplicates.

Recall must fold within a limited “budget”

There’s one more thing, unglamorous but unavoidable: the constraint of volume. There’s an upper limit to how many characters you can inject into a prompt (about 10,000 characters in this system). Memory grows every day. It doesn’t all fit.

So the ever-growing memory has to be folded down to a fixed amount (about 6,000 characters) and injected every time. Two techniques matter here.

One is to keep the content as close to identical as possible across turns. If the injected content is stable, the AI side’s input cache (a mechanism that reuses computation for the same preamble) kicks in and reduces waste. You don’t pay unlimited cost for the sake of recall.

The other is deciding a cut order in advance. When it can’t all fit and something must be trimmed, the first thing cut is the lower-priority content-based recall (④). What’s protected unconditionally is the absolute prohibitions (②) and recency (③) — cut what does the least harm if missing, keep what causes an incident if dropped, until the very last.

Finally, if the hook fails, the conversation doesn’t stop. Even if memory retrieval errors out, the input still passes through to the AI as-is. Recall is “a backing layer that helps when present,” and it’s not built so that the conversation can’t happen without it. A safety mechanism must never hold the main function hostage.

Recall is what turns memory into memory

What clicked for me after building this is that the value of memory isn’t determined by “how much you’ve stored” but by whether you can recall it. No matter how good the lessons you’ve stored are, if they don’t surface at the moment you need them, they aren’t functioning as memory.

Memory only works once it’s offered before being asked for. Don’t leave retrieval up to the other party’s will — make it recall automatically on every turn.
Separate content-based recall from recency. Keep both an old lesson relevant to the current topic and the recent context you can’t afford to drop, as two distinct systems.
Recall must fold within a limited budget. Keep it stable to let the cache work, decide the cut order in advance, and don’t let the main function stop even on failure.

The storage mechanism (RAG) was only half of memory. Pulling in what’s relevant to the current context before being asked — only with this recall layer does stored memory become “usable memory.” Humans can learn from experience not because we remember, but because we can recall when it matters. I wanted the AI to have that same recall.

The “storing” side of memory — raising retrieval accuracy with numbers instead of gut feel — is covered in Measurement-Driven Tuning of RAG Retrieval.
The overall design philosophy of this personal RAG is summarized in The night when AI corrected me 5 times — Design philosophy to make personal RAG your companion for 5 years.
Using the same hooks but for enforcement rather than recall — a mechanism that stops dangerous operations before they ever happen — is covered in Building a “Won’t Let You” Mechanism into an AI Coding Assistant — From Advice to Enforcement, and the Side Door Left Open in the Gate.