Vector search quietly mixes in whatever is "semantically close" — the trap of different catalog types bleeding into recommendations
Introduction
On my own, I am building an all-in-one platform (“EVERY DAYS”) that bundles video and music. It brings multiple services — video streaming (VOD), DVD and CD delivery rental, and so on — under the same membership and the same screen. The vision of growing a single rental business into a DMM-style all-in-one platform is described in From DVD rental to a DMM-style platform.
Because of this design, video works (VOD, DVD) and music (CD) live together inside the same system. And search, related recommendations, and personalization are all served by a single embedding index. Items are turned into vectors and kept around, and “nearby ones” are returned by cosine similarity — a lightweight build. Embeddings and semantic search themselves are covered in Raising RAG retrieval accuracy, measurement-driven. This article is about one trap that quietly hides in that setup.
The trap shows up when catalogs of different kinds coexist in the same index. When video works (VOD, DVD) and music CDs are in one index, an attempt to surface “things close to this video work” comes back with music CDs mixed in. To the embedding, the two are “semantically close.” But from the user’s side, they were looking for video and got handed something else.
What happens
In this setup, one single index has three exits.
┌─▶ ① Semantic search: a query string ────────▶ nearby items
one embedding index ───┼─▶ ② Related recs: a given item ─────────────▶ nearby items
(all items coexist) └─▶ ③ Personalization: centroid of history ───▶ nearby items
│
└─ heterogeneous catalogs (video / music …) coexist in one index
│
└─▶ another category slips into the top just by being "close"
│
(happens at all three exits ①②③)
The three features share the same index, so the crosstalk happens the same way at all three exits. Even if you notice it on one screen, the place to fix is not just one.
Why it happens
What embedding similarity measures is only “semantic closeness.” Business constraints — which category an item belongs to, whether it may be shown to this user, whether they have already seen it — are not inside the vector. So if you take from the top in order of similarity, things that are semantically near but must not be shown mix right in.
Put differently: “semantically close” does not equal “OK to show.” The embedding does not know the boundaries of the business. Those boundaries are knowledge that lives outside the embedding.
How to plug it
The countermeasure comes as a set of two. Either one alone does not work.
1. Apply the business filter at each exit
Place a filter that keeps only the categories you may show at each of the exits — search, related, and personalization. Whether you narrow the results or exclude on the query side does not matter. What matters is not settling it in one place just because the index is shared. The trap is at three exits, so the filter is needed in three places.
2. If you filter, widen the retrieval window
A filter reduces the count. Even if the goal is 10 items, retrieving just the top 10 first and then filtering leaves you short after exclusion. So retrieve several times the goal, anticipating what will disappear, then filter. For personalization you also exclude “already seen,” so retrieve extra by the size of the history on top of the goal.
[naive] take just top N ─▶ filter ─▶ falls short of the goal / another category remains
▲
(did not anticipate what disappears) ┘
[fix] take several×goal + history count ─▶ business filter ─▶ exclude already-seen ─▶ top goal count
│
retrieve wide so the goal count survives even if the top vanishes
And when the embedding cannot produce enough items, quietly fall back to another approach like popularity order (graceful degrade). Quality drops, but that is better than a result of zero and an empty screen.
What to place between “close” and “relevant”
“Semantically close” is not “OK to show.” Embedding similarity measures only semantic closeness. Category boundaries, and whether something may be shown, are business knowledge outside the embedding. Do not wave things through just because they resemble each other.
Apply the business filter at each exit, even with a shared index. The entrance (the index) may be one, but the exits are several. Do not feel fixed after fixing one place. Features that share the same index share the same trap. If you step on it at one, inspect the rest.
If you filter, widen the retrieval. On the premise that exclusion reduces the count, retrieve several times the goal plus the count you plan to exclude, then filter. Design the retrieval window so that “the goal count survives even if the top vanishes.”
Embeddings are powerful, but too honest. They bring back everything that resembles. Teaching them the business’s circumstances is the job of your own filter, placed outside the embedding.
Related articles
- Another pitfall stepped on the operations side of the same embedding / vector DB (breaking the index twice) is in Breaking my memory vector DB twice.
- Raising search accuracy by numbers rather than by feel is in Raising RAG retrieval accuracy, measurement-driven.
- Since “semantically close” is not “relevant,” sifting what you retrieve by relevance before handing it over is in Placing a small judge that tells “is this relevant?” into RAG.
- Implementing this semantic search without adding a dedicated vector DB in the first place is in Searching by “meaning” rather than “letters” — implementing semantic search without adding a dedicated vector DB.
- Using the same index for “recommendations,” recommending by the centroid of history while dropping already-seen items and CDs at the exit, is in Building recommendations from the “centroid vector” of viewing history.
- The overall picture of this platform’s customer app and an article map is in I’m building an end-user rental app.