Two layers so a transfer-completed notification is never sent twice — Idempotent Consumer, and the key granularity that nearly killed the legitimate second message
Introduction
I run a notification service on my homemade banking-integration system. It’s the part that receives events and sends things like “your transfer is complete” as email or push notifications.
In a distributed system, the same event arriving twice is a routine occurrence. Do nothing about it, and the user gets the same notification twice. This article is about how I stopped that with a double defense, and about a pitfall I nearly stepped into along the way in that design — the granularity of the dedup key.
The consistency story on the sending side — never losing money even if something dies mid-transfer across accounts (reliable delivery from the sender) — is written up in Keeping money from vanishing when a service dies mid-transfer. This article is the flip side of that: not processing the same event twice on the receiving side.
Why can a notification arrive twice?
The notification service receives events (like “transfer complete”) that other services emit, via a message platform (Kafka). What matters here is the delivery guarantee called at-least-once.
The message platform doesn’t guarantee “exactly once” — it guarantees “at least once, no matter what.” Put the other way around, it permits the same event to arrive twice because of network retries or rebalancing. Never dropping a single message comes at the cost of occasional duplicates — that’s at-least-once.
Sending service ──"transfer complete(event_id=A)"──▶ Message platform ──▶ Notification service
│
└─ a retry delivers the same event_id=A again
│
(if handled naively) two notifications go out
So the receiver has to remember for itself, “I’ve already processed this event_id,” and reject the second one. A receiver that behaves so that “however many times it receives something, the result is a single result’s worth” is called an Idempotent Consumer. Idempotent means the property that repeating the same operation any number of times doesn’t change the result.
The double defense of a receipt ledger (Inbox) and a DB constraint
The standard pattern for not processing twice is the Inbox pattern (a ledger recording received events; also called Transactional Inbox). You record the ID of every event you’ve processed in a dedicated table, and if the same ID comes in again, you treat it as “already processed” and discard it.
The key is that business processing and the Inbox record must always happen in the same transaction. Do them separately, and the moment “business processing finished but died before the Inbox record” occurs, the next retry lets the duplicate processing through. In a single transaction, there are only two outcomes: both succeed, or both fail.
event_id=A received
│
▼
Is A already in the Inbox?
├─ Yes ─▶ second time → discard (don't send again)
└─ No ─▶ ┌─ same transaction ──────────────────┐
│ ① create the notification │
│ ② record A in the Inbox │
└─ both succeed or both fail ───────────┘
Furthermore, I didn’t rely on that one layer alone. As the last line of defense in case something slips past the Inbox, I also put a constraint directly in the DB on the notification table itself: “no duplicate notifications from the same event.” Even if the application’s check leaks through in one place, the DB catches it in the end — defense in depth.
-- The notification table itself. A UNIQUE constraint on event_id as the last line of defense if something slips past the Inbox
CONSTRAINT notifications_event_id_uk UNIQUE (event_id)
Up to here, it’s all straightforward. The problem was in how the key for this last line of defense was chosen.
The pitfall: one event correctly turns into two messages
Notifications have a design that expands one event into multiple channels. Sending “transfer complete” by both email and push — this is called fan-out (expanding one thing into several). Here, a notification is “one record = one delivery to one channel,” so the same event_id correctly produces two rows: one for email, one for push.
Now, recall that UNIQUE (event_id) from before. It’s a constraint saying “only one notification row per event_id.” The moment the second row (for push) is INSERTed by fan-out, it gets rejected by the very constraint I set up myself, because the event_id is the same. The key meant to prevent duplicates kills a legitimate second message.
event_id=A (transfer complete)
├─ notification for email (event_id=A, channel=EMAIL) … INSERT succeeds
└─ notification for push (event_id=A, channel=PUSH) … ✗ fails on UNIQUE(event_id) violation
└─ rejected even though it's not a duplicate
This doesn’t surface at all at the stage before fan-out is implemented (a simplified version where one event maps to one channel). It only bares its fangs the moment multi-channel is added — it was a time bomb. It was a relief to catch it during a design review, before it surfaced in the implementation.
The fix is to change the granularity of the key. Drop the unit of dedup judgment from “per event” to “per event × channel.”
-- Fix: from event_id alone → a composite key of (event_id, channel)
ALTER TABLE notifications DROP CONSTRAINT notifications_event_id_uk;
ALTER TABLE notifications ADD CONSTRAINT notifications_event_id_channel_uk
UNIQUE (event_id, channel);
With this, “a duplicate of the same event and same channel” is still rejected by the DB just as before, and “the same event but a different channel” (fan-out) is now allowed as a separate row. It prevents only the duplicates you want to prevent, and lets a legitimate second message through. The intent of the double defense stays exactly the same — only the granularity of the key was corrected.
Want to prevent: a second occurrence of the same event × same channel → reject (duplicate)
Want to allow: the same event expanded to a different channel → allow (fan-out)
│
UNIQUE(event_id) … rejects both together (kills even a legitimate second message)
UNIQUE(event_id, channel) … rejects only the former, allows the latter ◀ this is the right granularity
Idempotency isn’t “exactly once” — it’s “once per what unit”
What sank in for me while building this is that the real difficulty of idempotency isn’t “not doing it twice” itself. The hard part is deciding what counts as “the same” — choosing the unit of dedup judgment (the granularity of the key) to match the reality of the business.
- If the granularity is too coarse (event_id alone), fan-out — which should count as different things — gets misjudged as “the same,” and even legitimate processing gets crushed.
- If the granularity is too fine, the duplicate you actually wanted to prevent slips through.
“Process an event exactly once” is correct. But once you translate it into an implementation, you have to decide “once, of what part of the event.” This time it was (event_id, channel). It’ll be a different granularity in a different situation. A dedup key isn’t something you pick on autopilot as the primary key or the event ID — it’s the very boundary line between the duplicate you want to prevent and the legitimate repetition you want to allow.
And one more thing. This misalignment of the boundary line, once fan-out was implemented, would have shown up as a production incident of “why does the second message fail for some reason.” Catching it before that, at the stage of re-reading the design, was possible because I asked not “does it work now” but “what breaks if I add this next”. The more something works, the easier it is to forget to ask that question.
Related articles
- The Outbox story on the sending side — reliably tying state changes and event delivery together without dropping anything — is written up in Keeping money from vanishing when a service dies mid-transfer — protecting consistency across accounts with Saga and Outbox. The Inbox in this article is its counterpart on the receiving side.