So money doesn't vanish if a service goes down mid-transfer — protecting distributed account consistency with Saga and Outbox
Introduction
I’m implementing transfers between accounts in a homemade bank-style system. I wrote about the overall shape of the system in What’s Actually Running in My Homemade Bank System.
A transfer, boiled down, is two moves: subtract from the source, add to the destination. This is the story of how I made sure those two moves never left money “disappearing or duplicating,” no matter how much the system fell over in the middle. Written naively, this is a spot that always has a hole in it — and I closed that hole with Saga and Outbox.
The moment you split into services, transfers become dangerous
In the first version, transfers were self-contained inside the accounts module. “Subtract” and “add” happened in a single transaction on the same database. So even if something crashed midway, both were rolled back together as if nothing happened. Safe.
But the moment accounts were carved out into an independent service, everything changed. “Subtract” and “add” split into separate services and separate transactions, and if something dies in between — it’s possible for money to be subtracted from the source but never arrive at the destination. The money vanishes into thin air.
Transfer = subtract from source + add to destination
【Before splitting: one transaction, one DB】
subtract + add ─▶ both succeed or both undone (safe)
【After splitting: services separated】
Payment service ─▶ subtract at account (source) ─▶ add at account (destination)
│ │
if this dies… it vanishes from the source and never reaches the destination ×
Not two-phase commit — Saga and compensation
There’s a classical way to bundle a distributed transaction: two-phase commit (2PC). This is a method of finalizing multiple databases all at once, “on the count of three.” First you ask everyone “ready?”, and only once everyone answers “OK” do you order “commit!” all together. It’s certain, but it locks and stalls everyone for the duration, so it’s slow and heavy, and if even one participant goes down or fails to respond, the whole thing freezes. It’s a poor fit for microservices.
Instead, I use Saga. Drop the “on the count of three,” and finalize each step, one database at a time, moving forward one commit at a time. If it fails partway through, undo the steps already completed, one at a time, with the reverse operation. This reverse operation is called compensation. The compensation for “subtract from the source” is “put it back to the source (add it back)” — not a rollback, but squaring the books with the opposite operation. So if “add” fails after “subtract” has already gone through, compensation cancels out “subtract.”
① Subtract from source (local TX) — succeeds
② Add to destination
├─ success ─────────▶ transfer complete
└─ failure ─▶ ③ compensate: put it back to source (undoing ①)
A timeout is not a “failure”
Here’s a hole that a naive implementation always falls into: when do you trigger compensation?
“If ‘add’ fails, compensate by putting it back” — sounds right. But “failure” comes in two kinds.
- Clear failure (a 4xx / 5xx came back): the other side definitely did not process it. It’s fine to reverse.
- Timeout (no response came back at all): this is not a failure. It’s an unknown outcome. The other side might actually have succeeded.
If you decide a timeout is a “failure” and fire compensation (put it back to the source), and it turns out the other side had actually succeeded, the destination got the addition and the source also gets it put back — the money is duplicated. It breaks in the opposite direction.
try {
accounts.deposit(toAccount, amount, idemKey); // add to the destination
} catch (AccountsClientException e) { // 4xx/5xx = clear failure
compensate(fromAccount, amount); // put it back to source (compensation is fine here)
} catch (AccountsUnavailableException e) { // timeout = unknown outcome
// Do not simply reverse it. Query the actual result and hold until it's confirmed.
markPendingForReconciliation(transferId);
}
Split “clear failure” from “unknown outcome” and decide whether to trigger compensation based on that. Lump them together and you break in both directions — money disappearing, and money duplicating.
Tying state changes and events together reliably with Outbox
There’s one more hole that distribution commonly falls into. How do you have “update the balance” and “tell other services ‘transfer complete’” both happen without dropping either one?
Done naively, you update the balance, then separately publish an event. But if something dies between those two steps, the balance moves while the event alone vanishes (no notification goes out, downstream processing never fires).
So I use Outbox. This isn’t some local table I invented myself — it’s a well-established pattern widely used in distributed systems, formally called Transactional Outbox.
Here’s how it works. Instead of firing the event straight out, you write a row into a dedicated table (outbox) for the event you want to send — “transfer complete” — inside the same single transaction as the balance update. This way, the DB update and the event record succeed together and fail together (they’re tied atomically). Then a separate process (the Relay) reads the “not yet sent” rows from that outbox table and delivers them reliably, marking each as sent once it succeeds. Even if something dies mid-way, the row stays in the outbox, so nothing gets dropped.
【Dangerous】update balance → (separately) publish event … if it dies in between, only the event vanishes
【Outbox】same transaction: update balance + write "transfer complete" to the outbox
→ a separate process (Relay) reads the outbox and delivers reliably (nothing dropped)
With this, transfers — even split across services, even if something goes down mid-way — leave money neither vanishing nor duplicating, and never drop a notification. Even distributed, consistency can be protected by design. Instead of binding everything tightly with two-phase commit, I moved forward with Saga, rolled back with compensation, and delivered reliably with Outbox. Never confuse a timeout with a failure. Those three things became the backbone of a distributed transfer.
Related articles
- The story of actually calling this transfer from the outside (payment on my homemade platform) and moving real money is in Connecting My Homemade App to My Homemade Bank Until Real Money Actually Moves — Adding Account Transfer to Payment, Confirmed End-to-End.
- A full tour of the system’s structure, screens, and backbone code is in What’s Actually Running in My Homemade Bank System.
- The paired mechanism on the receiving side, so events reliably sent via Outbox don’t get processed twice (Inbox, and the pitfall of idempotency-key granularity), is in Two layers so a transfer-completed notification is never sent twice — Idempotent Consumer, and the key granularity that nearly killed the legitimate second message.