The support ticket says the customer was charged for an order the system claims was never placed. Both things are true. The payments service recorded the charge; the orders service never heard about it. Somewhere between the two, a message got lost, and for three days nobody knew, because nothing errored. The systems didn’t crash. They just quietly stopped agreeing with each other.
This is the failure mode that defines event-driven microservices, and it’s a business problem before it’s a technical one. When you split one system into many, you trade a single source of truth for a set of services that each hold part of it and have to stay in agreement. When they drift, the customer finds out before you do, and the cost shows up as refunds, support load, and the slow erosion of trust in your own data. Change data capture is one of the more reliable ways to stop that drift. Worth understanding why, without needing a distributed-systems degree to follow it.
Why services fall out of sync in the first place
A service that changes something important usually has to do two things at once: record the change in its own database, and tell the rest of the world it happened. Record the payment, and publish a “payment received” event so the orders service, the email service, and the analytics pipeline all react.
The trap is that those are two separate systems: your database and your message broker. You can write to one and fail to reach the other. Save the payment, then crash before the event publishes, and the payment exists but no one downstream knows. Publish the event, then fail before the database commits, and now the rest of the company believes in a payment that isn’t real. Either way, two systems that should agree now disagree, and neither one is obviously wrong.
This is the dual-write problem, and it’s not a bug you can code your way around with more careful ordering. As long as “save it” and “announce it” are two independent operations, some fraction of the time one will happen without the other. At low volume you might never notice. At scale, “some fraction of the time” is a steady trickle of the exact tickets that opened this post.
The core problem isn’t that services fail. It’s that a save and an announcement can succeed independently, and a customer lives in the gap.
The fix: make the announcement part of the save
The reliable pattern turns two operations into one. Instead of writing to the database and then publishing an event, the service writes both to the database, in the same transaction: the actual change, and a record of the event it needs to send, dropped into an outbox table right next to the data. Because it’s one transaction, there’s no gap. Either both land or neither does. The database can’t end up holding a payment with no corresponding event, or an event with no payment.
That’s the transactional outbox pattern, and it’s where change data capture comes in. Once the event is sitting in a table, something has to notice it and actually send it. That something is CDC: it watches the outbox for new rows and relays each one to the broker as it appears. The service’s only job was to write locally and correctly. The job of reliably telling everyone else becomes CDC’s, running as a separate, observable step rather than a risky second write bolted onto the request.
The result is a system where a change and its announcement can’t diverge, because they were never two decisions. The service commits once. The event follows, guaranteed, even if the service crashes the moment after committing. For a business that runs on services agreeing about money, orders, and customer state, that guarantee is the whole point.
The outbox pattern doesn’t make services fail less. It makes their failures stop lying to your customers.
Order still matters, and so does seeing what happened
Two things decide whether this holds up in practice, and both are worth a non-engineer’s attention because both surface as customer-visible problems.
The first is order. Events often have to arrive in the sequence they happened. “Order created” then “order shipped” makes sense; the reverse describes a package that shipped before anyone ordered it. Keeping order sounds automatic and isn’t, especially once you scale a stream across parallel channels for throughput and events can overtake each other. Getting this wrong doesn’t throw an error. It produces a customer record that reads like nonsense, and a support agent who can’t explain it.
The second is visibility. When two services do disagree — and eventually one will — the only question that matters is how fast you can find out why. If answering “did this event send, and did it arrive” means logging into three services and correlating timestamps by hand, every incident is a half-day. If each event is a trace you can follow end to end, from the outbox write to the far-side handler, it’s a query. This is why teams that take event-driven architecture seriously insist their pipelines emit telemetry in a standard, queryable format rather than scattered logs. The pattern keeps services in sync; observability is how you prove they’re in sync, and how you recover fast when they briefly aren’t.
- Order preserved keeps customer records coherent instead of contradictory.
- End-to-end tracing turns a multi-service investigation into a single query.
- Replay lets you re-send events after an outage instead of reconstructing them by hand.
None of these is optional at scale. They’re the difference between an event-driven estate that’s an asset and one that generates a steady background hum of “why don’t these two systems match.”
The part teams underestimate
Any team can implement the outbox pattern for one service in an afternoon. The cost isn’t the first one. It’s that every service that publishes events needs the same machinery — the outbox, the relay, the ordering guarantees, the tracing, the replay — and hand-building it per service means every team reinvents the same plumbing slightly differently. Then you’re not operating one reliable pattern. You’re operating a dozen near-identical ones, each with its own quirks, and the reconciliation only one person understands.
This is the argument for treating the relay and everything around it as a composable pattern on a platform you already run, rather than bespoke code in every service. The service keeps its simple job: write locally, drop an event in the outbox. The reliable delivery, the ordering, the dead-letter handling, the replay, and the tracing become properties of the platform, configured once and applied everywhere, instead of a distributed-systems problem each team solves alone on a deadline.
Reliability that’s rebuilt per service isn’t reliability. It’s a dozen chances to build it slightly wrong.
Where the pattern doesn’t pay off
Be honest about scope. If your services are few and low-volume, and an occasional manual reconciliation is genuinely cheaper than the machinery, the outbox pattern can be more ceremony than the problem deserves. And at the far extreme — very high-throughput streaming where sub-second latency is the product — you’re into purpose-built streaming infrastructure that you staff deliberately, not a pattern you compose. The outbox-plus-CDC approach is for the wide middle: an event-driven estate where services genuinely have to agree, the volume is real, and silent drift has a real cost, but running a dedicated streaming platform per team is a heavier answer than the question needs.
Keeping services honest
Event-driven microservices give you independent teams shipping independently, which is why you split the system in the first place. The price is that those services have to keep agreeing about the facts, and the naive way of announcing changes lets them silently disagree. The outbox pattern, relayed by change data capture, closes that gap: a change and its announcement become one committed decision, and the reliable delivery becomes an observable step you can watch rather than a second write you hope succeeds.
The part worth carrying into your own planning is that this reliability isn’t something you build once and forget. Every service that publishes events needs the same guarantees, and the real choice is whether they come from shared, composable machinery or from a dozen slightly different hand-rolled versions. Services that agree about money, orders, and customer state are a business asset. Services that quietly disagree are a support queue. The outbox pattern is one of the most dependable ways to stay on the right side of that line.