The 2 a.m. page doesn’t say “API integration failed.” It says something broke downstream. An order didn’t process. A provisioning event didn’t fire. A webhook arrived, disappeared, and took no record with it. You trace it back through three systems and eventually locate something that called an external API, got a 504, and gave up. No retry logic. No dead-letter queue. No trace. Just a gap in the data and a customer who noticed before you did.
That’s the production reality of API integration. The theory is clean: services talk to each other through defined contracts. The practice is messier, because the diagram never shows timeouts, auth expiry, schema drift, or what happens when the downstream system is mid-deploy.
This post isn’t about what an API is — for the fuller definition of what API integration actually involves and where it fails, start there. This one is about why API integration keeps failing in production, what an honest model of it looks like, and what you actually have to build to make it hold.
The happy path is not the system
The definition-first version of API integration looks like this: Service A makes an HTTP request to Service B. Service B processes it and returns a response. Done.
That model is useful for a whiteboard and useless for a production incident. What the diagram omits is everything that happens when the request doesn’t complete cleanly — and in a distributed system under real load, that gap is where most of your actual integration work lives.
The production model of API integration is the failure-handling model, not the success path.
Start with the request itself. APIs fail for reasons that are not your code’s fault: the downstream service is rebooting, a load balancer is returning 503s, a rate limit was hit on the previous request and the window hasn’t reset. The first decision in any real integration is what to do when the call doesn’t return what you expected. Retry with backoff? Dead-letter and alert? Surface the failure to the calling service and let it handle it? Each choice has tradeoffs, and the wrong default — like a simple immediate retry on a service that’s already rate-limiting you — makes the failure worse.
Then there’s authentication. OAuth tokens expire. Service accounts get rotated. API keys get revoked in a security sweep. An integration that worked last Tuesday can fail this Tuesday because a credential hit its TTL and the rotation wasn’t threaded back through the integration layer. This is not an edge case. It’s one of the most common root causes of “the integration was fine and then it wasn’t,” and it’s entirely preventable if the credential lifecycle is managed as infrastructure rather than as a thing your integration assumes never changes.
Schema drift is slower and harder to catch. External APIs evolve. A field gets renamed, a nested object gets flattened, a date format changes across a version boundary. None of this breaks the HTTP call — the request still returns 200. What breaks is the data mapping layer downstream, usually silently, and usually in a way that produces records that look right until someone opens one and notices a null where there should be a value. By the time you catch it, the window of bad data is weeks wide.
What “integration” actually means in a distributed system
The vocabulary matters here, because how you name the thing shapes how you think about its failure modes.
An API call is a point-in-time synchronous action. One service asks another for something; the answer comes back in the same request or it doesn’t. This works well when the caller can wait, can retry, and doesn’t need the result to be guaranteed.
Event-driven integration is different: something happens, a message is published, and one or more consumers process it asynchronously. The coupling is looser — the producer doesn’t need to know who’s consuming, or whether they’re available right now. But you trade synchronous simplicity for a different class of problems: ordering, deduplication, idempotency, consumer lag, poison messages that keep failing and blocking the queue.
Most real integration layers use both patterns. Synchronous calls for things that need immediate answers, event-driven pipelines for things that can be processed offline. The integration logic you build is the layer that knows which pattern applies, routes accordingly, handles the failure modes for each, and provides some unified way to observe what’s actually happening.
The integration layer isn’t glue. It’s a system — with its own failure modes, its own operational surface, and its own debt if you don’t treat it like one.
This is where the gap between “API integration” as a category and what engineers actually build becomes important. The category framing suggests a solved problem: APIs are well-documented, the HTTP protocol is well-understood, your language has a client library. The production reality is that the interesting work isn’t the call — it’s everything that runs around the call to make it reliable, observable, and maintainable as the systems on both ends keep changing.
Why integration tooling keeps disappointing engineering teams
Most integration tooling was built around the happy-path model.
The early workflow automation platforms optimized for speed of wiring: drag a source, drop a destination, configure the mapping, done. That model works for low-stakes automations where a missed execution is an inconvenience. It doesn’t hold for production integrations where a missed event means a customer isn’t provisioned, a billing record is missing, or a compliance event wasn’t logged.
The specific failure isn’t that these tools can’t handle the error cases — most can, with enough configuration. The failure is that the defaults aren’t built for production. Retry logic is optional. Dead-letter queues are an advanced feature. Observability is a separate add-on or a UI-only audit log that doesn’t export into your existing monitoring stack. Schema validation is something you bolt on after the first bad-data incident.
When the defaults are built for the easy path, engineering teams spend most of their time adding the production scaffolding that should have been there from the start.
The deeper problem is operational ownership. An integration built in a low-code recipe editor lives in the tool’s database. When it breaks in production, you investigate in the tool’s UI — which shows you what the tool decided to log, in the format the tool picked, with the retention window the tool chose. You can’t query it with the same tools you use for the rest of production. You can’t put it in a runbook and reference the span ID. You can’t diff what changed between Tuesday and Wednesday.
Integration debt accrues the same way technical debt does: every shortcut that made the integration faster to build makes it harder to operate. The difference is that integration debt often sits in a different tool than the rest of your production stack, so it’s invisible to the normal debt-tracking mechanisms.
What a production-grade integration model looks like
If you’re building or evaluating an integration layer for production workloads, the questions that matter aren’t about connectors or the connector catalog. They’re operational.
Start with failure handling. When a transient upstream failure hits at 3 a.m. and nobody’s watching, what does the platform do? Retry with backoff? Dead-letter the event with full payload capture so you can replay it later? Or silently drop it and leave a gap in your data? Most platforms can be configured to do the right thing — the question is what happens with zero configuration.
Telemetry is where the gap between “integration tool” and “production infrastructure” shows up most clearly. Observability trapped in a vendor console is not operational observability. It’s a second investigation surface with a different login, different query syntax, and different retention than everything else you monitor. The integration layer should emit OpenTelemetry traces that land in your existing stack, so an integration failure is just another incident, debugged by whoever’s on-call with the tools they already know.
Credentials are a slower failure mode but just as common. API keys and OAuth tokens have lifecycles. An integration layer that treats credentials as one-time setup will eventually break on a security sweep, a rotation policy, or a token expiry nobody tracked. Rotation should be a vault operation that propagates to consuming workflows, not a deployment.
The change model is where the most integration debt quietly accumulates. If the only way to update a workflow is to log into the editor and click around, there’s no diff, no review, and no rollback beyond hoping the tool kept a backup. The change process should look like the rest of your software delivery: a versioned artifact, a branch, a review, a deploy.
The last variable is the pricing model, and it matters operationally. Usage-metered platforms charge per execution, which means retries, test runs, and loop iterations all count against the meter. When an upstream service degrades and your integration starts retrying aggressively, the bill climbs at exactly the moment you’re managing an incident. That’s the wrong incentive structure for production infrastructure.
The billing model should be indifferent to how hard your integration is working.
The billing model should be indifferent to how hard your integration is working.
Koodisi is built on feature-tiered pricing — no per-execution billing, no overage invoices — with OpenTelemetry traces on every tier from the free Community plan, real Git versioning with in-platform branching (PRs run through your GitHub or GitLab flow), and dead-letter capture with in-platform replay. The longer argument for why owning your integration layer matters, and what that actually looks like in practice, is in why unobservable integration stacks are the real hidden bill.
If you’re running production integrations and want to see what the operational surface looks like — traces, retry handling, dead-letter queues, credential management — the Community tier is free, no credit card required, and includes 1,000 executions per month. Start at koodisi.com.