Data integration patterns are proven, reusable designs for moving data between systems. Instead of designing every integration from scratch, teams pick the pattern that fits the job: copy data once (migration), push it from one system to many (broadcast), keep two systems in step (bi-directional sync), or combine data from several sources (aggregation). Naming the pattern up front makes integrations faster to build, easier to review, and less likely to fail in surprising ways.
This guide explains the core data integration patterns, when to use each, and how to choose between them. For how they fit into a wider architecture, see enterprise data integration.
The core data integration patterns
1. Migration
Move data from one system to another once, typically when replacing a system. Migration is about volume, data quality, and validation: extract, clean, transform, load, and reconcile. See data migration vs data integration.
2. Broadcast
Push data from one source to one or more targets, continuously or on a schedule, in one direction only. Example: product updates in the ERP sent to the ecommerce site, the warehouse system, and the marketplace feeds. The source is the single owner of the data.
3. Bi-directional sync
Keep the same records consistent in two systems that both allow edits. Example: contacts maintained in both the CRM and the support platform. It's powerful but needs rules for conflicts, such as which system wins for each field, and protection against update loops.
4. Correlation
Sync only the records that exist in both systems, matching them by a shared key. Example: update customer status in the billing system only for customers that also exist in the CRM, without creating new ones in either.
5. Aggregation
Combine data from several systems into one view or target. Example: a customer 360 record built from CRM, billing, support, and product usage data, or finance reporting that pulls from several ERPs.
6. Event-driven and streaming
React to business events or database changes as they happen. A system publishes an event (order placed, employee hired) and any number of consumers act on it; change data capture streams row-level changes from databases. This suits real-time needs and many consumers.
Choosing the right pattern
| Pattern | Direction | Timing | Source of truth | Use when |
|---|---|---|---|---|
| Migration | One-way, once | Project-based | Old system, then new | Replacing or consolidating systems |
| Broadcast | One-to-many | Real time or scheduled | One source | One system owns the data; others consume it |
| Bi-directional sync | Two-way | Near real time | Shared, with field rules | Two teams edit the same records in different tools |
| Correlation | Two-way, matched only | Scheduled or real time | Each system for its own records | Systems overlap but shouldn't create records in each other |
| Aggregation | Many-to-one | Scheduled or real time | Each source for its part | Reporting or a single combined view |
| Event-driven | One-to-many | Real time | The publishing system | Many consumers need to react quickly |
Patterns for data movement, not just direction
Two more choices shape every integration:
- Batch vs real time: batch moves data on a schedule and suits large volumes; real time moves each change as it happens. See real-time vs batch integration.
- ETL vs ELT: transform before loading into the target, or load raw data and transform it inside the warehouse. See ETL vs ELT.
Supporting patterns that make integrations reliable
- Idempotent writes: processing the same record twice changes nothing, so retries are safe.
- Upserts by key: create or update based on a stable identifier, avoiding duplicates.
- Dead-letter handling: records that fail validation are set aside for review instead of being dropped.
- Watermarks and checkpoints: track what was last processed, so a failed run resumes where it stopped.
- Canonical data model: map each system to a shared definition of key entities, so adding a system doesn't mean remapping every other one.
Frequently asked questions
What are data integration patterns?
They're reusable designs for moving and combining data between systems, such as migration, broadcast, bi-directional sync, correlation, aggregation, and event-driven integration.
What's the difference between broadcast and bi-directional sync?
Broadcast sends data one way from a single source to one or more targets. Bi-directional sync keeps records consistent in two systems that can both edit them, which needs conflict rules.
Which data integration pattern is best for real-time needs?
Event-driven integration and change data capture, because they process each change as it happens and can serve many consumers.
What is the aggregation pattern used for?
Combining data from several systems into one view or target, such as a customer 360 profile or consolidated financial reporting.
How do I avoid duplicates when integrating data?
Use upserts based on a stable key, make writes idempotent, and match records consistently across systems.
Pick the pattern, then the tool
Clear patterns make integrations predictable to build and run. To see these patterns working on Koodisi with your systems, book a demo.