Database Change Data Capture

CDC: polling a database for changes (SELECT WHERE updated_at > last_poll) misses deletes, has latency, and adds load. Log-based CDC reads the database's write-ahead log (WAL) or binary log, which captures every change as an ordered, low-latency event stream without additional database load.

Debezium: for PostgreSQL, Debezium uses logical replication slots and the pgoutput plugin. For MySQL, it reads the binary log as a replica. Each change event includes the before and after image of the row plus metadata (table, operation type, transaction ID), enabling downstream consumers to reconstruct the full change history.

Outbox pattern: the dual-write problem — writing to both a database and a message broker — risks partial failures. The outbox pattern writes the event to an outbox table atomically with the business data. CDC then streams the outbox rows to Kafka. Consumers process and delete outbox records, guaranteeing no event is lost even if the broker is temporarily unavailable.

Replication slot: without it, PostgreSQL could vacuum away WAL segments that a slow CDC consumer has not yet read, causing data loss. The slot retains WAL until the consumer confirms (via LSN advance) that it has processed changes. This guarantees no changes are skipped — but slots that fall behind can cause disk growth.

Database events as integration: when an order is updated in the orders database, CDC emits an event. The inventory service consumes it to adjust stock; the analytics service writes to a data warehouse; the search service updates its index — all independently, all decoupled. Adding a new consumer requires no change to the source system.