~/ cat notes/cdc-debezium.md

CDC, Debezium & the transactional outbox.

How a committed database row becomes a Kafka event without the application ever performing an unsafe dual write — and why the dangerous failure mode is not a lost event but a full database disk.

WAL → logical decoding1 connector taskat-least-once end to endslot lag is a disk risk

00The 30-second model

say this first

“The application writes business state and an event-intent row in the same PostgreSQL transaction, so they commit together or not at all. PostgreSQL writes that to the WAL. Logical decoding turns WAL into row-level changes, a replication slot holds the reader's position, and Debezium streams from that slot into Kafka. The API never waits for Kafka — it waits for the Postgres commit and nothing else.”

The problem being solved is the dual write: commit to the database, then publish to Kafka. Those are two systems and they cannot commit together, so a crash in between loses the event with no record anywhere that it should have existed. CDC does not make the publish atomic — it moves the atomic part into the database, where a transaction already exists.

01High-level design

The API waits for the Postgres commit and nothing else. Everything right of the slot is asynchronous.

Component	Owns
Application	The business transaction, including the outbox insert.
PostgreSQL	ACID, WAL, and exposing logical changes over the replication protocol.
Publication	Which tables are eligible to appear in the stream.
Replication slot	The durable read position — and the WAL retention that comes with it.
Debezium	Consumes the stream, converts to change events, hands them to Connect.
Kafka Connect	Runs the task and persists source offsets.
Consumers	Business effects — and idempotency, because replay is guaranteed to happen.

02What PostgreSQL is actually doing

Four facts worth being exact about, because they are the most common follow-up questions.

WAL is not a table. It is binary segment files under pg_wal.
Debezium does not read those files. PostgreSQL decodes WAL itself and streams logical changes out over its native replication protocol — a long-lived TCP connection, not REST or a WebSocket.
It is not polling. In steady state there is no repeated SELECT over business tables. Debezium still uses ordinary SQL connections for metadata, schema discovery and snapshots.
Debezium is not in the write path. It is a replication consumer hanging off the side of the database, not a proxy between the app and Postgres.

wal_level = logical
max_replication_slots = <connectors + reserve>
max_wal_senders       = <logical + physical replication needs>

CREATE PUBLICATION app_cdc FOR TABLE public.outbox_events;

03The outbox transaction

BEGIN;
  INSERT INTO orders (id, customer_id, amount, status)
  VALUES ('order-123', 'customer-7', 500, 'CREATED');

  INSERT INTO outbox_events (event_id, aggregate_type, aggregate_id, event_type, payload)
  VALUES ('evt-987', 'Order', 'order-123', 'ORDER_CREATED', '{"order_id":"order-123"}');
COMMIT;

Two outcomes only: both rows commit, or neither does. Debezium then captures the committed outbox insert and publishes it. The event can be late, but it cannot be missing.

the distinction people miss

The outbox does not give you API idempotency. It guarantees that a committed order produces an event. It does nothing about a retried POST /orders creating two orders — that needs an application-level idempotency key or a unique business constraint.

Outbox CDC vs raw table CDC

Raw table CDC	Outbox CDC
Every row mutation is an event.	The app emits explicit domain intent.
Great for replication, search indexing, analytics.	Great for service-to-service business events.
Consumers infer meaning from columns.	Meaning is explicit: ORDER_CANCELLED.
Internal maintenance updates leak into the event stream.	Only deliberate events are routed.

Also: the outbox table is not allowed to grow forever. Retention or time partitioning, planned once events are safely captured — and be careful that the cleanup itself does not become a second event stream if DELETEs are being routed.

04Offsets, LSN and why duplicates exist

Three positions, not one. The gap between read and persisted is why duplicates exist.

An LSN is a position in the WAL — a source-log position, not a business event ID. The reason replay is unavoidable is that the connector can have read further than Kafka Connect has durably persisted. Crash in that gap and records near the boundary come back.

Ordering: the Postgres stream is ordered, but Kafka only guarantees order within a partition. Pick a record key matching the entity whose order matters — aggregate_id or order_id. Never claim global ordering across partitions.

05Lag, slots and the disk

A stalled connector does not break the API. It fills the database disk, quietly, hours later.

This is the answer that separates people who have run CDC from people who have read about it. A stalled connector does not page you with errors — the API keeps working perfectly. What happens is that the replication slot keeps holding WAL that PostgreSQL is not allowed to reclaim, and some hours later the primary database runs out of disk.

SELECT slot_name, active, restart_lsn, confirmed_flush_lsn
FROM pg_replication_slots;

SELECT * FROM pg_stat_replication;

06Scaling — and the one-task constraint

the trap

You cannot set tasks.max=20 and get twenty parallel WAL readers. A PostgreSQL connector uses one task consuming an ordered replication stream. Parallelism lives downstream, in Kafka partitions and consumer groups.

Find the actual bottleneck first: WAL generation, decoding, connector queue, producer, broker, or consumers.
Scale Kafka partitions and consumer groups independently of the single source task.
Separate databases get separate connectors and separate slots.
If one connector genuinely cannot keep up, decompose the source — do not invent parallel WAL readers.

catch-up headroom matters more than steady-state throughput

  normal load        50k events/s
  connector capacity 60k events/s
  headroom           10k events/s

  a 30-minute outage = ~90M events of backlog
  at +10k/s of headroom → ~2.5 hours to drain

07Failure matrix

Failure	What happens	Response
App crashes before COMMIT	Neither order nor event exists	Client retry + API idempotency
App crashes after COMMIT	Both exist; response lost	Debezium captures it later; API still needs an idempotency key
Debezium crashes	DB fine, slot retains WAL	Restart; watch lag and disk; expect boundary replay
Kafka unavailable	No forward progress, backlog grows	Protect DB disk first, alert early, plan catch-up
Consumer crashes after side effect	Kafka redelivers	Dedupe on event_id
Postgres primary fails over	Slot continuity is version/platform dependent	Design slot failover deliberately, not as an afterthought
Schema-incompatible event	Connector or consumer loops on failure	Schema governance, DLQ policy, deploy compatibility before migrating

08Alternatives

Approach	Good for	Cost
App dual write	Simple, low delay	An unfixable failure window between two commits
Outbox + polling publisher	No Debezium dependency; simpler ops	Polling load, row claiming, batching and cleanup complexity
Outbox + Debezium	Atomic intent, log-based capture, decoupled API	Connect, slots, WAL retention, lag and failover to operate
Raw table CDC	Replication, indexing, analytics	DB changes are not domain events; couples consumers to schema
Managed CDC	Less operational burden	Vendor constraints and semantics you must verify yourself

When not to reach for Debezium: a small system where a polling outbox is simpler; a platform that cannot support logical replication safely; a workflow that genuinely needs synchronous propagation — in which case redesign the workflow rather than pretend CDC is synchronous.

09Question drill

Why not just publish to Kafka after the DB commit?

Because those are two systems with no shared transaction. The process can commit the order and die before publishing, and nothing anywhere records that an event was owed. The outbox moves the atomic part inside the database, where atomicity already exists.

Is WAL a table? Does Debezium read the files?

WAL is binary segment files in pg_wal, not a table. Debezium does not open them — PostgreSQL decodes WAL itself and streams logical changes over its replication protocol on a long-lived TCP connection.

Does Debezium continuously run SELECT queries?

Not in steady state — that is the point of log-based CDC. It does use ordinary SQL connections for schema discovery, metadata and snapshots, and an initial snapshot of a large table is a real database workload.

What is a publication? What is a replication slot?

A publication is the Postgres-side answer to “which tables may appear in this stream?”. A replication slot is a durable position in that stream — and, operationally, the thing that makes Postgres retain WAL the consumer has not yet confirmed.

Why can't I set tasks.max=20 for a Postgres connector?

Because it uses one task against one ordered replication stream. Twenty readers of one WAL stream would have no coherent ordering or offset story. Scale downstream instead — Kafka partitions and consumer groups.

What happens if Debezium is down for three days?

The application keeps working, which is exactly what makes this dangerous. The slot retains every WAL segment since the last confirmed position, pg_wal grows, and the risk is the primary database filling its disk. Alert on slot retained bytes and DB disk, not just connector state.

How do duplicates happen if Postgres is transactional?

The database is fine. The duplication comes from the gap between what the connector has read and what Kafka Connect has durably persisted as its source offset. Crash in that window and boundary records replay — so consumers dedupe on event_id.

How do you preserve ordering per order or per customer?

Key the Kafka record on the entity that needs ordering — aggregate_id. Kafka orders within a partition only; there is no global order, and claiming otherwise is a fast way to lose credibility.

Outbox guarantees the event. Does it stop duplicate orders?

No, and this is the distinction worth stating unprompted. The outbox guarantees a committed order produces an event. A retried POST /orders that creates two orders is a different problem, solved by an idempotency key or a unique business constraint.

How would a Postgres failover affect the slot?

It depends on version and platform — replication slot failover is not automatic everywhere. If slots do not survive, the connector can resume from a position that no longer exists, which means a resnapshot or a gap. Treat slot continuity as part of the failover design.

How do you bootstrap a 2 TB existing table?

Deliberately. A snapshot reads table contents and creates real I/O, CPU and long-transaction pressure. Plan snapshot mode, fetch sizes, incremental snapshotting, and whether it runs against a replica. Note that a fresh outbox table usually needs no historical snapshot at all.

What metrics warn you before the database disk fills?

Slot retained WAL bytes — the leading indicator.
Database disk free.
Connector milliseconds-behind-source and connector state.
Kafka produce errors and consumer lag downstream.

10What not to say

Do not say	Say instead
“Debezium reads the WAL files.”	Postgres decodes WAL and streams logical changes; Debezium is a replication client.
“CDC gives exactly-once.”	At-least-once end to end; consumers dedupe on event_id.
“The outbox solves duplicate orders.”	It solves the lost event. Duplicate orders need API-level idempotency.
“More tasks means more throughput.”	One task per Postgres connector; scale downstream.
“If Debezium dies we just restart it.”	True, but the clock is the database disk — that is what you are racing.

11Cheat sheet

Term	One line
CDC	Capture committed database changes and stream them downstream.
WAL	Postgres binary write-ahead log, stored as segment files.
Logical decoding	Turns storage-level WAL into logical row changes.
pgoutput	The built-in logical replication output plugin.
Publication	Which tables may appear in the logical stream.
Replication slot	Durable stream position — and the cause of WAL retention.
LSN	A position in the WAL. Not a business event ID.
Kafka Connect	Runs the connector task and persists source offsets.
Outbox	Commit business state and event intent atomically; publish async.
Slot lag	How far behind the consumer is — measured in retained WAL bytes.

Postgres → WAL → slot → Debezium → Kafka. The outbox is what makes the first arrow safe.

← all notes