Outbox vs Change Data Capture for Reliable Publishing

Both patterns solve the same problem: an event must be published if and only if a transaction commits, and no distributed transaction is available to make that true. Both solve it the same way — write the event into the database inside the transaction, then move it out afterwards. They differ entirely in how it gets moved. This decision guide sits under transaction scoping & atomic operations.

Prerequisites. A working transactional outbox, or at least an understanding of why publishing inside a transaction is unsafe.

---

Step 1 — What both guarantee

Neither is exactly-once. The polled publisher can send and crash before marking the row; the CDC connector can stream and crash before committing its offset. In both cases the event is published again on recovery, which is why the processed-events table on the consumer side is not optional in either design.

---

Step 2 — The polled outbox

CREATE TABLE outbox (
    id            bigserial PRIMARY KEY,
    event_id      text        NOT NULL UNIQUE,   -- ULID, the consumer's dedup key
    aggregate_id  text        NOT NULL,
    event_type    text        NOT NULL,
    payload       jsonb       NOT NULL,
    created_at    timestamptz NOT NULL DEFAULT now(),
    published_at  timestamptz
);
-- Partial index: the publisher only ever scans unpublished rows, and the index
-- shrinks to nothing as they are marked.
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

def publish_batch(conn, producer, batch=500) -> int:
    with conn.transaction():
        rows = conn.execute("""
            SELECT id, event_id, aggregate_id, event_type, payload
              FROM outbox
             WHERE published_at IS NULL
             ORDER BY id
             LIMIT %s
               FOR UPDATE SKIP LOCKED        -- lets N publishers run concurrently
        """, (batch,)).fetchall()

        for r in rows:
            producer.send("ledger.events", key=r.aggregate_id.encode(),
                          value=json.dumps(r.payload).encode(),
                          headers=[("event_id", r.event_id.encode())])
        producer.flush()                      # all sends acknowledged before marking

        conn.execute("UPDATE outbox SET published_at = now() WHERE id = ANY(%s)",
                     ([r.id for r in rows],))
    return len(rows)

FOR UPDATE SKIP LOCKED is what makes the publisher horizontally scalable: several instances poll the same table and never block one another. ORDER BY id preserves per-table insertion order; if you need strict per-aggregate order, partition the poll by aggregate_id hash across publishers.

The flush() before the UPDATE matters. Marking rows published before the broker has acknowledged them converts a broker outage into permanent data loss.

---

Step 3 — Log-based CDC

{
  "name": "outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "slot.name": "outbox_slot",
    "publication.autocreate.mode": "filtered",
    "table.include.list": "public.outbox",
    "transforms": "outbox",
    "transforms.outbox.type":
      "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.table.field.event.key": "aggregate_id",
    "transforms.outbox.table.field.event.payload": "payload",
    "transforms.outbox.route.by.field": "event_type"
  }
}

-- The slot is the mechanism AND the operational hazard: PostgreSQL retains WAL
-- for a slot indefinitely, so a stalled connector fills the disk.
SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
  FROM pg_replication_slots;

Debezium’s outbox event router reads the outbox table from the WAL and publishes without polling. Note the design: CDC is pointed at the outbox table, not at the business tables. That combination is the one most mature teams land on, because it gets log-based latency while keeping the published event contract intentional and decoupled from the business schema.

Pointing CDC directly at business tables removes the outbox but couples every consumer to your column names. A migration that renames a column then becomes a breaking change for services you do not own.

---

Step 4 — The comparison

Start with the polled outbox. It has no new infrastructure, its failure mode is a single visible query, and 250 ms of added latency is invisible in most event-driven systems. Move to CDC when latency genuinely matters or when the polling load becomes a measurable share of database capacity — and when you already operate Kafka Connect, because introducing it solely for this is a large operational step.

---

Step 5 — Deduplicate downstream either way

# The consumer looks identical regardless of which publisher fed the topic.
def handle(record, conn) -> str:
    event_id = dict(record.headers)["event_id"].decode()
    try:
        with conn.transaction():
            conn.execute(
                "INSERT INTO processed_events (consumer_group, event_id) VALUES (%s, %s)",
                (GROUP, event_id),
            )
            apply_effect(json.loads(record.value), conn)
        return "processed"
    except UniqueViolation:
        return "duplicate"

The event_id must be generated by the producing transaction and carried through unchanged — not derived from the broker offset, which differs between a first publication and a republication after a publisher crash.

---

Verification and testing

Confirm the outbox row and the business row commit together.

def test_atomic(db, killer):
    killer.after_statement("INSERT INTO outbox")
    with pytest.raises(ProcessKilled):
        create_charge(amount=4500)
    assert db.scalar("SELECT count(*) FROM charges") == 0
    assert db.scalar("SELECT count(*) FROM outbox") == 0     # neither, never one

Confirm the publisher never marks before the broker acknowledges.

# Stop the broker mid-run and assert nothing was marked published.
docker stop kafka && sleep 5
psql -c "SELECT count(*) FROM outbox WHERE published_at IS NOT NULL
          AND created_at > now() - interval '1 minute'"
# expect: 0
docker start kafka

Confirm the replication slot is not retaining unbounded WAL.

psql -tAc "SELECT slot_name,
             pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS bytes
             FROM pg_replication_slots WHERE slot_name='outbox_slot'"
# expect: well under a few hundred MB; a growing number means the connector is stalled

Confirm end-to-end latency.

psql -c "SELECT percentile_disc(0.99) WITHIN GROUP (ORDER BY published_at - created_at)
           FROM outbox WHERE published_at > now() - interval '10 minutes'"

---

Failure scenarios and debugging

---

SRE / observability checklist

1. outbox_unpublished_count — Gauge of count(*) WHERE published_at IS NULL. Alert above 10,000; it is the single clearest publisher-health signal.
2. outbox_publish_lag_seconds — Histogram of published_at - created_at. Alert if p99 exceeds 2 s for a polled outbox or 500 ms for CDC.
3. pg_replication_slots retained bytes — CDC only. Page above 10 GB; a stalled connector is a database-availability risk, not just an eventing one.
4. outbox_row_count — Gauge from the table’s row estimate. Growth outpacing the reaper predicts query slowdown.
5. events_handled_total{outcome="duplicate"} — Consumer-side counter. A rate above the expected republication baseline points at a publisher crash loop.
6. event_id propagated end to end — As a broker header and a log field, so an event can be traced from the producing transaction to the consuming effect.

---

Related

• Transaction Scoping & Atomic Operations — the parent page on keeping effects and their records inside one commit.
• Implementing the Transactional Outbox Pattern — the outbox itself, in full.
• Wrapping Database Transactions for Safe Retries — transaction boundaries on the write path that feeds the outbox.
• Kafka Transactions vs Consumer-Side Deduplication — what the consumer must do with the at-least-once stream either design produces.