Implementing the Transactional Outbox Pattern

Part of: Transaction Scoping & Atomic Operations

A service that writes a business row to its database and then separately publishes an event to a message broker has two independent failure points. If the process crashes after the database commit but before the publish, the event is lost forever — a downstream consumer never learns that the order was placed or the payment captured. If the publish succeeds but the database transaction later rolls back, the broker now carries an event for something that never happened. This is the dual-write problem, and it cannot be fixed by retrying harder or adding a try/catch around the publish call.

The transactional outbox pattern solves it by removing the second write entirely from the request path. The event is written to an outbox table in the same database transaction as the business row, so it either commits with the business state or not at all. A separate relay process — typically a change-data-capture connector like Debezium — tails that table and publishes to the broker asynchronously. This page assumes you already understand wrapping database transactions for safe retries and the guarantee model described in Transaction Scoping & Atomic Operations; it also assumes you have read the top-level Backend Implementation & Storage Patterns overview of durable dedup storage.

---

Problem statement and prerequisites

What you are implementing: an atomic write of business state and an outbound event, a durable relay from that event to a broker, and a consumer that treats broker delivery as at-least-once rather than exactly-once.

Prerequisites:

• A relational database that supports transactions and, ideally, logical replication or a write-ahead log a CDC tool can read (PostgreSQL is used throughout this page).
• A message broker — Kafka is used here, but the pattern applies equally to RabbitMQ or SNS/SQS.
• A downstream consumer you can modify to check an event ID before applying side effects, per idempotent consumer patterns for event streams.

The relay guarantees at-least-once delivery, never exactly-once — understand the distinction covered in exactly-once vs at-least-once delivery before you design the consumer side.

---

How the outbox flow fits together

---

Step-by-step implementation

Step 1 — Design the outbox table

Keep the outbox table in the same database and schema as the business tables so the insert participates in the same transaction. Store the event payload as jsonb so the relay can forward it without a second lookup.

CREATE TABLE outbox (
    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type TEXT NOT NULL,       -- e.g. 'payment'
    aggregate_id   TEXT NOT NULL,       -- e.g. the payment_id
    event_type     TEXT NOT NULL,       -- e.g. 'payment.captured'
    payload        JSONB NOT NULL,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_outbox_created_at ON outbox (created_at);

Step 2 — Write the business row and outbox event atomically

The insert into outbox must happen inside the exact same transaction as the business write. If either statement fails, both roll back.

@Service
public class PaymentService {

    @Autowired private JdbcTemplate jdbc;

    @Transactional
    public void capturePayment(String paymentId, BigDecimal amount) {
        jdbc.update(
            "UPDATE payments SET status = 'captured', amount = ? WHERE payment_id = ?",
            amount, paymentId
        );

        String payload = String.format(
            "{\"payment_id\": \"%s\", \"amount\": %s, \"event\": \"payment.captured\"}",
            paymentId, amount
        );

        jdbc.update(
            "INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload) " +
            "VALUES ('payment', ?, 'payment.captured', ?::jsonb)",
            paymentId, payload
        );
        // Both statements commit or roll back together — no separate broker call here.
    }
}

Step 3 — Deploy a CDC relay that tails the outbox

Debezium reads the PostgreSQL write-ahead log via logical replication and emits a change event per insert, with no polling load on the table.

name: outbox-connector
config:
  connector.class: io.debezium.connector.postgresql.PostgresConnector
  database.hostname: db.internal
  database.dbname: payments
  plugin.name: pgoutput
  slot.name: outbox_slot
  table.include.list: public.outbox
  transforms: outbox
  transforms.outbox.type: io.debezium.transforms.outbox.EventRouter
  transforms.outbox.route.by.field: aggregate_type
  transforms.outbox.table.field.event.key: aggregate_id
  transforms.outbox.table.field.event.payload: payload
  topic.prefix: payments-outbox

If you cannot run a CDC connector, a polling relay is an acceptable fallback below roughly 200 events/sec:

func pollAndPublish(ctx context.Context, db *sql.DB, producer *kafka.Writer) error {
    rows, err := db.QueryContext(ctx,
        `SELECT id, aggregate_id, event_type, payload FROM outbox
         ORDER BY created_at ASC LIMIT 100`)
    if err != nil {
        return err
    }
    defer rows.Close()

    var published []string
    for rows.Next() {
        var id, aggID, evType, payload string
        if err := rows.Scan(&id, &aggID, &evType, &payload); err != nil {
            return err
        }
        err := producer.WriteMessages(ctx, kafka.Message{
            Key:   []byte(aggID),
            Value: []byte(payload),
            Headers: []kafka.Header{
                {Key: "event_id", Value: []byte(id)},
                {Key: "event_type", Value: []byte(evType)},
            },
        })
        if err != nil {
            return err // do NOT mark as published — retry on next poll (500ms interval)
        }
        published = append(published, id)
    }
    return markPublished(ctx, db, published)
}

Run this poller on a 500 ms interval with a batch size of 100 rows — tight enough to bound latency, loose enough not to saturate the connection pool.

Step 4 — Mark or delete relayed rows without breaking at-least-once delivery

Only mark a row published after the broker acknowledges the write. If the process crashes between publish and mark, the row is republished on the next poll — this is the at-least-once guarantee, not a bug.

-- Age out published rows after a 48-hour retention window, not immediately,
-- so operators can replay recent events during an incident.
DELETE FROM outbox
WHERE published_at IS NOT NULL
  AND published_at < now() - interval '48 hours';

Step 5 — Build an idempotent consumer

The consumer must treat every delivery as a possible duplicate. Key the dedup check on the event_id header set in Step 3, not on message offset, since offsets are broker-local and not portable across a message queue deduplication failover.

def handle_message(msg):
    event_id = msg.headers["event_id"]
    with db.transaction():
        inserted = db.execute(
            "INSERT INTO processed_events (event_id) VALUES (%s) "
            "ON CONFLICT (event_id) DO NOTHING RETURNING event_id",
            (event_id,),
        )
        if inserted.rowcount == 0:
            # Already processed — ack and skip. This is the normal, expected path
            # for at-least-once delivery, not an error.
            return
        apply_side_effects(msg.value)

For a deeper treatment of this step see idempotent consumer patterns for event streams.

---

Verification and testing

Confirm atomicity of the dual write

# Force a rollback after the business update to prove the outbox insert rolls back too
psql payments -c "BEGIN; \
  UPDATE payments SET status='captured' WHERE payment_id='pay_1'; \
  INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload) \
    VALUES ('payment','pay_1','payment.captured','{}'::jsonb); \
  ROLLBACK;"
psql payments -c "SELECT count(*) FROM outbox WHERE aggregate_id = 'pay_1';"
# Expected: 0 — the outbox row never committed

Check the Debezium connector is streaming

curl -s localhost:8083/connectors/outbox-connector/status | jq '.connector.state'
# Expected: "RUNNING"
curl -s localhost:8083/connectors/outbox-connector/status | jq '.tasks[0].state'
# Expected: "RUNNING"

Simulate a duplicate delivery

# Replay the same event_id twice through the consumer topic
kafka-console-producer --topic payments-outbox --property "parse.key=true" \
  --property "key.separator=:" <<< "pay_1:{\"event_id\":\"evt_001\"}"
kafka-console-producer --topic payments-outbox --property "parse.key=true" \
  --property "key.separator=:" <<< "pay_1:{\"event_id\":\"evt_001\"}"
psql payments -c "SELECT count(*) FROM processed_events WHERE event_id = 'evt_001';"
# Expected: 1 — the second delivery was absorbed by the ON CONFLICT check

---

Failure scenarios and debugging

---

SRE / observability checklist

1. outbox_relay_lag_seconds — Gauge measuring time between an outbox row’s created_at and its publish to the broker. Alert if p99 exceeds 30 seconds.
2. outbox_table_row_count — Gauge on unpublished + retained rows. A steady climb indicates the relay or cleanup job has stopped.
3. outbox_republish_total — Counter for rows republished after a relay restart; expected to be non-zero occasionally, alert only if the rate exceeds 10/min sustained.
4. processed_events_conflict_total — Counter incremented every time a consumer’s ON CONFLICT fires, confirming the idempotency check is actually being exercised in production, not just in tests.
5. Debezium connector task state — poll GET /connectors/<name>/status every 60 seconds; page on any task leaving RUNNING.
6. Trace propagation — attach event_id and aggregate_id as span attributes on both the relay publish span and the consumer processing span so an event’s full lifecycle is queryable in one trace.

---

Related

• Transaction Scoping & Atomic Operations — parent page on transaction boundaries and atomic write guarantees.
• Wrapping Database Transactions for Safe Retries — the atomic-upsert foundations this pattern builds on.
• Message Queue Deduplication Patterns — broker-side dedup techniques to pair with an idempotent consumer.
• Idempotent Consumer Patterns for Event Streams — how to design the consumer-side dedup check used in Step 5.
• Exactly-Once vs At-Least-Once Delivery — the delivery-guarantee model this pattern relies on.