Advanced Distributed Systems #2PC #Saga #Outbox #idempotency

Distributed Transactions

5 exercises — master distributed transaction vocabulary: 2PC vs Saga trade-offs, compensating transactions, the Outbox pattern for reliable event publishing, lost updates, and idempotency key implementation.

0 / 5 completed

Distributed transactions quick reference

2PC — prepare + commit phases; ACID atomicity; blocking on coordinator failure. Use for co-located, short-lived XA resources.
Saga — sequence of local transactions + compensating transactions. Use for cross-service, long-lived flows.
Compensating transaction — application-level semantic undo for an already-committed Saga step.
Outbox pattern — write event to outbox table in same DB transaction; relay publishes to message broker. Solves dual-write.
Lost update — concurrent read-modify-write overwrites each other. Fix: optimistic locking (version), SELECT FOR UPDATE, or atomic DB operation.
Idempotency key — caller-generated UUID; service stores key → result; retries return cached result without re-executing.

1 / 5

An architect explains: "We use Two-Phase Commit (2PC) to coordinate our order and inventory updates. A colleague says '2PC is blocking — you should use the Saga pattern instead.' When is 2PC actually appropriate?"

2PC = synchronous, blocking, ACID atomicity for co-located resources. Saga = asynchronous, eventually consistent, with explicit compensation.

Two-Phase Commit protocol:

Phase 1: Prepare
  Coordinator → Participant A: "Can you commit?"
  Coordinator → Participant B: "Can you commit?"
  A → Coordinator: "Yes (VOTE_COMMIT)" or "No (VOTE_ABORT)"
  B → Coordinator: "Yes" or "No"

Phase 2: Commit (if all voted YES) / Abort (if any voted NO)
  Coordinator → A: COMMIT (or ABORT)
  Coordinator → B: COMMIT (or ABORT)
  A, B: execute and release locks

2PC blocking problem:
• If the coordinator crashes after Phase 1 but before Phase 2, participants are stuck holding locks
• They cannot proceed without hearing the coordinator's Phase 2 decision
• Resolution requires coordinator recovery (from stable storage) or manual intervention
• This is the "blocking" nature of 2PC — participants block waiting for the coordinator

When 2PC is appropriate:
• Same data centre: microsecond to millisecond network latency
• Under your operational control (you can recover the coordinator)
• XA-capable resources (databases, message brokers supporting XA: PostgreSQL, MySQL, Oracle, ActiveMQ)
• Short-lived transactions (seconds, not minutes)

When 2PC is problematic:
• Cross-datacenter (high latency + unreliable network = long blocking windows)
• Third-party systems (you can't guarantee they implement XA correctly or recover predictably)
• Long-lived operations (multi-step workflows that take seconds to minutes)
• High-throughput systems (lock holding during 2PC reduces concurrency)

Key vocabulary:
• Two-Phase Commit (2PC) — distributed transaction protocol ensuring atomicity across multiple participants via prepare and commit phases
• XA — open standard for distributed transaction management; implemented by databases and message brokers
• Coordinator — the node that drives the 2PC protocol and makes the final commit/abort decision
• Blocking — 2PC characteristic where participants hold locks indefinitely if the coordinator fails between phases

2 / 5

A team is implementing a Saga pattern for an e-commerce order flow: Reserve Stock → Charge Payment → Dispatch Notification → Create Shipment. The payment service fails halfway through.

How does the Saga handle this, and what is a compensating transaction?

Sagas achieve eventual consistency across services via forward progress + backward-propagating compensating transactions — not database-level rollbacks.

Order Saga failure scenario:

Steps:
  T1: Reserve Stock → SUCCESS ✓
  T2: Charge Payment → FAILURE ✗ (card declined)

Compensations (run in reverse order):
  C1: Release Stock Reservation (compensates T1)

Final state: order in FAILED state; stock released; no charge.

Properties of compensating transactions:
• Semantic undo, not database undo: "cancel reservation" creates a new event, doesn't rewind time
• Idempotent: can be re-executed safely if they fail mid-execution
• Business-meaningful: must handle side effects that have already occurred (e.g., a notification email was already sent — you can send a "sorry, order failed" follow-up, but you can't un-send the first email)

Two Saga implementation styles:

Style	How it works	Pros / Cons
Choreography	Each service publishes events; next service subscribes. No central coordinator.	+ Decoupled; − Hard to track state; cycling event chains
Orchestration	Central saga orchestrator commands each service step and coordinates compensation.	+ Clear flow; easier to debug; − Central point of coupling

Saga vs 2PC — when to choose:
• Short-lived, same-datacenter, ACID required → 2PC
• Cross-service, long-running, third-party services → Saga
• Can define meaningful compensations → Saga works well
• "No compensation possible" (e.g., rocket launched) → design the Saga step as the last irreversible action in the chain

Key vocabulary:
• Saga pattern — long-lived transaction pattern decomposing a distributed workflow into a sequence of local transactions with compensating transactions for failure recovery
• Compensating transaction — an application-level operation that semantically reverses the effect of a completed Saga step
• Choreography Saga — each service reacts to events published by the previous step; no central coordinator
• Orchestration Saga — a central saga orchestrator directs each service step and manages compensation on failure

3 / 5

A distributed systems engineer mentions: "We're using the Outbox pattern to ensure our database write and the event publication to Kafka are atomic. Without it, we had dual-write problems."

What is the dual-write problem and how does the Outbox pattern solve it?

Dual-write problem: two separate write operations (DB + message broker) with no atomicity guarantee. Outbox pattern: piggyback message on the DB transaction, then relay reliably.

The dual-write failure scenarios:

Scenario A (partial failure):
  1. UPDATE orders SET status='PAID'     ← SUCCESS
  2. kafka.send('order.paid', payload)   ← PROCESS CRASH
  Result: DB says PAID; downstream services never heard about it

Scenario B (message without DB commit):
  1. kafka.send('order.paid', payload)   ← SUCCESS
  2. COMMIT transaction                  ← DB TRANSACTION ROLLBACK
  Result: Kafka has the event; DB does not have the change

Outbox pattern solution:

BEGIN TRANSACTION;
  UPDATE orders SET status='PAID' WHERE id=123;
  INSERT INTO outbox (aggregate_id, event_type, payload, published)
    VALUES (123, 'OrderPaid', '{"orderId":123,...}', false);
COMMIT;  ← Both writes are atomic

Relay process (CDC or polling):
  SELECT * FROM outbox WHERE published = false;
  → Publish each event to Kafka
  → UPDATE outbox SET published=true, published_at=now()

Two relay approaches:

Approach	Mechanism	Trade-offs
CDC (Debezium)	Reads WAL/binlog; captures outbox INSERT as a change event	+ Near real-time; no polling overhead − Requires CDC setup; WAL access
Polling	Background job polls outbox every N ms for unpublished events	+ Simpler; − Polling latency; load on DB

At-least-once delivery:
The relay may publish the same event twice (process crash after Kafka publish but before outbox mark). Consumers must be idempotent — they deduplicate using the event ID.

Key vocabulary:
• Dual-write problem — inconsistency risk when writing to two separate systems without an atomic umbrella
• Outbox pattern — write the event to an outbox table in the same DB transaction as the business data; relay publishes it separately
• CDC (Change Data Capture) — captures database changes from the transaction log (WAL, binlog) in near real-time
• At-least-once delivery — event may be delivered more than once; consumers must handle idempotency

4 / 5

A team is reviewing their Saga orchestration and a senior engineer flags: "We have a lost update problem in our inventory service. When we get concurrent reservations, some are being lost."

What causes this and how is it fixed?

Lost update = read-modify-write cycle without atomicity. Three standard solutions: optimistic locking, pessimistic locking, or atomic operations.

The lost update race condition:

T1: SELECT stock FROM inventory WHERE id=5  → stock=100
T2: SELECT stock FROM inventory WHERE id=5  → stock=100

T2: UPDATE inventory SET stock=99 WHERE id=5  ← COMMIT
T1: UPDATE inventory SET stock=99 WHERE id=5  ← COMMIT (overwrites T2!)

Result: stock=99 but 2 items were reserved. 1 reservation lost.

Solution 1: Optimistic locking (version column)

T1: SELECT stock, version FROM inventory → {stock:100, version:5}
T2: SELECT stock, version FROM inventory → {stock:100, version:5}

T2: UPDATE SET stock=99, version=6 WHERE id=5 AND version=5  → 1 row affected ✓
T1: UPDATE SET stock=99, version=6 WHERE id=5 AND version=5  → 0 rows (version mismatch!)
T1: RETRY → re-read version=6, re-compute stock=98, update with version=7

• No locks held; high concurrency; retries on conflict
• Best for low-contention scenarios

Solution 2: Pessimistic locking (SELECT FOR UPDATE)

T1: SELECT stock FROM inventory WHERE id=5 FOR UPDATE  ← acquires row lock
T2: SELECT stock FROM inventory WHERE id=5 FOR UPDATE  ← BLOCKS until T1 commits
T1: UPDATE SET stock=99; COMMIT → releases lock
T2: re-reads stock=99, UPDATE SET stock=98; COMMIT

• Prevents lost updates; but holds locks = lower concurrency
• Risk of deadlock if T1 and T2 each hold locks the other needs

Solution 3: Atomic database operation

UPDATE inventory SET stock = stock - 1
WHERE id = 5 AND stock > 0

Returns rows affected:
  1 → success (atomically decremented)
  0 → stock was 0; reservation fails

• Simplest; database handles atomicity at the statement level
• Works well when the computation is simple arithmetic

Distributed context (multi-service Saga):
In a Saga, each service maintains its own inventory. The reservation step should use optimistic locking or atomic decrement + check. If it returns 0 (stock unavailable), the Saga step fails and compensation begins.

Key vocabulary:
• Lost update — anomaly where two concurrent read-modify-write cycles overwrite each other's changes
• Optimistic locking — detects write conflicts using a version field; conflicting writer must retry
• Pessimistic locking (SELECT FOR UPDATE) — acquires a row-level lock at read time, preventing concurrent writes
• Atomic operation — a database operation that reads and writes in a single, indivisible step

5 / 5

An architect reviews a system using the Saga pattern and says: "We need to handle idempotency carefully here. The payment service might receive the same payment request twice if our Saga retries."

What does idempotency mean in this context, and how is it implemented?

Idempotency = safe to replay. In distributed systems with retries, at-least-once delivery, and failure recovery, idempotency is a critical correctness requirement.

Why retries happen:
• Network timeout: the request was received and processed, but the response was lost → caller retries → double processing
• Saga compensation + re-execution: a Saga step is retried after temporary failure
• Message queue at-least-once delivery: the same Kafka message may be delivered twice (exactly-once requires Kafka transactions)

Idempotency key implementation:

-- Idempotency store table
CREATE TABLE idempotency_records (
  idempotency_key VARCHAR(128) PRIMARY KEY,
  result_status   VARCHAR(50),
  result_payload  JSONB,
  created_at      TIMESTAMPTZ DEFAULT now(),
  expires_at      TIMESTAMPTZ
);

-- Payment service handler
BEGIN;
  -- Check if already processed
  SELECT result_status, result_payload
  FROM idempotency_records
  WHERE idempotency_key = :key;

  IF found → return stored result (early exit)
  ELSE:
    → Execute payment
    → INSERT INTO idempotency_records VALUES (:key, 'SUCCESS', ...)
    → UPDATE payments SET charged = true WHERE order_id = :order_id
COMMIT;

The key-result store must be written atomically with the business operation — otherwise a crash between payment and record insertion creates a window where retries would re-charge.

Idempotency key best practices:
• Generated by the caller (not the service): UUID or hash(entity_id + operation_type + version)
• Scoped to the operation: key for "charge order 123" is different from "refund order 123"
• TTL: expire after max retry window (e.g., 24 hours for payment retries)
• Returned in response header: Idempotency-Key for RESTful APIs (Stripe, Adyen pattern)

Real-world examples:
• Stripe API: requires Idempotency-Key header for all POST requests; returns cached response for duplicate keys within 24h
• Kafka transactions: producer exactly-once semantics use a producer ID + sequence number to deduplicate
• AWS SQS: message deduplication ID for FIFO queues provides 5-minute deduplication window

Key vocabulary:
• Idempotency — property of an operation that can be applied multiple times without changing the result beyond the first application
• Idempotency key — a client-generated unique identifier for a specific request, used to detect and short-circuit duplicate submissions
• At-least-once delivery — messaging guarantee that every message is delivered at least once; may be delivered more than once
• Exactly-once semantics — delivery guarantee ensuring a message is processed exactly once; requires coordination overhead