Advanced Distributed Systems #split-brain #fencing #leases #ZooKeeper

Split-Brain & Network Partitions

5 exercises — master network partition vocabulary: split-brain causes and prevention, fencing tokens, lease timeouts, minority partition behaviour, epoch numbers, and ZooKeeper distributed lock patterns.

0 / 5 completed

Split-brain & partition quick reference

Split-brain — two nodes simultaneously believe they are primary; caused by network partition.
Prevention: quorum (majority must vote) + fencing tokens + STONITH.
Fencing token — monotonically increasing epoch; storage rejects writes from lower-token leaders.
Lease — time-bounded authority grant; expires automatically; follower must wait full timeout before electing new leader.
Minority partition — cannot form quorum → enters read-only mode (CP choice).
Epoch / generation — higher epoch supersedes lower; old leader's writes rejected if epoch is stale.
ZooKeeper ephemeral node — auto-deleted on client crash; crash-safe lock primitive.
Watch-predecessor pattern — each lock waiter watches only its immediate predecessor; prevents herd effect.

1 / 5

A site reliability engineer describes an incident: "We had a split-brain scenario in our database cluster. Both nodes thought they were the primary and accepted writes. The result was diverged data that took 4 hours to reconcile."

What causes split-brain and how is it prevented?

Split-brain = two nodes both acting as primary. Root cause: network partition. Solution: quorum + fencing to ensure only one can succeed.

How split-brain occurs:

Normal: Client → Primary (Node A) → Replica (Node B)

Network partition:
[Node A] ................... [Node B]
Node B can't see A → "A must be dead" → B elects itself primary
Node A is still running and accepting writes

Result:
  Client 1 → Node A: writes X=1
  Client 2 → Node B: writes X=2
  Partition heals: X=1 and X=2 in conflict

Prevention 1: Quorum-enforced writes
• In a 3-node cluster, require 2 nodes to acknowledge writes
• During partition: {A} has 1 node − cannot form quorum → stops accepting writes
• {B, C} has 2 nodes − forms quorum → continues operation
• Only one side can form a majority quorum; split-brain prevented

Prevention 2: Fencing tokens
• When a new leader is elected, it gets a monotonically increasing fencing token (e.g., epoch/generation number)
• All writes to storage include the fencing token
• Storage layer rejects writes from any leader with a lower token than it has seen
• Old primary (unaware it was replaced) attempts writes → storage sees lower token → rejects → old primary cannot cause split-brain damage

Prevention 3: STONITH / node fencing
• When new leader believes old leader might still be running, it actively powers it off via IPMI/iDRAC/PDU
• Ensures the old leader is definitively dead before the new leader begins accepting writes
• Used by Pacemaker/Corosync cluster managers

Key vocabulary:
• Split-brain — cluster state where two nodes simultaneously believe they are the active primary; causes data divergence
• Network partition — network failure that prevents communication between cluster nodes while they remain individually operational
• Fencing token — monotonically increasing epoch number; storage layer rejects writes from superseded leaders
• STONITH — "Shoot The Other Node In The Head"; proactive node termination to prevent split-brain writes

2 / 5

An engineer explains: "We use a fencing token when our distributed lock manager grants leases. The token monotonically increases with each lock acquisition."

Describe a scenario where a fencing token prevents data corruption that a naive lock implementation would allow.

The classic GC pause scenario shows why a distributed lock alone is insufficient — fencing tokens provide the safety guarantee.

The failure scenario without fencing:

Client 1 (C1) acquires lock, gets lease T=10s
C1 begins processing (reading, computing...)
C1 experiences GC pause for 15 seconds  ← lease expires at 10s
Lock server: C1's lease expired → grants lock to C2 (token=2)
C2 (token=2) writes: SET user_balance = 100
C1 wakes up after GC pause, still thinks it has the lock!
C1 (token=1, lease expired) writes: SET user_balance = 50  ← CORRUPTS C2's write

With fencing tokens:

C1 acquires lock → receives token=1
C2 acquires lock (C1's lease expired) → receives token=2
C2 writes to storage: {token:2, user_balance:100} ← storage records max_seen_token=2
C1 wakes, writes: {token:1, user_balance:50}
Storage: token(1) < max_seen(2) → REJECT C1's write

Requirements for effective fencing:
• The storage layer must check and enforce fencing tokens — a lock service alone cannot help if storage doesn't cooperate
• Tokens must be monotonically increasing — new grants always get a higher token
• Storage must persist the highest seen token durably — survives reboots
• The check must be atomic with the write (prevents TOCTOU races)

Real-world implementations:
• ZooKeeper znodes: each znode has a version number; conditional updates include expected version (acts as a fencing token)
• etcd transactions: ModRevision check ensures writes only apply if revision hasn't changed since you read
• Optimistic locking in databases: version column acts as a per-record fencing token

Why lock timeouts alone don't solve this:
A shorter lease reduces the window but doesn't eliminate it — GC pauses and OS scheduling can be unpredictable. A 1-second GC pause defeats a 500ms lease. Fencing tokens provide a safety net that works regardless of timing.

Key vocabulary:
• Fencing token — monotonically increasing value granted with a lock; storage rejects writes from lower-token holders
• Lease — a time-limited grant of a distributed lock; expires automatically if the holder doesn't renew
• GC pause — a stop-the-world garbage collection event that can stop a JVM process for hundreds of milliseconds
• TOCTOU (Time-of-Check to Time-of-Use) — race condition where the state changes between when it is checked and when it is acted upon

3 / 5

A distributed systems architect describes their cluster's behaviour: "When we detect a network partition, our cluster automatically enters read-only mode on the minority partition. Some engineers think this is too conservative."

Why is this the correct default behaviour?

Minority partition read-only = explicit CP trade-off. The alternative is AP (accept writes, risk divergence), which is appropriate for different use cases.

The partition scenario:

5-node cluster: {A, B, C, D, E}
Partition: {A, B} | {C, D, E}

{A, B} — minority partition (2 nodes, cannot form quorum of 3)
{C, D, E} — majority partition (3 nodes, forms quorum)

CP choice (Raft, etcd, ZooKeeper, most RDBMS clusters):
  {A, B}: stops accepting writes (read-only)
  {C, D, E}: continues accepting writes (forms new quorum if needed)
  Consistency preserved; availability reduced on {A, B}

AP choice (Cassandra, DynamoDB eventually consistent, Riak):
  {A, B}: continues accepting writes (W=1 on this partition)
  {C, D, E}: continues accepting writes
  Availability preserved; conflict resolution required on partition heal

When to choose CP vs AP:

Factor	Choose CP	Choose AP
Data type	Financial, inventory, user credentials	Social likes, view counts, shopping cart
Conflict handling	No safe merge strategy exists	LWW or CRDT merge is semantically correct
Availability priority	Stale is worse than unavailable	Unavailable is worse than stale
Example systems	etcd, ZooKeeper, PostgreSQL HA, CockroachDB	Cassandra, DynamoDB (eventual), Riak

ZooKeeper's approach:
ZooKeeper explicitly refuses operations without a quorum — a minority partition returns errors on write and on some read types. This is intentional: ZooKeeper is a coordination service where "partial" writes would be dangerous.

Key vocabulary:
• Minority partition — the partition with fewer than quorum-count nodes; cannot safely accept writes in a CP system
• CP (Consistent + Partition-tolerant) — CAP trade-off: prefers consistency, sacrifices availability during partition
• AP (Available + Partition-tolerant) — CAP trade-off: prefers availability, allows diverging reads/writes during partition
• Read-only mode — cluster state where a partition serves reads from its local state but rejects all writes

4 / 5

A senior engineer describes their system's approach to distributed leadership: "Instead of a strict lease-based system, we use a leader heartbeat with failure detector and epoch-based authority. When the leader fails, followers wait for the lease timeout before electing a new one — and the new leader issues a higher epoch."

Why must followers wait for the full lease timeout before electing a new leader?

The lease timeout creates a safety gap: new leader's authority begins only after the old leader's authority has definitely ended.

Lease-based leadership safety invariant:

Time →
Leader A: [----lease valid (10s)----][leader knows: "my lease expired"]
                                     ↑
                            Followers must not elect a new leader before this point

Followers:
  t+0: stop receiving A's heartbeat
  t+10s: A's lease has definitively expired → safe to elect new leader
  t+??: Leader B elected, issues epoch=2

If followers elected B at t+5s:
  A (lease still valid until t+10s) might still be writing → split-brain window of 5s

What makes timeout-based leases safe:
• Leader A knows its lease expires at T
• Leader A stops acting as leader at T (or earlier if it voluntarily steps down)
• Followers wait until T before starting new leader election
• Gap between "last heartbeat from A" and "T" is the detection window, not a splitbrain window

Epoch (generation/term) numbers:
• When new leader B is elected, it gets epoch=2 (A had epoch=1)
• B announces epoch=2 in all messages
• Any node receiving a message with epoch < 2 from A knows A is a stale leader
• Writes attempted by A with epoch=1 are rejected by storage (fencing token enforcement)

Practical challenges:
• Clock skew: lease timeouts rely on wall clocks, which can drift. Safer: use logical monotonic clocks or add a clock skew safety margin to the timeout
• Asymmetric partitions: leader can receive messages but not send them — it might not renew its lease even though it's "alive"
• Google Chubby solution: leaders use "conservative" lease times with a built-in uncertainty buffer

Key vocabulary:
• Lease — time-bounded grant of authority; leader's authority is valid until the lease expiry time
• Epoch / Generation number — monotonically increasing authority identifier; higher epoch supersedes lower
• Lease timeout — the duration after which a leader's authority definitively expires even without an explicit resign message
• Clock skew — difference between wall clock times on different nodes; can undermine lease-based assumptions

5 / 5

A team uses ZooKeeper for distributed coordination. An engineer asks: "We need a distributed lock that is automatically released if the lock holder crashes, and we want herd effect to be avoided. How does ZooKeeper support this?"

ZooKeeper recipe: sequential ephemeral nodes + watch-predecessor pattern = crash-safe + no herd effect.

Naive distributed lock (herd effect problem):

All clients watch /lock node for deletion
Client 1 acquires lock, creates /lock
Clients 2, 3, 4, 5 all watch /lock

Client 1 releases or crashes → /lock deleted
All 4 watchers notified simultaneously → herd effect!
All 4 try to create /lock simultaneously → thundering herd

Better recipe: sequential ephemeral nodes

Each client creates /lock/lock-{sequential} (ephemeral + sequential)
Results: /lock/lock-0001 (C1), /lock/lock-0002 (C2), /lock/lock-0003 (C3)

C1 has lowest number → C1 holds the lock
C2 watches /lock/lock-0001 (the node just before it)
C3 watches /lock/lock-0002 (the node just before it)

C1 crashes → /lock/lock-0001 disappears (ephemeral node)
ONLY C2 is notified (it was watching 0001)
C2 checks: am I the lowest? Yes → C2 acquires lock
C3 is not notified; herd effect avoided

Ephemeral node properties:
• Created by a client session; automatically deleted by ZooKeeper when the session ends (crash or explicit disconnect)
• Session is maintained via heartbeats; if client stops heartbeating for session timeout, ZooKeeper deletes all ephemeral nodes owned by that session
• This is the ZooKeeper mechanism for distributed lock safety: no explicit unlock needed for crash recovery

ZooKeeper watches:
• One-time event notification: fire once when the watched node changes, then must be re-registered
• Watches are notifications, not polling: no spinning, efficient
• On watch trigger, client re-reads state to determine current situation (don't rely solely on the trigger)

Key vocabulary:
• Ephemeral node — a ZooKeeper node that is automatically deleted when the creating client session ends; used for crash-safe locks and leader election
• Sequential node — a ZooKeeper node created with a monotonically increasing suffix; used for ordered lock queuing
• Herd effect / Thundering herd — all waiting clients wake up simultaneously when a resource becomes available; creates contention spike
• Watch-predecessor pattern — each waiter in a ZooKeeper lock queue watches only the next-lower sequential node, preventing herd effect