⚖️ Reading: System Design Trade-offs — CAP Theorem

CAP in practice: the real choice is C vs A

The CAP theorem says you can have at most two of: Consistency, Availability, Partition tolerance. This sounds like three options (CA, CP, AP). But the article makes a crucial point: partition tolerance is non-negotiable in any real distributed system.

Why? Networks fail. Routers drop packets. Data centres lose connectivity. Cloud providers have zone outages. A distributed system that assumes a perfect network will fail catastrophically when — not if — the network misbehaves. Giving up P means your system becomes a single node (no distribution at all).

The real choice collapses to:

• CP (Consistency + Partition tolerance): When a partition occurs, refuse to serve requests rather than risk returning stale data. The system becomes temporarily unavailable to preserve data correctness. Example: a banking ledger must never show a balance that hasn't been confirmed across all nodes.
• AP (Availability + Partition tolerance): When a partition occurs, continue serving requests even if some nodes have stale data. Eventually the data converges when the partition heals. Example: a social media feed can show posts from a few minutes ago without causing real harm.

The insight for system design interviews and real architecture decisions: Before choosing a database, ask: "What is worse for my users — getting an error, or getting slightly stale data?" This maps directly to C vs A.

Vocabulary note: "Partition" in distributed systems means a network split — two or more groups of nodes cannot communicate with each other. It is NOT about database table partitioning. Context determines the meaning.

Eventual consistency — a deliberate design choice, not a flaw

"Eventual consistency" is a consistency model that guarantees: if no new updates are made to a data item, eventually all accesses to that item will return the last updated value.

The key word is "eventually" — there is no time bound guarantee, but the system does converge.

Contrast with strong (linearisable) consistency:

• Strong consistency: Every read sees the most recent write, regardless of which node you query. This requires coordination (quorum reads/writes, distributed transactions) — adding latency and reducing availability during partitions.
• Eventual consistency: Different nodes may temporarily return different values for the same key. After all writes propagate, all nodes agree. No coordination needed — better performance and availability.

When eventual consistency is completely acceptable:

• User profile "last seen" timestamps — being 30 seconds behind is invisible to users
• Shopping cart item counts in a catalogue — approximate stock numbers are fine until checkout
• Social media like counts — 10,312 vs 10,318 likes makes no real difference
• DNS propagation — a classic real-world example of eventual consistency that everyone accepts

When eventual consistency is NOT acceptable:

• Bank account balances — you cannot allow two withdrawals from the same £100 to both succeed
• Inventory reservation — two customers cannot both "successfully" book the last hotel room
• Distributed locks — the whole point is that exactly one process holds the lock at a time

The takeaway: Eventual consistency is not a compromise forced on engineers — it is a deliberate, valid choice that enables higher availability and better performance for the correct category of data.

Tunable consistency — the pragmatic evolution beyond binary CAP

Modern distributed databases (most notably Apache Cassandra) implement tunable consistency: the ability to choose, per query, how many replicas must agree before returning a result.

In Cassandra, consistency levels include:

• ONE — only 1 replica must respond. Fastest, lowest consistency guarantee. Best for: non-critical reads where staleness is acceptable.
• QUORUM — majority of replicas must respond (e.g., 2 of 3). Balanced consistency and availability. Best for: most application reads/writes where some consistency is needed.
• ALL — every replica must respond. Strongest consistency. Best for: critical financial data, where correctness outweighs any latency or availability cost.

The article's example made concrete:

• "Financial transactions" → use QUORUM or ALL — you cannot afford stale balance reads
• "User activity feeds" → use ONE — if a user's feed is 5 seconds behind, nobody notices or cares

Why this matters architecturally: Tunable consistency means you don't have to pick a different database for different consistency requirements. A single Cassandra cluster can handle both your strict-consistency payment records and your eventually-consistent activity stream — you express the requirement at the application layer per query.

The key English phrase from the article: "dial the trade-off" uses a metaphor of a physical dial (a knob that has a continuous range, not a binary switch). This communicates that consistency is a spectrum, not a binary choice — a nuance that's important both for technical accuracy and for system design interviews where demonstrating this understanding distinguishes strong candidates.