CAP Theorem

In the previous lesson we saw that consistency and availability pull against each other when a network partition occurs. CAP Theorem formalises this into a proof: a distributed system can guarantee at most two of three properties simultaneously.

The Three Properties

Consistency means every read returns the most recent write. If you write a value to Node A, a read from Node B immediately after will return that same value. All nodes in the cluster see the same data at the same time.

Availability means every request gets a response. The system does not return an error or time out — it always responds, even if the data it returns might not be the absolute latest. The system stays up and answering.

Partition Tolerance means the system keeps functioning even when nodes cannot communicate with each other due to a network failure. Messages between servers get dropped, delayed, or lost, but the system continues operating.

Why Partition Tolerance Is Not Optional

Networks fail. This is not a hypothetical — it is a certainty at scale. A fiber cable gets cut, a switch reboots, a datacenter loses connectivity to another datacenter for 30 seconds. In large distributed systems, partial network failures happen regularly.

A network partition creates a "split brain" scenario: two groups of nodes can no longer talk to each other, and each group only knows its own state. You cannot design around this by writing better code. The network is outside your control.

This means partition tolerance is not a choice you get to opt out of. Any distributed system running across multiple machines must tolerate partitions or it is not a distributed system — it is a single machine with very expensive failure modes. So the real decision CAP forces on you is: when a partition happens, do you sacrifice consistency or availability?

CP Systems — Consistency Over Availability

CP systems choose to stop responding rather than return potentially wrong data. When a partition occurs and nodes cannot confirm they have the latest state, the system refuses the request. You get an error instead of a stale answer.

This sounds harsh, but in some domains it is the only acceptable behavior. Examples of CP systems include MongoDB in its default write concern configuration, HBase, ZooKeeper, and Redis Cluster. ZooKeeper is worth noting explicitly — it is widely used as a coordination service precisely because it never lies to you about data state.

The use cases where CP is the right call are those where wrong data is worse than no data: bank transfers, inventory management, seat reservations. If your bank shows you a $500 balance and two simultaneous requests both try to withdraw $400, you cannot afford eventual consistency. One of those requests must fail, and the system must refuse to answer rather than allow both to proceed.

AP Systems — Availability Over Consistency

AP systems keep responding even when they cannot guarantee the data is current. During a partition, each side of the split continues serving reads and writes independently. When the partition heals, the nodes reconcile and converge on the same state — this is what "eventually consistent" means.

The tradeoff is that between the partition and the reconciliation, different clients might see different versions of the data. Examples of AP systems include Cassandra, DynamoDB, CouchDB, and DNS. DNS is a classic example most engineers interact with daily — when you update a DNS record, it takes time to propagate globally, and during that window some users resolve to the old IP and some to the new one.

AP systems are appropriate when being slightly stale is acceptable: social media feeds, shopping carts, user profile reads, product catalog pages. If your Twitter timeline shows a tweet as having 1,200 likes when it actually has 1,201, no one is harmed.

Real World Examples

Amazon's shopping cart is one of the most famous AP system decisions in engineering history, described in Amazon's Dynamo paper from 2007. Amazon chose availability explicitly — they decided it was always better to let a customer add items to their cart than to show an error. Cart merges happen on checkout. A stale cart is recoverable; a broken cart costs a sale.

Bank transfers are the canonical CP example. If your database cannot confirm both the debit and the credit have been written consistently, the transaction must not proceed. A wrong balance is not an inconvenience — it is a compliance failure and a trust disaster.

Google Spanner is an interesting outlier. It attempts to behave as a CA system by using atomic clocks and GPS receivers to establish a global time ordering of transactions with extremely tight bounds. This effectively compresses the partition window so small that the system can maintain strong consistency while remaining highly available in practice. It does not violate CAP — it just makes partitions vanishingly rare through extraordinary hardware investment.

The Nuance — CAP Is a Spectrum

The original CAP theorem is a binary proof, but real systems are not binary. Most modern distributed databases let you tune consistency per operation rather than making a single system-wide choice.

Cassandra is the clearest example. When you issue a query, you specify a consistency level: ONE means one replica needs to respond (fast, potentially stale), QUORUM means a majority of replicas must agree (slower, stronger consistency), ALL means every replica must respond (slowest, strongest consistency). You can make a single Cassandra cluster behave closer to CP for critical writes and closer to AP for high-volume reads, depending on what the operation requires.

// Cassandra consistency levels — you choose per query
SELECT * FROM orders WHERE id = ? -- consistency: QUORUM
INSERT INTO events ... -- consistency: ONE

This flexibility is why the real skill in distributed systems design is not memorizing which category a database falls into — it is understanding which consistency guarantees each operation in your system actually needs.

Key Takeaways

• CAP Theorem states a distributed system can guarantee at most two of Consistency, Availability, and Partition Tolerance simultaneously.
• Partition Tolerance is not optional in any real distributed system — network failures are inevitable, so the real choice is between C and A.
• CP systems refuse to respond during a partition rather than return potentially stale data — the right choice for financial systems and coordination services.
• AP systems keep responding during a partition and reconcile later — the right choice for feeds, carts, and anything where eventual consistency is acceptable.
• Most production databases let you tune consistency levels per operation, so you are rarely locked into a single position on the spectrum.
• The key engineering judgment is knowing which operations in your system genuinely require strong consistency and which can tolerate staleness.

In the next lesson we look at the two strategies for handling more load — scaling your server up versus scaling your servers out.