Replication and Consistency

Replication means keeping copies of your data on multiple nodes. It gives you fault tolerance (one node dies, others have the data) and read performance (distribute reads across replicas). It also introduces consistency challenges.

Why Replicate

There are two core reasons to replicate data. The first is fault tolerance: if your primary database node goes down, a replica can take over and your system keeps running. Without replication, a single hardware failure takes your entire database offline.

The second is read scaling: instead of routing every read query to one machine, you spread them across multiple replicas. A system with one primary and three replicas can handle roughly four times the read throughput of a single node. For read-heavy workloads — think social media feeds, product catalogues, dashboards — this matters a lot.

Primary-Replica Replication

The most common replication pattern is primary-replica (also called leader-follower or master-slave in older documentation). One node is designated the primary. It accepts all writes. Every change made on the primary is replicated to one or more replica nodes.

Replicas handle read queries. Your application sends writes to the primary and reads to any replica. This is what PostgreSQL streaming replication, MySQL binlog replication, and MongoDB replica sets all implement under the hood.

The topology is simple enough to reason about: one source of truth for writes, many copies for reads. The complexity creeps in when you look at how those copies stay in sync.

Synchronous vs Asynchronous Replication

The key question in replication is: when does the primary tell the client that a write succeeded?

Synchronous replication means the primary waits for at least one replica to confirm it has received and written the data before acknowledging the client. You get a strong durability guarantee — if the primary crashes immediately after, the replica has the data.

# PostgreSQL synchronous replication config (postgresql.conf)
synchronous_commit = on
synchronous_standby_names = 'replica1'

The tradeoff is latency. Every write now waits for a round trip to the replica. If the replica is slow or the network has hiccups, your writes slow down too. A struggling replica becomes a drag on your entire write path.

Asynchronous replication means the primary acknowledges the client immediately and replicates in the background. Writes are fast. The tradeoff is that if the primary crashes between acknowledging the client and the replica receiving the data, those recent writes are lost.

# PostgreSQL asynchronous replication config (postgresql.conf)
synchronous_commit = off
# Primary does not wait for replica confirmation

Most production systems use asynchronous replication and accept the small window of potential data loss in exchange for write performance. For anything where data loss is unacceptable — financial transactions, audit logs — synchronous replication (or a synchronous standby for at least one replica) is worth the latency cost.

Replication Lag

In asynchronous replication, there's always a delay between a write landing on the primary and it appearing on replicas. This is replication lag. On a healthy, lightly loaded system it might be milliseconds. Under heavy write load or network congestion it can stretch to seconds.

This creates a subtle class of bugs. A user writes a post, then immediately navigates to their profile page. The read goes to a lagging replica. The post isn't there yet. From the user's perspective, their write disappeared.

The standard solution is read-your-own-writes consistency: for a short window after a user writes something, route their reads to the primary instead of a replica. You can implement this with a timestamp-based approach — if the user wrote within the last two seconds, send their reads to primary.

def read_post(user_id, post_id, last_write_ts):
    # If user wrote recently, read from primary to avoid lag
    if time.now() - last_write_ts < 2:
        return primary.query(post_id)
    return replica.query(post_id)

Another approach is sticky reads: once a session has seen a particular version of data, always route that session to a replica that is at least as up-to-date. More complex to implement but avoids the "am I the same user as the one who wrote this?" problem.

Failover

When the primary node goes down, something needs to promote a replica to take its place. This is failover.

Automatic failover relies on the replicas monitoring the primary (using heartbeats) and electing a new leader when the primary stops responding. Tools like Patroni for PostgreSQL and MySQL InnoDB Cluster handle this. Recovery time is typically seconds to a few minutes.

Manual failover requires a human operator to detect the failure and promote a replica. Slower, but gives you more control over which replica gets promoted and when.

The dangerous failure mode is split brain: two nodes both believe they are the primary and start accepting writes independently. When the network partition heals, you have two diverged histories with no clear way to reconcile them. Preventing split brain requires a quorum mechanism — a node only becomes primary if a majority of the cluster agrees.

# Split brain scenario (what you want to prevent)
Node A: primary, accepts writes, thinks Node B is dead
Node B: also thinks it's primary, accepts writes
Network partition heals → conflicting data on both nodes

Multi-Primary Replication

Multi-primary replication (also called multi-master) allows multiple nodes to accept writes simultaneously. This maximises write availability — even if one node goes down, others keep accepting writes. It's the pattern used for multi-region deployments where you want writes to land in the closest data centre.

The complexity is conflict resolution. If two users in different regions update the same record at the same time, both primaries accept the write. When those changes replicate to each other, the system has to decide which one wins. Common strategies are last-write-wins (based on timestamp) or application-defined merge logic.

Last-write-wins sounds reasonable until you realise that clocks across distributed systems are never perfectly synchronised. A write with a slightly later timestamp might actually have happened first in wall-clock time. For most use cases, conflicts are rare enough that last-write-wins is acceptable. For anything involving money or inventory, you need more careful conflict handling.

Key Takeaways

• Replication serves two purposes: fault tolerance when a node fails, and read scaling by distributing reads across replicas
• Primary-replica is the most common pattern — one node accepts writes, replicas serve reads
• Synchronous replication guarantees no data loss but slows writes; asynchronous replication is fast but risks losing recent writes if the primary crashes
• Replication lag in async setups can cause reads to return stale data — read-your-own-writes consistency is the standard mitigation
• Split brain — two nodes both acting as primary — is the most dangerous failover failure mode; quorum-based election prevents it
• Multi-primary replication maximises write availability but requires explicit conflict resolution logic

Next up: Rate Limiting — how systems protect themselves from being overwhelmed by too many requests, and the algorithms behind it.