Database Replication Strategies for Distributed Systems

Imagine you run an online store with a single database server sitting in one data center. Two things go wrong as the business grows. First, every user outside that region pays the round trip distance in load time, no matter how fast the server itself is. Second, that one server is now the whole business. A disk failure, a bad deploy, a power outage in that data center, and the store is down until someone fixes it.

Replication is the fix for both problems. Keep more than one copy of the data, in more than one place, so no single machine can take the whole system down and users can be served from a copy near them.

What is database replication

Database replication means keeping multiple copies of the same data on different servers, called replicas, so that if one server goes down the others keep serving requests without interruption.

If the data never changed, replication would be trivial, copy it once to every node and stop. Nearly all the difficulty in replication comes from handling ongoing writes, keeping every copy current without slowing the system down or losing data along the way.

Why we need it

  • Fault tolerance. One server failing should not take the whole application down with it.
  • Latency. A replica placed close to a user answers faster than a database on the other side of the world. This is the same principle a CDN uses for static content.
  • Read scalability. Read traffic can spread across many replicas instead of hammering one server, which matters once traffic grows past what a single machine can serve.

There are three architectures used to implement this, each with a different answer to the question of who is allowed to accept a write.

  • Single leader replication (also called Active-Passive or Master-Slave)
  • Multi leader replication (also called Active-Active or Master-Master)
  • Leaderless replication (also called No-Leader replication)

Single leader replication

One node is designated the leader. Every write goes to the leader first, and the leader streams those changes out to its followers. Reads can be served from the leader or from any follower, which is where the read scaling benefit comes from.

Client Leader accepts writes Follower 1 reads Follower 2 reads Follower 3 reads write replication read (optional, can be stale)

Used in: PostgreSQL, MySQL, SQL Server, MongoDB, Kafka.

Advantages: Simple mental model. There is exactly one place writes happen, so there is never a question of which copy is correct. Strong read scalability, since followers can absorb read traffic without touching the leader.

Disadvantages: The leader is a single point of failure for writes. If it goes down, no new writes are accepted until a follower is promoted. Followers can lag behind the leader, so a read from a follower can return stale data. This delay is called replication lag.

Synchronous vs asynchronous replication

Inside single leader replication there is a second decision, how long the leader waits before telling the client a write succeeded.

Synchronous Client Leader Follower write replicate ack success client waits for the full round tripAsynchronous Client Leader Follower write success (immediate) replicate later client never waits on the follower

Synchronous replication has the leader wait for a follower to confirm the write before responding to the client. The advantage is strong consistency, that follower is guaranteed to be caught up. The disadvantage is that a slow or unreachable follower blocks every write, which gets painful fast once you have more than a couple of followers.

Asynchronous replication has the leader respond as soon as its own copy is updated, without waiting on any follower. Writes stay fast regardless of how many followers exist. The risk is durability, if the leader crashes before a follower catches up, whatever was written in that gap is lost permanently. This is why asynchronous replication is still the default for most systems, the write latency it saves is usually worth more than the small window of risk it introduces.

Handling node failures

Follower failure is the easy case. The follower reconnects to the leader and asks for everything it missed since it went down. This is usually called catchup recovery, and it does not require any coordination beyond the leader and that one follower.

Leader failure is the hard case. One of the followers has to be promoted to leader, a process called failover. The question of which follower gets promoted is decided by a consensus algorithm, where a quorum of the remaining nodes votes on the new leader. Getting this vote wrong, or having two nodes both believe they are the leader at once, is known as the split brain problem, and it is most of why leader election is treated as its own hard problem in distributed systems rather than a footnote.

Multi leader replication

Single leader replication has one structural weakness, every write has to physically reach one machine. Multi leader replication removes that constraint by allowing more than one leader to accept writes. Each leader also acts as a follower of the other leaders, so writes eventually reach every node.

REGION A Leader A Follower A1 Follower A2 REGION B Leader B Follower B1 Follower B2 cross replication (both directions)

Used in: systems that need a leader per geographic region, such as multi region deployments of MySQL and PostgreSQL, and CouchDB.

Advantages: Writes can be accepted close to the user in every region, which cuts write latency dramatically for a global user base. No single leader failure can block writes everywhere, only in the region whose leader went down.

Disadvantages: Conflicts. If two leaders accept a write to the same record before either hears from the other, something has to decide which write wins, last write wins, a merge function, or pushing the decision to the application. More moving parts to operate and reason about, since every leader now needs to track and resolve conflicts with every other leader.

Leaderless replication

The third option removes the leader concept entirely. Clients send writes to several nodes in parallel and read from several nodes in parallel too. No node is special, and there is no election or promotion to manage.

Client Node 1 v=5 Node 2 v=5 Node 3 v=4 (stale) reads and writes go to several nodes at once majority says v=5, so Node 3 gets a repair write

Used in: Amazon DynamoDB, Apache Cassandra, Riak.

Without a leader coordinating things, nodes can end up holding different values for the same key. The read path does the work a leader would normally do, through a technique called read repair. A client reads a key from several nodes at once, and if the responses do not agree, whichever value the majority of nodes hold wins, then a write goes out to the outlier to bring it back in line.

Advantages: No single point of failure for either reads or writes, since any node can serve either. High availability, because there is no leader whose downtime blocks anything.

Disadvantages: No single, clear answer to “what is the current value” at any given instant, only what most nodes agree on. Conflict handling is pushed into every read, which adds latency and complexity that a single leader system does not have to deal with.

Choosing between them

Single leaderMulti leaderLeaderless
Write pathOne leader onlySeveral leadersAny node
ConsistencyStrong (sync) or eventual (async)Eventual, conflicts possibleEventual, resolved by read repair
Write availabilityBlocked if leader is downHigh, regional leadersHighest, no leader to lose
Operational complexityLowMedium to highMedium to high
Common use caseMost relational databasesMulti region write-heavy appsHigh write throughput, no strict ordering needed

The takeaway

There is no universally correct replication strategy, only the right one for what you are optimizing for. Single leader is the default because it is the easiest to reason about, and most applications never need more than that. Reach for multi leader when write latency across regions matters more than avoiding conflicts. Reach for leaderless when write availability matters more than having one clear answer to what the data currently is. Every one of these trades some consistency for some combination of speed and availability, and naming which one your system actually needs is most of the design decision.