How a keenPG cluster fails over in under four seconds

Most of the time a database failover is invisible. A disk dies at 3 a.m., a replica takes over and the on-call engineer reads about it over breakfast. But "invisible" hides a lot of careful engineering. If any one of the five steps above is skipped or rushed, you get the two outcomes everyone fears: downtime or, worse, two primaries writing different data.

Here is the full timeline of a typical failover on a keenPG cluster:

Notice

Each node in the cluster writes a heartbeat to a small consensus store every second. The heartbeat carries more than "I'm alive". It includes the node's role, its replication position and its timeline, which gives the rest of the cluster everything it needs to make a decision later.

Why wait for three missed beats instead of one? Because networks hiccup. A single lost packet shouldn't trigger a failover, which has a cost of its own: every connection drops. Three seconds is the balance we've found between reacting quickly and not reacting to noise.

Agree

A replica that can't see the primary doesn't simply promote itself. The primary might be fine, and the replica might be the one that's cut off. Instead the remaining nodes need a majority: in a three-node cluster, two nodes must agree that the primary is gone.

This is the single most important rule in high availability. A node that loses its majority can never become, or stay, the primary. It is the rule that makes split-brain impossible. You can watch it happen in the "Cut the network" scenario of our failure simulator.

Fence

Before anything is promoted, the old primary has to be made harmless. If it's merely slow rather than dead, it could wake up and accept writes that the new primary never sees. So it gets fenced: its lease in the consensus store expires, and it demotes itself to read-only the moment it notices. If it can't be reached at all, the platform stops its instance.

Promote

Now the cluster picks the replica with the most data. With synchronous replication, at least one replica is guaranteed to have every committed transaction, because the primary waited for it before telling your app "OK". That replica is chosen and promoted:

-- what the platform runs on the chosen replica
SELECT pg_promote(wait => true);

You can check how up-to-date each replica is at any moment:

SELECT application_name, sync_state, replay_lag
FROM   pg_stat_replication;

 application_name | sync_state | replay_lag
 pg-b             | sync       | 00:00:00.0004
 pg-c             | async      | 00:00:00.0120

Reconnect

A promoted primary is no use if your applications are still knocking on the old door. Every keenPG cluster sits behind a connection proxy that watches the consensus store. When the leader changes, the proxy routes new connections to it. Apps don't need a config change or a restart, only the reconnect logic that every database driver already has.

What you can tune

The defaults suit most workloads, but every cluster can be adjusted. On keenPG on Kubernetes (KPK), it's part of the cluster spec:

spec:
  instances: 3
  failover:
    heartbeatInterval: 1s      # how often nodes check in
    missedBeats: 3             # beats before failover starts
    synchronousStandbys: 1     # 0 = faster commits, 1 = zero data loss

A word of caution: lowering missedBeats to 1 makes failover faster on paper, but it also makes it trigger on every network blip. We rarely recommend it.