Split brain in clusters: how quorum and fencing prevent it

Cluster quorum: the majority of votes decides

Quorum is the minimum number of votes a part of the cluster needs before it is allowed to act. In Proxmox VE, every node has one vote by default. Only a group of nodes holding more than half of all votes may change the state of the cluster. Without that majority, the cluster switches to read-only mode.

The nodes communicate through the Corosync Cluster Engine. Its votequorum service counts the votes and, according to its documentation, exists specifically to avoid split brain. Terms such as Corosync and HA Manager are explained in our high availability glossary.

The majority rule: ⌊n/2⌋ + 1

Corosync uses a simple majority: half of the votes plus one, rounded down to whole votes. For n nodes with one vote each, the majority is ⌊n/2⌋ + 1, and at most n − (⌊n/2⌋ + 1) nodes can fail at the same time.

One detail matters for planning: the majority refers to all expected votes, which Corosync sets at startup, not only to the votes currently reachable. If one node of a three-node cluster fails, the remaining two still need two votes. A second failure leaves the last node without a majority.

Why an even number of nodes works against you

A fourth node adds capacity, yet it does not add failure tolerance: four nodes need three votes and, like three nodes, survive only one failure. And if a four-node cluster breaks into two halves of two nodes each, neither half has a majority, and both lose quorum. For even node counts, Proxmox therefore supports an external quorum device whose extra vote breaks the tie.

Split brain in a cluster: what happens without quorum

A split brain cluster has fallen apart into parts that can no longer reach each other and keep running independently. Each part assumes the other has failed and takes over its work. The same virtual machines then run twice and write to the same data at the same time.

A network split is enough to cause it, for example a faulty switch or a broken link. The nodes keep running; only their connection to each other is gone. From the inside, this looks exactly like the other nodes failing.

Quorum resolves the conflict with a simple rule: only the part holding a majority of votes may carry on. Because a majority means more than half, only one such part can exist. If the nodes split exactly in half, the cluster stays consistent, though it can no longer act.

In Proxmox VE, losing quorum makes the cluster file system pmxcfs, which holds the configuration of all nodes, read-only. The node waits for a new quorum. If highly available VMs are running on it, it fences itself.

Cluster fencing: isolating a failed node safely

Fencing guarantees that a failed or cut-off node is really offline before its services start on another node. Only then is it certain that no VM runs twice. Without fencing, the node would remain in an unknown state and might still have access to shared storage.

Quorum and fencing work as a pair. Quorum decides which part of the cluster may act. Fencing makes sure the other part has actually stopped before the acting part takes over its services.

How Proxmox VE fences: self-fencing with a watchdog

A node can be fenced with an external power switch, by cutting off all of its traffic at the network switch, or by self-fencing with a watchdog timer. Proxmox VE uses self-fencing because it needs no additional hardware; according to the documentation, external fence devices would be critical components in their own right.

During normal operation, the HA Manager resets the watchdog regularly. A node without quorum can no longer do so. If active highly available services run on it, the documentation states that the watchdog reboots the server after 60 seconds. By default, Proxmox VE uses the Linux kernel's software watchdog (softdog); a hardware watchdog has to be configured explicitly. The vendor describes the software watchdog as reliable, though not independent of the server hardware. Only once the cluster resource manager has acquired the node's lock does it count as fenced, and its VMs restart on the remaining nodes.

The load balancer removes a failed node from rotation within seconds. The node's virtual machines restart on another node after fencing; how long that takes depends on fencing and on the guest system's boot time.

Proxmox two node cluster: only with an external quorum device

A two-node cluster has two votes, and its majority is also two. If either node fails, the remaining node loses quorum and the cluster can no longer act. A Proxmox two node cluster becomes highly available only with a third vote from outside: the external quorum device, or QDevice.

With a QDevice, the corosync-qdevice service runs on every node, and the corosync-qnetd arbitrator runs on an independent server outside the cluster. According to the Proxmox documentation, the arbitrator gives its vote to only one part of the cluster at a time, and only to a part that reaches a majority with it. It connects over TCP/IP, without the strict latency requirements of the cluster network. If the link between the two nodes breaks while both still reach the QDevice, only one of them gets the vote. According to the Corosync documentation, a tie is settled last by the tiebreaker, which by default favors the node with the lowest node ID.

Limits of a two-node cluster with a QDevice

• Separate location: the QDevice runs on an independent server; otherwise it would fail with the very node it protects.
• Its own failure: if the QDevice fails, the next node failure costs quorum, so it belongs in your monitoring.
• One node carries everything: after a failure, the remaining node needs capacity for the load of both nodes.
• Replication at intervals: without three nodes for Ceph, we usually recommend ZFS replication for compact setups, which syncs at fixed intervals; after a failover, Proxmox notes, data since the last sync may be lost. See our Ceph vs ZFS comparison.
• Discouraged for odd node counts: there the QDevice would receive n − 1 votes and, according to Proxmox, act almost as a single point of failure; the vendor currently advises against it.

Two-node setups are possible with an external quorum device. For business-critical applications, we recommend three or more nodes.

Clusters across two sites: the deciding vote belongs at a third location

If you spread a cluster's nodes evenly across two sites, neither side holds a majority after a split. Four nodes, two per site, need three votes; each side has only two. Both sides lose quorum, and nodes running highly available VMs fence themselves. The fix is an arbitrator at a third site that gives its vote to exactly one side.

In Proxmox VE, the QDevice plays that role. With four nodes it adds the fifth vote, so the majority becomes three of five. When the sites split, the Corosync documentation states that the side with the most active nodes gets the vote; in a tie, the tiebreaker decides last. If an entire site fails, the other site also reaches three votes with the QDevice.

The third location is what makes this work: if the QDevice stood at one of the two sites, it would fail along with it, leaving the other side with two of five votes and no quorum. The same principle applies to any even node count split evenly across two sites.

The same principle applies to distributed storage. For two data centers, the Ceph documentation has each site hold all of the data. A tiebreaker monitor in a third zone decides which side carries on when the sites lose contact.

How we implement quorum and fencing in a managed Proxmox cluster

By default, our managed Proxmox clusters consist of three nodes, which form quorum through Corosync. Quorum planning and split brain protection are part of the written redundancy plan we prepare during the design phase. Before go-live, we run a full failover test, followed by regular tests documented in a written log.

• N+1 design: the remaining nodes have headroom for the VMs of a failed node
• Fencing before restart: the HA Manager restarts VMs on another node only afterwards
• Ceph with three replicas of every block on different nodes, standard in Essentials and Business
• Redundant network: bonding/LACP per node, redundant switches, separate networks for management, storage and public traffic
• 24/7 monitoring with on-call alerting and a post-incident report on cause and remediation
• Written documentation of architecture, configuration and emergency plan, which belongs to you

If an environment is to span more than one site, the location of the deciding vote belongs in that plan from day one. We describe the process in high availability cluster design and setup and failover testing and emergency runbooks.