Hook – Why an HA-Cluster without Quorum is like a Tower of Babel
Imagine building a house of cards. Each card represents a node in your Proxmox High Availability (HA) cluster. If one card tumbles (a node fails), the whole structure becomes unstable. Now, imagine those remaining cards start debating amongst themselves who is actually in charge of holding up the remaining cards. This is precisely what happens in a Proxmox HA cluster when quorum isn't correctly configured: a 'split-brain' scenario. This leads to data loss, inconsistent states, and sleepless nights. This article guides you on how to stabilize your cluster's foundation, set up quorum correctly, and effectively eliminate split-brain issues, using practical commands and configuration examples.
What is a Proxmox HA-Cluster and Why is Quorum its Heart?
A High Availability (HA) cluster in Proxmox Virtual Environment (PVE) is designed to ensure that your virtual machines (VMs) and containers (CTs) remain accessible even if one or more physical nodes in the cluster fail. It achieves this by distributing resources and management across multiple nodes. The core technology enabling this coordination is Corosync, which uses a consensus algorithm to ensure all nodes agree on the cluster's state. Quorum is the mechanism that determines whether enough nodes are communicating to make decisions. In a cluster, a majority of nodes must agree to form a quorum. If a node loses communication with the majority, it assumes it's in a split-brain state and ceases operations to prevent data corruption. This is crucial: a cluster needs a clear leader, or at least a clear consensus, to function correctly. Without it, nodes might act independently, leading to conflicting actions and data divergence.
Understanding Split-Brain Scenarios
A split-brain situation occurs when network partitioning isolates a subset of nodes from the rest of the cluster. Each isolated partition believes it is the only valid part of the cluster and can continue to operate, potentially making conflicting decisions about the same resources. For example, if a VM is running on Node A and Node B gets isolated, Node B might think Node A is down and try to start the VM on itself. If Node A is still running the VM, you now have two instances of the same VM running, leading to data corruption when they try to write to shared storage. This is catastrophic for data integrity and application availability. The primary goal of quorum is to prevent this by ensuring that only a group of nodes that can communicate with a majority of the cluster can continue operating. If a node cannot reach a majority, it must shut down its HA services to avoid becoming part of a minority partition.
The Role of Corosync and Quorum Configuration
Corosync is the underlying cluster communication layer used by Proxmox VE. It manages node membership and ensures reliable message passing. Quorum is a critical parameter within Corosync's configuration. For a cluster to function, a majority of nodes must be able to communicate. In a typical setup with an odd number of nodes (e.g., 3), quorum is simply `(N/2) + 1`. So, in a 3-node cluster, 2 nodes are needed for quorum. In a 5-node cluster, 3 nodes are needed.
However, this simple majority rule can be problematic in specific scenarios, especially with an even number of nodes or when network failures are common. For instance, in a 2-node cluster, each node needs the other to operate, meaning if one fails, the cluster has no quorum and stops. This defeats the purpose of HA. To address this, Proxmox VE allows for specific quorum configurations, including the use of a 'vote disk' or a 'quorum device'.
Configuring Quorum: The VoteDisk Method
The vote disk, or quorum device, is a shared storage resource (like an iSCSI LUN, a small NAS share, or even a dedicated block device) that all cluster nodes can access. Corosync uses this device to maintain a token, ensuring that only one partition of the cluster can hold the token at any given time. This effectively breaks ties in even-numbered clusters or scenarios where network partitions might occur.
Steps to configure a vote disk:
- Prepare the Vote Disk: Create a small, shared storage LUN or disk image (e.g., 1GB is usually sufficient). Ensure it's accessible by all nodes that will participate in the HA cluster.
- Configure Corosync: Edit the Corosync configuration file, typically located at `/etc/pve/corosync.conf`. You need to add the `quorum.device` parameter, specifying the path to your vote disk.
Here’s an example snippet for `/etc/pve/corosync.conf`:
quorum {
provider: corosync_votequorum
device: /path/to/your/vote/disk
}
Replace `/path/to/your/vote/disk` with the actual device path (e.g., `/dev/sdX1` or an iSCSI device path).
Important considerations for the vote disk:
- Accessibility: All nodes must be able to read and write to it.
- Exclusivity: Only one node should actively write to it at any given moment. Corosync handles this by managing a token on the device.
- Reliability: The storage hosting the vote disk must be highly available itself. If the vote disk becomes unavailable, the entire cluster may lose quorum.
Alternative: The No-Quorum-Device Option (Use with Caution)
In some specific, often small, setups, you might consider disabling the quorum device. This is generally not recommended for production environments, especially for clusters with an even number of nodes, as it significantly increases the risk of split-brain. However, Proxmox allows you to configure `quorum.votes: 1` and `no_quorum_policy: ignore` in `corosync.conf`. The `no_quorum_policy: ignore` setting tells Corosync to continue operating even if quorum is lost. This is dangerous because it allows all nodes to remain active, even when partitioned, leading to split-brain. This setting is typically only suitable for specific testing or lab environments where data consistency is not paramount or where manual intervention is guaranteed upon network failure.
Practical Steps for Configuration and Verification
After making changes to `/etc/pve/corosync.conf`, you must reload the Corosync service for the changes to take effect. This is typically done via the Proxmox VE web interface (Datacenter -> Cluster -> Corosync configuration -> Reload) or by manually restarting the Corosync service on all nodes (though this can briefly interrupt cluster communication). It's best to perform this during a maintenance window.
Verification:
- Check cluster status: Use the command `pvecm status` on any node. This will show the current quorum status, active nodes, and configuration details. Look for the `Quorum: [...]` line.
- Monitor logs: Regularly check `/var/log/syslog` and `/var/log/corosync/corosync.log` for any Corosync-related errors or warnings, especially regarding quorum loss or network issues.
If you have configured a vote disk, ensure that `pvecm status` reports that quorum is achieved and that the vote disk is actively being used. If you encounter issues, review your `/etc/pve/corosync.conf` file for syntax errors and verify network connectivity between all nodes, including any necessary firewall rules.
When to Use an Odd Number of Nodes
The simplest and most robust way to avoid quorum issues, especially in smaller clusters, is to always use an odd number of nodes. A 3-node cluster is the minimum recommended for true HA. In a 3-node setup, two nodes are required for quorum. If one node fails, the remaining two can still form a quorum. If two nodes fail, the cluster loses quorum and stops accepting HA actions, but the remaining active VM on the single surviving node continues to run. This is a much safer default than trying to manage quorum in an even-numbered cluster without a vote disk.
Conclusion: Stability Through Consensus
Configuring quorum correctly is not an optional step for Proxmox HA clusters; it is fundamental to their stability and reliability. A split-brain scenario is the nightmare of any cluster administrator, leading to data loss and prolonged downtime. By understanding the role of Corosync, implementing a vote disk for even-numbered clusters, and always preferring an odd number of nodes where possible, you build a resilient infrastructure. Treat your quorum configuration with the seriousness it deserves – it's the silent guardian that prevents chaos when the network flickers.