Quorum/Witness Server Support Package for LifeKeeper
Feature Summary
The Quorum/Witness Server Support Package for LifeKeeper (steeleye-lkQWK, hereinafter “Quorum/Witness Package”) combined with the existing failover process of the LifeKeeper core allows system failover to occur with a greater degree of confidence in situations where total network failure could be common. This effectively means that local site failovers and failovers to nodes across a WAN can be done while greatly reducing the risk of split-brain situations.
In a distributed system that takes network partitioning into account, there is a concept called quorum to obtain consensus across the cluster. A node having quorum is a node that can obtain consensus of all the clusters and is allowed to bring resources in service. On the other hand, a node not having quorum is a node that cannot obtain consensus of all the clusters and it is not allowed to bring resources in service. This will prevent split brain from happening. To check whether a node has quorum is called quorum check. It is expressed as “quorum check succeeded” if it has quorum, and “quorum check failed” if it does not have quorum.
In case of a communication failure, using one node where failure occurred and another multiple nodes (or other devices) will allow a node to get a “second opinion” on the status of the failing node. The node to get a “second opinion” is called a witness node (or a witness device), and getting a “second opinion” is called witness checking. When determining when to fail over, the witness node (the witness device) allows resources to be brought in service on a backup server only in cases where it verifies the primary server has failed and is no longer part of the cluster. This will prevent failovers from happening due to simple communication failures between nodes when those failures don’t affect the overall access to, and performance of, the in-service node. During actual operation, the witness node (the witness device) will be consulted when LifeKeeper is started or the failed communication path is restored. Witness checking can only be performed for nodes having quorum.
Package Installation and Configuration
The Quorum/Witness Server Support Package for LifeKeeper will need to be installed on every node in the cluster that uses quorum/witness functionality, including a witness-only node. The only configuration requirement for the witness node is to create appropriate comm paths. When using a quorum mode with tcp_remote, LifeKeeper does not need to be installed on the host which was set as QUORUM_MODE in /etc/default/LifeKeeper configuration file.
The general process for setting up quorum/witness functionality will involve the following steps:
- Set up the server and make sure that it can communicate with other servers.
- Install LifeKeeper on the server. During the installation, enable “Use Quorum / Witness functions” with the setup command and install the quorum/witness package as well.
- Create appropriate communication paths between the nodes including witness-only nodes.
- Configure quorum/witness.
When the above steps are completed, the quorum/witness functions will be activated in the cluster and quorum checking and witness checking will be performed before failovers are allowed.
See the Configurable Components section below for additional configuration options.
Configurable Components
The quorum/witness package contains two configurable modes: quorum and witness. By default, installing the quorum/witness package will enable both quorum and witness modes.
The behavior of these modes can be customized via the /etc/default/LifeKeeper configuration file, and the quorum and witness modes can be individually adjusted. The package installs default settings into the configuration file when it is installed, majority being the default quorum mode and remote_verify being the default witness mode. An example is shown below:
QUORUM_MODE=majority
WITNESS_MODE=remote_verify
Available Quorum Modes
Four quorum checking modes are available which can be set via the QUORUM_MODE setting in /etc/default/LifeKeeper.
| majority (default) | With majority as the quorum mode setting quorum checks occur via SPS for Linux communication paths. A node has quorum when it is able to communicate with the majority of the nodes in the cluster. This quorum mode is available on clusters with three or more nodes. A witness – only node needs to be added when using a two-node configuration. See “majority mode” for details. |
| tcp_remote | Checks the connection to the TCP/IP service on the specified port for the host independent from the communication path. It is determined that the node has quorum when it is able to communicate with the majority of the nodes in the cluster. A host for connection checking is required separately. See “tcp_remote mode” for details. |
| storage | With storage as the quorum mode setting quorum checks occur using a “shared storage” device. A node has quorum when it is able to access the shared storage device and update its own quorum object. A cluster is considered to have quorum consensus with this mode when each node is able to access the shared storage device This mode can be used for 2, 3 or 4 node clusters. Shared storage devices can be a block storage device shared to all nodes, a file accessed via a NFS share on all nodes or an Amazon S3 storage object. The “shared storage” used for this solution must be obtained separately. See “storage mode” for details. When storage is selected for the quorum mode, the witness mode, which is describe later, must also be set to storage. |
| none/off | Quorum checking is disabled. With this configuration, quorum checking is always determined to be successful. |
Available Witness Modes
Three witness modes are available which can be set via the WITNESS_MODE setting in /etc/default/LifeKeeper.
| remote_verify (default) | Consults all the other nodes in the cluster about their view of the status of a node which appears to be failing. If any node determines that there is no failure, witness checking determines that there is no failure. If all the nodes determine that there is failure, witness checking determines that the node is failing. |
| storage | A witness mode where shared storage is used as a witness device. The share storage device is used to “share” status information between nodes in the cluster. Each node updates its own information and reads the other nodes information. If a node detects that information for another node is not being updated then that node will be considered failed. See “storage mode” for details. When storage is selected for the witness mode, then storage must be selected for quorum mode. See above. |
| none/off | In this mode, witness checking is disabled. With this setting, it is always determined that there is no failure. |
Supported Combinations of a Quorum Mode and Witness Mode
LifeKeeper supports the following combinations.
| Supported 3 or more nodes |
Supported 3 or more nodes |
Not supported | Supported 3 or more nodes |
||
| Not Supported | Not Supported | Supported Between 2 and 4 nodes |
Not supported | ||
| Supported 3 or more nodes |
Supported 2 or more nodes |
Not supported | Supported | ||
Available Actions When Quorum is Lost
The quorum/witness package offers three different options for how the system should react if quorum is lost — “fastboot”, “fastkill” and “osu”. These options can be selected via the QUORUM_LOSS_ACTION setting in /etc/default/LifeKeeper. All three options take the system’s resources out of service; however, they each allow a different behavior.
| fastboot | The system will be immediately rebooted when a loss of quorum is detected (from a communication path failure). Although this is an aggressive option, it ensures that the system will be disconnected from any external resources right away. In many cases, such as with storage-level replication, this immediate release of resources is desired. Two important notes on this option are:
|
| fastkill (default) | The fastkill option is very similar to the fastboot option, but instead of a hard reboot, the system will immediately halt when quorum is lost. As with the fastboot option, no tasks are performed (disk syncing, etc.), and the system will then need to be manually started and will come back up performing normal startup routines, including negotiating storage and resource access, etc. |
| osu | This is the least aggressive option, leaving the system operational but taking resources out of service on the system where quorum is lost. In some cluster configurations, this is all that is needed, but it may not be strong enough or fast enough in others. |
Post your comment on this topic.