STONITH in VMware vSphere Environments

ESXCLI is a command-line interface supported by Broadcom for managing virtual infrastructure, including ESXi hosts and virtual machines. You can select appropriate ESXCLI commands to implement LifeKeeper STONITH functionality between VMware virtual machines.
In this example, the VM to be powered off is lk-node1, and the VM to execute STONITH is lk-node2.

Requirements

Package

• ESXCLI package version 8 (Note: For vSphere 8 or later, the vSphere CLI package and vSphere SDK for Perl package are no longer supported). *Please refer to Broadcom’s ESXCLI.

Virtual machines to be monitored

• VMware Tools

Preliminary Verification

Confirm the procedure for powering off a VM using the esxcli command.
The following parameters are used when executing the esxcli command. Information other than <world_id> must be collected in advance.

The steps required to power off the VM are as follows.

1. Retrieve a list of running VMs managed by <esxi_host> to obtain the World ID of the target VM.
   

   # esxcli ––server=<esxi_host> ––username=<username> ––password=<password> --thumbprint=<thumbprint> vm process list

2. From the retrieved VM list, identify the World ID of the VM you want to power off. For example, if targeting lk-node1, the information for this VM would be displayed as follows (values are examples):
   

   lk-node1
      World ID: 2699949
      Process ID: 0
      VMX Cartel ID: 2699948
      UUID: 42 25 89 28 3d 58 94 0c-74 bc fd 21 6a 01 bc 32
      Display Name: lk-node1
      Config File: /vmfs/volumes/5305d4dc-5c59219f-b2a1-2c44fd8341b4/lk-node1/lk-node1.vmx

   
   The displayed World ID is used in the following command.

3. Power off the target VM using the identified World ID.
   

   # esxcli ––server=<esxi_host> ––username=<username> ––password=<password> --thumbprint=<thumbprint> vm process kill ––type=hard ––world-id=<world_id>

   
   Confirm that the VM with <world_id> has been powered off.

Installing STONITH

Execute the following command to install the LifeKeeper STONITH scripts on each server where LifeKeeper is installed and communication paths are configured to all servers.

# /opt/LifeKeeper/samples/STONITH/stonith-install

Creating esxcli-stonith.pl

To simplify the implementation when extracting the VM World ID, create esxcli-stonith.pl using Perl. Create the following STONITH script and save it under /usr/local/bin/esxcli-stonith.pl.

#!/usr/bin/perl
$ENV{'HOME'} = `cd ~; pwd`;
chomp($ENV{'HOME'});
my $esxcli_cmd="esxcli --server=$ARGV[0] --thumbprint=$ARGV[1] --username=$ARGV[2] --password=$ARGV[3]";
my $vm_process_list=`$esxcli_cmd vm process list`;
$vm_process_list =~ /[\n]*$ARGV[4]\n\s+World\s+ID:\s*(\d+)\n/s;
my $world_id = $1;
`$esxcli_cmd vm process kill --type=hard --world-id=$world_id`;

Notes:

• When executed via LifeKeeper, the esxcli command fails because the environment variable HOME is not configured. For this reason, it is configured in the script.
  
  
• The World ID changes when the VM is restarted. It is necessary to dynamically extract the World ID when STONITH is executed.
  
• The script uses regular expressions to extract the World ID, but if the VM name contains characters other than alphabets, numbers, and hyphens (“-”), accurate extraction may not be possible. In such cases, please modify the logic to ensure proper extraction of the World ID.
  
  

Grant execution permission.

# chmod 755 /usr/local/bin/esxcli-stonith.pl

Updating /opt/LifeKeeper/config/stonith.conf

Assuming the cluster nodes are lk-node1 and lk-node2, configure stonith.conf on each node as follows:

lk-node1 /usr/local/bin/esxcli-stonith.pl <esxi_host(has lk-node1)> <thumbprint> <username> <password> lk-node1
lk-node2 /usr/local/bin/esxcli-stonith.pl <esxi_host(has lk-node2)> <thumbprint> <username> <password> lk-node2