A Node in a DataKeeper Cluster Has Become Permanently Unreachable

Evict a Node from a DataKeeper Cluster that has become Permanently Unreachable

Symptom

‘delallsys.ps1’ is a support script that can be used to evict a node from a DataKeeper Cluster that has become ‘permanently unreachable’. If a node in a DataKeeper cluster becomes permanently unreachable, this script can be used to delete all DataKeeper Job information about that node on the remaining nodes in the cluster.

Normal mirror deletion is encouraged when possible in DataKeeper. This support script is used as a last resort when it comes to stabilizing a DataKeeper Cluster’s State with a permanently unreachable node.

Solution

This script should be run individually on each remaining DataKeeper Cluster node when utilized. It evicts a node in relation to DataKeeper and DataKeeper Mirrors.

*Note: If a High-Availability (HA) product is being used as well, such as LifeKeeper for Windows or Windows Server Failover Cluster, additional steps will need to be taken to evict the node for the respective software.

This script should not be utilized if restoration of the node is possible.

!If a node is restored and comes back online after being previously evicted by delallsys.ps1, run the following four cleanup commands on the previously evicted node to refresh its state before attempting to reconnect with the surviving cluster.

%extmirrbase%\Support\Cleanupmirror.cmd <volume letter>
%extmirrbase%\emcmd.exe . GETJOBINFO
%extmirrbase%\emcmd.exe . DELETEJOB <JOB ID>  // obtained from running the GETJOBINFO command.
%extmirrbase%\emcmd.exe . UNLOCKVOLUME <volume letter>

This script should be used to clear the state of the restored node before attempting to re-create a DataKeeper mirror after using this command.

Notes:

• This is not the normal way of deleting a DataKeeper mirror.
• This is a ‘last resort’ script to use if a node becomes permanently lost.
• If this is a 1×1×1 or 2×1 or 1×1×1×1 cluster (any multi-server cluster configuration) then all machines configured in the cluster need to run the delallsys.ps1 command.
• If a node that was evicted with this script does manage to come back online, there are additional commands required to refresh the previously evicted node’s state before it is able to successfully reconnect.

‘delallsys.ps1’ Script

NAME
    %extmirrbase%\Support\delallsys.ps1
SYNOPSIS
    This script will safely evict a node from a DataKeeper Cluster
SYNTAX
    %extmirrbase%\Support\delallsys.ps1 [-sys] <String> [-execute] [<CommonParameters>]
DESCRIPTION
    This script will safely evict a node from a DataKeeper Cluster by handling the removal of DataKeeper mirrors.
    -sys <name-of-system-to-evict>  Name of system to be evicted
    -execute                        Execute the removal of the node from the cluster
    -verbose                        Enables additional logging to the terminal
PARAMETERS
    -sys <String>
        Required?                    true
        Position?                    1
        Default value                None
    -execute [<SwitchParameter>]
        Required?                    false
        Position?                    named
        Default value                False
    -------------------------- EXAMPLE 1 --------------------------
    delallsys.ps1 -sys <name-of-system-to-evict>
    -------------------------- EXAMPLE 2 --------------------------
    delallsys.ps1 -sys <name-of-system-to-evict> -execute
    -------------------------- EXAMPLE 3 --------------------------
    delallsys.ps1 -sys <name-of-system-to-evict> -execute -verbose