Test Switchover and Failover

In this section we will perform basic tests to verify the expected behavior of the SAP-SPS_ASCS10 and SAP-SPS_ERS20 resource hierarchies on switchover and failover. It is important to test that the enqueue server process in the ASCS10 instance is able to successfully recover the enqueue lock table from the ERS20 instance after switchover or failover.

1. Verify that the SAP-SPS_ASCS10 resource state is currently Active on node-a and Standby on node-b, and that the SAP-SPS_ERS20 resource state is currently Active on node-b and Standby on node-a.

2. Execute the following commands to verify that the ASCS10 and ERS20 instances are running successfully on node-a and node-b, respectively:

[root@node-a ~]# su - spsadm -c "sapcontrol -nr 10 -function GetProcessList"
04.03.2021 20:24:12
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
msg_server, MessageServer, GREEN, Running, 2020 12 21 16:53:00, 1755:31:12, 11497
enq_server, Enqueue Server 2, GREEN, Running, 2020 12 21 16:53:00, 1755:31:12, 11498

[root@node-b ~]# su - spsadm -c "sapcontrol -nr 20 -function GetProcessList"
04.03.2021 20:24:22
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
enq_replicator, Enqueue Replicator 2, GREEN, Running, 2021 02 22 16:55:17, 243:29:05, 30028

3. Execute the following command on node-a to write 100 exclusive non-cumulative locks labeled 0-99 to the lock table maintained by the enqueue server.
   

[root@node-a ~]# su - spsadm -c "enq_admin --set_locks=100:X:DIAG::TAB:%u pf=/usr/sap/SPS/SYS/profile/SPS_ASCS10_sps-ascs"
Enqueue Server 2

2021-03-04 20:32:16; OK; 'Set Locks'; Response=41496 usec
==============================================================

Execute the following commands to verify that the locks have been successfully stored in the lock table on node-a and replicated to the enqueue replication server on node-b:

[root@node-a ~]# su - spsadm -c "sapcontrol -nr 10 -function EnqGetStatistic" | grep locks_now
locks_now: 100

[root@node-b ~]# su - spsadm -c "sapcontrol -nr 20 -function EnqGetStatistic" | grep locks_now
locks_now: 100

4. Perform a switchover of the ASCS resource hierarchy by right-clicking the SAP-SPS_ASCS10 resource on node-b and choosing the In-Service… operation. Click In Service to begin the switchover.
   

Once the ASCS resource hierarchy has successfully come in-service on node-b and the enqueue server process has obtained the copy of the backup enqueue lock table from the enqueue replication server process, LifeKeeper will automatically relocate the SAP-SPS_ERS20 resource to node-a to provide lock table redundancy across cluster nodes.

5. Once LifeKeeper has relocated the SAP-SPS_ERS20 resource back to node-a, execute the following commands to verify that the ASCS10 and ERS20 instances are running successfully on node-b and node-a, respectively, and that they both still hold the 100 locks written in step 3.

[root@node-a ~]# su - spsadm -c "sapcontrol -nr 20 -function GetProcessList"
04.03.2021 20:58:57
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
enq_replicator, Enqueue Replicator 2, GREEN, Running, 2021 03 04 20:57:34, 0:01:23, 21967

[root@node-a ~]# su - spsadm -c "sapcontrol -nr 20 -function EnqGetStatistic" | grep locks_now
locks_now: 100

[root@node-b ~]# su - spsadm -c "sapcontrol -nr 10 -function GetProcessList"
04.03.2021 20:56:56
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
msg_server, MessageServer, GREEN, Running, 2021 03 04 20:54:47, 0:02:09, 17074
enq_server, Enqueue Server 2, GREEN, Running, 2021 03 04 20:54:47, 0:02:09, 17075

[root@node-b ~]# su - spsadm -c "sapcontrol -nr 10 -function EnqGetStatistic" | grep locks_now
locks_now: 100

6. Execute the following command to forcefully reboot node-b:
   

[root@node-b ~]# echo b > /proc/sysrq-trigger

Once LifeKeeper has detected that node-b has been powered off, the status of node-b updates to “Unknown” in the LifeKeeper GUI and LifeKeeper initiates failover of the SAP-SPS_ASCS10 resource hierarchy back to node-a.

Once node-b is back online, LifeKeeper will automatically relocate the SAP-SPS_ERS20 resource hierarchy back to node-b. Once this process is complete, the SAP-SPS_ASCS10 and SAP-SPS_ERS20 resource hierarchies will be back in-service on node-a and node-b, respectively.

7. Execute the sapcontrol commands given in steps 2 and 3 again to verify the expected state on each node.

8. Execute the following command on node-a to release the 100 locks that were written in step 3:

[root@node-a ~]# su - spsadm -c "enq_admin --release_locks=100:X:DIAG::TAB:%u pf=/usr/sap/SPS/SYS/profile/SPS_ASCS10_sps-ascs"
Enqueue Server 2

2021-03-04 21:10:22; OK; 'Release Locks'; Response=36883 usec
===============================================================

We have now verified the basic switchover and failover functionality of the ASCS and ERS resource hierarchies.