Test Switchover and Failover

In this section we will perform basic tests to verify the expected behavior of the HANA-SPS_HDB00 resource hierarchy on switchover and failover.

1. Verify that the HANA-SPS_HDB00 resource state is currently “Active” on node-a and “Standby – In-Sync” on node-b.

2. Execute the following commands on node-a and node-b to verify that the HDB00 instance is running successfully on both nodes, node-a and node-b are both registered appropriately in SAP HANA System Replication, and that a connection to the primary database can be established through the virtual hostname sps-hana. The replication and operation modes should be primary/primary and sync/logreplay on node-a and node-b, respectively.

*In a Google Cloud deployment, the steps given in the Disable IP Forwarding section of Google Cloud – Using an Internal Load Balancer must be implemented in order to allow connections from node-b to the primary database instance on node-a using the virtual host name sps-hana.

[root@node-a ~]# sudo -iu spsadm sapcontrol -nr 00 -function GetProcessList
04.03.2021 21:31:50
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2021 03 04 19:44:27, 1:47:23, 19236
hdbcompileserver, HDB Compileserver, GREEN, Running, 2021 03 04 19:45:46, 1:46:04, 19618
hdbindexserver, HDB Indexserver-SPS, GREEN, Running, 2021 03 04 19:45:47, 1:46:03, 19666
hdbnameserver, HDB Nameserver, GREEN, Running, 2021 03 04 19:44:27, 1:47:23, 19254
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2021 03 04 19:45:46, 1:46:04, 19621
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2021 03 04 19:46:15, 1:45:35, 20241
hdbxsengine, HDB XSEngine-SPS, GREEN, Running, 2021 03 04 19:45:47, 1:46:03, 19669

[root@node-a ~]# sudo -iu spsadm hdbnsutil -sr_state
System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: SiteA

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false

Host Mappings:
~~~~~~~~~~~~~~

node-a -> [SiteB] node-b
node-a -> [SiteA] node-a

Site Mappings:
~~~~~~~~~~~~~~
SiteA (primary/primary)
    |---SiteB (sync/logreplay)

Tier of SiteA: 1
Tier of SiteB: 2


Replication mode of SiteA: primary
Replication mode of SiteB: sync

Operation mode of SiteA: primary
Operation mode of SiteB: logreplay

Mapping: SiteA -> SiteB
done.

[root@node-b ~]# sudo -iu spsadm sapcontrol -nr 00 -function GetProcessList
04.03.2021 21:38:10
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2021 03 04 19:48:50, 1:49:20, 13383
hdbcompileserver, HDB Compileserver, GREEN, Running, 2021 03 04 19:49:00, 1:49:10, 13747
hdbindexserver, HDB Indexserver-SPS, GREEN, Running, 2021 03 04 19:49:00, 1:49:10, 13794
hdbnameserver, HDB Nameserver, GREEN, Running, 2021 03 04 19:48:51, 1:49:19, 13401
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2021 03 04 19:49:00, 1:49:10, 13750
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2021 03 04 19:49:42, 1:48:28, 15167
hdbxsengine, HDB XSEngine-SPS, GREEN, Running, 2021 03 04 19:49:00, 1:49:10, 13797

[root@node-b ~]# sudo -iu spsadm hdbsql -n sps-hana -i 00 -u SYSTEM -p <SYSTEM user password> -d SPS '\s'
host          : sps-hana:30013
sid           : SPS
dbname        : SPS
user          : SYSTEM
kernel version: 2.00.046.00.1581325702
SQLDBC version:        libSQLDBCHDB 2.04.182.1579711187
autocommit    : ON
locale        : en_US.UTF-8
input encoding: UTF8
sql port      : node-a:30015

3. Perform a switchover of the SAP HANA resource hierarchy by right-clicking the HANA-SPS_HDB00 resource on node-b and choosing the In-Service… operation. Click In Service to begin the switchover.
   

Allow time for the switchover to complete and for the resource states to switch to “Standby – In Sync” on node-a and “Active” on node-b. Note that it may take up to two minutes for the resource state to transition from “Standby – HDB Running” to “Standby – In-Sync” after the standby database instance is running.

4. Execute the following commands on node-a and node-b to verify that the HDB00 instance is running successfully on both nodes, node-a and node-b are both registered appropriately in SAP HANA System Replication, and that a connection to the primary database can be established through the virtual hostname sps-hana. The replication and operation modes should now be sync/logreplay and primary/primary on node-a and node-b, respectively.

[root@node-a ~]# sudo -iu spsadm sapcontrol -nr 00 -function GetProcessList
04.03.2021 21:50:20
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2021 03 04 21:43:51, 0:06:29, 156845
hdbcompileserver, HDB Compileserver, GREEN, Running, 2021 03 04 21:43:58, 0:06:22, 156970
hdbindexserver, HDB Indexserver-SPS, GREEN, Running, 2021 03 04 21:43:58, 0:06:22, 157101
hdbnameserver, HDB Nameserver, GREEN, Running, 2021 03 04 21:43:52, 0:06:28, 156864
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2021 03 04 21:43:58, 0:06:22, 156973
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2021 03 04 21:44:36, 0:05:44, 157429
hdbxsengine, HDB XSEngine-SPS, GREEN, Running, 2021 03 04 21:43:58, 0:06:22, 157104

[root@node-a ~]# sudo -iu spsadm hdbsql -n sps-hana -i 00 -u SYSTEM -p <SYSTEM user password> -d SPS '\s'
host          : sps-hana:30013
sid           : SPS
dbname        : SPS
user          : SYSTEM
kernel version: 2.00.046.00.1581325702
SQLDBC version:        libSQLDBCHDB 2.04.182.1579711187
autocommit    : ON
locale        : en_US.UTF-8
input encoding: UTF8
sql port      : node-b:30015

[root@node-b ~]# sudo -iu spsadm sapcontrol -nr 00 -function GetProcessList
04.03.2021 21:52:36
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2021 03 04 19:48:50, 2:03:46, 13383
hdbcompileserver, HDB Compileserver, GREEN, Running, 2021 03 04 19:49:00, 2:03:36, 13747
hdbindexserver, HDB Indexserver-SPS, GREEN, Running, 2021 03 04 19:49:00, 2:03:36, 13794
hdbnameserver, HDB Nameserver, GREEN, Running, 2021 03 04 19:48:51, 2:03:45, 13401
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2021 03 04 19:49:00, 2:03:36, 13750
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2021 03 04 19:49:42, 2:02:54, 15167
hdbxsengine, HDB XSEngine-SPS, GREEN, Running, 2021 03 04 19:49:00, 2:03:36, 13797

[root@node-b ~]# sudo -iu spsadm hdbnsutil -sr_state
System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: SiteB

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false

Host Mappings:
~~~~~~~~~~~~~~

node-b -> [SiteB] node-b
node-b -> [SiteA] node-a

Site Mappings:
~~~~~~~~~~~~~~
SiteB (primary/primary)
    |---SiteA (sync/logreplay)

Tier of SiteB: 1
Tier of SiteA: 2

Replication mode of SiteB: primary
Replication mode of SiteA: sync

Operation mode of SiteB: primary
Operation mode of SiteA: logreplay

Mapping: SiteB -> SiteA
done.

The following are examples of how the above commands would look if using ‘su’ in place of ‘sudo’.

su - spsadm -c "sapcontrol -nr 00 -function GetProcessList"
su - spsadm -c "hdbnsutil -sr_state"
su - spsadm -c "hdbsql -n sps-hana -i 00 -u SYSTEM -p <SYSTEM user password> -d SPS '\s'"

5. Execute the following command to forcefully reboot node-b:
   

[root@node-b ~]# echo b > /proc/sysrq-trigger

Once LifeKeeper has detected that node-b has been powered off, the status of node-b updates to “Unknown” in the LifeKeeper GUI.

At this point, LifeKeeper automatically initiates failover of the HANA-SPS_HDB00 resource hierarchy back to node-a.

Once node-b is back online, LifeKeeper will automatically re-register it as a secondary replication site in SAP HANA System Replication and restart the secondary database instance. This process may take several minutes to complete. Once this process is complete, the resource states will transition back to “Active” on node-a and “Standby – In-Sync” on node-b.

6. Execute the commands given in step 2 again to verify that the database is functioning as expected on both nodes.

We have now verified the basic switchover and failover functionality of the SAP HANA resource hierarchy.