Test Switchover and Failover

In this section we will perform basic tests to verify the expected behavior of the HANA-SPS_HDB00 resource hierarchy on switchover and failover.

1. Verify that the HANA-SPS_HDB00 resource state is currently “Active” on node-a and “Standby – In-Sync” on node-b.

2. Execute the following commands on node-a and node-b to verify that the HDB00 instance is running successfully on both nodes, node-a and node-b are both registered appropriately in SAP HANA System Replication, and that a connection to the primary database can be established through the virtual hostname sps-hana. The replication and operation modes should be primary/primary and sync/logreplay on node-a and node-b, respectively.

[root@node-a ~]# su - spsadm -c "sapcontrol -nr 00 -function GetProcessList"
04.03.2021 21:31:50
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2021 03 04 19:44:27, 1:47:23, 19236
hdbcompileserver, HDB Compileserver, GREEN, Running, 2021 03 04 19:45:46, 1:46:04, 19618
hdbindexserver, HDB Indexserver-SPS, GREEN, Running, 2021 03 04 19:45:47, 1:46:03, 19666
hdbnameserver, HDB Nameserver, GREEN, Running, 2021 03 04 19:44:27, 1:47:23, 19254
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2021 03 04 19:45:46, 1:46:04, 19621
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2021 03 04 19:46:15, 1:45:35, 20241
hdbxsengine, HDB XSEngine-SPS, GREEN, Running, 2021 03 04 19:45:47, 1:46:03, 19669

[root@node-a ~]# su - spsadm -c "hdbnsutil -sr_state"
System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 1
site name: SiteA

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false

Host Mappings:
~~~~~~~~~~~~~~

node-a -> [SiteB] node-b
node-a -> [SiteA] node-a

Site Mappings:
~~~~~~~~~~~~~~
SiteA (primary/primary)
    |---SiteB (sync/logreplay)

Tier of SiteA: 1
Tier of SiteB: 2


Replication mode of SiteA: primary
Replication mode of SiteB: sync

Operation mode of SiteA: primary
Operation mode of SiteB: logreplay

Mapping: SiteA -> SiteB
done.

[root@node-b ~]# su - spsadm -c "sapcontrol -nr 00 -function GetProcessList"
04.03.2021 21:38:10
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2021 03 04 19:48:50, 1:49:20, 13383
hdbcompileserver, HDB Compileserver, GREEN, Running, 2021 03 04 19:49:00, 1:49:10, 13747
hdbindexserver, HDB Indexserver-SPS, GREEN, Running, 2021 03 04 19:49:00, 1:49:10, 13794
hdbnameserver, HDB Nameserver, GREEN, Running, 2021 03 04 19:48:51, 1:49:19, 13401
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2021 03 04 19:49:00, 1:49:10, 13750
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2021 03 04 19:49:42, 1:48:28, 15167
hdbxsengine, HDB XSEngine-SPS, GREEN, Running, 2021 03 04 19:49:00, 1:49:10, 13797

[root@node-b ~]# su - spsadm -c "hdbsql -n sps-hana -i 00 -u SYSTEM -p <SYSTEM user password> -d SPS '\s'"
host          : sps-hana:30013
sid           : SPS
dbname        : SPS
user          : SYSTEM
kernel version: 2.00.046.00.1581325702
SQLDBC version:        libSQLDBCHDB 2.04.182.1579711187
autocommit    : ON
locale        : en_US.UTF-8
input encoding: UTF8
sql port      : node-a:30015

3. Perform a switchover of the SAP HANA resource hierarchy by right-clicking the HANA-SPS_HDB00 resource on node-b and choosing the In-Service… operation. Click In Service to begin the switchover.
   

Allow time for the switchover to complete and for the resource states to switch to “Standby – In Sync” on node-a and “Active” on node-b. Note that it may take up to two minutes for the resource state to transition from “Standby – HDB Running” to “Standby – In-Sync” after the standby database instance is running.

4. Execute the following commands on node-a and node-b to verify that the HDB00 instance is running successfully on both nodes, node-a and node-b are both registered appropriately in SAP HANA System Replication, and that a connection to the primary database can be established through the virtual hostname sps-hana. The replication and operation modes should now be sync/logreplay and primary/primary on node-a and node-b, respectively.

[root@node-a ~]# su - spsadm -c "sapcontrol -nr 00 -function GetProcessList"
04.03.2021 21:50:20
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2021 03 04 21:43:51, 0:06:29, 156845
hdbcompileserver, HDB Compileserver, GREEN, Running, 2021 03 04 21:43:58, 0:06:22, 156970
hdbindexserver, HDB Indexserver-SPS, GREEN, Running, 2021 03 04 21:43:58, 0:06:22, 157101
hdbnameserver, HDB Nameserver, GREEN, Running, 2021 03 04 21:43:52, 0:06:28, 156864
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2021 03 04 21:43:58, 0:06:22, 156973
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2021 03 04 21:44:36, 0:05:44, 157429
hdbxsengine, HDB XSEngine-SPS, GREEN, Running, 2021 03 04 21:43:58, 0:06:22, 157104

[root@node-a ~]# su - spsadm -c "hdbsql -n sps-hana -i 00 -u SYSTEM -p <SYSTEM user password> -d SPS '\s'"
host          : sps-hana:30013
sid           : SPS
dbname        : SPS
user          : SYSTEM
kernel version: 2.00.046.00.1581325702
SQLDBC version:        libSQLDBCHDB 2.04.182.1579711187
autocommit    : ON
locale        : en_US.UTF-8
input encoding: UTF8
sql port      : node-b:30015

[root@node-b ~]# su - spsadm -c "sapcontrol -nr 00 -function GetProcessList"
04.03.2021 21:52:36
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2021 03 04 19:48:50, 2:03:46, 13383
hdbcompileserver, HDB Compileserver, GREEN, Running, 2021 03 04 19:49:00, 2:03:36, 13747
hdbindexserver, HDB Indexserver-SPS, GREEN, Running, 2021 03 04 19:49:00, 2:03:36, 13794
hdbnameserver, HDB Nameserver, GREEN, Running, 2021 03 04 19:48:51, 2:03:45, 13401
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2021 03 04 19:49:00, 2:03:36, 13750
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2021 03 04 19:49:42, 2:02:54, 15167
hdbxsengine, HDB XSEngine-SPS, GREEN, Running, 2021 03 04 19:49:00, 2:03:36, 13797

[root@node-b ~]# su - spsadm -c "hdbnsutil -sr_state"
System Replication State
~~~~~~~~~~~~~~~~~~~~~~~~

online: true

mode: primary
operation mode: primary
site id: 2
site name: SiteB

is source system: true
is secondary/consumer system: false
has secondaries/consumers attached: true
is a takeover active: false

Host Mappings:
~~~~~~~~~~~~~~

node-b -> [SiteB] node-b
node-b -> [SiteA] node-a

Site Mappings:
~~~~~~~~~~~~~~
SiteB (primary/primary)
    |---SiteA (sync/logreplay)

Tier of SiteB: 1
Tier of SiteA: 2

Replication mode of SiteB: primary
Replication mode of SiteA: sync

Operation mode of SiteB: primary
Operation mode of SiteA: logreplay

Mapping: SiteB -> SiteA
done.

5. Execute the following command to forcefully reboot node-b:
   

[root@node-b ~]# echo b > /proc/sysrq-trigger

Once LifeKeeper has detected that node-b has been powered off, the status of node-b updates to “Unknown” in the LifeKeeper GUI and LifeKeeper initiates failover of the HANA-SPS_HDB00 resource hierarchy back to node-a.

Once node-b is back online, LifeKeeper will automatically re-register it as a secondary replication site in SAP HANA System Replication and restart it. Once this process is complete, the resource states will transition back to “Active” on node-a and “Standby – In-Sync” on node-b.

6. Execute the commands given in step 2 again to verify that the database is functioning as expected on both nodes.

We have now verified the basic switchover and failover functionality of the SAP HANA resource hierarchy.