Test Switchover and Failover

In this section we will perform basic tests to verify the expected behavior of the pgsql-5432 resource hierarchy on switchover and failover.

1. Verify that the pgsql-5432 resource state is currently “Active” on node-a and “Standby” on node-b.

2. Execute the following commands on node-a to verify that the PostgreSQL database is running on the primary and the scsi target is mounted in the /datakeeper/pgsql directory.

[root@node-a ~]# su - postgres -c "/usr/pgsql-15/bin/pg_ctl status -D /datakeeper/pgsql/data -l /datakeeper/pgsql/pgstartup.log -o '-p 5432' -w"
pg_ctl: server is running (PID: 15282)
/usr/pgsql-15/bin/postgres "-D" "/datakeeper/pgsql/data" "-i" "-k" "/tmp" "-p" "5432"

[root@node-a ~]# df | grep datakeeper
/dev/sda2        5095040   49916   4782980   2% /datakeeper/pgsql

[root@node-a ~]# mount | grep datakeeper
/dev/sda2 on /datakeeper/pgsql type ext3 (rw,relatime,stripe=32)

3. Additionally, execute the following commands on node-b to verify that the PostgreSQL database is not running on the secondary and the scsi target is not mounted in the /datakeeper/pgsql directory.

[root@node-b ~]# su - postgres -c "/usr/pgsql-15/bin/pg_ctl status -D /datakeeper/pgsql/data -l /datakeeper/pgsql/pgstartup.log -o '-p 5432' -w"
pg_ctl: directory "/datakeeper/pgsql/data" does not exist

[root@node-b ~]# df | grep datakeeper
[root@node-b ~]#

[root@node-b ~]# mount | grep datakeeper
[root@node-b ~]#

4. Perform a switchover of the PostgreSQL resource hierarchy by right-clicking the pgsql-5432 resource on node-b and choosing the In-Service… operation. Click In Service to begin the switchover.

First, the resource and all child dependencies will be transitioned to “StandBy” on node-a.

Allow time for the switchover to complete and for the resource states to switch to “Active” on node-b. Note that it may take a couple minutes for the resource to come in service on node-b.

5. Execute the following commands on node-b to verify that the PostgreSQL database is running on the new primary and the scsi target is mounted in the /datakeeper/pgsql directory.

[root@node-b ~]# su - postgres -c "/usr/pgsql-15/bin/pg_ctl status -D /datakeeper/pgsql/data -l /datakeeper/pgsql/pgstartup.log -o '-p 5432' -w"
pg_ctl: server is running (PID: 10333)
/usr/pgsql-15/bin/postgres "-D" "/datakeeper/pgsql/data" "-i" "-k" "/tmp" "-p" "5432"

[root@node-b ~]# df | grep datakeeper
/dev/sda2        5095040   49924   4782972   2% /datakeeper/pgsql

[root@node-b ~]# mount | grep datakeeper
/dev/sda2 on /datakeeper/pgsql type ext3 (rw,relatime,stripe=32)

6. Execute the same commands on node-a to verify that the PostgreSQL database is not running on the new secondary and nothing is mounted in the /datakeeeper/pgsql directory.

[root@node-a ~]# su - postgres -c "/usr/pgsql-15/bin/pg_ctl status -D /datakeeper/pgsql/data -l /datakeeper/pgsql/pgstartup.log -o '-p 5432' -w"
pg_ctl: directory "/datakeeper/pgsql/data" does not exist

[root@node-a ~]# df | grep datakeeper
[root@node-a ~]#

[root@node-a ~]# mount | grep datakeeper
[root@node-a ~]#

7. Now to simulate a failover, execute the following command to forcefully reboot node-b:

[root@node-b ~]# echo b > /proc/sysrq-trigger

Once LifeKeeper has detected that node-b has been powered off, the status of node-b updates to “Unknown” in the LifeKeeper GUI.

At this point, LifeKeeper automatically initiates failover of the pgsql-5432 resource hierarchy back to node-a.

Once node-b is back online, LifeKeeper will automatically re-register it as a secondary node. This process may take several minutes to complete. Once this process is complete, the resource states will transition back to “Active” on node-a and “Standby” on node-b.

8. Execute the commands given in steps 2 and 3 to verify that the database is functioning as expected on both nodes.

We have now verified the basic switchover and failover functionality of the PostgreSQL resource hierarchy.