Test Your Environment – PostgreSQL

The following test scenarios have been included to guide you as you get started evaluating SIOS Protection Suite for Linux. Before beginning these tests, make sure the data replication resources are in the mirroring state.

Note: For these test examples, the Primary Server is referred to as LINUXPRIMARY. The Backup or Secondary Server is referred to as LINUXSECONDARY.

Manual Switchover of the PostgreSQL Hierarchy to Secondary Server

Procedure:

• From the LifeKeeper GUI, right click on the PostgreSQL resource on the Secondary Server (LINUXSECONDARY) and choose IN SERVICE.
• Click “In Service” in the window that pops up

Expected Result:

• Beginning with the PostgreSQL resource, all resources will be removed from service on the Active Server (LINUXPRIMARY).
• Beginning with the dependent resources (IP and Shared Volume), all resources will be brought in service on LINUXSECONDARY

Tests/Verification:

• Using the LifeKeeper GUI, verify that the PostgreSQL and dependent resources are active on LINUXSECONDARY.
• Run “ifconfig –a” on LINUXSECONDARY to validate that the IP Address 192.168.197.151 is active on LINUXSECONDARY
• Run “df –h” to verify that the /var/lib/pgsql shared iSCSI filesystem is mounted on LINUXSECONDARY
• Verify the PostgresSQL services are running on LINUXSECONDARY by running “ps –ef | grep –i postgres”
• On LINUXSECONDARY run the following command to verify client connectivity to the PostgreSQL database:
  
  
  • # su – postgres
    
  • # psql
  
  
• Verify that the SCSI reservation has properly locked the iSCSI LUN to the currently active cluster node. On LINUXPRIMARY, run “mount /dev/sdc1 /var/lib/pgsql”. This should FAIL
  because LINUXPRIMARY does not own the SCSI reservation on this LUN.

Manual Switchover of the PostgreSQL Hierarchy back to Primary Server

Procedure:

• From the LifeKeeper GUI, right click on the PostgreSQL resource on the Primary Server (LINUXPRIMARY) and choose IN SERVICE.
• Click “In Service” in the window that pops up

Expected Result

• Beginning with the PostgreSQL resource, all resources will be removed from service on the Active Server (LINUXSECONDARY).
• Beginning with the dependent resources (IP and Shared Volume), all resources will be brought in service on LINUXPRIMARY

Tests/Verification:

• Using the LifeKeeper GUI, verify that the PostgreSQL and dependent resources are active on LINUXPRIMARY.
• Run “ifconfig –a” on LINUXPRIMARY to validate that the IP Address 192.168.197.151 is active on LINUXPRIMARY
• Run “df –h” to verify that the /var/lib/pgsql shared iSCSI filesystem is mounted on LINUXPRIMARY
• Verify the PostgreSQL services are running on LINUXPRIMARY by running “ps –ef | grep –i postgres”
• On LINUXPRIMARY run the following command to verify client connectivity to the PostgreSQL database:
  
  
  • # su – postgres
  • #35; psql
  
  
• Verify that the SCSI reservation has properly locked the iSCSI LUN to the currently active cluster node. On LINUXSECONDARY, run “mount /dev/sdc1 /var/lib/pgsql”. This should FAIL because LINUXSECONDARY does not own the SCSI reservation on this LUN.

Simulate a network failure on the Primary Server by failing the IP resource

*IMPORTANT NOTE: Only perform this test if you have more than one communications path configured.

If you perform this test and have only one communications path configured, your system will enter a split-brain scenario as described in the LifeKeeper Administration Guide found here. Refer to this document for more information or contact SIOS presales technical support for assistance in resolving this condition.

Procedure

• On LINUXPRIMARY, pull the network cable attached to the NIC that the virtual IP address is configured on

Expected Result:

• The IP Resource should fail first.
• The entire hierarchy should failover to LINUXSECONDARY

Tests/Verification:

• Check the LifeKeeper Log to verify the IP resource failed – “/opt/LifeKeeper/bin/lk_log log”
• Using the LifeKeeper GUI, verify the PostgreSQL resource hierarchy fails over successfully to LINUXSECONDARY
• After this test has been completed, re-connect the network cable on LINUXPRIMARY

Hard failover of the resource from the Secondary Server back to the Primary Server

Procedure:

• Pull the power cord on LINUXSECONDARY, as this is the server with all resources currently In Service.

Expected Result:

• After failure has been detected, beginning with the dependent resources (IP and Volume), all resources will be brought in service on LINUXPRIMARY.

Tests/Verification:

• Using the LifeKeeper GUI, verify the mirror has reversed and is in a Resync Pending state waiting for LINUXSECONDARY to come back on line.
• Verify the PostgreSQL Server services are running on LINUXPRIMARY.

Bring Failed Server back on line

Procedure:

• Plug the power cord back into LINUXSECONDARY and boot it up.

Expected Result:

• Using the LifeKeeper GUI, verify that LINUXSECONDARY is coming back up and has become the Standby Server.

Tests/Verification:

• Verify the PostgreSQL Hierarchy is in service on LINUXPRIMARY and standby on LINUXSECONDARY.

Verify Local Recovery of PostgreSQL Server

Procedure:

• Kill the PostgreSQL processes via the command line:
• # ps –ef | grep postgres
• # (kill -9 the PIDs returned)
• run “ps –ef | grep postgres once again to verify that the processes no longer exist

Expected Result: (Assumes Local Recovery for SQL resource is set to YES)

• The PostgreSQL Server service should stop.
• The PostgreSQL quickcheck process will automatically restart the PostgreSQL Server Service when it runs periodically.
• No failure of PostgreSQL should occur.

Tests/Verification:

• Execute “ps –ef | grep postgres” once again to verify that the postgresql processes have been restored locally on LINUXPRIMARY.
• Verify connectivity to the postgreSQL database by running:
  
  
  
  • # su – postgres
  • # psql
  
  
• If you inspect the LifeKeeper logs, you will see information indicating that LifeKeeper dectected the failure of the PostgreSQL service and recovered it locally. Run /opt/LifeKeeper/bin/lk_log log for more information.