Control File Switchover Failure

If the $ORACLE_HOME directory does not recover, the database control files may not be set up properly. For automatic switchover, the control files need to be configured on a shared device during the database creation. If you keep the control files on separate servers, you must manually update both servers when you need to implement changes.

Truncated Output

Some versions of Oracle truncate the output when executing the show parameters control_files in sqldba mode. If your version of Oracle exhibits this behavior, verify the following:

  • controlfile parameter. Verify that the controlfile parameter resides in the
    $Oracle_HOME/dbs/init#SID.ora file.
  • controlfile devices. Verify that the controlfile devices are on a continuous line, with no new lines, and with each device being separated by a comma.

If an Oracle-related device does not configure properly, then the device can be configured manually using the file system applications available under LifeKeeper Application management.

Flash Recovery Destination Located on a Shared Drive

As noted in the configuration section of this document, it is important that the Flash Recovery destination is located on a shared drive. To see where Oracle believes the Flash Recovery Area is, issue the following query (as SYSDBA):

SQL> SELECT substr(Name,1,30) Name,

  (SPACE_LIMIT/1024/1024/1024) Space_Limit_GB,

       SPACE_USED/1024/1024/1024 Space_Used_GB,

       SPACE_RECLAIMABLE, NUMBER_OF_FILES

FROM V$RECOVERY_FILE_DEST;

NAME SPACE_LIMIT_GB SPACE_USED_GB
SPACE_RECLAIMABLE

————————— ——————— ——————- ————————- ———————— ——————-

NUMBER_OF_FILES

———————-

/U01/flash_recovery_area 3.76171875 .156448364 0

4

Following is an example of how to make a change in $ORACLE_HOME/dbs/spfile<sid> to complete this task:

SQL> ALTER SYSTEM SET
 DB_RECOVERY_FILE_DEST=’/oracledb/oracle/flash_recovery_area’ scope=both;

System altered.

SQL> show parameter DB_RECOVERY_FILE_DEST;

NAME TYPE VALUE

———————————————————————————————————

db_recovery_file_dest string /oracledb/oracle/flash_recovery_area

db_recovery_file_dest_size big integer 2G

SQL> commit;

Prevent Failover When Unable to Connect to the Database

When resource health checks are performed, the Oracle ARK checks for running database processes and attempts to connect to the database. To prevent a health check failure caused by reaching the maximum allowed connections, set the following in /etc/default/LifeKeeper:

LK_ORA_NICE=1

Note: Setting LK_ORA_NICE can mask other types of connection errors caused by a non-functioning database. Use this setting with caution.

Non-Traditonal Location for oratab

By default, the Oracle ARK looks for the oratab file in /etc followed by /var/opt/oracle. If the oratab file is not located in one of these default locations, then ORACLE_ORATABLOC must be set in /etc/default/LifeKeeper to the directory containing oratab.

NFS Version 4 Not Supported

The Oracle Recovery Kit supports NFSv3 for shared database storage. NFSv4 is not supported at this time due to NFSv4 file locking mechanisms.

Creating Oracle Database Hierarchy After Installing Oracle Binaries on Local Storage

If you have elected to install the Oracle binaries ($ORACLE_BASE) on local storage on each of your LifeKeeper cluster nodes, you will see a message similar to the following when you create your Oracle database hierarchy in the LifeKeeper GUI.

BEGIN create of <SID> on server <server1>
Creating resource instance <SID> on server <server1>
Setting resource state for <SID> on server <server1> to “ISP”.

ORACLE_HOME “/opt/oracle/app/oracle/product/11.2.0/dbhome_1” does not reside on a shared file system. Please be sure that the ORACLE_HOME directory and associated files are identical on all servers. Refer to the LifeKeeper Oracle Recovery Kit documentation for more information.
Creating dependent file system resource “/u00” on <server1>.
Creating dependency between Oracle database “SID (SID)” and the dependent resource “/u00” on <server1>.
Creating dependency between Oracle database “SID (SID)” and the listener resource “LSNR.LISTENER” on <server1>.
Performing in-service of new Oracle resource tag=< SID > on <server1>.
END successful create of on server <server1>

You will also find a similar warning in the LifeKeeper log. If this warning is not heeded and you continue on by extending your hierarchy and then try to bring the database resource in service on another node, you will get a message similar to the following in the LifeKeeper GUI dialog:

Put resource “OST” in-service
BEGIN restore of “OST” on server “cae-qa-v39”
Begin the “start [ start.normal ]” of the database “OST” on “cae-qa-v39”.
The “start [ start.normal ]” attempt of the database “OST” appears to have failed on “cae-qa-v39”.
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file ‘/opt/oracle/app/oracle/product/11.2.0/dbhome_1/dbs/initOST.ora’
Begin the “start [ start.force ]” of the database “OST” on “cae-qa-v39”.
The “start [ start.force ]” attempt of the database “OST” appears to have failed on “cae-qa-v39”.
select ‘alter database datafile ‘||file#||’ end backup;’ from v\$backup where status = ‘ACTIVE’

It is also possible to get a message similar to the following in the dialog box:

Put resource “OST” in-service
BEGIN restore of “OST” on server “cae-qa-v39”
Begin the “start [ start.normal ]” of the database “OST” on “cae-qa-v39”.
Initial inspection of “start.normal” failed, verifying failure or success of received output.
Logon failed with “” for “OST” on “cae-qa-v39”. Please check username/password and privileges.
The “start [ start.normal ]” attempt of the database “OST” appears to have failed on “cae-qa-v39”.
ERROR:

ORA-01031: insufficient privileges

Enter password:

ERROR:

ORA-01005: null password given; logon denied

To solve this issue, copy $ORACLE_BASE/admin from the primary system where the database instance was created to the backup system (where the hierarchy was extended to) $ORACLE_BASE/admin. Also change ownership of this directory to your Oracle username and Oracle group (typically oracle:oinstall).

Also copy all *{$ORACLE_SID}* (OST in this example) files from the primary system in $ORACLE_BASE/product/11.2.0/dbhome_1/dbs to ORACLE_BASE/product/11.2.0/dbhome_1/dbs on the backup system.

For example, these were the files that were copied from a primary system to the backup, and the ORACLE SID was OST.

-rw-r——- 1 oracle oinstall 1544 2012-05-09 11:02 hc_OST.dat
-rw-r——- 1 oracle oinstall 24 2012-01-31 10:22 lkOST
-rw-r——- 1 oracle oinstall 1536 2012-03-05 09:02 orapwOST
-rw-r——- 1 oracle oinstall 2560 2012-05-09 10:58
spfileOST.ora

In another example, where the SID was ORA01, the following files were copied:

-rw-r——- 1 oracle oinstall 1536 2010-09-08 18:25 orapwORA01
-rw-r——- 1 oracle oinstall 24 2010-09-08 18:25 lkORA01
-rw-r——- 1 oracle oinstall 2560 2010-09-08 18:30 spfileORA01.ora
-rw-r——- 1 oracle oinstall 1544 2010-09-08 18:30
hc_ORA01.dat

and a directory

peshm_ORA01_0/:

Oracle Listener Stays in Service on Primary Server After Failover

Network failures may result in the listener process remaining active on the primary server after an application failover to the backup server.

Feedback

Was this helpful?

Yes No
You indicated this topic was not helpful to you ...
Could you please leave a comment telling us why? Thank you!
Thanks for your feedback.

Post your comment on this topic.

Post Comment