Apache Hierarchy Restore, Remove, and Recover Messages and Errors

The following information and error messages are printed to the LifeKeeper error log.

They may be viewed by typing “lk_log log”.

Bringing an Apache Resource In Service (Restore)

“LifeKeeper: RESTORE: APACHE: RESTORING $TAG TO SERVICE START AT: <date>”

Informational message. Records when the restore begins. Logged at the start of every restore.

“LifeKeeper: RESTORE APACHE RESOURCE $TAG END err=$err AT: <date>”

Informational message. Records when the restore completes. Logged at the end of every restore. If any errors occur during the restore, additional messages will be logged between these two messages and the value displayed for err=$err will be non-zero.

“Apache: No instance information found for Tag=$TAG.”

Error: Indicates no instance is defined with the tag value passed to the “restore” script. Unlikely to occur with the GUI, since only tags known to LifeKeeper are available as choices for the In Service and Out of Service actions.

“LifeKeeper: RESTORE: Apache: Tag=$TAG already running.”

Informational message. Indicates that the instance appeared to already be up and running.

“LifeKeeper: RESTORE: Apache: Existing processes terminated for ID=$ID.”

Informational message. Existing httpd processes were found running for this instance ID, but the PidFile is either missing or invalid. Therefore, the running processes were terminated.

“LifeKeeper: RESTORE: Apache: Invalid PidFile=$PIDFILE has been deleted.”

Informational message. An existing PidFile was found for this instance, but its contents were invalid. Therefore, the PidFile was deleted.

“LifeKeeper: RESTORE: Apache: Tag=$TAG is being restarted.”

Informational message. Indicates that the instance appeared to partially running, but needed to be restarted. If a PidFile still exists (which contains the process ID of the parent httpd process), the instance is restarted with a HUP signal. If the PidFile is missing, the instance is completely stopped and restarted.

“LifeKeeper: RESTORE: *ERROR* Apache: Error in web server configuration file $CONF_FILE for instance $ID.”

“LifeKeeper: RESTORE: *ERROR* Apache: Execute the following command to check the syntax of this file:”

“LifeKeeper: RESTORE: *ERROR* Apache: $HTTPD_PATH -t -d $SERVER_ROOT -f $CONF_FILE.”

Prior to instance startup, the syntax of the configuration file is checked using the httpd -t option. The -d option checks the ServerRoot directory. Additional options related to modules may also be displayed if you have configured Apache to use modules. Any syntax errors caught during hierarchy creation are displayed in the LifeKeeper GUI, but syntax errors introduced later will not be displayed in the GUI or the LifeKeeper logs. You must manually run the following command to determine what is wrong with your configuration (add additional options for modules, if applicable):

$HTTPD_PATH -t -d $SERVER_ROOT -f $CONF_FILE

“LifeKeeper: RESTORE: *ERROR* Apache: Error starting web server instance $INSTANCE.”

“LifeKeeper: RESTORE: *ERROR* Apache: Restore of tag $TAG failed.”

“LifeKeeper: RESTORE: *ERROR* Apache: Examine the Apache error log at $ERROR_LOG”

“LifeKeeper: RESTORE: *ERROR* Apache: to determine the cause of the problem.”

An error occurred executing the httpd daemon with the parameters specified. Check the httpd executable being used, configuration file, and general configuration for possible problems.

“LifeKeeper: RESTORE: *ERROR* Apache: Web server instance $ID did not start correctly.”

“LifeKeeper: RESTORE: *ERROR* Apache: Restore of tag $TAG failed.”

“LifeKeeper: RESTORE: *ERROR* Apache: Examine the Apache error log at $ERROR_LOG”

“LifeKeeper: RESTORE: *ERROR* Apache: to determine the cause of the problem.”

Note that in many cases the httpd daemon will appear to start even if its web sites don’t respond as expected. The restore script checks all IP/port combinations used to make sure all sites configured are fully functional. If they are not, this message is printed and the restore fails.

Although the site is left in the Out of Service state, one/more httpd processes may still be left running. (This is intentional, since one/more web sites may be operational and we don’t want to kill them off). You should resolve the problem as soon as possible and bring the instance In Service. If you don’t, LifeKeeper will eventually attempt to recover the instance and restore it to service automatically. If it can’t, it will fail over the hierarchy to another server.

Taking an Apache Resource Out of Service (Remove)

“LifeKeeper: REMOVE: APACHE: REMOVE $TAG FROM SERVICE START AT: <date>”

Informational message. Records when the remove begins. Logged at the start of every remove.

“LifeKeeper: REMOVE APACHE RESOURCE $TAG END err=$err AT: <date>”

Informational message. Records when the remove completes. Logged at the end of every remove.

If any errors occur during the remove, additional messages will be logged between these two messages and the value displayed for err=$err will be non-zero.

“LifeKeeper: REMOVE: *WARNING* APACHE: Error attempting to kill parent process for INSTANCE=$INSTANCE.”

There was an error attempting to kill the parent httpd process (whose process ID is stored in the Pidfile).

“LifeKeeper: REMOVE: *ERROR* APACHE: Error attempting to kill all processes for INSTANCE=$INSTANCE.”

Although the parent httpd process appeared to be killed successfully, one/more processes for this instance are still running. Normally the remove will be able to terminate any/all processes for this instance. When it cannot, this message is printed and the remove fails.

Bringing an Apache Resource Back In Service (Recover)

The LifeKeeper core periodically checks the health of every Apache instance In Service on the local server by running an Apache “quickCheck” script, which checks the web sites using the same scripts used to check the state of the instance during restore and remove. If the instance is not fully functional, a “recover” script is invoked to attempt to restart the instance. This simply logs the first message shown below, invokes “restore,” prints the final error or success message shown below—depending on error or success of the “restore” script—and returns the same result as “restore.” If restore/recover fails, this instance is failed over to another server.

“LifeKeeper: RECOVER: APACHE: Invoking restore for Apache instance “$ID” at: <date>”

“LifeKeeper: RECOVER: APACHE: Restore for Apache instance $ID returned error $RET at: <date>”

“LifeKeeper: RECOVER: APACHE: Restore for Apache instance $ID successful at: <date>”

Copyright © 2024 SIOS TECHNOLOGY CORP.