Logging With syslog

Beginning with LifeKeeper v8.0, logging is done through the standard syslog facility. LifeKeeper supports rsyslog, which is an extension of the original syslog protocol. During package installation, syslog will be configured to use the “local6” facility for all LifeKeeper log messages (if “local6” is already in use another local should be used). The syslog configuration file /etc/rsyslog.conf is modified to include LifeKeeper-specific routing sending all LifeKeeper log messages to /var/log/lifekeeper.log (the original configuration file will be backed up with the same name ending in “~”).

!Important: DO NOT edit LifeKeeper’s unique setup steps manually or upgrading and uninstalling may not function correctly.

The facility can be changed after installation by using the lklogconfig tool located in /opt/LifeKeeper/bin. For example, changing the facility to local5, run the following command.

lkstop -f
lklogconfig --action=update --facility=local5
lkstart

See the lklogconfig(8) manpage on a system with LifeKeeper installed for more details on this tool.

If a generic resource script puts a message into /opt/LifeKeeper/out/log directly, LifeKeeper will send a log message with the error severity level as the default into /var/log/lifekeeper.log. The severity level can be changed to information level by adding the following parameter into /etc/default/LifeKeeper, LOGMGR_LOGLEVEL=LK_INFO.

*Note: When LifeKeeper is removed from a server, the LifeKeeper-specific syslog configuration will be removed.

* Note: The LifeKeeper configuration files that control the log rotation can be found in /etc/logrotate.d as lifekeeper for the lifekeeper.log file and lifekeeper-err for lifekeeper.err log file. By default, log rotation will only occur when the size of the log file reaches 100MB.

Changing rsyslog configuration

If rsyslog is using the imjournal module, after a forced shutdown of the OS and subsequent reboot, the log updates of /var/log/lifekeeper.log may stop or past logs may be output. To prevent these, please follow the steps below.

1. Add the following setting option to the imjournal module settings in /etc/rsyslog.conf:

For RHEL 8 and later, add IgnorePreviousMessages="on" as a setting for the imjournal module

module(load="imjournal"             # provides access to the systemd journal
      UsePid="system" # PID nummber is retrieved as the ID of the process the journal entry originates from
      IgnorePreviousMessages="on"
StateFile="imjournal.state") # File to store the position in the journal

If you are using legacy notations such as RHEL 7, SLES 12, or SLES 15, add:

$ImjournalIgnorePreviousMessages on

$ModLoad imjournal # provides access to the systemd journal
...
$IMJournalStateFile imjournal.state
$ImjournalIgnorePreviousMessages on

2. If logging has stopped, delete /var/lib/rsyslog/imjournal.state

3. Run systemctl restart rsyslog.service

Checking logs with journalctl

If there are missing logs due to OS forced termination, the journalctl command can be used to view logs up to just before the forced shutdown. Specifying the following options will filter the output to the equivalent of what is output in /var/log/lifekeeper.log. (The value 22 is the facility code corresponding to the facility local6 used by LifeKeeper. If local5 is specified, the value would be 21.)

journalctl SYSLOG_FACILITY=22

When using the -b option, you can pull a log from the specified previous startup. See man journalctl for details.