Use Control F to search for a specific error code in each catalog. To search for any error code, select the Search button at the top right of the screen.


Code Severity Message Cause/Action
138001 ERROR $line

Cause: A DRBD API command failed. The output from the API is logged in this message.

Action: Following this message will be a specific message detailing any action needed.

138002 ERROR Failed to execute DRBD API command "%s" with arguments "%s" for resource %s: %s

Cause: A DRBD API command that failed is listed along with the arguments used.

Action: Following this message will be a specific message detailing any action needed.

138003 FATAL $msg

Cause: Log the stack trace for this process.

Action: None.

138005 ERROR Unable to create DRBD object ($sys, $tag).

Cause: The DRBD object could not be created due to either $sys or $tag not being defined.

Action:
Retry command with correct tag and system.

138006 WARN The "%s_wait_for_peer" flag is set in "%s/subsys/scsi/resources/drbd/" on system "%s". The resource is being forced online.

Cause: All systems are not available to determine where up-to-date data exists. The "Force Mirror Online" is being used to bring the DRBD resource online while all systems are not available.

Action: None

138007 ERROR The "%s_wait_for_peer" flag is set in "%s/subsys/scsi/resources/drbd/" on system "%s". To avoid using outdated data, LifeKeeper will not restore the resource until all systems are accessible.

Cause: All systems are not available to determine where up-to-date data exists.

Action: Start LifeKeeper on all systems to allow LifeKeeper and DRBD to use the latest data available. Use the "Force Mirror Online’‘ option in the GUI or lkcli to bring the DRBD resource in-service immediately. WARNING: this may result in committed data loss if more up-to-date data is on a system that is down.

138008 ERROR Failed to get resource information: %s.

Cause: LifeKeeper failed to retrieve the information from the resource database file.

Action: Further information on the failure will be listed and additional error messages in /var/log/lifekeeper.log will provide more details on how to resolve.

138009 FATAL UUID "%s" in "%s" does not match LifeKeeper ID "%s".

Cause: The UUID found in the resource database file does not match the ID in the LifeKeeper instance.

Action: Bring the resource in-service on the other system in the cluster. If the in-service (restore) is successful, the ID’s in the resource DB file and LifeKeeper instance match. Synchronize the resource files and the ID in the instance on the failed system to the system where the in-service was successful.

138010 ERROR Resource %s role on %s is "Primary". Set %s to "Secondary" using "drbdadm secondary %s" on %s or bring the resource in-service on %s.

Cause: An out of service (standby) system has the resource "up" with the role of "Primary". LifeKeeper expected the specified resource to be "Secondary". This is similar to a split-brain situation where there may be divergent data.

Action: Determine which system has the latest or best data. If the standby system that is "Primary" does not have the latest data then manually stop all activity on that resource and set the state to "Secondary". Bring the resource in-service on the system with the latest or best data.

138011 ERROR Resource %s on %s is "Diskless". Will try to re-attach by taking resource "down" and back "up".

Cause: The DRBD resource is "up" with the disk NOT attached. This may be due to an IO error or a manual action. LifeKeeper will try to attach the disk.

Action: Review /var/log/lifekeeper.log and /var/log/messages to determine the cause of the volume not being attached. If due to an IO error replace the faulty disk.

138012 ERROR Re-attach failed to repair "Diskless" Resource %s on %s.

Cause: LifeKeeper was not able to get the disk to attach to the resource.

Action: Review DRBD driver messages logged in /var/log/messages to determine the cause of the failure. If due to an IO error replace the faulty disk.

138013 ERROR Skipping connection to %s, %s is not configured for this resource.

Cause: There is a system configured in the DRBD resource file that is not configured for this resource in LifeKeeper.

Action: Remove the unexpected system/connection using ‘lkcli drbd removesys -sys <sys> -tag <tag>’.

138015 WARN Resource %s is not connected to %s. If %s has divergent data then it will not automatically connect and will require using the GUI "Resume Replication" option or "lkcli drbd resume —sys %s —tag %s"

Cause: The peer-role for the resource on the target is "Unknown". DRBD could not determine the state of the resource or consistency of the data.

Action: When DRBD is able to connect to the target and determine the state of the resource it will automatically connect if there is NOT divergent data. If there is divergent data then use the command in the message to resume replication to the target.

138016 EMERG Split-brain detected, %s is in-service and %s is in Primary role. Manual intervention is required in order to minimize the risk of data loss. To resolve this situation, take resource %s out-of-service on the server(s) with invalid data and resume replication using the GUI or lkcli.

Cause: The resource is in-service and "Primary" on multiple systems causing divergent data.

Action: Take the resource out-of-service on the server(s) with invalid data and resume replication using the GUI or lkcli.

138017 WARN Resource %s, target %s connection state is "%s".

Cause: The DRBD resource connection-state was expected to be "Connected".

Action: None. LifeKeeper will automatically repair the connection.

138018 ERROR Resource %s is not ready to use in %s seconds (see "DRBD_WAIT_FOR_USABLE_TIMEOUT" in /etc/default/LifeKeeper).

Cause: During restore of the resource the DRBD API ‘wait_for_usable’ is called to verify the DRBD driver has added the new resource and it is ready to use. More detailed errors from the DRBD API ‘wait_for_usable’ command should precede this message.

Action: The default setting for DRBD_WAIT_FOR_USABLE_TIMEOUT is 30 seconds, which should be sufficient for the DRBD driver to configure the resource. Check /var/log/messages for further errors. Increase the default setting if necessary.

138019 WARN DRBD detected the network connection between %s and %s is congested. The primary node %s will "pull ahead" of %s, temporarily going out of sync. When more bandwidth becomes available, replication will automatically resume and a background synchronization will take place

Cause: The change rate is higher than the network bandwidth available.

Action: DRBD will automatically recover when bandwidth is available. Decrease change rate or increase bandwidth.

138020 WARN Resource %s on target %s is "Diskless".

Cause: The DRBD resource is "up" on the target with the disk NOT attached. This may be due to an IO error or a manual action. LifeKeeper will try to attach the disk.

Action: Review /var/log/lifekeeper.log and /var/log/messages to determine the cause of the volume not being attached. If due to an IO error replace the faulty disk.

138021 FATAL "drbdadm down %s" on %s failed, this prevents forcing %s online: %s.

Cause: A "Force Mirror Online" requires taking down the resource on all other systems to be stopped (taken down) to force the correct synchronization.

Action: Review /var/log/lifekeeper.log and /var/log/messages on the specified system to determine why the resource can not be stopped (taken down). Take down the resource on the specified system and retry the force online.

138022 ERROR The underlying disk is mounted on %s and must be unmounted for DRBD to use the disk. Changes made directly to the underlying disk are not tracked by DRBD. A full resync will automatically occur when the resource is brought "up" and "connects", overwriting any changes made on %s. After unmounting the disk the "Force Resource Online (Primary)" GUI option or "lkcli drbd force —sys %s —tag %s" can force the resource online on %s where a full resync will occur to the other servers.

Cause: The underlying disk is directly mounted on the specified system.

Action: Unmount the file system. A full resync will occur overwriting the data on the server that had the underlying disk mounted. To preserve the data on the server that has the underlying disk mounted, after unmounting the file system then use the “force online” to the server that had the underlying disk mounted.

138023 WARN Resynchronization is in-progress for Resource %s (%s out-of-sync blocks).

Cause: Out of sync blocks are being synchronized.

Action: None.

138024 ERROR Failed to create UUID.

Cause: During create of a resource the DRBD API did not generate a UUID.

Action: Check /var/log/messages for further messages from the DRBD driver.

138025 ERROR Failed to write resource file %s.

Cause: During create of a resource the resource file could not be written. More detailed errors from the DRBD API that writes the file will precede this message. The file system that /etc/drbd.d is on may be read only or out of space.

Action: Check /var/log/messages for further errors.

138026 ERROR Failed to store resource DB file %s.

Cause: The DRBD API ‘store’ failed. More detailed errors from the DRBD API that writes the file will precede this message. The file system that /etc/drbd.d is on may be read only or out of space.

Action: Check /var/log/messages for further errors.

138027 ERROR Failed to create resource %s.

Cause: The ‘create_md’ DRBD API command failed. More detailed errors from the DRBD API ‘create_md’ command will precede this message.

Action: Check /var/log/messages for further errors.

138028 WARN Resynchronization is paused for Resource %s.

Cause: This may be due to a dependency on the completion of another synchronization process, or due to synchronization having been manually interrupted by ‘drbdadm pause-sync <res>’.

Action: If manually paused then resume synchronization using ‘drbdadm resume-sync <res>’.

138029 ERROR Resource %s is not ready to use in %s seconds (see "DRBD_WAIT_FOR_USABLE_TIMEOUT" in /etc/default/LifeKeeper).

Cause: During create of the resource the DRBD API ‘wait_for_usable’ is called to verify the DRBD driver has added the new resource and it is ready to use. More detailed errors from the DRBD API ‘wait_for_usable’ command should precede this message.

Action: The default setting for DRBD_WAIT_FOR_USABLE_TIMEOUT is 30 seconds, which should be sufficient for the DRBD driver to configure the resource. Check /var/log/messages for further errors. Increase the default setting if necessary.

138030 WARN Resynchronization is starting for Resource %s (transient state %s).

Cause: DRBD is in the transient state starting to resynchronize a resource.

Action: None.

138031 ERROR Failed to "up" resource %s on %s.

Cause: The DRBD in-service was not able to “up” the resource. This may be due to a failure on the underlying disk or the underlying disk is open by a process.

Action: Further information on the failure will be listed in /var/log/messages. Resolve the errors and if the data has been modified on the underlying disk outside of DRBD then force a full resync using ‘drbdadm invalidate <res>’.

138034 ERROR Failed to copy $self->{‘resource_file’} to $sys: $out

Cause: LifeKeeper synchronizes the DRBD resource files in /etc/drbd.d on all systems. The file system that /etc/drbd.d is on may be read only or out of space.

Action: Check the details provided in the message for the specific error. After correcting the error, retry the operation.

138035 ERROR Unable to open file "%s".

Cause: The specified file could not be opened to determine the list of mounts.

Action: Further information on the failure will be listed in /var/log/messages. Resolve the errors and retry the command.

138037 WARN Resource %s has out of sync blocks, resynchronization is not in-progress.
138038 WARN Remove %s on %s: resource on %s is in state %s.

Cause: The DRBD resource is being stopped when not ‘UpToDate’ on a target system. The DRBD resource will not be allowed in-service on the target unless the ‘UpToDate’ server is available.

Action: Bring the resource in-service when the ‘UpToDate’ data is available.

138039 WARN Remove $self->{‘tag’} on $self->{‘sys’}: resource is in state $res_status->{devices}0{‘disk-state’}.

Cause: The DRBD resource is being stopped with the in-service resource not "UpToDate". The DRBD resource will not be allowed in-service on this server unless the "UpToDate" server is available.

Action: Bring the resource in-service when the "UpToDate" data is available.

138041 ERROR Failed to set resource %s to secondary on %s.

Cause: LifeKeeper is stopping the resource by first setting the resource to "Secondary". If the resource is mounted, open or busy it will not be able to set the state to "Secondary". LifeKeeper will log a message if the resource is mounted.

Action: Stop all activity to the resource. If the resource is mounted then unmount it.

138043 ERROR "drbdadm down %s" failed on %s: %s.

Cause: LifeKeeper was not able to stop, or take down, the DRBD resource on the specified system. The log message will contain more details on the failure.

Action: Check /var/log/messages on the specified system for more details. Manually stop the resource on the specified system using ‘drbdsetup down <resource>’.

138045 ERROR Failed to "down" resource %s on %s.

Cause: LifeKeeper was not able to stop, or take down, the DRBD resource on the specified system. The log message will contain more details on the failure.

Action: Check /var/log/messages on the specified system for more details. Manually stop the resource on the specified system using ‘drbdsetup down <resource>’.

138051 ERROR Unable to open file "%s".

Cause: The specified file could not be opened to determine the list of mounts.

Action: Further information on the failure will be listed in /var/log/messages. Resolve the errors and retry the command.

138052 ERROR Failed to "invalidate" resource %s on %s to force a full resync. Replicated data may not be consistent or may be out-of-date. You should force a full resync using "drbdadm invalidate %s" on the system with inconsistent or out-of-date data.

Cause: The DRBD invalidate command failed. The specified server has the underlying device mounted. To make sure the data is valid a full resync is required.

Action: Further information on the failure will be listed in /var/log/messages. Resolve the errors and force a full resync with the command listed in the message.

138053 ERROR Failed to remove %s from the resource %s on %s. Run "lkcli drbd removesys —sys %s —tag %s" on %s to remove %s from the configuration.

Cause: During a "Delete Resource Hierarchy" the DRBD resource file was not updated properly to remove the system.

Action: Check the errors preceding this message and /var/log/messages. Run the command listed in the message to retry the operation.

138054 ERROR Unable to locate a server with resource %s to update configuration. Run "lkcli drbd removesys —sys %s —tag %s" on another server with %s configured to remove %s from the configuration.

Cause: During a "Delete Resource Hierarchy" the DRBD resource file was not updated properly to remove the system.

Action: Check the errors preceding this message and /var/log/messages. Run the command listed in the message to retry the operation.

138055 ERROR Tag %s is not a valid DRBD resource.

Cause: The tag passed to remove a system from the DRBD configuration was not for a DRBD resource.

Action: Retry the command with the correct tag.

138056 ERROR Tag %s is extended to %s, resource configuration can not be removed.

Cause: A system can only be removed from a configuration if the system is no longer extended to the system to be removed.

Action: Use "Unextend Resource Hierarchy" in the GUI or lkcli to remove the system.

138057 ERROR Failed \"drbdadm invalidate $resource —force\" on $sys: $out

Cause: The DRBD invalidate command failed. The specified server has the underlying device mounted. To make sure the data is valid a full resync is required.

Action: Following this message will be a specific message detailing any action needed.

138059 ERROR Failed to write resource file %s.

Cause: While removing a system from the DRBD resource configuration, the resource file could not be saved.

Action: Check the errors preceding this message and /var/log/messages. Retry remove after correcting the errors.

138060 ERROR Failed to store resource DB file %s.

Cause: While removing a system from the DRBD resource configuration, the resource DB file could not be saved.

Action: Check the errors preceding this message and /var/log/messages. Retry remove after correcting the errors.

138065 ERROR Failed to save original %s for %s: %s

Cause: While trying to recover from a failure the original resource file listed was not able to be saved.

Action: Check the error listed in the message and check /var/log/messages. Retry failed operation.

138066 ERROR Failed to copy $self->{‘resource_db_file’} to $sys: $out

Cause: LifeKeeper synchronizes the DRBD resource DB files in /etc/drbd.d on all systems

Action: Check the details provided in the message for the specific error. After correcting the error, retry the operation.

138067 ERROR Failed to update resource files for resource %s on %s.

Cause: Resource files were copied to the specified system in a temporary location and were to be updated to the final location. The update failed.

Action: Check the errors preceding this message and /var/log/messages. Retry the operation that failed after correcting the errors.

138068 ERROR System %s is not configured in resource file %s.

Cause: The specified system was being removed from the DRBD configuration but could not be found in the DRBD configuration for this resource.

Action: Check that the system specified is the correct name listed in sys_list.

138069 ERROR Failed to copy $file to $sys: $out

Cause: The LifeKeeper ‘lcdrcp’ utility failed when copying the file to the specified system. The error information is included in the message.

Action: Check the error in the log message and /var/log/messages on the specified system. Correct any errors and retry the operation.

138070 ERROR Failed to open and lock %s on %s: %s.

Cause: The resource database file is locked to prevent an unextend from the GUI and lkcli operations from making simultaneous operations.

Action: Check the error listed in the message and check /var/log/messages. Retry failed operation.

138071 ERROR Failed to open and lock %s on %s: %s.

Cause: The resource database file is locked to prevent an unextend from the GUI and lkcli operations from making simultaneous operations.

Action: Check the error listed in the message and check /var/log/messages. Retry failed operation.

138072 ERROR Failed to open and lock %s on %s: %s.

Cause: The resource database file is locked to prevent an extend from the GUI and lkcli operations from making simultaneous operations.

Action: Check the error listed in the message and check /var/log/messages. Retry failed operation.

138074 ERROR Tag %s is not a valid DRBD resource.

Cause: The tag passed in to update resource files is not for a DRBD resource.

Action: Retry the command with the correct tag.

138075 ERROR Failed to save original %s for %s: %s

Cause: While trying to recover from a failure the original resource database file listed was not able to be saved. The file system that /etc/drbd.d is on may be read only or out of space.

Action: Check the error listed in the message and check /var/log/messages. Retry failed operation.

138078 ERROR Failed to update %s for %s: %s

Cause: The resource file could not be renamed. The system error is included in the message. The file system that /etc/drbd.d is on may be read only or out of space.

Action: Check the error listed in the message and check /var/log/messages.

138079 ERROR Failed to update %s for %s: %s

Cause: The resource database file could not be renamed. The system error string is included in the message. The file system that /etc/drbd.d is on may be read only or out of space.

Action: Check the error listed in the message and check /var/log/messages.

138083 ERROR ERROR: There is no equivalence to system %s that is configured in %s. Delete system %s using "lkcli drbd removesys —sys %s —tag %s" on %s

Cause: The DRBD resource configuration contains a system that the LifeKeeper instance is not extended to.

Action: Remove the system configured in the DRBD resource configuration using the lkcli command supplied in the error message.

138084 ERROR Resource %s is configured on %d servers, the DRBD Recovery Kit is restricted to %d servers.

Cause: The DRBD resource is restricted to 2 servers only.

Action: None.

138085 ERROR Tag %s is not a valid DRBD resource.

Cause: The tag passed in to update resource files not for a DRBD resource.

Action: Retry the command with the correct tag.

138086 EMERG Split-brain detected, %s servers (%s) are in-service. Manual intervention is required in order to minimize the risk of data loss. To resolve this situation, take resource %s out-of-service on the server(s) with invalid data and resume replication using the GUI or lkcli.

Cause: The resource is in-service on multiple systems causing divergent data most likely caused by a communication error.

Action: Take the resource out-of-service on the server(s) with invalid data and resume replication using the GUI or lkcli.

138087 ERROR Tag %s is not a valid DRBD resource.

Cause: The tag and ID passed to quickCheck are not for a DRBD resource.

Action: Retry quickCheck with the correct tag and ID.

138090 ERROR Resource %s is not "Primary" on %s.

Cause: The in-service DRBD resource is not in the "Primary" state. A local recovery will automatically try to make the resource "Primary".

Action: None.

138093 ERROR Resource %s on %s is "Diskless", check for IO errors on %s. Data will be accessed from a target.

Cause: An in-service DRBD resource is "Primary" with the volume not connected to storage, "Diskless". If "DRBD_ALLOW_DISKLESS" is set to 1 in /etc/default/LifeKeeper then the resource will remain in-service, accessing the data from the target.

Action: Check /var/log/lifekeeper.log and /var/log/messages for errors on the resource and its volume.

138094 ERROR Diskless mode not allowed (see DRBD_ALLOW_DISKLESS in /etc/default/LifeKeeper), switchover being initiated.

Cause: An in-service DRBD resource is "Primary" with the volume not connected to storage, "Diskless". If "DRBD_ALLOW_DISKLESS" is set to 0 in /etc/default/LifeKeeper a switchover to the standby system will occur.

Action: Check /var/log/lifekeeper.log and /var/log/messages for errors on the resource and its volume.

138095 ERROR Tag %s is not a valid DRBD resource.

Cause: The tag passed in to recover issues quickCheck found with a resource is not for a DRBD resource.

Action: Check /var/log/lifekeeper.log and /var/log/messages for errors.

138096 ERROR Tag %s is not a valid DRBD resource on %s.
138097 FATAL Pause requires tag ($tag) and server ($sys).

Cause: The tag and server is required to pause replication.

Action: Specify the tag and server that should be paused.

138099 ERROR Tag %s is not a valid DRBD resource on %s.

Cause: The tag passed in to pause replication is not for a DRBD resource.

Action: Retry the command with the correct tag.

138100 ERROR Failed to pause %s on %s.

Cause: The DRBD resource was not successfully paused on the specified system.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the specified system to determine the failure.

138102 ERROR Tag %s is not a valid DRBD resource on %s.
138103 ERROR %s must be in-service to pause replication.

Cause: The DRBD resource is not in-service. Only in-service resources can be paused.

Action: Bring the mirror in-service.

138105 ERROR Failed to pause resource %s.

Cause: The DRBD API "pause" operation failed.

Action: The DRBD API to pause the mirror failed. Check the preceding error messages and /var/log/messages for the root cause.

138106 ERROR Failed to create "paused" flag file on target "%s". Pause aborted.

Cause: LifeKeeper requires a "paused" flag to track when a resource has been paused. The file system that /opt/LifeKeeper is on may be read only or out of space.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the target to determine why the paused flag was not created.

138107 ERROR Failed to initial_sync resource %s.

Cause: The DRBD API ‘initial_sync’ operation failed. To pause a DRBD resource the DRBD API ‘initial_sync’ is used to force the resource to ‘Primary’.

Action: Check the preceding error messages and /var/log/messages for the root cause.

138108 ERROR Resource %s is not ready to use in %s seconds (see "DRBD_WAIT_FOR_USABLE_TIMEOUT" in /etc/default/LifeKeeper).

Cause: During a pause operation, the DRBD API ‘wait_for_usable’ is called to verify the DRBD driver has added the resource and it is ready to use.

Action: More detailed errors from the DRBD API ‘wait_for_usable’ command should precede this message. The default setting for DRBD_WAIT_FOR_USABLE_TIMEOUT is 30 seconds, which should be sufficient for the DRBD driver to configure the resource. Check /var/log/messages for further errors. Increase the default setting if necessary.

138109 ERROR Resource %s is not "Primary" on %s.

Cause: During a local recovery the DRBD resource is not in the ‘Primary’ state as it should be.

Action: None, local recovery will set the resource to ‘Primary’.

138110 ERROR Failed to set resource %s to "Primary" on %s.

Cause: The DRBP API ‘primary’ failed during local recovery.

Action: None. Local recovery will fail and initiate a switchover.

138111 ERROR "drbdadm down %s" on %s failed: %s.

Cause: Local recovery was not able to ‘down’ the DRBD resource on $sys.

Action: None. Local recovery will continue to repair the DRBD resource.

138112 FATAL Resume requires tag ($tag) and server ($sys).

Cause: Resume replication to a disconnected target requires the tag of the resource and the server to connect. Both of these are not set.

Action: Retry the operation with a specific tag and server to resume.

138114 ERROR Tag %s is not a valid DRBD resource on %s.

Cause: The tag passed in to resume replication is not for a DRBD resource.

Action: Retry the command with the correct tag.

138115 ERROR Failed to resume %s to %s.

Cause: The resume operation failed on the target system listed.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the target system listed to determine why the resume failed.

138117 ERROR "drbdadm adjust %s" on %s failed. This may cause connection issues for resource %s: %s.

Cause: The DRBD API ‘adjust’ will synchronize the configuration between all systems in the configuration.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the target system listed to determine why the adjust failed.

138118 ERROR "drbdadm up %s" on %s failed: %s.

Cause: Local recovery was not able to ‘up’ the DRBD resource on the specified system.

Action: None. Local Recovery will continue to repair the DRBD resource.

138119 ERROR Resource %s is not ready to use in %s seconds (see "DRBD_WAIT_FOR_USABLE_TIMEOUT" in /etc/default/LifeKeeper).

Cause: During resume of the resource the DRBD API ‘wait_for_usable’ is called to verify the DRBD driver has added the new resource and it is ready to use. More detailed errors from the DRBD API ‘wait_for_usable’ command should precede this message.

Action: The default setting for DRBD_WAIT_FOR_USABLE_TIMEOUT is 30 seconds, which should be sufficient for the DRBD driver to configure the resource. Check /var/log/messages for further errors. Increase the default setting if necessary.

138120 WARN If %s has divergent data then it will not automatically connect and will require using the GUI "Resume Replication" option or "lkcli drbd resume —sys %s —tag %s"

Cause: The peer-role for the resource on the target is ‘Unknown’. DRBD could not determine the state of the resource or consistency of the data.

Action: When DRBD is able to connect to the target and determine the state of the resource it will automatically connect if there is NOT divergent data. If there is divergent data then use the command in the message to resume replication to the target.

138121 ERROR Failed to set resource %s to secondary on %s.

Cause: Resume on the specified system failed to set the resource to ‘secondary’. This is most likely due to the resource on the specified system being open or busy.

Action: More detailed errors from the DRBD API that writes the file will precede this message. Check /var/log/messages on the specified system for further errors.

138123 ERROR Failed to repair "Diskless" Resource %s on %s.

Cause: Resource is ‘up’ with the disk NOT attached. This may be due to an IO error or a manual action.

Action: Check /var/log/messages on the specified system for further errors.

138124 ERROR Failed to find the in-service server.

Cause: Resume replication requires a system in the cluster to be in-service and in the ‘Primary’ state.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the in-service system for errors.

138125 ERROR "drbdadm adjust %s" on %s failed. This may cause connection issues for resource %s: %s.

Cause: The DRBD API ‘adjust’ will synchronize the configuration between all systems in the configuration.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the specified system to determine why the ‘adjust’ failed.

138126 ERROR Resource $self->{‘tag’} must be paused on $self->{‘sys’} in order to enable access on the target.

Cause: The specified system is in-service or the ‘paused’ flag does not exist on the specified system.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the specified system to determine why the ‘paused’ flag failed.

138127 ERROR Parent file system resource not found.

Cause: A file system resource could not be found in the hierarchy.

Action: Verify there is a file system resource that is a parent of the DRBD resource. If there is a parent file system resource then check /var/log/lifekeeper.log for errors.

138128 ERROR Parent file system resource is mounted on $dev that is not the DRBD device $self->{‘drbd’}.

Cause: The file system resource is not mounted on the correct DRBD device node.

Action: Unmount the device listed in the error message, ‘unmount /dev/drbd<num>’. Retry pause.

138129 ERROR Failed to set up temporary read/write access to data for $self->{‘tag’}. Error: $ret

Cause: The parent file system was not successfully mounted.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the local system to determine why the mount failed.

138131 ERROR Parent file system resource not found.

Cause: A file system resource could not be found in the hierarchy.

Action: Verify there is a file system resource that is a parent of the DRBD resource. If there is a parent file system resource then check /var/log/lifekeeper.log for errors.

138132 ERROR Failed to undo temporary access for $self->{‘tag’} on $self->{‘sys’}. Error: $ret. Please verify that $fsid is not mounted on server $self->{‘sys’}.

Cause: The resume of a paused DRBD resource failed to unmount the file system on the specified system. A process that is not killable may have a file open on the file system.

Action: Unmount the file system with ‘umount <mount_point>’. If this fails check /var/log/messages for errors.

138133 ERROR %s must be in-service to resume replication.

Cause: A DRBD resource must be in-service on a system in the cluster before replication can be resumed.

Action: Bring the resource in-service on a system and retry your resume.

138134 ERROR Split-brain detected, %s servers are in-service. Take the resource out-of-service on the server with invalid data.

Cause: During a ‘resume’ operation a split-brain was detected where more than one system has a DRBD resource in-service.

Action: Determine which system has the latest or best data. Take the hierarchy with DRBD out-of-service on the system that is in-service that does NOT have the latest data. Make sure the DRBD resource with the latest or best data is in-service. Resume replication to all systems.

138135 ERROR Split-brain detected, %s servers are in-service. Take the resource out-of-service on the server with invalid data.

Cause: During a ‘pause’ operation a split-brain was detected where more than one system has a DRBD resource in-service.

Action: Determine which system has the latest or best data. Take the hierarchy with DRBD out-of-service on the system that is in-service that does NOT have the latest data. Make sure the DRBD resource with the latest or best data is in-service. Resume replication to all systems.

138136 ERROR Pause does not support pausing the in-service server.

Cause: The system that is in-service is being paused.

Action: The ‘pause’ operation should be run on the DRBD resource that is not in-service.

138137 ERROR Resume does not support resuming the in-service server.

Cause: The ‘resume’ operation is being run on the in-service system.

Action: The ‘resume’ operation should be run on the DRBD resource that is ‘paused’, aka not in-service.

138138 FATAL Tag %s is paused on server %s, a force online is required to in-service a paused target. Use "Force Resource Online (Primary)" option in the GUI or "lkcli drbd force —sys %s —tag %s".

Cause: A ‘restore’ operation was done on a DRBD resource that is in the ‘paused’ state.

Action: A ‘force online’ is required to in-service a paused resource. Use the command in the message to force the resource in-service.

138139 ERROR Tag %s is not a valid DRBD resource.

Cause: The tag passed to ‘lkcli’ was not for a DRBD resource.

Action: Retry the command with the correct tag.

138140 ERROR LifeKeeper is not ALIVE on server %s, all servers must be ALIVE and at least 1 communication path UP to update options.

Cause: All systems must have LifeKeeper running and communication paths up before using ‘lkcli’.

Action: Make sure LifeKeeper is running on all systems and communication paths are up.

138141 ERROR Resource is in split-brain, resolve split-brain before modifying options.

Cause: During a ‘lkcli’ command a split-brain was detected where more than one system has a DRBD resource in-service.

Action: Determine which system has the latest or best data. Take the hierarchy with DRBD out-of-service on the system that is in-service that does NOT have the latest data. Make sure the DRBD resource with the latest or best data is in-service. Resume replication to all systems. After resolving the split-brain the ‘lkcli’ command can be retried.

138142 ERROR Failed to update option %s for %s on %s.

Cause: The ‘lkcli’ command failed on the specified system.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the specified system to determine why the ‘lkcli’ command failed.

138146 ERROR Failed to validate entry "%s": %s

Cause: The DRBD API ‘validate_drbd_option’ returned that the entry passed in was not a valid entry.

Action: Valid entries and values are listed in ‘lkcli drbd options -h’.

138147 ERROR Failed to get XML config information for resource %s.

Cause: The ‘drbdadm dump-xml <res>’ command failed.

Action: Check /var/log/lifekeeper.log and /var/log/messages to determine why the command failed.

138148 ERROR Both host1 (%s) and host2 (%s) must be defined.

Cause: When making changes to the connections in a DRBD resource with the ‘lkcli’ both hosts for the connection must be defined.

Action: See the ‘lkcli drbd options’ section of the DRBD User’s Guide for more information on required parameters.

138149 ERROR Failed %s %s: %s

Cause: The DRBD API command listed failed causing the ‘lkcli’ command to fail.

Action: More detailed errors from the DRBD API call will precede this message. Check /var/log/messages for further errors.

138150 ERROR Failed to find connection for host1 (%s) and host2 (%s).

Cause: The DRBD API ‘get_connection’ was not able to find a connection for the hosts provided in the DRBD configuration file.

Action: Run ‘drbdadm status’ to see the list of connections for each resource. Each host name or system listed is the same format as the systems configured in LifeKeeper. Use sys_list(1M) to see the list of systems configured in LifeKeeper.

138151 ERROR Failed %s %s: %s

Cause: The DRBD API for the specified command failed causing the ‘lkcli’ command to fail.

Action: More detailed errors from the DRBD API will precede this message. Check /var/log/messages for further errors.

138152 ERROR Failed %s %s: %s

Cause: The DRBD API command listed failed.

Action: More detailed errors from the DRBD API will precede this message. Check /var/log/messages for further errors.

138153 ERROR Failed to write resource file %s.

Cause: The resource file could not be written. The file system that /etc/drbd.d is on may be read only or out of space.

Action: More detailed errors from the DRBD API ‘write_resource_file’ will precede this message. Check /var/log/messages for further errors.

138154 ERROR Failed to write resource DB file %s.

Cause: The DRBD API ‘store’ failed. The file system that /etc/drbd.d is on may be read only or out of space.

Action: More detailed errors from the DRBD API ‘store’ will precede this message. Check /var/log/messages for errors on the target system.

138155 ERROR Failed to copy resource files to %s.

Cause: Resource files were not copied to the specified system.

Action: More detailed errors from the copy failure will precede this message.

138156 ERROR Failed to copy resource files to %s.
138157 ERROR Failed to "adjust" resource %s after copying original files back.

Cause: The DRBD API ‘adjust’ will synchronize the configuration between all systems in the configuration.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the local system to determine why the adjust failed.

138158 ERROR Failed to copy original resource files to %s.

Cause: While copying configuration file a failure occurred and while the original resources were being restored there was a failure.

Action: DRBD requires that all resources files are identical on each server. The local server most likely has the most up to date or best files. Copy the most up to date or best files to all servers. Run ‘drbdadm adjust <res>’ on each server to synchronize the configuration.

138159 ERROR Failed to validate entry "%s", value %s: %s

Cause: The DRBD API ‘validate_drbd_option’ returned that the entry or value was not valid.

Action: Valid entries and values are listed in ‘lkcli drbd options -h’.

138160 ERROR Failed to find volume.

Cause: During a ‘lkcli drbd options’ command where the ‘entry’ changed the volume of a resource, the volume was not found in the configuration. The file system that /etc/drbd.d is on may be read only or out of space.

Action: The configuration file may be corrupted.

138161 ERROR "-v" requires an entry in the "disk" section, however entry "%s" is for section "%s".

Cause: The ‘lkcli drbd options -v’ is for an ‘entry’ that is not intended for the ‘disk’ section of a resource configuration.

Action: Valid entries and values are listed in ‘lkcli drbd options -h’.

138162 ERROR "-c" requires an entry in the "net" section, however entry "%s" is for section "%s".

Cause: The ‘lkcli drbd options -c’ is for an ‘entry’ that is not intended for the ‘net’ section of a resource configuration.

Action: Valid entries and values are listed in ‘lkcli drbd options -h’.

138163 ERROR Failed to recover original %s for %s: %s

Cause: After an error the rename of the original resource file failed.

Action: Check the error listed in the message and check /var/log/messages.

138164 ERROR Failed to recover original %s for %s: %s

Cause: After an error the rename of the original resource database file failed.

Action: Check the error listed in the message and check /var/log/messages.

138165 ERROR Failed to recover original %s for %s: %s

Cause: After an error the rename of the original resource file failed.

Action: Check the error listed in the message and check /var/log/messages.

138166 ERROR Failed to recover original %s for %s: %s

Cause: After an error the rename of the original resource database file failed.

Action: Check the error listed in the message and check /var/log/messages.

138167 ERROR Failed to recover original %s for %s: %s

Cause: After an error the rename of the original resource file failed.

Action: Check the error listed in the message and check /var/log/messages.

138168 ERROR Failed to recover original %s for %s: %s

Cause: After an error the rename of the original resource database file failed.

Action: Check the error listed in the message and check /var/log/messages.

138169 ERROR Failed to recover original %s for %s: %s

Cause: After an error the rename of the original resource file failed.

Action: Check the error listed in the message and check /var/log/messages.

138170 ERROR Failed to recover original %s for %s: %s

Cause: After an error the rename of the original resource database file failed.

Action: Check the error listed in the message and check /var/log/messages.

138171 ERROR Failed: drbdsetup xml-help %s: %s

Cause: The ‘drbdsetup xml-help <res>’ command failed.

Action: The failure output is included in the message.

138172 ERROR Failed: drbdsetup xml-help %s: %s

Cause: The ‘drbdsetup xml-help <group>’ command failed.

Action: The failure output is included in the message.

138202 ERROR cmd::Usage: $usage

Cause: The usage ‘restore -t <tag> -i <id>’. The <tag> or <id> are not defined.

Action: Retry restore with <tag> and <id>.

138204 ERROR cmd:$tag:Error getting resource information for $tag on server $drbd::me.

Cause: The DRBD restore was not able to match the $tag with a DRBD resource.

Action: Verify that $tag is a DRBD resource on server $sys and that the ID on the command line matches the resource ID.

138212 ERROR cmd:$tag:Usage: $usage

Cause: The usage is ‘remove -t <tag> [-i <id>] [-d]’. The <tag> is not defined.

Action: Retry remove with a valid DRBD tag.

138214 ERROR cmd:$tag:Error getting resource information for $tag on server $drbd::me.

Cause: The DRBD remove was not able to match the $tag with a DRBD resource.

Action: Verify that $tag is a DRBD resource on server $sys and if the ID was passed in that it matches the resource ID.

138222 ERROR cmd:$tag:Usage: $usage

Cause: The usage is ‘delete -t <tag> -i <id> [-U]’. The <tag> or <id> are not defined.

Action: Retry delete with <tag> and <id>.

138224 ERROR cmd:$tag:Error getting resource information for $tag on server $drbd::me.

Cause: The DRBD delete was not able to match the $tag with a DRBD resource.

Action: Verify that $tag is a DRBD resource on server $sys and that the ID on the command line matches the resource ID.

138230 ERROR cmd::Usage: %s <event>, event is not defined.

Cause: DRBD event handler called drbd_event without an <event>.

Action: Verify ‘handlers’ defined in the DRBD resource configuration are valid.

138231 ERROR cmd::Environment variable DRBD_RESOURCE not found when handling event $event (this is normally passed in by drbdadm).

Cause: The DRBD event handler did not set the DRBD_RESOURCE environment variable.

Action: Verify ‘handlers’ defined in the DRBD resource configuration are valid.

138232 ERROR cmd::Failed to retrieve $resource_db_file when handling event $event.

Cause: The drbd_event utility to handle DRBD events was not able to retrieve or open the resource database file that matches the resource.

Action: Verify ‘handlers’ defined in the DRBD resource configuration are valid.

138233 ERROR cmd::Failed to find \"LifeKeeper UUID\" in the resource DB file $resource_db_file when handling event $event.

Cause: The resource database file does not contain the LifeKeeper UUID.

Action: Verify the configuration file is not corrupt by comparing with the other systems it is extended with.

138234 ERROR cmd:$tag:DRBD detected a split-brain on resource $resource, data resynchronization will not occur. MANUAL INTERVENTION IS REQUIRED. In order to initiate data resynchronization, you should take tag $tag out of service on $drbd::me or $peer. If replication does not automatically resume use the resume option in the GUI or lkcli to force replication to resume.

Cause: DRBD event handler detected multiple systems with a resource in the ‘Primary’ state.

Action: Follow instructions in the message.

138235 ERROR cmd:$tag:DRBD detected \"out-of-sync\" blocks on resource $resource.

Cause: DRBD event handler reported block out-of-sync. This may be caused by a resync, replication not being able to keep up with the change rate, a verify miscompare, etc.

Action: Increase network bandwidth or decrease change rate if the change rate is exceeding the network bandwidth. DRBD will automatically synchronize the out-of-sync block.

138236 WARN cmd:$tag:DRBD is beginning to resync (source) resource $resource.

Cause: DRBD event handler reports that a resync has begun on the source of the resynchronization.

Action: None.

138237 WARN cmd:$tag:DRBD is beginning to resync (target) resource $resource.

Cause: DRBD event handler reports that a resync has begun on the target.

Action: None.

138240 ERROR cmd:$tag:DRBD detected an IO error on resource $resource on $drbd::me.

Cause: DRBD event handler reported an IO error on the volume of the DRBD resource. This will cause the resource to become ‘diskless’.

Action: Check /var/log/messages for details on the IO error. LifeKeeper will automatically reattach the disk. If the IO error persists, replace the faulty drive.

138241 WARN cmd::Failed to find instance for ID $uuid when handling event $event. This message can be ignored during extend as a DRBD event may occur before the instance is created.

Cause: The ‘LifeKeeper_UUID’ entry in the resource file does not match any LifeKeeper resource currently configured.

Action: The event is for a device no longer configured in LifeKeeper. Restore the LifeKeeper configuration from a lkbackup or remove the DRBD resource and re-add it with LifeKeeper.

138250 ERROR snmptrap returned $err for Trap 210
138251 ERROR COMMAND OUTPUT: cat /tmp/err$$
138253 ERROR [$SUBJECT event] mail returned $err
138254 ERROR COMMAND OUTPUT: cat /tmp/err$$
138256 ERROR snmptrap returned $err for Trap 211
138257 ERROR COMMAND OUTPUT: cat /tmp/err$$
138259 ERROR [$SUBJECT event] mail returned $err
138260 ERROR COMMAND OUTPUT: cat /tmp/err$$
138330 ERROR cmd::Usage: $usage

Cause: The usage is ‘quickCheck -t <tag> -i <id>’. The <tag> or <id> are not defined.

Action: Retry quickCheck with <tag> and <id>.

138400 FATAL cmd::Insufficient input parameters.\n%s

Cause: The usage is ‘create -t <tag> -n <mount> -f <fs_tag> -y <fs_type> -p <local_part> -s <switchback>’. All required parameters were not provided.

Action: Retry create with all parameters defined properly.

138401 ERROR cmd:$tag:END failed create of %s on system %s with signal $sig.

Cause: The LifeKeeper DRBD create failed. More detailed errors will precede this message.

Action: Resolve errors and retry.

138405 FATAL cmd:$tag:The requested filesystem type "%s" is not supported by LifeKeeper. LifeKeeper supported file system types are listed under "%s/lkadm/subsys/gen/filesys/supported_fs".

Cause: An unsupported file system type was entered.

Action: Retry with a supported file system type.

138406 FATAL cmd:$tag:Failed to find a unique disk name for $local_part. The DRBD kit requires devices with unique disk names in \"/dev/disk/by-\*\".

Cause: The LifeKeeper DRBD recovery kit requires the device to be used as a volume disk have a unique disk name listed in "/dev/disk/by-*".

Action: Select a disk or partition that has a unique disk name. Partitions with a GUID will have a unique name, see gdisk(8) as one way to create a GUID name.

138408 FATAL cmd:$self->{‘tag’}:Failed restore of tag %s on system %s

Cause: The LifeKeeper restore operation failed.

Action: More detailed errors will precede this message.

138410 FATAL cmd:$tag:Cannot make the %s filesystem on "%s" (%d)

Cause: The ‘mkfs’ command failed with the error code listed. Additional error information will be logged following this message.

Action: Check /var/log/messages for more details on the error.

138411 FATAL cmd:$tag:%s

Cause: The ‘mkfs’ command failed and this log message is the internal perl error.

Action: None.

138412 FATAL cmd:$tag:Cannot mount %s on "%s" (%d)

Cause: The ‘mount’ command failed with the error code listed. Additional error information will be logged following this message.

Action: Check /var/log/messages for more details on the error.

138413 FATAL cmd:$tag:%s

Cause: The ‘mount’ command failed and this log message is the internal perl error.

Action: None.

138414 FATAL cmd:$tag:Cannot create filesys hierarchy "%s".

Cause: The LifeKeeper /opt/LifeKeeper/bin/creFShier utility failed to create the file system resource.

Action: More detailed errors will precede this message.

138453 ERROR cmd:$target_tag:END failed extend of resource %s on system %s with signal $sig.

Cause: The LifeKeeper extend failed.

Action: More detailed errors will precede this message.

138454 FATAL cmd:$target_tag:Resource %s is configured on %d servers, the DRBD Recovery Kit is restricted to %d servers.

Cause: An attempt was made to extend a DRBD resource to a third server.

Action: None.

138455 FATAL cmd:$target_tag:IPv6 detected, this is not supported at this time.
138456 FATAL cmd:$target_tag:Failed to find a unique disk name for $target_part. The DRBD kit requires devices with unique disk names in \"/dev/disk/by-\*\".

Cause: The LifeKeeper DRBD recovery kit requires the device to be used as a volume disk have a unique disk name listed in "/dev/disk/by-*".

Action: Select a disk or partition that has a unique disk name. Partitions with a GUID will have a unique name, see gdisk(8) as one way to create a GUID name.

138457 FATAL cmd:$target_tag:IPv6 detected, this is not supported at this time.

Cause: The selected network is IPV6.

Action: Use IPV4 network.

138458 ERROR cmd:$target_tag:Invalid protocol setting %s for %s, valid settings are "Asynchronous/Synchronous/Memory Synchronous".

Cause: The LifeKeeper DRBD recovery kit only supports ‘asynchronous’, ‘asynchronous’, and ‘memory synchronous’ protocols.

Action: Enter the correct protocol.

138459 ERROR cmd:$target_tag:Invalid parameter: template_sys=%s, template_tag=%s, target_tag=%s, target_part=%s.
138460 FATAL cmd:$target_tag:Failed to copy $template_self->{‘resource_db_file’} from $template_sys.

Cause: The extend of a DRBD resource was not able to copy the resource database file from the template system to the target. The file system that /etc/drbd.d is on may be read only or out of space.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the template and target system to determine why the copy failed. Resolve the problem and retry extend.

138464 FATAL cmd:$target_tag:Failed to write resource file %s.

Cause: The extend of a DRBD resource could not write the resource file to the target system. The file system that /etc/drbd.d is on may be read only or out of space.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the target system to determine why the write failed. Resolve the problem and retry extend.

138465 FATAL cmd:$target_tag:Failed to store resource DB file %s.

Cause: The DRBD API ‘store’ failed. The file system that /etc/drbd.d is on may be read only or out of space.

Action: More detailed errors from the DRBD API ‘store’ will precede this message. Check /var/log/messages for errors on the target system. Retry extend when errors are resolved.

138466 FATAL cmd:$target_tag:Failed to create resource %s.

Cause: The DRBD API ‘create_md’ failed.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the target system for errors.

138467 FATAL cmd:$target_tag:Failed to update the configuration for resource %s on %s.
138468 FATAL cmd:$target_tag:Failed to update the configuration for resource %s on %s.

Cause: The extend of a DRBD resource was not able to copy the resource files to the template system. The resource files must be synchronized on all systems.

Action: Check /var/log/lifekeeper.log and /var/log/messages on the target and template systems for errors.

138469 ERROR cmd:$target_tag:Failed to remove %s from the resource %s on %s. Run "lkcli drbd removesys —sys %s —tag %s" on %s to remove %s from the configuration.

Cause: The LifeKeeper instance create failed during the extend of a DRBD resource. During cleanup of the DRBD resource configuration the system specified was not removed from the resource database file.

Action: Run the command listed in the message to remove the system from the DRBD configuration. This will be necessary before retrying the extend.

138470 FATAL cmd:$target_tag:Error creating resource $target_tag on system $target_sys.

Cause: The extend of a DRBD resource was not able to create the LifeKeeper instance on the target server.

Action: Check /var/log/lifekeeper.log for errors.

138471 FATAL cmd:$target_tag:Network endpoints \"$template_ip/$target_ip\" do not match a configured communication path\n
138472 FATAL cmd:$target_tag:Network endpoints \"$node_ip/$target_ip\" do not match a configured communication path\n
138500 ERROR cmd:$template_tag:Cannot extend resource "%s" to server "%s".

Cause: The extend of a DRBD resource to the server listed failed. More detailed errors will precede this message.

Action: Check /var/log/messages for errors on the target system. Retry extend when errors are resolved.

138501 ERROR cmd:$template_tag:Invalid parameter. %s.

Cause: The usage is ‘canextend <template server> <template tag>’. The ‘template server’ or ‘template tag’ are not defined.

Action: Retry the extend of the DRBD resource with the correct template server name and the template tag for the DRBD resource.

138502 ERROR cmd:$template_tag:The target and the template server are the same. Please specify the correct value for the template server. %s

Cause: During the extend of a DRBD resource the canextend utility was run on the target.

Action: Run the extend of the DRBD resource on the target.

138503 ERROR cmd:$template_tag:Failed to load the drbd module. %s

Cause: During the extend of a DRBD resource ‘modprobe drbd’ failed. The error from modprobe is included in the message.

Action: Verify ‘kmod-drbd’ package is installed for the kernel running on the target.

138550 WARN $REM_MACH is not alive, create flag ${FLAGTAG}_wait_for_peer.

Cause: During startup of LifeKeeper the system listed did not establish communications. LifeKeeper will not automatically resume or restore DRBD resources until all servers are available to assure the most up-to-date data is available.

Action: Make sure LifeKeeper is running on all systems and resolve any communication issues.

138561 ERROR Failed to "down" resource %s: %s

Cause: The DRBD API ‘down’ failed. The output from the API is included in the message.

Action: Unmount the file system if it is mounted on the DRBD resource. Stop or kill any process that has the DRBD resource open. Run ‘drbdadm down <res>’ when the processes are stopped and the file system unmounted.

138570 FATAL cmd::The name of the machine was not specified.

Cause: The machine name was not provided to the comm_up script.

Action: Check /var/log/lifekeeper.log for errors.

138600 ERROR cmd::Usage: $usage

Cause: The usage is ‘recover -d <tag> -n <id>’. The <tag> or <id> are not defined.

Action: Check /var/log/lifekeeper.log for further error messages.

Feedback

Was this helpful?

Yes No
You indicated this topic was not helpful to you ...
Could you please leave a comment telling us why? Thank you!
Thanks for your feedback.

Post your comment on this topic.

Post Comment