Using Network Attached Storage

There are a couple of special considerations to take into account when configuring LifeKeeper to use an NFS file server (Network Attached Storage) as cluster storage.

Use the NAS Recovery Kit

The optional Network Attached Storage (NAS) recovery kit is required when using an NFS server as a shared storage array with LifeKeeper for Linux. Install the NAS recovery kit (and a license) on each cluster node. See the NAS Recovery Kit documentation for more details.

Possible Error Message

When using Network Attached Storage (NAS) with MySQL, you may experience MySQL instances not restarting following a failover due to a system crash. The MySQL error log should indicate the cause of the error.

MySQL 5.0

110523 22:10:58 mysqld started
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
110523 22:10:58 InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.

MySQL 5.5

110524 10:52:20 InnoDB: The InnoDB memory heap is disabled
110524 10:52:20 InnoDB: Mutexes and rw_locks use GCC atomic builtins
110524 10:52:20 InnoDB: Compressed tables use zlib 1.2.3
110524 10:52:20 InnoDB: Initializing buffer pool, size = 128.0M
110524 10:52:20 InnoDB: Completed initialization of buffer pool
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
110524 10:52:20 InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.

This indicates that the MySQL mysqld process has set an NFS lock on the file “ibdata1” on the NFS file system that is being controlled by LifeKeeper. The lock was not cleared by the system crash, so LifeKeeper is unable to bring the MySQL instance back into service. MySQL thinks that some other process is using the ibdata1 file.

Solution

To fix this, mount the NFS file system that will hold ibdata1 with the “nolock” NFS option before the File System resource is created. By default, NFS allows file locks to be set. If the “nolock” option is used before resource creation, LifeKeeper will pick up this option and use it each time it brings the file system resource in service. Since LifeKeeper will be controlling access (from the cluster nodes) to the file system containing ibdata1, the lock is not typically critical. The NFS mount options used during testing were “rw,sync,tcp,nfsvers=3,nolock”.

It is not necessary to use the “nolock” on other file systems used by the MySQL resource hierarchy such as the file system where the MySQL binaries are located.

If the NAS File System resource has already been created without the “nolock” option set, use the following procedure to change the mount option:

1. Using the LifeKeeper GUI, take the file system resource that needs to be changed out of service. This can be done from the LifeKeeper GUI putting the pointer on the file system resource and doing a right mouse click, and select Out of Service from the drop-down menu. This action may take parent resources out of service as well.

2. Confirm the Out of Service action and allow the process to complete.

3. Once the file system resource is out of service, you can put the pointer on the resource and do another right mouse click, and from the drop-down menu select Change Mount Options.

4. In the popup window, add nolock to the line of options, and click Set Value. You will need to repeat steps 3 and 4 for each node in the cluster.

5. Bring the NAS File System resource back in service by doing a right mouse click, and selecting In Service.

6. The File System resource’s property panel should now reflect that “nolock” is one of the current mount options.