Storage and Adapter Configuration

There are several multipath I/O solutions either already available or currently being developed for the Linux environment.  SIOS Technology Corp. is actively working with a number of server vendors, storage vendors, adapter vendors and driver maintainers to enable LifeKeeper to work with their multipath I/O solutions.  LifeKeeper’s use of SCSI reservations to protect data integrity presents some special requirements that frequently are not met by the initial implementation of these solutions.

Refer to the technical notes below for supported disk arrays to determine if a given array is supported with multiple paths and with a particular multipath solution.  Unless an array is specifically listed as being supported by LifeKeeper with multiple paths and with a particular multipath solution, it must be assumed that it is not.

In multipath configurations, performing heavy I/O while paths are being manipulated can cause a system to temporarily appear to be unresponsive.  When the multipath software moves the access of a LUN from one path to another, it must also move any outstanding I/Os to the new path.  The rerouting of the I/Os can cause a delay in the response times for these I/Os.  If additional I/Os continue to be issued during this time, they will be queued in the system and can cause a system to run out of memory available to any process.  Under very heavy I/O loads, these delays and low memory conditions can cause the system to be unresponsive such that LifeKeeper may detect a server as down and initiate a failover.

There are many factors that will affect the frequency at which this issue may be seen.

• The speed of the processor will affect how fast I/Os can be queued.  A faster processor may cause the failure to be seen more frequently.
  
• The amount of system memory will affect how many I/Os can be queued before the system becomes unresponsive.  A system with more memory may cause the failure to be seen less frequently.
  
• The number of LUNs in use will affect the amount of I/O that can be queued.
  
• Characteristics of the I/O activity will affect the volume of I/O queued.  In test cases where the problem has been seen, the test was writing an unlimited amount of data to the disk.  Most applications will both read and write data.  As the reads are blocked waiting on the failover, writes will also be throttled, decreasing the I/O rate such that the failure may be seen less frequently.
  

For example, during testing of the IBM DS4000 multipath configuration with RDAC, when the I/O throughput to the DS4000 was greater than 190 MB per second and path failures were simulated, LifeKeeper would (falsely) detect a failed server approximately one time out of twelve.  The servers used in this test were IBM x345 servers with dual Xeon 2.8GHz processors and 2 GB of memory connected to a DS4400 with 8 volumes (LUNs) per server in use.  To avoid the failovers, the LifeKeeper parameter LCMNUMHBEATS (in /etc/default/LifeKeeper) was increased to 16.  The change to this parameter results in LifeKeeper waiting approximately 80 seconds before determining that an unresponsive system is dead, rather than the default wait time of approximately 15 seconds.

With some large storage configurations (for example, multiple logical volume groups with 10 or more LUNs in each volume group), LifeKeeper may not be able to complete a sendevent within the default timeout of 300 seconds when a failure is detected.  This results in the switchover to the backup system failing.  All resources are not brought in-service and an error message is logged in the LifeKeeper log.

The recommendation with large storage configurations is to change SCSIERROR from “event” to “halt” in the /etc/default/LifeKeeper file.  This will cause LifeKeeper to perform a “halt” on a SCSI error.  LifeKeeper will then perform a successful failover to the backup system.

The HP 3PAR StoreServ 7200 was tested by a SIOS Technology Corp. partner with the following configurations:

HP 3PAR StoreServ 7200 (Firmware (HP 3PAR OS) version 3.1.2) using QLogic QMH2572 8Gb FC HBA for HP BladeSystem c-Class (Firmware version 5.06.02 (90d5)), driver version 8.03.07.05.06.2-k (RHEL bundled) with DMMP (device-mapper-1.02.66-6, device-mapper-multipath-0.4.9-46.el6).

The test was performed with LifeKeeper for Linux v8.1.1 using RHEL 6.2 (x86_64).

Note: 3PAR StoreServ 7200 returns a reservation conflict with the default path checker. To avoid this conflict, set the following parameter in “/etc/default/LifeKeeper”:

DMMP_REGISTRATION_TYPE=hba

The HP 3PAR StoreServ 7400 was tested by a SIOS Technology Corp. partner with the following configurations:

HP 3PAR StoreServ 7400 (Firmware (HP 3PAR OS) version 3.1.2) with HP DL380p Gen8 with Emulex LightPulse Fibre Channel SCSI HBA (driver version 8.3.5.45.4p) with DMMP (device-mapper-1.02.66-6, device-mapper-multipath-0.4.9-46.el6).

The test was performed with LifeKeeper for Linux v8.1.1 using RHEL 6.2 (x86_64).

Note: 3PAR StoreServ 7400 returns a reservation conflict with the default path checker. To avoid this conflict, set the following parameter in “/etc/default/LifeKeeper”:

DMMP_REGISTRATION_TYPE=hba

And user friendly device mapping are not supported. Set the following parameter in “multipath.conf”

“user_friendly_names no”

The HP 3PAR StoreServ 7400 was tested by a SIOS Technology Corp. partner with the following configurations:

HP 3PAR StoreServ 7400 (Firmware (HP 3PAR OS) version 3.1.3) using HP Ethernet 10Gb 2-port 560SFP+ Adapter (Networkdriver ixgbe-3.22.0.2) with iSCSI (iscsi-initiator-utils-6.2.0.873-10.el6.x86_64), DMMP (device-mapper-1.02.79-8.el6,device-mapper-multipath-0.4.9-72.el6).

Note: 3PAR StoreServ 7400 iSCSI returns a reservation conflict. To avoid this conflict, set the following parameter in “/etc/default/LifeKeeper”:

DMMP_REGISTER_IGNORE=TRUE

The HP 3PAR StoreServ 7400 was tested by a SIOS Technology Corp. partner with the following configurations:

iSCSI (iscsi-initiator-utils-6.2.0.872-21.el6.x86_64), DMMP (device-mapper-multipath-0.4.9-41, device-mapper-1.02.62-3), nx_nic v4.0.588.

DMMP with the DMMP Recovery Kit on RHEL 6.1 — must be used with the combination of Quorum/Witness Server Kit and STONITH. To disable SCSI reservation, set RESERVATIONS=none in“/etc/default/LifeKeeper”. Server must have interface based on IPMI 2.0.

The HP 3PAR StoreServ 10800 FC was tested by a SIOS Technology Corp. partner with the following configurations:

Firmware (HP 3PAR OS) version 3.1.2 with HP DL380p Gen8 with Emulex LightPulse Fibre Channel HBA (driver version 8.3.5.45.4p) with DMMP (device-mapper-1.02.66-6, device-mapper-multipath-0.4.9-46 el6). The test was performed with SPS for Linux v8.1.2 using RHEL 6.2 (x86_64).

Note: 3PAR StoreServ 10800 FC returns a reservation conflict with the default path checker. To avoid this conflict, set the following parameter in “/etc/default/LifeKeeper”:

DMMP_REGISTRATION_TYPE=hba

And user friendly device mapping are not supported. Set the following parameter in “multipath.conf”

“user_friendly_names no”

The HP MSA 2040 Storage FC was tested by a SIOS Technology Corp. partner with the following configurations:

HP MSA 2040 Storage FC (Firmware GL101R002) using HP SN1000Q 16Gb 2P FC HBA QW972A (Firmware version 6.07.02, driver version 8.04.00.12.06.0-k2 (RHEL bundled)) with DMMP (device-mapper-1.02.74-10, device-mapper-multipath-0.4.9-56).

The test was performed with LifeKeeper for Linux v8.1.2 using RHEL 6.3 (X86_64).

Certified by Hewlett-Packard Company using SIOS LifeKeeper for Linux v7.2 or later.  Model tested was the HP P9500/XP and has been qualified to work with LifeKeeper on the following:

• Red Hat Enterprise for 32-bit, x64 (64-bit; Opteron and Intel EMT64)
  RHEL 5.3, RHEL 5.4, RHEL 5.5
• SuSE Enterprise Server for 32-bit, x64 (64-bit; Opteron and Intel EMT64)
  SLES 10 SP3, SLES 11, SLES 11 SP1
• Native or Inbox Clustering Solutions RHCS and SLE HA
  

The HP StoreVirtual 4330 was tested by a SIOS Technology Corp. partner with the following configurations:

HP StoreVirtual 4330 (Firmware HP LeftHand OS 10.5) using HP Ethernet 1Gb 4-port 331FLR (Networkdriver tg3-3.125g) with iSCSI (iscsi-initiator-utils-6.2.0.872-41.el6.x86_64), DMMP (device-mapper-1.02.74-10.el6,device-mapper-multipath-0.4.9-56.el6).

OS (SAN/iQ) version 11.00 is supported in HP StoreVirtual (LeftHand) storage. All StoreVirtual series are supported, including StoreVirtual VSA as the virtual storage appliance. This storage was tested with the following configurations:

StoreVirtual VSA + RHEL 6.4(x86_64) ＋ DMMP

The HP StoreVirtual 4730 was tested by a SIOS Technology Corp. partner with the following configurations:

HP StoreVirtual 4730 (Firmware HP LeftHand OS 11.5) using HP FlexFabric 10Gb 2-port 536FLB Adapter (Networkdriver bnx2×-1.710.40) with iSCSI (iscsi-initiator-utils-6.2.0.873-10.el6.x86_64), DMMP (device-mapper-1.02.79-8.el6,device-mapper-multipath-0.4.9-72.el6).

LeftHand OS version 11.5 is supported in HP StoreVirtual (LeftHand) storage. All StoreVirtual series are supported, including StoreVirtual VSA as the virtual storage appliance. This storage was tested with the following configurations:

StoreVirtual 4730(11.5.00.0673.0) + RHEL 6.5(x86_64) ＋ DMMP (device-mapper-1.02.79-8.el6.x86_64, device-mapper-multipath-0.4.9-72.el6.x86_64)

The HP StoreVirtual 4330 was tested by a SIOS Technology Corp. partner with the following configurations:

HP StoreVirtual 4330 (Firmware HP LeftHand OS 10.5) using iSCSI (iscsi-initiator-utils-6.2.0.872-41.el6.x86_64), bonding(version: 3.6.0),tg3(version: 3.125g)

To disable SCSI reservation, set RESERVATIONS=none in “/etc/default/LifeKeeper”.

Certified by partner testing in a single path configuration.  Certified by SIOS Technology Corp. in multipath configurations using the Device Mapper Multipath Recovery Kit.

The IBM Storwize V7000 (Firmware Version 6.3.0.1) has been certified by partner testing using iSCSI (iscsi-initiator-utils-6.2.0.872-34.el6.x86_64) with DMMP (device-mapper-1.02.66-6.el6, device-mapper-multipath-0.4.9-46.el6). The test was performed with LifeKeeper for Linux v7.5 using RHEL 6.2.

Restriction: IBM Storwize V7000 must be used in combination with the Quorum/Witness Server Kit and STONITH. Disable SCSI reservation by setting the following in /etc/default/LifeKeeper:

• RESERVATIONS=none

Certified by partner testing in only multipath configuration on Red Hat Enterprise Linux Release 5.6, HBA: NEC N8190-127 Single CH 4Gbps (Emulex LPe1150 equivalent), XIV Host Attachement Kit: Version 1.7.0.

Note: If you have to create over 32 LUNs on IBM XIV Storage System with LifeKeeper, please contact your IBM sales representative for details.

Fujitsu

ETERNUS DX60 S2 / DX80 S2 / DX90 S2 iSCSI

ETERNUS DX410 S2 / DX440 S2 iSCSI

ETERNUS DX8100 S2/DX8700 S2 iSCSI

ETERNUS DX100 S3/DX200 S3 iSCSI

ETERNUS DX500 S3/DX600 S3 iSCSI

ETERNUS DX200F iSCSI

ETERNUS DX60 S3 iSCSI

ETERNUS AF250 / AF650 iSCSI

ETERNUS DX60 S4 / DX100 S4 / DX200 S4 iSCSI

ETERNUS DX500 S4 / DX600 S4 / DX8900S4 iSCSI

ETERNUS AF250 S2 / AF650 S2 iSCSI

ETERNUS DX60 S5 / DX100 S5 / DX200 S5 iSCSI

ETERNUS DX500 S5 / DX600 S5 / DX900S5 iSCSI

ETERNUS AF150 S3 / AF250 S3 / AF650 S3 iSCSI

When using LifeKeeper DMMP ARK for multipath configuration it is necessary to set the following parameters to /etc/multipath.conf.

prio alua

path_grouping_policy group_by_prio

failback immediate

no_path_retry 10

Path_checker tur

Fujitsu

ETERNUS DX60 S2 / DX80 S2 / DX90 S2

• FC, single path and multipath configurations

ETERNUS DX410 S2 / DX440 S2

• FC, single path and multipath configurations

ETERNUS DX8100 S2/DX8700 S2

• FC, single path and multipath configurations

ETERNUS DX100 S3/DX200 S3/DX500 S3/DX600S3

• FC, single path and multipath configurations

ETERNUS DX200F

• FC, single path and multipath configurations

ETERNUS DX60 S3

• FC, single path and multipath configurations

ETERNUS DX8700 S3 / DX8900 S3

• FC, single path and multipath configurations

ETERNUS AF250 / AF650

ETERNUS DX60 S4 / DX100 S4 / DX200 S4

ETERNUS DX500 S4 / DX600 S4 / DX8900S4

ETERNUS AF250 S2 / AF650 S2

ETERNUS DX60 S5 / DX100 S5 / DX200 S5

ETERNUS DX500 S5 / DX600 S5 / DX900S5

ETERNUS AF150 S3 / AF250 S3 / AF650 S3

• iSCSI, single path and multipath configurations

When using LifeKeeper DMMP ARK for multipath configuration it is necessary to set the following parameters to /etc/multipath.conf.

prio alua

path_grouping_policy group_by_prio

failback immediate

no_path_retry 10

Path_checker tur

When using ETERNUS Multipath Driver for multipath configuration, it is no need to set parameters to any configure file.

The NEC iStorage M10e iSCSI was tested by a SIOS Technology Corp. partner with the following configurations:

NEC iStorage M10e iSCSI + 1GbE NIC + iSCSI (iscsi-initiator-utils-6.2.0.873-10.el6.x86_64),SPS (sps-utils-5.3.0-0.el6,sps-driver-E-5.3.0-2.6.32.431.el6)

Protecting Applications and File Systems That Use Multipath Devices: In order for SPS to configure and protect applications or file systems that use SPS devices, the SPS recovery kit must be installed.

Once the SPS kit is installed, simply creating an application hierarchy that uses one or more of the multipath device nodes will automatically incorporate the new resource types provided by the SPS kit.

Multipath Device Nodes: To use the SPS kit, any file systems and raw devices must be mounted or configured on the multipath device nodes (/dev/dd*) rather than on the native /dev/sd* device nodes.

Use of SCSI-3 Persistent Reservations: The SPS kit uses SCSI-3 persistent reservations with a “Write Exclusive” reservation type. This means that devices reserved by one node in the cluster will remain read-accessible to other nodes in the cluster, but those other nodes will be unable to write to the device. Note that this does not mean that you can expect to be able to mount file systems on those other nodes for ongoing read-only access.

LifeKeeper uses the sg_persist utility to issue and monitor persistent reservations. If necessary, LifeKeeper will install the sg_persist(8) utility.

Tested Environment: The SPS kit has been tested and certified with the NEC iStorage disk array using Emulex HBAs and Emulex lpfc driver. This kit is expected to work equally well with other NEC iStorage D, S and M supported by SPS.

[Tested Emulex HBA]

iStorage D-10
=======================
LP952
LP9802
LP1050
LP1150
=======================

iStorage M100
=======================
LPe1150
LPe11002
LPe1250
LPe12002
LPe1105
LPe1205
=======================

Multipath Software Requirements: The SPS kit has been tested with SPS for Linux 3.3.001. There are no known dependencies on the version of the SPS package installed.

Installation Requirements: SPS software must be installed prior to installing the SPS recovery kit.

Adding or Repairing SPS Paths: When LifeKeeper brings an SPS resource into service, it establishes a persistent reservation registered to each path that was active at that time. If new paths are added after the initial reservation, or if failed paths are repaired and SPS automatically reactivates them, those paths will not be registered as a part of the reservation until the next LifeKeeper quickCheck execution for the SPS resource. If SPS allows any writes to that path prior to that point in time, reservation conflicts that occur will be logged to the system message file. The SPS driver will retry these IOs on the registered path resulting in no observable failures to the application. Once quickCheck registers the path, subsequent writes will be successful.

Protecting Applications and File Systems That Use Multipath Devices:
In order for LifeKeeper to configure and protect applications or file systems that use NEC HDLM devices, the HDLM Recovery Kit must be installed.
Once the HDLM Recovery Kit is installed, simply creating an application hierarchy that uses one or more of the multipath device nodes will automatically incorporate the new resource types provided by the HDLM Recovery Kit.

Multipath Device Nodes:
To use the HDLM Kit, any file systems and raw devices must be mounted or configured on the multipath device nodes (/dev/sddlm*) rather than on the native /dev/sd* device nodes.

Use of SCSI-3 Persistent Reservations:
The HDLM Recovery Kit uses SCSI-3 persistent reservations with a “Write Exclusive” reservation type. This means that devices reserved by one node in the cluster will remain read-accessible to other nodes in the cluster, but those other nodes will be unable to write to the device. Note that this does not mean that you can expect to be able to mount file systems on those other nodes for ongoing read-only access. LifeKeeper uses the sg_persist utility to issue and monitor persistent reservations. If necessary, LifeKeeper will install the sg_persist(8) utility.

Hardware Requirements:
The HBA and the HBA driver must be supported by iStorage V10e/V100/V300.

Multipath Software Requirements:
The HDLM kit has been tested with NEC HDLM for Linux as follows:

8.7.9-04、8.7.9-05、8.8.6-00、8.8.6-03

There are no known dependencies on the version of the HDLM package installed.
Note: If using LVM, the LVM settings must be changed. Refer to the NEC HA Dynamic Link Manager manual available on the NEC Support Portal for more information

Linux Distribution Requirements:
Refer to the iStorage V series software Supported OS information on the NEC Support Portal for the supported Linux kernel and the distribution.

Installation Requirements:
NEC HDLM software must be installed prior to installing the HDLM recovery kit. Also, customers wanting to transfer their environment from SCSI devices to NEC HDLM devices must run the Installation setup script after configuring the NEC HDLM environment. therwise, sg3_utils will not be installed.

Adding or Repairing HDLM Paths:
When LifeKeeper brings an NEC HDLM resource into service, it establishes a persistent reservation registered to each path that was active at that time. If new paths are added after the initial reservation, or if failed paths are repaired and NEC HDLM automatically reactivates them, those paths will not be registered as a part of the reservation until the next LifeKeeper quickCheck execution for the NEC HDLM resource. If NEC HDLM allows any writes to that path prior to that point in time, reservation conflicts that occur will be logged to the system message file. The NEC HDLM driver will retry these IOs on the registered path resulting in no observable failures to the application. Once quickCheck registers the path, subsequent writes will be successful. The status will be changed to “Offline(E)” if quickCheck detects a reservation conflict. If the status is “Offline(E)”, customers will need to manually change the status to “Online” using the online NEC HDLM command.

For a fresh installation of a multiple path cluster that uses Secure Path, perform these steps:

1. Install the OS of choice on each server.
2. Install the clustering hardware:  FCA2214 adapters, storage, switches and cables.
3. Install the HP Platform Kit.
4. Install the HP Secure Path software.  This will require a reboot of the system.  Verify that Secure Path has properly configured both paths to the storage.  See Secure Path documentation for further details.
5. Install LifeKeeper.
   

Secure Path supports “persistent” device nodes that are in the form of /dev/spdev/spXX where XX is the device name.  These nodes are symbolic links to the specific SCSI device nodes /dev/sdXX.  LifeKeeper v4.3.0 or later will recognize these devices as if they were the “normal” SCSI device nodes /dev/sdXX.  LifeKeeper maintains its own device name persistence, both across reboots and across cluster nodes, by directly detecting if a device is /dev/sda1 or /dev/sdq1, and then directly using the correct device node.

Note:  Support for symbolic links to SCSI device nodes was added in LifeKeeper v4.3.0.

In order for LifeKeeper to configure and protect applications or file systems that use  devices, the HDLM Recovery Kit must be installed.

Once the HDLM Kit is installed, simply creating an application hierarchy that uses one or more of the multipath device nodes will automatically incorporate the new resource types provided by the HDLM Kit.

The HDLM Kit uses SCSI-3 persistent reservations with a “Write Exclusive” reservation type. This means that devices reserved by one node in the cluster will remain read-accessible to other nodes in the cluster, but those other nodes will be unable to write to the device. Note that this does not mean that you can expect to be able to mount file systems on those other nodes for ongoing read-only access.

LifeKeeper uses the sg_persist utility to issue and monitor persistent reservations. If necessary, LifeKeeper will install the sg_persist(8) utility.

The HDLM Kit has been tested and certified with the Hitachi SANRISE AMS1000 disk array using QLogic qla2432 HBAs and 8.02.00-k5-rhel5.2-04 driver and Silkworm3800 FC switch. This kit is expected to work equally well with other Hitachi disk arrays. The HDLM Kit has also been certified with the SANRISE AMS series, SANRISE USP and Hitachi VSP. The HBA and the HBA driver must be supported by HDLM.

BR1200 is certified by Hitachi Data Systems. Both single path and multipath configuration require the RDAC driver. Only the BR1200 configuration using the RDAC driver is supported, and the BR1200 configuration using HDLM (HDLM ARK) is not supported.

The HDLM kit has been tested with HDLM for Linux as follows:

8.0.1, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.2.0, 8.2.1, 8.4.0, 8.5.0, 8.5.1, 8.5.2, 8.5.3, 8.6.0, 8.6.1, 8.6.2, 8.6.4, 8.6.5, 8.7.0, 8.7.1, 8.7.2, 8.7.3, 8.7.4, 8.7.6, 8.7.7, 8.7.8, 8.8.0, 8.8.1, 8.8.3, ,8.8.4

There are no known dependencies on the version of the HDLM package installed.

Note: If using LVM with HDLM, the version supported by HDLM is necessary. Also, a filter setting should be added to /etc/lvm/lvm.conf to ensure that the system does not detect the /dev/sd* corresponding to the /dev/sddlm*. If you need more information, please see “LVM Configuration” in the HDLM manual.

Linux Distribution Requirements

HDLM is supported in the following distributions:

RHEL 7, 7.1, 7.2, 7.3, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9 (x86_64(*1) non-English version)

RHEL 8.1, 8.2, 8.3, 8.4 (x86_64(*1) non-English version)

(*1) AMD Opteron(Single Core，Dual Core) or Intel EM64T architecture CPU with x86_64 kernel.

(*2) The following kernels are supported
x86_64：3.10.0-123.13.2.el7.x86_64, 3.10.0-123.20.1.el7.x86_64

(*3) The following kernels are supported
x86_64:3：10.0-229.4.2.el7.x86_64,3.10.0-229.20.1.el7.x86_64, 3.10.0-229.34.1.el7.x86_64

(*4) The following kernels are supported
x86_64：3.10.0-327.4.4.el7.x86_64, 3.10.0-327.4.5.el7.x86_64, 3.10.0-327.10.1.el7.x86_64, 3.10.0-327.18.2.el7.x86_64,
3.10.0-327.22.2.el7.x86_64, 3.10.0-327.36.1.el7.x86_64, 3.10.0-327.36.3.el7.x86_64, 3.10.0-327.44.2.el7.x86_64, 3.10.0-327.46.1.el7.x86_64, 3.10.0-327.49.2.el7.x86_64，3.10.0-327.55.2.el7.x86_64，3.10.0-327.55.3.el7.x86_64，3.10.0-327.58.1.el7.x86_64, 3.10.0-327.62.1.el7.x86_64，3.10.0-327.62.4.el7.x86_64, 3.10.0-327.64.1.el7.x86_64, 3.10.0-327.93.1.el7.x86_64, 3.10. 0 327. 9 6 .1 .el 7 .x86_64, 3.10.0-327.98.2.el7.x86_64，3.10.0-327.102.1.el7.x86_64

(*5)The following kernels are supported
x86_64：3.10.0-514.6.1.el7.x86_64，3.10.0-514.10.2.el7.x86_64，3.10.0-514.16.1.el7.x86_64，3.10.0-514.21.1.el7.x86_64，3.10.0-514.26.2.el7.x86_64，3.10.0-514.36.5.el7.x86_64，3.10.0-514.44.1.el7.x86_64, 3.10.0-514.51.1.el7.x86_64, 3.10.0-514.55.4.el7.x86_64

(*6)The following kernels are supported
x86_64：3.10.0-693.1.1.el7.x86_64, 3.10.0-693.5.2.el7.x86_64, 3.10.0-693.11.1.el7.x86_64, 3.10.0-693.11.6.el7.x86_64, 3.10.0-693.21.1.el7.x86_64, 3.10.0-693.43.1.el7.x86_64, 3.10.0-693.33.1.el7.x86_64, 3.10.0-693.46.1.el7.x86_64, 3.10.0-693.62.1.el7.x86_64

(*7)The following kernels are supported
x86_64： 3.10.0-862.3.2.el7.x86_64,  3.10.0-862.14.4.el7.x86_64

(*8)The following kernels are supported
x86_64：3.10.0-957.10.1.el7.x86_64，3.10.0-957.12.2.el7.x86_64,
3.10.0-957.21.3.el7.x86_64，3.10.0-957.27.2.el7.x86_64

(*9)The following kernels are supported
x86_64：3.10.0-1062.1.1.el7.x86_64, 3.10.0-1062.9.1.el7.x86_64, 3.10.0-1062.18.1.el7.x86_64, 3.10.0-1062.56.1.el7.x86_64, 3.10.0-1062.60.1.el7.x86_64, 3.10.0-1062.63.1.el7.x86_64, 3.10.0-1062.67.1.el7.x86_64

(*10)The following kernels are supported
x86_64：4.18.0-147.5.1.el8_1.x86_64, x86_64：4.18.0-147.8.1.el8_1.x86_64

(*11)The following kernels are supported
x86_64：4.18.0-193.13.2.el8_2.x86_64，4.18.0-193.28.1.el8_2.x86_64 , 4.18.0-193.79.1.el8_2.x86_64, 4.18.0-193.95.1.el8_2.x86_64

(*12) Only iSCSI environments are supported

(*13)The following kernels are supported
x86_64：4.18.0-240.22.1.el8_3.x86_6

(*14) The following kernels are supported
x86_64：3.10.0-1160.15.2.el7.x86_64, 3.10.0-1160.31.1.el7.x86_64 ×86_64, 3.10.0-1160.32.2.el7.x86_64, 3.10.0-1160.59.1.el7.x86_64, 3.10.0-1160.76.1.el7.x86_64, 3.10.0-1160.80.1.el7.x86_64， 3.10.0-1160.83.1.el7.x86_64

(*15) The following kernels are supported
x86_64：4.18.0-305.3.1.el8_4.x86_64，4.18.0-305.19.1.el8_4.x86_64，4.18.0-305.25.1.el8_4.x86_64