P2000 SAS | Linux | Multipath | HPSA driver - not creating devices for all paths

[StewartM]

Components:

1x C7000 blade enclosure, fitted with 2x HP 3G SAS blade switches
1x P2000 SAS storage enclosure with 2x storage controllers
Multiple BL460c G1 blades fitted with P712M SAS controllers in mezzanine slot 1
OS = Debian Squeeze - running 2.6.32-5-amd64

Basic issue:

Getting multipath running with 4 paths to the storage. Dependent on which driver we use, the server can see 4 paths to a given volume but we see different problems when trying to manage multiple paths.

Underlying config:

Zoning/Storage config is congfigured so that the test blade we are trying to get running can see a single volume via 4 paths - a connection to each switch which has a connection to each storage controller

Previous History:

Initially, we tried to get the config running with the CCISS driver:

• The driver creates 4x devices - /dev/cciss/c1d[0-3] (in addition to /dev/cciss/c0d0 for the blades internal disks)
• In sysfs the driver creates 4x directories - /sys/block/cciss!c1d[0-3] (in addition to /sys/block/cciss!c0d0 for the blades internal disks)
• The nature of the device names in sysfs, breaks the basic functionality of the default multipath 'getuid_callout' function but can be fixed using a custom script to collect the volume UID

At this point `multipath -v3` displays all 4 paths, and a multipath device has been created, however it is unusable as multipathd regards all 4 paths as being 'faulty'. They aren't and judging by `strace` the userland multipath tools are not getting as far as even issuing any calls to the storage volume to test it (irrespective of which 'path_checker' method we configure in multipath.conf). We believe that this is again because of the curious naming format in sysfs for the devices (cciss!c1d[0-3])

Current Issue:

We noticed that Debian have backported the HPSA driver into the 2.6.32 kernel and so we have tried using this instead, passing 'cciss.cciss_allow_hpsa=1' as a boot parameter, to ensure that the HPSA driver is used for the P712M, attached to the P2000 storage. This gives with one hand and takes with another.

• As described in the HPSA driver docs, the driver creates device files at /dev/sd[a-z]
• As described in the HPSA driver docs, the driver creates directories at /sys/block/sd[a-z]
• However, the driver is only creating a device file for the 1st of the 4 paths (/dev/sda) which is shown as 'ready' rather than 'faulty' by multipath.

Tried passing 'hpsa_simple_mode=1' in the boot parameters. Nothing in the docs suggested that this would have any bearing, but it was easy to test and rule out and made no difference.

This behaviour was also seen in Centos 6.2 using a more current 2.6.34 kernel.

So I guess the questions are:

1. Does the HPSA driver support multiple paths? (I see nothing in the docs to suggest that it doesn't or requires any special config)
2. If 1 is yes, how do I persuade it to create devices for the other 3 missing paths?

If we can clear that hurdle, then we should finally have a fully operational multipath setup.

[Sebastian.Koehler]

I've not seen it myself, but I heard the issue is related to the SAS switch in that case. Only the 6G SAS switch is compatible and working with the 6G HBA and the P2000 G3 which is also 6G. Just checked SPOCK to confirm this.

Regards,
Sebastian

[StewartM · Accepted solution]

We have managed to fix this and is caused by a bug in the HPSA driver. A patch has been added to the bleeding edge kernel source, but has not made it back into the stable versions running under Debian (and, I think, Redhat/Centos).

The fix for the bug was discovered here:

http://www.spinics.net/lists/linux-scsi/msg56690.html

but was not the end of the story. A little way before the function that was patched in the link above, is an array that specifies a series of HP products for which that specific code should be executed. The 'P2000 G3 SAS' model name needed adding in to that array.  So the complete patch is:

--- hpsa.c      2012-02-24 09:38:14.000000000 +0000
+++ hpsa.patched        2012-02-24 09:38:43.000000000 +0000
@@ -1483,6 +1483,7 @@
        "MSA2024",
        "MSA2312",
        "MSA2324",
+       "P2000 G3 SAS",
        NULL,
 };
 
@@ -1512,30 +1513,26 @@
 
        if (is_logical_dev_addr_mode(lunaddrbytes)) {
                /* logical device */
-               if (unlikely(is_scsi_rev_5(h))) {
-                       /* p1210m, logical drives lun assignments
-                        * match SCSI REPORT LUNS data.
+               lunid = le32_to_cpu(*((__le32 *) lunaddrbytes));
+               if (is_msa2xxx(h, device)) {
+                       /* msa2xxx way, put logicals on bus 1
+                        * and match target/lun numbers box
+                        * reports.
                         */
-                       lunid = le32_to_cpu(*((__le32 *) lunaddrbytes));
-                       *bus = 0;
-                       *target = 0;
-                       *lun = (lunid & 0x3fff) + 1;
+                       *bus = 1;
+                       *target = (lunid >> 16) & 0x3fff;
+                       *lun = lunid & 0x00ff;
                } else {
-                       /* not p1210m... */
-                       lunid = le32_to_cpu(*((__le32 *) lunaddrbytes));
-                       if (is_msa2xxx(h, device)) {
-                               /* msa2xxx way, put logicals on bus 1
-                                * and match target/lun numbers box
-                                * reports.
-                                */
-                               *bus = 1;
-                               *target = (lunid >> 16) & 0x3fff;
-                               *lun = lunid & 0x00ff;
+                       if (likely(is_scsi_rev_5(h))) {
+                               /* All current smart arrays (circa 2011) */
+                               *bus = 0;
+                               *target = 0;
+                               *lun = (lunid & 0x3fff) + 1;
                        } else {
-                               /* Traditional smart array way. */
+                               /* Traditional old smart array way. */
                                *bus = 0;
-                               *lun = 0;
                                *target = lunid & 0x3fff;
+                               *lun = 0;
                        }
                }
        } else {

 This was applied to the HPSA driver in the standard Debian kernel sources and after a kernel compile and reboot, the driver & mutlipath are working correctly.