topic Re: Trying to get multipath working with BL460c & MSA2324sa in HPE EVA Storage

Trying to get multipath working with BL460c & MSA2324sa

BradV — Tue, 18 Jan 2011 13:07:27 GMT

Hello,

I'm using RHEL 6 on the BL460c. I have 8 connections to the MSA and can see the 8 devices, /dev/cciss/c0d[0-7] for the volume presented. I can't seem to get multipathd to create a multipath device for it.

If I do a multipath -v3 -ll for example on one of the devices I see:

cciss!c0d7 not found in pathved
cciss!c0d7 mask = 0x5
cciss!c0d7 dev_t = 104:112
cciss!c0d7 size = 83886080
cciss!c0d7 subsystem = cciss
cciss!c0d7 h:b:t:l = 0:0:7:0
cciss!c0d7 vendor = HP
cciss!c0d7 product = MSA2324sa
cciss!c0d7 revision = M110
cciss!c0d7 get_state
cciss!c0d7 path_checker = tur (controller setting)
cciss!c0d7 checker timeout = 300000 ms (internal default)
cciss!c0d7 path offline
...
==== paths list ====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
0:0:0:0 cciss!c0d0 104:0 -1 undef faulty HP,MSA2324sa running
0:0:1:0 cciss!c0d1 104:16 -1 undef faulty HP,MSA2324sa running
0:0:2:0 cciss!c0d2 104:32 -1 undef faulty HP,MSA2324sa running
0:0:3:0 cciss!c0d3 104:48 -1 undef faulty HP,MSA2324sa running
0:0:4:0 cciss!c0d4 104:64 -1 undef faulty HP,MSA2324sa running
0:0:5:0 cciss!c0d5 104:80 -1 undef faulty HP,MSA2324sa running
0:0:6:0 cciss!c0d6 104:96 -1 undef faulty HP,MSA2324sa running
0:0:7:0 cciss!c0d7 104:112 -1 undef faulty HP,MSA2324sa running
0:0:0:0 sda 8:0 -1 undef ready HP,LOGICAL VOLUME running

This is a brand new installation.

Any thoughts as to what I might be missing?

Regards,

Brad

Re: Trying to get multipath working with BL460c & MSA2324sa

Bhavnish Kamboj (BK) — Tue, 18 Jan 2011 13:43:05 GMT

Hi Brad,

Failover support for Linux is with Device Mapper 4.0 or higher.

Try installing and then check.

Re: Trying to get multipath working with BL460c & MSA2324sa

BradV — Tue, 18 Jan 2011 14:08:41 GMT

Hi BK,

Not sure I am following. This is a fresh install of RHEL 6. The devive-mapper-multipath rpm is listed as 0.4.9-31.e16.x86_64. I would assume since this OS is only a couple of months old that this version of multipath is quite recent.

Regards,

Brad

Re: Trying to get multipath working with BL460c & MSA2324sa

jpmartin — Wed, 19 Jan 2011 06:36:34 GMT

Brad,

install the last HP Device Mapper kit, but the link seems to be missing, or edit the /etc/multipath.conf and insert the following:

device
{
vendor "HP"
product "MSA2012sa|MSA2312sa|MSA2324sa"
getuid_callout "/sbin/hp_scsi_id -g -u -n -s /block/%n"
prio_callout "/sbin/mpath_prio_alua /dev/%n"
hardware_handler "0"
path_selector "round-robin 0"
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
no_path_retry 18
rr_min_io 100
path_checker tur

}

then restart the multipath daemon.

The multipath -ll would be fine.

Jean-Pierre

Re: Trying to get multipath working with BL460c & MSA2324sa

BradV — Wed, 19 Jan 2011 10:50:47 GMT

Hi Jean-Pierre,

Well, the install didn't work because this is RHEL 6. However, I had copied over the multipath.conf file from one of my RHEL 5 systems. I adjusted the blacklist so that it removed the internal disks from consideration. The only difference in the device section you listed and what I have is the getuid_callout line. The installation script uses the built-in scsi_id utility instead of hp_scsi_id starting with RHEL 5.4. So, my line is:

getuid_callout "/sbin/scsi_id --whitelisted --replace-whitespace --device=/block/%n"

I have the kit, I'll copy the hp_scsi_id into /sbin and change that line to see if it makes a difference.

Regards,

Brad

Re: Trying to get multipath working with BL460c & MSA2324sa

BradV — Wed, 19 Jan 2011 10:58:18 GMT

Darn,

Didn't work. I see:

/sbin/hp_scsi_id exited with 1
error calling out /sbin/hp_scsi_id -g -u -n -s /block/cciss/c0d0

for each of the devices.

Regards,

Brad

Re: Trying to get multipath working with BL460c & MSA2324sa

BradV — Wed, 19 Jan 2011 11:42:38 GMT

Well, apparently HP stopped updating the enablement kit after RHEL 5.4. The latest version was 4.4.1. They have a document that explains how to use standard Linux device-mapper to work with HP storage devices. It is available at:

http://h20272.www2.hp.com/utility/document.aspx?docurl=Shared%20Documents/an/solutions_linux/Device_Mapper_Multipath_RG_2010_08_02.pdf

However, it is outdated. Because for the 2324sa, it says to use "/sbin/scsi_id -g -u ..." and those parameters are not supported for this version of scsi_id.

Regards,

Brad

Re: Trying to get multipath working with BL460c & MSA2324sa

Zygmunt Krawczyk — Thu, 20 Jan 2011 11:08:54 GMT

For this configuration (P700m, SAS switch, MSA2324sa) the only working multipathing solutions are for Windows Server (MPIO) and for VMware. VMware uses hpsa driver for P700m, not cciss. HP develops hpsa driver to be widely available with verious linux, but unfortunately they gave up P700m support in hpsa.

Re: Trying to get multipath working with BL460c & MSA2324sa

BradV — Thu, 20 Jan 2011 12:22:05 GMT

Darn, I wonder why the sales engineer recommended that we go with SAS then instead of FC? That means all the redundant paths are pretty much useless?

Thanks for the information!

Regards,

Brad

Re: Trying to get multipath working with BL460c & MSA2324sa

BradV — Wed, 09 Feb 2011 12:54:40 GMT

OK, I have an update. Apparently RHEL 6 isn't certified on the MSA2324sa yet. They hope in a few months. In the meantime, I was trying with RHEL 5. That wasn't working. After installing RHEL 5.6 and then installing the HPDMultipathEnablementKit 4.4.1, it still didn't work. I then applied the PSP 8.60 for Linux and the BL460C G6 and it started working! So, install the latest enablement kit and the latest PSP for your setup. :)

Re: Trying to get multipath working with BL460c & MSA2324sa

BradV — Wed, 09 Feb 2011 12:56:33 GMT

See my last post.

Re: Trying to get multipath working with BL460c & MSA2324sa

Timo J — Tue, 08 Mar 2011 07:12:30 GMT

Hi Brad,

We have similar HW (BL460c G7 + P700 + MSA2324sa) and SW (RHEL 5.6) combo as you have. We also have similar kind of multipath problems as you did.

It was nice to read that you managed to get the combo working - Unfortunately we are still stuck with the same problem. Could you be so kind and share the multipath config here or/and share other tricks you needed to do to get the combo working?

Cheers, Timo

Re: Trying to get multipath working with BL460c & MSA2324sa

BradV — Tue, 08 Mar 2011 10:57:46 GMT

Hi Timo,

I think I might have miss-stated just a little. I was able to get RHEL 5.5 working, not RHEL 5.6 or RHEL 6.0. After installing 5.5, I then installed psp 8.60 (I think 8.62 is the most recent) followed by installing HPDMMultipathEnablementKit vs 4.4.1.

Vs 4.4.1 essentially just creates the multipath.conf file. The real trick the psp which builds some new kernels for you.

HP told me that they are working on hard on getting RHEL 6.0 certified, but did not give me a timetable. :(

Regards,

Brad

Re: Trying to get multipath working with BL460c & MSA2324sa

Timo J — Tue, 08 Mar 2011 16:44:21 GMT

Thanks for the answers Brad,

Sorry to bother you again/still but we have to get the multipath working. As you seem to be only one who has managed to crack the problem here are few more questions:

1) Could you tell a bit more about your RHEL 5.5 installation? As a reference, we made a basic kickstart installation with only basic packages included:

%packages
@admin-tools
@base
@core
@development-tools
@development-libs
@system-tools
@base-x
@editors

2) Did you have the device-mapper-multipath or related package(s) installed in advance (before PSP and HPDMmultipath)?

3) Did you run yum update during the OS installation (before/after installing PSP, etc)?

4) While installing PSP we droppped all fiber channel related stuff - Did you do the same or install all suggested updates?

5) While installing the HPDMmultipath-4.4.1 kit we noticed that the install script didn't install the hp_scsi_id - did you copy it manually to /sbin/ or leave it as it is?

6) Any possibility to get to see your multipath.conf file?

Cheers, Timo

Re: Trying to get multipath working with BL460c & MSA2324sa

BradV — Tue, 08 Mar 2011 17:15:25 GMT

Re: Trying to get multipath working with BL460c & MSA2324sa

BradV — Tue, 08 Mar 2011 17:37:59 GMT

Hi Timo,

1) My kickstart %packages section:
%packages
@admin-tools
@base
@core
@development-libs
@development-tools
@gnome-desktop
@gnome-software-development
@graphical-internet
@java
@kvm
@legacy-software-development
@mail-sever
@network-server
@office
@x-software-development
@base-x
kexec-tools
iscsi-initiator-utils
bridge-utils
fipscheck
x86info
device-mapper-multipath
sgpio
imake
kvm-tools
vnc-server
dnsmasq
mesa-libGLU-devel
xorg-x11-utils
xorg-x11-server-Xnext
xorg-x11-server-Xvfb
expect
libnl
lm_sensors
net-snmp
-system-config-netboot
-readadhead
-krb5-workstation
-pcmciautils
-pam_smb
-pkinit-nss
-dhcpv6-client
-rsh
-rp-pppoe
-pam_krb5
-irda-utils
-ypbind
-bluez-utils
-evolution-data-server-devel
-dovecot
-spamassin

2) yes
3) no
4) PSP had it's own prerequisites. I installed all of them it asked for (had to re-run the install a couple of times) until all suggested packages were installed.
5) That is correct. The only thing it does under 5.5 is setup the multipath.conf. It doesn't use the hp_scsi_id application anymore.
6) In the mulipath.conf, I added to the blacklist:

devnode "hd[a-z][[0-9]*]"

to keep it from reading the on-board disks.

Other than that, it is just as the enablement kit made it. I have:

device {
vendor "HP"
product "MSA2012sa|MSA2312sa|MSA2324sa"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -n -s /block/%n"
path_checker tur
path_selector "round-robin 0"
prio_callout "/sbin/mpath_prio_alua %d"
rr_weight uniform
failback immediate
hardware_handler "0"
no_path_retry 18
rr_min_io 100
}

There may some typos in the above. :)

Regards,

Brad