topic Re: RHEL 5.5 multipath problem (P700m + MSA2324sa) in Operating System - Linux

RHEL 5.5 multipath problem (P700m + MSA2324sa)

Timo J — Mon, 07 Mar 2011 14:25:22 GMT

Hello,

We are having problems configuring multipath with RHEL 5.5.

multipath -ll doesn't show anything and -v5 produces the following errors:

cciss!c0d0: not found in pathvec
cciss!c0d0: mask = 0x1f
cciss!c0d0: bus = 4
cciss!c0d0: dev_t = 104:0
cciss!c0d0: size = 234374976
cciss!c1d0: not found in pathvec
cciss!c1d0: mask = 0x1f
cciss!c1d0: bus = 4
cciss!c1d0: dev_t = 105:0

Collected below are the other background details of the installation.

--

HW configuration:
--------------------------------------------
BladeSystem c7000 Enclosure G2
ProLiant BL460c G7 (603588-B21)
Smart Array P700m (508226-B21)
3Gb SAS Switch (AJ865A)
MSA2324sa (AJ807A Rev B)

Base Install:
--------------------------------------
RHEL5 Update 5

Installed packages:
--------------------------------------
PSP 8.60
device-mapper-multipath-0.4.7-42.el5.i386.rpm
kpartx-0.4.7-42.el5.i386.rpm
HPDMmultipath-4.4.1.tar.gz

# rpm -qa --last
device-mapper-multipath-0.4.7-42.el5 kpartx-0.4.7-42.el5 mptlinux-4.00.13.07-5 hpvca-6.2.0-8 hponcfg-3.1.0-0 hpmouse-1.2.0-96 hpdiags-8.6.0-18 hpacucli-8.60-8.0 hp-smh-templates-8.6.0.14-16
hp-snmp-agents-8.6.0.18-18 hpsmh-6.2.0-12 hp-ilo-8.5.0-1.rhel5 hp-health-8.6.0.24-27 hp-OpenIPMI-8.5.0-1.rhel5 cpqacuxe-8.60-7.0
cpq_cciss-3.6.26-5

# /etc/multipath.conf
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy failover
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/bin/true"
path_checker tur
rr_min_io 100
rr_weight uniform
failback immediate
no_path_retry 12
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z][[0-9]*]"
}
devices {
device {
vendor "HP"
product "MSA2012sa|MSA2312sa|MSA2324sa"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -n -s /block/%n"
path_checker tur
path_selector "round-robin 0"
prio_callout "/sbin/mpath_prio_alua %d"
rr_weight uniform
failback immediate
hardware_handler "0"
no_path_retry 18
rr_min_io 100
}
}

# /var/log/dmesg (When only one cable connected)
SCSI subsystem initialized
HP CISS Driver (v 3.6.26-5)
cciss: Device 0x3230 has been found at bus 6 dev 0 func 0
ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 30 (level, low) -> IRQ 122
cciss0: Placing controller into performant mode
cciss/c0d0: p1 p2 p3 p4 < p5 >
cciss: Device 0x323a has been found at bus 12 dev 0 func 0
ACPI: PCI Interrupt 0000:0c:00.0[A] -> GSI 16 (level, low) -> IRQ 169
cciss1: Placing controller into performant mode
Fusion MPT SAS Host driver 4.00.13.07
Fusion MPT SPI Host driver 4.00.13.07
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
usb 6-1: new full speed USB device using uhci_hcd and address 2
device-mapper: dm-raid45: initialized v0.2594l

# multipath -ll

# multipath -v5
cciss!c0d0: not found in pathvec
cciss!c0d0: mask = 0x1f
cciss!c0d0: bus = 4
cciss!c0d0: dev_t = 104:0
cciss!c0d0: size = 234374976
cciss!c1d0: not found in pathvec
cciss!c1d0: mask = 0x1f
cciss!c1d0: bus = 4
cciss!c1d0: dev_t = 105:0
dm-0: blacklisted
dm-1: blacklisted
dm-2: blacklisted
dm-3: blacklisted
dm-4: blacklisted
dm-5: blacklisted
dm-6: blacklisted
dm-7: blacklisted
dm-8: blacklisted
dm-9: blacklisted
loop0: blacklisted
loop1: blacklisted
loop2: blacklisted
loop3: blacklisted
loop4: blacklisted
loop5: blacklisted
loop6: blacklisted
loop7: blacklisted
md0: blacklisted
ram0: blacklisted
ram10: blacklisted
ram11: blacklisted
ram12: blacklisted
ram13: blacklisted
ram14: blacklisted
ram15: blacklisted
ram1: blacklisted
ram2: blacklisted
ram3: blacklisted
ram4: blacklisted
ram5: blacklisted
ram6: blacklisted
ram7: blacklisted
ram8: blacklisted
ram9: blacklisted
===== no paths =====
libdevmapper: ioctl/libdm-iface.c(1740): dm names NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-tmp NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-home NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-log NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-mcx NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-usr NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-var NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-archive NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-cdr NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-opt NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-ready NF [16384]

# /sbin/hp_scsi_id -g -u -n -s /block/cciss\!c0d0
3600c0ff0001047c653f06b4d01000000

We need urgent help to get this problem solved - has anyone managed to get ths HW+SW combo working?

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Alzhy — Mon, 07 Mar 2011 15:29:40 GMT

Do you have the latest HP Device Mapper Multipath kit installed?

Can you pst your complete /etc/multipath.conf?

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Timo J — Mon, 07 Mar 2011 16:35:50 GMT

Yes, we are using the latest HPDMmultipath-4.4.1 kit. Multipath config file is the stock one that HPDMultipath installer modifies during installation (see attachment).

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Alzhy — Mon, 07 Mar 2011 16:40:10 GMT

Try uncommenting cciss in the the blacklist section:

blacklist {
# wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z][[0-9]*]"
devnode "^cciss!c[0-9]d[0-9]*"
# device {
# vendor "HP"
# product "OPEN-*"
# }
}

Then multipath -F
Then do issue_lip and scan on each of your fc hosts.

Do you have lsscsi installed? Do the MAS luns even appear at all?

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Timo J — Mon, 07 Mar 2011 17:04:03 GMT

For additional info we are using SAS connection to the disk array - not fibre channel.

Hence I have to keed the cciss part commented or else all devices are blacklisted.

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Alzhy — Mon, 07 Mar 2011 18:00:26 GMT

Hmmm...
I never had any HP Array that's SAS connected... do you have documentation handy for your MSA array and possibly a HOWTO in a Linux environment?

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Timo J — Tue, 08 Mar 2011 18:01:44 GMT

Managed to do so far...

1) Installed RHEL 5.5 again with following base install:

%packages
@admin-tools
@base
@core
@development-tools
@development-libs
@system-tools
@base-x
@editors

2) Installed PSP 8.60 with following packages:

cpq_cciss-3.6.26-5.rhel5.i686.rpm
mptlinux-4.00.13.07-5.rhel5.i686.rpm
hp-OpenIPMI-8.5.0-1.rhel5.i386.rpm
hp-ilo-8.5.0-1.rhel5.i386.rpm
hp-health-8.6.0.24-27.rhel5.i386.rpm
hp-snmp-agents-8.6.0.18-18.rhel5.i386.rpm
hpmouse-1.2.0-96.noarch.rpm
hpsmh-6.2.0-12.i386.rpm
hp-smh-templates-8.6.0.14-16.noarch.rpm
hpvca-6.2.0-8.linux.rpm
cpqacuxe-8.60-7.0.noarch.rpm
hpacucli-8.60-8.0.noarch.rpm
hpdiags-8.6.0-18.linux.i586.rpm
hponcfg-3.1.0-0.noarch.rpm

3) Installed HPDMmultipath-4.4.1

Situation after this:

# multipath -v3

...
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st
3600c0ff0001021929bb9744d01000000 #:#:#:# cciss!c0d0 104:0 50 [undef][ready]
3600c0ff0001021929bb9744d01000000 #:#:#:# cciss!c0d1 104:16 10 [undef][ready]
3600c0ff0001021929bb9744d01000000 #:#:#:# cciss!c0d2 104:32 10 [undef][ready]
3600c0ff0001021929bb9744d01000000 #:#:#:# cciss!c0d3 104:48 50 [undef][ready]
Found matching wwid [3600c0ff0001021929bb9744d01000000] in bindings file.
Setting alias to mpath0
cciss!c0d0: ownership set to mpath0
cciss!c0d0: not found in pathvec
cciss!c0d0: mask = 0xc
cciss!c0d0: state = 2
cciss!c0d0: prio = 50
cciss!c0d1: ownership set to mpath0
cciss!c0d1: not found in pathvec
cciss!c0d1: mask = 0xc
cciss!c0d1: state = 2
cciss!c0d1: prio = 10
cciss!c0d2: ownership set to mpath0
cciss!c0d2: not found in pathvec
cciss!c0d2: mask = 0xc
cciss!c0d2: state = 2
cciss!c0d2: prio = 10
cciss!c0d3: ownership set to mpath0
cciss!c0d3: not found in pathvec
cciss!c0d3: mask = 0xc
cciss!c0d3: state = 2
cciss!c0d3: prio = 50
mpath0: pgfailback = -2 (controller setting)
mpath0: pgpolicy = group_by_prio (controller setting)
mpath0: selector = round-robin 0 (controller setting)
mpath0: features = 0 (internal default)
mpath0: hwhandler = 0 (controller setting)
mpath0: rr_weight = 1 (internal default)
mpath0: minio = 100 (controller setting)
mpath0: no_path_retry = 18 (controller setting)
pg_timeout = NONE (internal default)
mpath0: set ACT_CREATE (map does not exist)
mpath0: failed to load map (a path might be in use)
mpath0: domap (0) failure for create/reload map
...

and

# multipath -d
create: mpath0 (3600c0ff0001021929bb9744d01000000) HP,MSA2324sa
[size=121G][features=0][hwhandler=0][n/a]
\_ round-robin 0 [prio=100][undef]
\_ #:#:#:# cciss!c0d0 104:0 [undef][ready]
\_ #:#:#:# cciss!c0d3 104:48 [undef][ready]
\_ round-robin 0 [prio=20][undef]
\_ #:#:#:# cciss!c0d1 104:16 [undef][ready]
\_ #:#:#:# cciss!c0d2 104:32 [undef][ready]

note!

# multipath
# multipath -ll >> doesn't return anything

Can anybody help me forward from here on???

-Timo

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

markpag — Tue, 12 Apr 2011 12:45:58 GMT

I am having the same issue but with RedHat 5.6.....Let me know if you get any help.

Thanks

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Michael Moir_1 — Fri, 22 Apr 2011 14:41:06 GMT

Same trouble here. Looking for solution please.

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

markpag — Fri, 22 Apr 2011 14:53:55 GMT

Hi have opened a case with RedHat on this problem.

HP support has verified that the problem exists, at least with my RedHat5.6 version. I am now trying to resolve with RedHat. HP has no know resolution.

I will update this thread as I receive results.

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Ralph Grothe — Wed, 27 Apr 2011 09:12:25 GMT

Don't know if this helps?

On a RHEL 5.6 box that I administer and that also uses the cciss driver to access the internal disks it helped to blacklist them in order to make multipath errors disappear.
I suspect the driver uses deprecated ioctl calls or is otherwise broken.
I'm not sure if this really matters, but I escaped the bang in the blacklist regex for it to work.

These are my vgroot/md1 internal disks:

# vgdisplay -v vgroot 2>/dev/null|awk '/PV Name/{system("mdadm -QD "$NF"|grep cciss")}'
0 104 2 0 active sync /dev/cciss/c0d0p2
1 104 18 1 active sync /dev/cciss/c0d1p2

# grep -A2 ^blacklist /etc/multipath.conf
blacklist {
devnode "^cciss\!c0d[01]"
}

Here I so far have only one SAN LUN attached.
And this is all multipath lists.

# multipath -ll
mpath0 (360060e801530d700000130d700000052) dm-8 HP,OPEN-V
[size=7.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 0:0:0:0 sda 8:0 [active][ready]
\_ 1:0:0:0 sdb 8:16 [active][ready]

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

DeafFrog — Thu, 25 Aug 2011 20:21:57 GMT

Hi Timo ,

not sure , but check this thread , this might be of some help :

http://h30499.www3.hp.com/t5/HP-BladeSystem/SAS-Switch-Port-Mapping/m-p/4516011#M7777

Rehards,

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Michael Moir_1 — Wed, 27 Apr 2011 13:17:14 GMT

Still unsuccessful mapping multiple paths to storage.
DL360 G6, MSA 2324sa
RHEL 5.6 -- 2.6.18-238.9.1.el5 x86_64
System is entirely up to date including:
device-mapper-multipath-0.4.7-42.el5_6.2, HPDMmultipath-4.4.1, PSP 8.70

# multipath -ll -v5
cciss!c1d0: not found in pathvec
cciss!c1d0: mask = 0x5
cciss!c1d0: bus = 4
cciss!c1d0: dev_t = 105:0
cciss!c1d0: size = 1432149760
cciss!c1d1: not found in pathvec
cciss!c1d1: mask = 0x5
cciss!c1d1: bus = 4
cciss!c1d1: dev_t = 105:16
cciss!c1d1: size = 1432149760
===== no paths =====

A) Wouldn't be appropriate to blacklist cciss from multipath.conf,
as that's driver for P800 SAS controller connected to MAS2324sa.
B) No SAS switches or Blade chassis involved here.
Server direct attached to storage.
C) Vendor and Model strings are contained in /sys/block/cciss!cXdX/device,
but not detected by multipath command?

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Michael Moir_1 — Tue, 10 May 2011 17:41:26 GMT

Out of my support case with Red Hat (00456892), I was provided test patches for device-mapper-multipath and kpartx packages which resolved this issue for me.

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

seinyan — Tue, 28 Feb 2012 05:43:52 GMT

There is bug in the device mapper multipah

https://bugzilla.redhat.com/show_bug.cgi?id=742906

Last that i recieved from RH is:

Ben Marzinski 2012-01-12 13:30:54 EST
"Personally, I favor leaving the bug open and fixing this in RHEL 5.9. "

I have same problems with rhel 5 update 6, but with blade servers P712m and P2000 sa g3.

http://h30499.www3.hp.com/t5/Disk-Array/P2000-G3-SAS-P712m-blade-and-multipath-Redhat-Linux-5/td-p/5350499