multipathed Storage Devcies not loaded on boot -> multipath not working on boot?

[Janezic Milan]

Hi,

I'm trying to setup the /etc/fstab with some multipathed Storage Disks that will be automatically mounted on boot .

The Root Device is a local Raid1 Disk (LVM) and I've also mounted 3 Virtual Storage Disks (Ext3, Raid5)

The multipath Devices are correctly set, everything is working fine. "multipath -ll" shows me the correct Disks, paths and the aliases.

I've added the virtual disks to to /etc/fstab. With the command "mount -a" everything gets mounted.

But when I restart/sart the Server with the mulitpath (SAN Vdisks) entries in fstab, I'm getting an Error that the Disk isn't available.

The Boot procedure stops and a Read-Only Terminal is started.

There the multipath isn't loaded -> DM multipath kernel driver not loaded. "multipathd" Daemon isn't started.

Also in "/dev/mapper/" the disks are missing. I can't start from there the OS, the only thing I can do is to reboot and mount the whole system Disk with the Installation DVD (Rescue Option) to uncomment the multipath Devices in /etc/fstab to get the System started, which takes a lot of time.

I've already rebuilded the RAM Disk (initrd) after saving all the multipath configuration.

So I'm not sure if I'm doing anything wrong, but I've invested a lot of time searching through the Internet, but didn't find anythingto solve my problem, I hope someone can help me here.

Here the detailed Information from our Environment:

Server: HP ProLiant 460c G1 BladeServer

OS: Oracle Linux 5.6  x64 (same as RedHat Linux 5.6) / 2.6.18-238.12.1.0.1.el5

HBA/FC: QLogic QMH2462 4Gb FC HBA (2 Ports)
HBA/FC Driver: QLogic-HP-FC-installer-rhel5u5-rhel5u6-20110413-1 -> 8.03.07.03.5.6-1 (newest HP Driver)

Local Disk: Raid1 LVM, 70GB

External Disk: 3 Attached SAN VirtualDisks (EVA 4000 Storage System, 2 Controllers)

Installed: HP PSP 8.73 (hp-fc-enablement-1.2-5)

[root@apy-db-dev ~]# cat /etc/fstab
/dev/VolGroup00/LogVol00 /                      ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
sysfs                   /sys                    sysfs   defaults        0 0
/dev/VolGroup00/LogVol01 swap                   swap    defaults        0 0
/dev/mapper/datap1      /data                   ext3    defaults        1 2
/dev/mapper/data01p1    /data01                 ext3    defaults        1 2
/dev/mapper/data02p1    /data02                 ext3    defaults        1 2

[root@apy-db-dev /]# cat /etc/multipath.conf
blacklist {
       devnode "^cciss!c[0-9]d[0-9]*"
       devnode "^vg*"
}

defaults {
    user_friendly_names yes
}

devices {
    device
    {
        vendor "HP.
        product "HSV2[01]0|HSV300|HSV4[05]0"
        getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout "/sbin/mpath_prio_alua /dev/%n"
        hardware_handler "0"
        path_selector "round-robin 0"
        path_grouping_policy group_by_prio
        failback immediate
        rr_weight uniform
        no_path_retry 18
        rr_min_io 100
        path_checker tur
    }
}

[root@apy-db-dev /]# cat /etc/modprobe.conf
alias eth0 bnx2
alias eth1 bnx2
alias scsi_hostadapter cciss
alias scsi_hostadapter2 qla2xxx
options qla2xxx ql2xmaxqdepth=16 ql2xloginretrycount=30 qlport_down_retry=64
options lpfc lpfc_lun_queue_depth=16 lpfc_nodev_tmo=30 lpfc_discovery_threads=32
blacklist netxen_nic
blacklist nx_xport
install netxen_nic /bin/true
install nx_xport /bin/true

[root@apy-db-dev ~]# multipath -ll
data01 (3600508b40006af870000500000780000) dm-4 HP,HSV200
size=150G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 1:0:1:2 sdn 8:208 active ready running
|-+- policy='round-robin 0' prio=50 status=enabled
| `- 0:0:1:2 sdf 8:80  active ready running
|-+- policy='round-robin 0' prio=10 status=enabled
| `- 1:0:0:2 sdj 8:144 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 0:0:0:2 sdb 8:16  active ready running
data (3600508b40006af8700005000003b0000) dm-5 HP,HSV200
size=300G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 1:0:0:1 sdi 8:128 active ready running
|-+- policy='round-robin 0' prio=50 status=enabled
| `- 0:0:0:1 sda 8:0   active ready running
|-+- policy='round-robin 0' prio=10 status=enabled
| `- 1:0:1:1 sdm 8:192 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 0:0:1:1 sde 8:64  active ready running
data02 (3600508b40006af870000500000d30000) dm-3 HP,HSV200
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 1:0:1:3 sdo 8:224 active ready running
| `- 0:0:1:3 sdg 8:96  active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:0:3 sdk 8:160 active ready running
  `- 0:0:0:3 sdc 8:32  active ready running

[root@apy-db-dev ~]# rpm - qa |grep -i qla
kmod-hp-qla4xxx-5.01.01.04-2
kmod-hpqlgc-qla2xxx-8.03.07.03.5.6-1

[Matti_Kurkela]

First, make sure that the multipathd service has been configured to start at normal system start-up:

chkconfig --list multipathd
chkconfig multipathd on

Another possibility is that the probing and detection of SAN disks simply takes so much time that the operation is not yet finished when the system tries to mount the filesystems located on SAN.

Ideally, the start-up sequence of a RHEL 5 (and presumably also OEL 5) system essentially works like this:

1. check and mount the root filesystem
2. switch from initrd to real root filesystem
3. start up multipathing
4. check and mount other local filesystems
5. start up networking, iSCSI initiator (if applicable)
6. check and mount remote filesystems (FibreChannel/iSCSI/NFS). NFS filesystems won't need to be checked.

The problem is, how can the system tell which filesystems are local and which are remote? NFS filesystems can be identified by the filesystem type, but iSCSI and FibreChannel have no such identification. This causes the system to attempt mounting the SAN filesystems in step 4, which always fails with iSCSI and may or may not fail with FibreChannel.

To delay the mounting of the SAN filesystems to step 6, and thus allow the FC subsystem more time to initialize, you can add the special option "_netdev" to the /etc/fstab lines of the SAN filesystems, i.e. change the last three lines of your /etc/fstab to:

/dev/mapper/datap1      /data                   ext3    defaults,_netdev  1 2
/dev/mapper/data01p1    /data01                 ext3    defaults,_netdev  1 2
/dev/mapper/data02p1    /data02                 ext3    defaults,_netdev  1 2

The "_netdev" option simply causes the filesystem checking & mounting to be delayed to step 6 of the start-up sequence instead of step 4. It was originally developed for iSCSI filesystems, but it has been found useful for FibreChannel setups too.

By the way:

The RHEL 4 and 5 (and probably also OEL 5) have a little design mistake in dm-multipath configuration. If /var is configured as a separate filesystem, it won't be mounted yet when the dm-multipath subsystem starts up. This will cause the /var/lib/multipath/bindings file to be unavailable when the SAN disk devices are probed and identified, and as a result, the multipath subsystem cannot refer to the persistent table of friendly names and will assign them in simple detection order. This may lead to conflicts if the SAN configuration has been changed since the last time the system was rebooted.

The recommended workaround is to move the bindings file to /etc/multipath/bindings and add the necessary configuration entry to /etc/multipath.conf:

[...]
defaults {
   use_friendly_names yes
   bindings_file /etc/multipath/bindings
}
[...]

Since your /var seems to be included to the root filesystem, this may not concern you; but I thought you should be aware of this, just in case. The fix for this design mistake has appeared in RHEL 6.