topic Re: mdadm Multipath raid configuration in Operating System - Linux

mdadm Multipath raid configuration

Nabil_11 — Sat, 23 Jul 2005 05:39:27 GMT

Hi All,
I have to DL380 G4 HP servers
two HW RAID Controllers pluged in
with MSA500 G2 storage
I connect one of the raid controller as HW RAID 0+1 for OS Redhat AS 3.0 update 3
on the local hard drives, olso this RAID controller connceted to the HW raid 0+1 on MSA500 storage
and there is another RAID controller connected to the storage
So: one RAID controller connected to
/dev/cciss/c0d0 (OS local drives RAID)
and /dev/cciss/c0d1 (raid on storage)
and the other controller
connected to /dev/cciss/c1d0
inside linux /dev/cciss/c0d1 and /dev/cciss/c1d0 the same drive with multipath
I configured /etc/mdadm.conf
DEVICE /dev/cciss/c0d1 /dev/cciss/c1d0
ARRAY /dev/md0 devices=/dev/cciss/c0d1, /dev/cciss/c1d0
then I execute
#mdadm -C /dev/md0 --level=multipath --raid-devices=2 /dev/cciss/c0d1 /dev/cciss/c1d0
its created successfully as /dev/md0
but I lost it after reboot ??????????
so I have to exectue
#mdadm -A /dev/md0
after every reboot
also the important think that I cant access
any defined partion inside this md0
when I execute fdisk /dev/md0
its display /dev/md0p1 p2 p3 p4 p5 p6
but I cant access thease partions
CUZ there is NO special device file for these
device there is no /dev/md0p1 and so

Please Advice

Kind Regards

Re: mdadm Multipath raid configuration

Ivan Ferreira — Sat, 23 Jul 2005 15:31:24 GMT

From the red hat guide, it seems that the multipath devices must be created on a per partition basis.

See this example:

# mdadm -C /dev/md0 --level=multipath --raid-devices=4 /dev/sda1 /dev/sdb1
/dev/sdc1 /dev/sdd1
Continue creating array? yes
mdadm: array /dev/md0 started.

# mdadm --detail /dev/md0

So, i think you need to partition the /dev/cciss/c1d0 device, and then create the multipath device.

Let me know if this works.

Regards.

Re: mdadm Multipath raid configuration

Nabil_11 — Sun, 24 Jul 2005 00:38:21 GMT

Thanks for your relply,

I already did it, I fdisk /dev/cciss/c1d0
then create /dev/md0
it work fine also #mdadm --detail /dev/md0
it gives excellent output.
BUT
1. When I reboot my machine I have to execute
#mdadm -A /dev/md0
I read on one site saying that I have to create scsripts for start and shutdown md0
and link it to appropriate run level
its OK I can do it,

2. The main question when I execute fdisk /dev/md0 it display six partions inside md0
/dev/md0p1
/dev/md0p2
.
.
/dev/md0p6
But I cant access any of them CUZ there is no
special device file for these partitions
I make trick by make
ls -l /dev/cciss/c1d0
it gives me maj/min 106 0
so I execute
#mknod /dev/md0p6 b 106 6
and I mount this file system it works fine
but only with second path

Regards

Nabil

Re: mdadm Multipath raid configuration

Matti_Kurkela — Mon, 25 Jul 2005 07:20:41 GMT

Your multipath setup looks OK to me.

Remember that the native Linux multipath is active/passive: it uses only one path at a time. If the active path fails, it switches over to another path. So, it does not give extra bandwidth, just fault tolerance.

For active/active multipath configuration (load balancing between paths, for example) you need additional software that's compatible with your storage. You probably have to pay for that (I don't know of any free software like thati).

You can try disconnecting the active path: by default, it will take 10s or so to confirm the active path is gone, then it automatically retries the active disk operations on the other path. You also get some messages on syslog (and probably on the console, too) about a lost path.

To see the state of the paths at any time, check the file /proc/mdstat.

Re: mdadm Multipath raid configuration

Nabil_11 — Tue, 26 Jul 2005 03:17:41 GMT

Thanks,

I know that my configuration is correct BUT
in my cluster suite I thing the correct way to use
/dev/md0p1 as primary quorum
/dev/md0p2 as shadow quorum
/dev/md0p5 as /u01
/dev/md0p6 as /u02

I cant access thease devices
I'm still using /dev/cciss/c1d0

any help,

Regards

Nabil

Re: mdadm Multipath raid configuration

Matti_Kurkela — Tue, 26 Jul 2005 07:58:23 GMT

Oops, I didn't notice this at first...

The /dev/md0 devices cannot be split into partitions like normal disks. Fdisk might show the partitions, but the kernel can't access them like that.

Normal disks like /dev/cciss/c0d* have gaps in their device numbers for partitions: /dev/cciss/c0d0 is major 104, minor 0 and /dev/cciss/c0d1 is major 104, minor 16. This leaves minor numbers 1-15 for partitions /dev/cciss/c0d0p*.

However, /dev/md0 is major 9 minor 0 and
/dev/md1 is major 9 minor 1. There is no gap in the minor numbers for /dev/md0p* -style partitions. The major and minor numbers are hardcoded in kernel, so you can't change them easily.

You are supposed to make /dev/cciss/c0d1p1 and /dev/cciss/c1d0p1 into /dev/md0, /dev/cciss/c0d1p2 and /dev/cciss/c1d0p2 into /dev/md1 and so forth. Does not make much sense for a multipath configuration, but that's how it is.

There is a way, though: you can use LVM.
Make your current multipathed /dev/md0 into a LVM physical volume (pvcreate /dev/md0), then create a volume group (vgcreate vgsomething /dev/md0) and create logical volumes (lvcreate -L somesize vgsomething) instead of partitions. Then the logical volumes are accessible as /dev/vgsomething/lvol* instead of /dev/md0p*.

To start up this construct, you need to script the following things to happen at boot-up (unless RedHat AS already does them):

mdadm --assemble --scan
(this autodetects the multipath components and assembles them into /dev/md* device or devices)

vgscan
(this detects the LVM volume groups on all disks, including /dev/md0. It's like a fsck for the LVM configuration.)

vgchange -a y vgsomething
(this activates your volume group: necessary before mounting the logical volumes)

After this, you can mount the logical volumes /dev/vgsomething/lvol* as normal disks.

To shut down cleanly, after umounting all the logical volumes you should do:
vgchange -a n vgsomething
mdadm --stop /dev/md0

It seems to me that Linux LVM does not get overly upset if you forget to do that, though...

(You can name the volume group as you wish: my use of "vgsomething" as an example is a sign I've been spending a lot of time with HP-UX LVM...)