topic Linux Raw Devices under RHEL 5.4 and Multipathing in Operating System - Linux

Linux Raw Devices under RHEL 5.4 and Multipathing

Alzhy — Wed, 07 Oct 2009 01:33:52 GMT

RHEL 5.4 suppsedly restored support for raw devices (probably for Oracle RAW/ASM and other database storage needs).

However /dev/dm-NN devices seem to be not fixed and can change accross reboots -- how does one map them to raw devices.

With ASMLib - this is not a problem and /dev/dm-NNN can change all it wants and OacleASM will always find the disks it had tagged. But with RAW -- how does one map the multipath devices?

My experience with dm devices is when a disk is provisioned and a mutlipath device is allocated - on reboot the orginal /dev/dm-NNN changes.

TIA.

Re: Linux Raw Devices under RHEL 5.4 and Multipathing

Matti_Kurkela — Wed, 07 Oct 2009 10:09:58 GMT

Basically, map the /dev/raw/raw* devices to appropriate /dev/mapper/mpath* devices.

The /dev/mapper/mpath* names are bound to disk WWIDs reported by the storage system, using the information in /var/lib/multipath/bindings file.

Therefore, once a /dev/mapper/mpath* name has been associated with a particular multipathed disk, the name stays with that disk until the multipath binding is explicitly cleared (with "multipath -f " or "multipath -F).

The /dev/dm-NN names are essentially for device-mapper subsystem's internal use only and should not be used by the applications.

RedHat's recommendations:

http://kbase.redhat.com/faq/docs/DOC-10164

Please see method 5, "Creating persistent raw devices for multipathed LUNs or LVM device" near the end of the document linked above.

MK

Re: Linux Raw Devices under RHEL 5.4 and Multipathing

Kapil Jha — Wed, 07 Oct 2009 10:26:48 GMT

try moving /var/lib/multipath/bindings file to /etc/multipath/binding
and make a soft link to /var/lib/multipath/bindings

this would help
and after system comes up
#multipath -v2 -d
if this does not return anything you are good.

BR,
Kapil+

Re: Linux Raw Devices under RHEL 5.4 and Multipathing

Alzhy — Wed, 07 Oct 2009 12:08:56 GMT

Matti,

Selamat Tembang (Malay for Thanks!)

The KBDOC it seems has been updated too for 5.4.

Muchas Gracias!

Re: Linux Raw Devices under RHEL 5.4 and Multipathing

Alzhy — Wed, 07 Oct 2009 12:30:57 GMT

Matti,

"Unfortunately it is not possible to write udev rules for creating raw devices on multipath devices (/dev/dm-*) without manipulating existing udev rules. Modifying existing rules for this purpose could cause unforeseen problems and is not supported by Red Hat Global Support Services. If absolutely necessary, an alternate method for creating raw devices on top of multipath devices could be to create the raw devices in /etc/rc.d/rc.local, so long as the raw device is not required before rc.local is executed."

So what is "SUPORTED"? Surely Linux DB servers that hook up to multi-pathed SAN arrays are prevalent these days. Does that mean that RAW devices under Oracle, Sybase, etc. are "pase" already and ASMLib is the norm for Linux environments?

/dev/dm-NN devices work beatufifully under ASMlib and even LVM -- both are agnostic to device name changes. But I've a DBA who insists on going with RAW devices under ASM. And personally - I prefer Oracle to simply use ASM-less RAW storage even.

Re: Linux Raw Devices under RHEL 5.4 and Multipathing

Alzhy — Wed, 07 Oct 2009 13:03:12 GMT

Also, should I have /dev/mpath/mpathNNN for every /dev/dm-NNN?

All I see under /dev/mpath are 34 character entries (hex) linked to /dev/dm-NNN devices like:

[root@redhat mpath]# ls -l 3600508b4000ce3920003c00005c80000
lrwxrwxrwx 1 root root 8 Oct 6 09:37 3600508b4000ce3920003c00005c80000 -> ../dm-61

I have the Storageworks multipath.conf from the HP package.

Re: Linux Raw Devices under RHEL 5.4 and Multipathing

Matti_Kurkela — Wed, 07 Oct 2009 15:10:03 GMT

RedHat seems to say "Do not mess with the udev rules: modifying the existing rules is not supported by RH." They seem to be happy with adding the necessary commands to /etc/rc.d/rc.local.

But why ask me? As you're using RedHat 5.4, you should have a valid RHN account and could send your question to RedHat support instead. You would get your answer "straight from the horse's mouth", so to speak.

> /dev/dm-NN devices work beautifully under ASMlib and even LVM -- both are agnostic to device name changes. But I've a DBA who insists on going with RAW devices under ASM.

Yes, the device nodes /dev/dm-NN have just the same major & minor numbers as the corresponding devices with persistent names in /dev/mapper. The kernel does not care which device node you use: feel free to use the /dev/dm-NN devices with ASM if you like.

My point was that the /dev/dm-NN devices definitely seem to be not fixed - because they are not *designed* to be fixed at all.

If you use ASM, you can just set up your system to create RAW devices for each disk and let ASM sort them out if you wish.

Your 34-character hex strings under /dev/mpath/* are a direct representation of the WWIDs of the disks - apparently the Storageworks multipath.conf sets FriendlyNames=Off. In RedHat's default configuration it is On, which causes the /var/lib/multipath/bindings file to be used and the names to appear in the shorter /dev/mapper/mpath* form.

The long hex string is somewhat inconvenient, but it uniquely identifies each multipathed disk with no need for a mapping file. The WWIDs are allocated to disks and storage systems like MAC addresses to network cards: you can be pretty sure you cannot find two disks with matching WWIDs unless the disk manufacturer has made a mistake with their WWID allocation.

MK

Re: Linux Raw Devices under RHEL 5.4 and Multipathing

Alzhy — Wed, 07 Oct 2009 15:26:26 GMT

MK,

So are you saying I am free to use /dev/mapper/XXXXXXXXXXXXXXXXXXXXX in my ASMlib configs? Also to map it to raw too?

The multipth.conf blocks for the EVA and XP arrays I have hoked up to this Redhat machine does not explicitly declare FriendlyNames in each block. The Default section declares it as Yes though.

For EVA mapper entries - there is an extra "3" before the EVA VDISK WWID. For XP -- I still ave to deciper where in the 34 character string CU:LDEV is depicted.

We have both StorageWorks and Redhat Support but I wanted to consult this esteemed forum in the hope this post aso helps sould out there who are into the same adventure with BIG Linux as we are.

Thanks very much good sir.