Multipath - Eva 4000 and Blade Serie C

[Miguel Carabano_1]

Hello everyone,

I have the following problem:
EVA 4000 with Blade Serie C.

root@servertest:~# fdisk -l

Disk /dev/cciss/c0d0: 73.3 GB, 73372631040 bytes
255 heads, 32 sectors/track, 17562 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 1 1961 8000864 82 Linux swap
/dev/cciss/c0d0p2 1962 17562 63652080 83 Linux

Disk /dev/sda: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 1009 2095662 8e Linux LVM

Disk /dev/sdb: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 1009 2095662 8e Linux LVM

Disk /dev/sdc: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 1009 2095662 8e Linux LVM

Disk /dev/sdd: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes

Device Boot Start End Blocks Id System
/dev/sdd1 1 1009 2095662 8e Linux LVM
root@servertest:~# vgcreate vg01 /dev/sda1
Found duplicate PV ohrp2eO0VfILlOytcEq9WNy13wKBJz8a: using /dev/sdb1 not /dev/sda1
Found duplicate PV ohrp2eO0VfILlOytcEq9WNy13wKBJz8a: using /dev/sdc1 not /dev/sdb1
Found duplicate PV ohrp2eO0VfILlOytcEq9WNy13wKBJz8a: using /dev/sdd1 not /dev/sdc1
Found duplicate PV ohrp2eO0VfILlOytcEq9WNy13wKBJz8a: using /dev/sda1 not /dev/sdd1
Found duplicate PV ohrp2eO0VfILlOytcEq9WNy13wKBJz8a: using /dev/sdb1 not /dev/sda1
Found duplicate PV ohrp2eO0VfILlOytcEq9WNy13wKBJz8a: using /dev/sdc1 not /dev/sdb1
Found duplicate PV ohrp2eO0VfILlOytcEq9WNy13wKBJz8a: using /dev/sdd1 not /dev/sdc1
Found duplicate PV ohrp2eO0VfILlOytcEq9WNy13wKBJz8a: using /dev/sda1 not /dev/sdd1
Physical volume '/dev/sda1' is already in volume group 'vg00'
Unable to add physical volume '/dev/sda1' to volume group 'vg01'.

How can I manage this?

Thanks for any help you may give me.

Miguel

[Jimmy Vance]

You need to enable multipath in the driver. Since you have 4 paths on the EVA your seeing 4 paths to the same LUN. Not sure if your using the QLogic, or Emulex based HBA.

If your running the QLogic HBA, load the latest HP Qlogic driver and enable multipath

There is something similar for the Emulex based adapter but I'm more familiar with the Qlogic cards.

[Miguel Carabano_1]

Hi,

how i can do??

[Miguel Carabano_1]

hello everyone....

Recently i just did something that i want to share with all of you... This is a procedure to make multipath filesystem when you've got a SAN or just a server connected to a virtual array by 2 fiber channel ways...

Situation:
Well this is the situation... I have a server HP Blade Serie C and a EVA 4000.. Everyone who knows that type of hardware knows that you can create LUNs through them and present them dinamically to your server and make many things with that.. So I installed RedHat Advance Server 3 in my server.

But the problem was when i wanted to create a new filesystem. for example, i created a new LUN and because i have two fiber channels (EMULEX) i see to differents devices. for example:

#cat /proc/scsi/scsi
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: MegaRAID Model: LD0 RAID1 34731R Rev: H
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: HP Model: A6189A Rev: HP19
Type: Direct-Access ANSI SCSI revision: 03
Host: scsi4 Channel: 00 Id: 00 Lun: 00
Vendor: HP Model: A6189A Rev: HP19
Type: Direct-Access ANSI SCSI revision: 03


#dmesg | grep Attach
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Attached scsi disk sdb at scsi3, channel 0, id 0, lun 0
Attached scsi disk sdc at scsi4, channel 0, id 0, lun 0

Now you may observe that my new LUN 0 is /dev/sdb and /dev/sdc..

So if i want to make a new filesystem and if i want to use my two devices i need a multipath way to do that.. But the problem is not ending right here..
You see, when you present many other LUNS on your system this is gonna be very annoying because the device files change when you reboot the system... for example...

#dmesg | grep Attach
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Attached scsi disk sdb at scsi3, channel 0, id 0, lun 0
Attached scsi disk sdc at scsi3, channel 0, id 0, lun 1
Attached scsi disk sdd at scsi3, channel 0, id 0, lun 2
Attached scsi disk sde at scsi4, channel 0, id 0, lun 0
Attached scsi disk sdf at scsi4, channel 0, id 0, lun 1
Attached scsi disk sdg at scsi4, channel 0, id 0, lun 2

Now my LUN 0 is /dev/sdb and change from /dev/sdc to /dev/sde.. you see??

There are many ways to resolve this problem...but i didn't do any of them because demand changes and patches.. They are good but i really want to resolve my problem!!!..

So this is what i do...

The begining of the solution:

First of all i create the multipath device using Software Raid... now you say what???? really???... well yes, software Raid have the tools to support multipath levels when you construct a device... this is the way:

Supose you have LUN 0 created and no other else... so...

# mdadm -C /dev/md0 -l multipath -n2 /dev/sdb /dev/sdc

Yes, this is the magic command to do that... -l or --level tells to mdadm what kind of level your going to use. raid 1, raid 5 or in this case, multipath...

So my first problem is solved... now i have one device to refer the last two...

write: mdadm -D /dev/md0 to see the properties of the device...

You see... there are many software in the market, for example SecurePath for HP... and all of them works the same way!!.. you create a virtual device and you need a software to understand which one is the principal and which one is the spare in case of failure...

Now that you create it, you need to do other thing... write

#echo "DEVICES /dev/sd[a-z]*" >> /etc/mdadm.conf

and

# mdadm -Es
ARRAY /dev/md0 level=multipath num-devices=1 UUID=d3261354:a885a17d:8bf8448b:1e84030c
devices=/dev/sdc,/dev/sdb

well you need this information to activate this devices at boot time... Many literatures tells that you have to do this action:
mdadm -Es >> /etc/mdadm.conf...
but in this particular case you must not do that... You see, mdadm -Es gives you all the basic information you need for /dev/md0 but if you read the file carefully /etc/mdadm.conf, you will find that you don't need all of this options.. in this case the only thing you need is: ARRAY /dev/md0 level=multipath num-devices=1 UUID=d3261354:a885a17d:8bf8448b:1e84030c

now you are asking... why are you worry about it??... well the answer is very simple.. in the future when you create more LUNs probably you are going to have troubles activating your md devices because there is a tag called "devices" in this file that reference to a device that maybe was changed.... REMEMBER???... when you reboot your system.. your devices will change is you have the conventional way to represent them...!!!.

so we only add to our file /dev/mdadm.conf the information of mdadm -Es without the devices option...

ok... until now i think this going fine.. now if you want to use LVM and i really really recommend it do use your md device:

#pvcreate /dev/md0
#vgcreate vg00 /dev/md0
#lvcreate -L 5000 /dev/vg00
#mkdir /test
#mkfs.ext3 /dev/vg00/lvol1
#mount /dev/vg00/lvol1 /test

Edit your /etc/fstab to add your new filesystem...

Ok... Almost finish... there is only one thing you have to do... I don't know if there is applicable to everry distribution of Linux but in my case i use RedHat... so i will explain it...

I realized when a rebooted my system that my md devices didn't activate.. well i was worry!!!.. because all what i told you before is right!! why is this happenning??... well i realized that LVM is very veryyyy inteligent.. and my filesystems came up and mounted but on one of the hard devices... not on my md devices... for example.. when i did an vgdisplay -v vg00 the disk device that i found was /dev/sdc not /dev/md0..!!! the root of the problem was in the initialization of the system... my md devices doesn't start.. so LVM was activated on the disk directly...

My particular solution was.. editing /etc/rc.d/rc.sysinit file..

I found an error of the scripting in /etc/rc.d/rc.sysinit in RedHat (I don't know in other systems..) in the initialization of Raid devices... There is an if[ -f /etc/raidtab] but in RedHat raidtab file doesn't exist.. in Fact, you really really don't need it.. but i touched /etc/raidtab... # touch /etc/raidtab beacuse i didn't want to change the rc.sysinit file to much, and only add the lines that resolve my problem...

Considere that my changes are only for solve... and is only in the limits of if[ -f /etc/raidtab]; !!!


Other thing that i did was adding two little lines... after if [ -f /proc/mdstat ]; then

# Line that i add.....
/sbin/mdadm -As

This was to force init my md devices... because the script is written for the old Raid Software and the old sintax for raidtab..

and this lines... after if [ -e /proc/lvm -a -x /sbin/vgchange -a -f /etc/lvmtab ]; then

action $"Unsetting up Logical Volume Management:" /sbin/vgscan && /sbin/vgchange -a n
action $"Resetting up Logical Volume Management:" /sbin/vgscan && /sbin/vgchange -a y

Because... i need to reload the LVM devices now i have my md devices started...

I save the file and exit...

NOW... FINALLY... I FINISH.......... you can reboot your system to probe it... and add new LUNS and create new filesystems without reboot with multipath software and of course use it with LVM or hard filesystem ways... it's your decision... and reboot your system many times you want..!!! it's not necesary...!!!

You don't need to worry if your system reboot and your devices change!!! always they are gonna be found because you started your md devices using your UUID number that is unique for a particular md device!!!...

I hope this help to every one... maybe there is an easy way to do it.. But i found this...

Greetings..

[Miguel Carabano_1]

Thanks