LUN persistence with Failover mode QLogic driver

[yyghp]

LUN persistence with Failover mode QLogic driver

I presented 2 LUNs on 2 different EVA5000 to a server ( running Red Hat Enterprise Linux 3, with Kernel 2.4 ), and used QLogic driver as "Failover mode" to combine 4 paths to 1, then I can see 2 LUNs ( each LUN is at different location. ):

# lssd
sda 0,0,0,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-569c
sdb 0,0,2,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5448

The problem is that the mapping of "sda -- 5000-1fe1-5004-569c" and "sdb -- 5000-1fe1-5004-5448" may change after reboot, because which LUN maps to sda/sdb depends on which LUN the server can detect first! And as requested by our DBA to setup RAC, I HAVE TO make the mapping like this:

sda 5000-1fe1-5004-5448
sdb 5000-1fe1-5004-569c

So, it should be crossover! Now, I wonder whether it's possible for me to make the LUN persistent to such crossover mapping, even the server reboots, sda has to point to LUN 5000-1fe1-5004-5448.

The problem is that such LUN order is controlled by the QLogic driver with Failover mode automatically, I can't find any config file to change such order, it seems the order is out of my control!

I read the last document from HP for LUN persistence, I attach the utiliy files with the doc ( application notes: HP StorageWorks LUN migration and persistence utilities 1.1 ) within this email (LMPutilsv1.1.tar.gz), you can take a look, there is a solution for Linux with 2.4 kernel.

So, I did the following:

Change the QLogic driver back to Single Path mode, reboot, then

After reboot:

[root@srs121 /]# lssd
sda 0,0,0,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-569c
sdb 0,0,1,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5699
sdc 0,0,2,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5448
sdd 0,0,3,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-544d
sde 1,0,0,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5698
sdf 1,0,1,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-569d
sdg 1,0,2,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-544c
sdh 1,0,3,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5449

[root@srs121 backup]# ll -lrt
...
-rw-r--r-- 1 root root 330038 Jun 21 08:52 LMPutilsv1.1.tar.gz

[root@srs121 backup]# tar -zxvf LMPutilsv1.1.tar.gz
LMPutils/
LMPutils/HPMigrationPersistenceUtil-1.1-1.noarch.rpm
LMPutils/LUN Migration Application Notes.pdf
LMPutils/hpudev-cfg-1.1-1.noarch.rpm
LMPutils/install.sh

[root@srs121 backup]# cd LMPutils
[root@srs121 LMPutils]# pwd
/backup/LMPutils

[root@srs121 LMPutils]# ./install.sh -i

Please wait ...



Welcome to the Linux Lun Migration and Lun Persistence instal

Version 1.1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ 1. Lun Migration with Lun Persistency
~ 2. Lun Persistency only (2.4 Linux Kernel)
~ 3. Lun Persistency only (2.6 Linux Kernel)
~ 4. Exit
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Enter your Choice (1/2/3/4) : 2

Please wait, checking the system...


Installing Lun Persistence Utility Please Wait....

Preparing... ########################################### [100%]
1:HPMigrationPersistenceU########################################### [100%]

Lun Persistence utility Installed successfully


Lun Migration and Persistence utility is installed successfully

INSTALL QLOGIC DRIVER IN FAILOVER MODE,AND FOLLOW THE INSTRUCTIONS IN LUN MIGRATION
& PERSISTENCE UTILITY APPLICATION NOTES FOR MIGRATION AND ARRAY UPGRADE PROCESS

[root@srs121 LMPutils]# cd /opt/hp/src/hp_qla2x00src/
[root@srs121 hp_qla2x00src]# ./set_parm


Driver Parameters
-----------------------

1) XP/MSA/EVA Failover
2) Secure Path
3) Single Path
4) Set Paramters Manually
5) Quit

Choice: 1
Writing new /etc/hp_qla2x00.conf...done
adding line to /etc/modules.conf: alias scsi_hostadapter1 qla2200
adding line to /etc/modules.conf: alias scsi_hostadapter2 qla2300_conf
adding line to /etc/modules.conf: alias scsi_hostadapter3 qla2300
adding line to /etc/modules.conf: alias scsi_hostadapter4 sg
adding line to /etc/modules.conf: options qla2200 ql2xmaxqdepth=16 qlport_down_retry=30 qlogin_retry_count=16 ql2xfailover=1 ql2xlbType=1 ql2xexcludemodel=0x0
adding line to /etc/modules.conf: options qla2300 ql2xmaxqdepth=16 qlport_down_retry=30 qlogin_retry_count=16 ql2xfailover=1 ql2xlbType=1 ql2xexcludemodel=0x0

Would you like to create a new initrd to reflect the changes in the parameters (Y/n)? Y
Creating new initrd - initrd-2.4.21-27.ELsmp.img

---Press ENTER to continue---

Quit...

# reboot

After reboot:

[root@srs121 /]# lssd
sda 0,0,0,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-569c
sdb 0,0,2,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5448


Now, the LUN order is still wrong, how can I use "hpdevlabel" to make it crossover, and make it persistent?
It has two ways to create symbolic links:

hpdevlabel add -d /dev/sdb1 -s /dev/myalias
hpdevlabel add -u UUID -s /dev/myalias

But the problem is that I want to do the mapping between "disk" and "LUN", not between "partition" and "LUN", and I can't use this:

[root@srs121 root]# scsi_unique_id /dev/sda
model: COMPAQ HSV110 (C)COMPAQ
page80: 503538343945314141514530344a
page83 type3: 600508b4001010640000c00000090000

[root@srs121 hp-devlabel]# hpdevlabel add -u 503538343945314141514530344a -s /dev/sdb
The file /dev/sdb already exists.
Failure. Could not create a symlink.

Yeah, that's true, sda/sdb was occupied!
So, how can I FORCE the mapping as I want before the QLogic automatically assign the mapping? Any config file(s) for that?

Thanks a lot!

[yyghp]

NOTE: I have to keep using Kernel 2.4, instead of 2.6.

Thanks!

[yyghp]

Any idea for this?
Thanks!

[yyghp]

The reason why I want to change the LUN order is that our DBA asked me to make it the same LUN mapping as the other two production RAC servers, just for his configuration convenience, not a must technically.
However, even now I don't care about such crossover LUN order request by our DBA, I still have to find out a way to ensure the LUN persistence under the QLogic Failover mode. Because QLogic driver with failover mode, which combines 4 paths to 1, assigns /dev/sda to the first LUN that the server detects, since now we have two LUNs at the same EVA5000, we have to make sure /dev/sda always point to the same LUN. That is, Server A has /dev/sda and /dev/sdb point to two LUNs on the same EVA5000, for example, /dev/sda points to LUN#1, and /dev/sdb points to LUN#2, which is controlled by QLogic driver with failover mode automatically when system restarts. If during next reboot, Server A detects LUN#2 first, then it will assign /dev/sda to it, instead of /dev/sdb, then I think Oracle RAC will encounter the wrong raw device files. So, we have to find out a way to make sure that the raw device file ( sda or sdb ) always points to the right LUN.
I had the similar issue before on our production RAC servers, however, because those two LUNs are located in two different EVA5000, so, either server normally dectected the local LUN first, then the remote LUN, which helped to ensure the persistent order, but not guarantee.
As I mentioned in my post above, the problem is that the LUN order is controlled by the QLogic driver automatically, but how can we control the order manually?

Thanks again!

[Simon Hargrave]

I know this thread is out of date, but a tip if anyone searches for similar: -

Instead of worrying about LUN persistence, use filesystem labels. Okay it doesn't work for Oracle using RAW filesystems, but it does for ext3 filesystems, and also ocfs (Oracle filesystem).

If you use OCFS for RAC, you can mount the filesystems by LABEL and not specify /dev/sda etc. That way if the disks are reordered it doesn't matter.

One point to note, it seems you need to create a partition sda1 as the full size of the LUN and use this rather than the "whole disk" - the mount facility searching for labels looks on partitions and not on disks for matching labels.