LUN persistence with Failover mode QLogic driver

[yyghp]

I presented 2 LUNs on 2 different EVA5000 to a server ( running Red Hat Enterprise Linux 3, with Kernel 2.4 ), and used QLogic driver as "Failover mode" to combine 4 paths to 1, then I can see 2 LUNs ( each LUN is at different location. ):

# lssd
sda 0,0,0,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-569c
sdb 0,0,2,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5448

The problem is that the mapping of "sda -- 5000-1fe1-5004-569c" and "sdb -- 5000-1fe1-5004-5448" may change after reboot, because which LUN maps to sda/sdb depends on which LUN the server can detect first! And as requested by our DBA to setup RAC, I HAVE TO make the mapping like this:

sda 5000-1fe1-5004-5448
sdb 5000-1fe1-5004-569c

So, it should be crossover! Now, I wonder whether it's possible for me to make the LUN persistent to such crossover mapping, even the server reboots, sda has to point to LUN 5000-1fe1-5004-5448.

The problem is that such LUN order is controlled by the QLogic driver with Failover mode automatically, I can't find any config file to change such order, it seems the order is out of my control!

I read the last document from HP for LUN persistence, I attach the utiliy files with the doc ( application notes: HP StorageWorks LUN migration and persistence utilities 1.1 ) within this email (LMPutilsv1.1.tar.gz), you can take a look, there is a solution for Linux with 2.4 kernel.

So, I did the following:

Change the QLogic driver back to Single Path mode, reboot, then

After reboot:

[root@srs121 /]# lssd
sda 0,0,0,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-569c
sdb 0,0,1,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5699
sdc 0,0,2,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5448
sdd 0,0,3,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-544d
sde 1,0,0,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5698
sdf 1,0,1,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-569d
sdg 1,0,2,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-544c
sdh 1,0,3,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5449

[root@srs121 backup]# ll -lrt
...
-rw-r--r-- 1 root root 330038 Jun 21 08:52 LMPutilsv1.1.tar.gz

[root@srs121 backup]# tar -zxvf LMPutilsv1.1.tar.gz
LMPutils/
LMPutils/HPMigrationPersistenceUtil-1.1-1.noarch.rpm
LMPutils/LUN Migration Application Notes.pdf
LMPutils/hpudev-cfg-1.1-1.noarch.rpm
LMPutils/install.sh

[root@srs121 backup]# cd LMPutils
[root@srs121 LMPutils]# pwd
/backup/LMPutils

[root@srs121 LMPutils]# ./install.sh -i

Please wait ...



Welcome to the Linux Lun Migration and Lun Persistence instal

Version 1.1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ 1. Lun Migration with Lun Persistency
~ 2. Lun Persistency only (2.4 Linux Kernel)
~ 3. Lun Persistency only (2.6 Linux Kernel)
~ 4. Exit
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Enter your Choice (1/2/3/4) : 2

Please wait, checking the system...


Installing Lun Persistence Utility Please Wait....

Preparing... ########################################### [100%]
1:HPMigrationPersistenceU########################################### [100%]

Lun Persistence utility Installed successfully


Lun Migration and Persistence utility is installed successfully

INSTALL QLOGIC DRIVER IN FAILOVER MODE,AND FOLLOW THE INSTRUCTIONS IN LUN MIGRATION
& PERSISTENCE UTILITY APPLICATION NOTES FOR MIGRATION AND ARRAY UPGRADE PROCESS

[root@srs121 LMPutils]# cd /opt/hp/src/hp_qla2x00src/
[root@srs121 hp_qla2x00src]# ./set_parm


Driver Parameters
-----------------------

1) XP/MSA/EVA Failover
2) Secure Path
3) Single Path
4) Set Paramters Manually
5) Quit

Choice: 1
Writing new /etc/hp_qla2x00.conf...done
adding line to /etc/modules.conf: alias scsi_hostadapter1 qla2200
adding line to /etc/modules.conf: alias scsi_hostadapter2 qla2300_conf
adding line to /etc/modules.conf: alias scsi_hostadapter3 qla2300
adding line to /etc/modules.conf: alias scsi_hostadapter4 sg
adding line to /etc/modules.conf: options qla2200 ql2xmaxqdepth=16 qlport_down_retry=30 qlogin_retry_count=16 ql2xfailover=1 ql2xlbType=1 ql2xexcludemodel=0x0
adding line to /etc/modules.conf: options qla2300 ql2xmaxqdepth=16 qlport_down_retry=30 qlogin_retry_count=16 ql2xfailover=1 ql2xlbType=1 ql2xexcludemodel=0x0

Would you like to create a new initrd to reflect the changes in the parameters (Y/n)? Y
Creating new initrd - initrd-2.4.21-27.ELsmp.img

---Press ENTER to continue---

Quit...

# reboot

After reboot:

[root@srs121 /]# lssd
sda 0,0,0,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-569c
sdb 0,0,2,1 COMPAQ HSV110 (C)COMPAQ 3020 5000-1fe1-5004-5448


Now, the LUN order is still wrong, how can I use "hpdevlabel" to make it crossover, and make it persistent?
It has two ways to create symbolic links:

hpdevlabel add -d /dev/sdb1 -s /dev/myalias
hpdevlabel add -u UUID -s /dev/myalias

But the problem is that I want to do the mapping between "disk" and "LUN", not between "partition" and "LUN", and I can't use this:

[root@srs121 root]# scsi_unique_id /dev/sda
model: COMPAQ HSV110 (C)COMPAQ
page80: 503538343945314141514530344a
page83 type3: 600508b4001010640000c00000090000

[root@srs121 hp-devlabel]# hpdevlabel add -u 503538343945314141514530344a -s /dev/sdb
The file /dev/sdb already exists.
Failure. Could not create a symlink.

Yeah, that's true, sda/sdb was occupied!
So, how can I FORCE the mapping as I want before the QLogic automatically assign the mapping? Any config file(s) for that?

NOTE: I have to keep using Kernel 2.4, instead of 2.6.

Thanks a lot!

[Ivan Ferreira]

I readed half of your post, what a detailed information. But to make devices persistent, try using the "devlabel" command.

he devlabel software attempts to address the device naming issue in a different manner than file system labels. The devlabel software is run by Red Hat Enterprise Linux whenever the system reboots (and whenever hotpluggable devices are inserted or removed).

When devlabel runs, it reads its configuration file (/etc/sysconfig/devlabel) to obtain the list of devices for which it is responsible. For each device on the list, there is a symbolic link (chosen by the system administrator) and the device's UUID (Universal Unique IDentifier).

The devlabel command makes sure the symbolic link always refers to the originally-specified device â even if that device's name has changed. In this way, a system administrator can configure a system to refer to /dev/projdisk instead of /dev/sda12, for example.

Because the UUID is obtained directly from the device, devlabel must only search the system for the matching UUID and update the symbolic link appropriately.

# devlabel add -s /dev/my_stable_disk_name -d /dev/sda

[yyghp]

Thanks Ivan, but you haven't given me any solution according to my situation.
I know that "devlabel"(by Redhat) or "hpdevlabel"(by QLogic) can be used to ensure LUN persistence, but it DOES NOT work when we are using QLogic's "Failover mode", which combines 4 paths to 1 path.
Because I set the QLogic driver to use "Failover mode", it assigns the device names(/dev/sda,/dev/sdb) to the LUNs AUTOMATICALLY when system starts.
Our DBA asked to use "/dev/sda" to point to specific LUN, instead of using any other alias/symbolic link.
I think there must be a way to manually control QLogic to assign the device names to the LUNs. That is, I need the exactly mapping between device name and LUN, like:

/dev/sda ----------- LUN@site1
/dev/sdb ----------- LUN@site2

but now:
/dev/sdb ----------- LUN@site1
/dev/sda ----------- LUN@site2
which doesn't fit the requirement.

[Ivan Ferreira]

So, the problem is that the DBA does not like it, because the DEVLABEL uses the UUID (which is uniq) and the device will always point to the same device. Why your DBA only wants to use /dev/sda? The device name should not matter if you can access the same data all the time.

[yyghp]

How can I use DEVLABEL to ensure the LUN persistance when I am using failover mode QLogic driver?
And yes, the DBA requested to use /dev/sda for specific LUN, because that's the requirement to setup Oracle RAC ( two boxes use "crossover" two LUNs. ), and in order to make sure the new RAC environment has exactly same configuration as the other RAC environments, the DBA requested this. If no, I wouldn't have posted this for help.
Thanks!

[Darrin St. Amant]

I've set up many RAC environ. not needing this mapping requirement. As far as Oracle RAC is concerned you only need shared devices for Oracle CSR services and the DB files. Now you can use ASM/RAW, ASM/ASMLib, or OCFS for DB files which matains the persistance and you can use raw dev for CSR. Why do you want to change the order? why does it matter?

ds

[yyghp]

Hi Darrin,

Thanks for your reply!
The reason why I want to change the LUN order is that our DBA asked me to make it the same LUN mapping as the other two production RAC servers, just for his configuration convenience, not a must technically.
However, even now I don't care about such crossover LUN order request by our DBA, I still have to find out a way to ensure the LUN persistence under the QLogic Failover mode. Because QLogic driver with failover mode, which combines 4 paths to 1, assigns /dev/sda to the first LUN that the server detects, since now we have two LUNs at the same EVA5000, we have to make sure /dev/sda always point to the same LUN. That is, Server A has /dev/sda and /dev/sdb point to two LUNs on the same EVA5000, for example, /dev/sda points to LUN#1, and /dev/sdb points to LUN#2, which is controlled by QLogic driver with failover mode automatically when system restarts. If during next reboot, Server A detects LUN#2 first, then it will assign /dev/sda to it, instead of /dev/sdb, then I think Oracle RAC will encounter the wrong raw device files. So, we have to find out a way to make sure that the raw device file ( sda or sdb ) always points to the right LUN.
I had the similar issue before on our production RAC servers, however, because those two LUNs are located in two different EVA5000, so, either server normally dectected the local LUN first, then the remote LUN, which helped to ensure the persistent order, but not guarantee.
As I mentioned in my post above, the problem is that the LUN order is controlled by the QLogic driver automatically, but how can we control the order manually?

Thanks again!

[yyghp]

Anyone could share his experience about this issue?
Thanks!