Xen Multipath Virtualization

[Gilbert Standen_1]

Hi, I recently did a 64-bit Oracle 2-node RAC production deployment on RHEL 5.3 Linux for an HP consulting client (I am an HP consultant). I prototyped the system on a laptop for the client mostly to work through any software issues on the OS or Oracle software before doing the actual install on the production blades. The prototype was on CentOS 5.3 running the Xen kernel (2 Centos 5.3 DomU's) on intel dual core T3400 EM64T running 4Gb memory.

My question is this: Underneath the Oracle installation (which I did as Oracle DBA) we enter the world of how the multipathing can be designed, about which as an Oracle DBA I know not so much as I do about Oracle.

First, we did the devices using (deprecated) raw devices by making use of the various files in the /etc/udev/rules.d directory since there is no more "service rawdevices start" etc. We could have used dm-multipath though, and I'd like to do some prototyping of that on my Xen VM prototypes of our system using dm-multipath instead of rawdevices approach.

So what I am wondering about is this: I have done my shared disk on the domU's for rac this way: they are files on disk of this presentation type to the domU's:

disk = [ "tap:aio:/vtdev/rac1.img,xvda,w", "tap:aio:/vtdev/hdb,xvdb,!", "tap:aio:/vtdev/hdc,xvdc,!", "tap:aio:/vtdev/hdd,xvdd,!", "tap:aio:/vtdev/hde,xvde,!" ]

The disks with "!" as the last entry are the shared disks (the domU's access these as "shared raw"). This works fine for the RAC prototype.

Ok finally I think we come to what I am wondering about. First of all, what in fact is the "path" to these "raw disks" ? I'm trying to get a hand on the the rockledge of "multipath" so I'd like to start by understanding what is my "path" to these simple "shared disks". I am guessing these are "singlepath" devices and are not multipath. How, in the language of linux, can we describe, view the "path" to these simple "devices" (e.g. /dev/xvdd, /dev/xvde, etc.). Are there linux commands which can "show" me how we describe the "path" to these devices?

My next question is, can I somehow simulate with what I already have, "multipath" devices. Can I build "multiple paths" to these "shared files".

Also, have you any suggestions on how to create a "replica" of a "SAN" or any links you could point me to so that I could build under these RAC nodes "virtual" HBA's, SAN's, controllers, etc. that would allow us to emulate our actual physical hardware.

Currently ( I discussed this with the Linux Administrator) we are using IBM "RDAC" for our "multipath" and I verified using lsmod and "service dm_multipath status" that although dm_multipath rpm's are installed, the service is not on and all boot levels (chkconfig) have dm-multipath off so we are definitely not using dm for anything now.

I apologize for this kind of vague message. I'm just looking for general pointers where I can go to learn more about this and start to move in the direction of building a more authentic "virtual" structure underneath the domU's than just these files which are emulating raw devices.

Thanks in Advance,
Gil Standen
HP CWO Consultant
NYC

[Gilbert Standen_1]

Well doing some additional dogged research turned up this paper from IBM which seems to really give a pretty darn good roadmap to do some of the things I am asking about:

http://www.ibm.com/developerworks/linux/library/l-multipath-xen/?ca=dgr-lnxw16RedHatLinuxXenp&S_TACT=105AGX59&S_CMP=GR

Thoughts? The only thing remaining really given this great roadmap at the above URL is to create a "simulated" IBM DS4800 in the sense of just simulating the existence of the 3 LUNS. Any thoughts on that? Thanks!

[Matt Palmer_2 · Accepted solution]

HI, with regards to replica's,etc. I guess this depends somewhat on what your SAN/disk infrastructure is built up of. In my environment we have used DRBD when we have needed to create DR replicas that are either on SAN, but we need copying to local disk.

In other scenarios we have used SSSU to create snapclones/snapshots of datasets. Instead of using dm-multipath, we have used Qlogic FC HBAs with their built-in failover mode, and setup multiple paths to the disks at the switch/SAN level. - This has worked really well for us, and has also given us the added benefit of being able to use Continous Access to replicate data between 'live' and DR site with no service interruptions.

Hope this helps.

regards

Matt

[Matt Palmer_2]

Hi,

I've looked over the link you provided, and the tests that the guy does are basically the same ones I created for the environment I have here. The only difference in my configuration is that I'm running SLES 10 SP2 not redhat. I found a link on google that details an issue with using the failover mode on a Qlogic FC HBA in conjuntion with dm-multipath. What I saw happening was that the failover only appeared to work one way, and then once the path was available again the filesystem remained locked as readonly and required a reboot to cure this. Not sure if this problem is particular to just SLES10/Qlogic/dm-multipath, but thought it may be worth a mention.

regards

Matt

[Gilbert Standen_1]

Here is an URL http://communities.vmware.com/message/107057 which describes how "multipath" was set up on VMware and here is a snip from that URL of how it was set up in the .vmx file:

# INTERNAL SCSI HBA
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"

# HBAs attached to "shared disks"
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "virtual"
scsi2.present = "TRUE"
scsi2.virtualDev = "lsilogic"
scsi2.sharedBus = "virtual"

# shared bus setting
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"
disk.locking=FALSE

# internal disk 1
scsi0:0.present = "TRUE"
scsi0:0.fileName = "dsk0-000001.vmdk"
scsi0:0.redo = ""
# internal disk 2
scsi0:1.present = "TRUE"
scsi0:1.fileName = "dsk1-000001.vmdk"
scsi0:1.redo = ""

# shared disk 1, via path1
scsi1:0.present = "TRUE"
scsi1:0.fileName = "D:\share\shared1.vmdk"
scsi1:0.redo = ""
scsi1:0.mode = "independent-persistent"
scsi1:0.deviceType = "disk"
# shared disk 1, via path2
scsi2:0.present = "TRUE"
scsi2:0.fileName = "D:\share\shared1.vmdk"
scsi2:0.redo = ""
scsi2:0.mode = "independent-persistent"
scsi2:0.deviceType = "disk"

[Matt Palmer_2]

Hi,

this approach looks fine from the guest point of view. My only question (not being from an oracle house) is does it only need to be the raw devices that you need have multipathing setup on? are there filesystems that need to have a diverse route set to them?

regards

Matt