lvm and virtual array on multiple hosts

[Randy Tarrier]

Can anyone enlighten me on how the va7400 works with multiple hosts?
My system is an rp8400 w/2 hdw partitions connected to the va7400 via 2 brocade switches. Each host has redundant connections to each switch; each switch has 2 connections to each controller. Everything displays ok w/various arm cmds. I do not have the Secure Mgr option on my system.

I'm having a major brain cramp with the concept of sharing luns between hosts. I tried vgexport/vgimport with the -s -m options to get one host to "see" volume groups created on the other. Unfortunately, that requires identical hdw paths. So far, when I do vgexport/vgimport without -s, I get a brand new vg, which means I have to newfs it to mount it.
It looks to me like there is no way for 2 hosts to "see" the same filesystems created on the va (unless you resort to nfs). I'm guessing this is because the phys vol's are on different hdw paths, so LVM doesn't realize it's the same patch of real estate.
I'm thinking that the safest way to access the same data from either host is to define it in separate luns- waste of space! At least this works with Business Copy for backups, but why am I able to see the data from host 1 on a bc lun on host 2?

Thanks in advance for any virtual bufferins!

[A. Clay Stephenson]

Actually you can vgimport without -s and specify the pv_paths as appropriate to each host. They don't have to be identical BUT that's not your problem. If you are going to share a filesystem between hosts then both MUST be read-only. The buffer cache of hostA will have no idea what's going on in hostB's buffer cache and absolute chaos is guaranteed.


Veritas does make a product called 'File Replicator' that you might interest you.

[Ian Dennison_1]

Randy,

You have discovered one of the major drawbacks of a SAN Device as opposed to a NAS device; SAN just makes disks available, whereas NAS shares File Systems on top of the Disks.

As long as you ensure that only 1 server has the disks in a volume group, you can swap them back and forth between servers at will.

While it may be possible to mount a disk on 2 servers as read only, I would not do it myself personally, but would take the hit on using NFS. How dynamic is the data? Why not 2 copies of the same data?

Can you provide more information on what you are trying to achieve? It sounds like a Split-Mirror backup solution.

Share and Enjoy! Ian

[Randy Tarrier]

Clay,
I understand the issues with respect to changes to data; what I was thinking was host 1 would see it as read-only, and periodically export/import the vg to see the latest changes wrought by host 2.
My problem is that I can't get host 1 to see anything at all that is present on host 2's file system.

For sake of example, host 2 has Omniback, host 1 has the Oracle db. If I do a business copy of host 1's Oracle lun on host 2, then vgimport host 1's vg & lvols (from a mapfile), I'm defining host 1's on a separate lun for LVM, and it works. Now, if I want to recover my db from a backup to host 2, then vgexport/vgimport it back to host 1, I can't get host 1 to see the files on its vg, defined to the same lun that host 2 has.

My process on host 1:
ioscan -fnC disk
disk 7 1/0/8/0/0.1.1.42.0.0.1 sdisk CLAIMED DEVICE HP A6189A
/dev/dsk/c4t0d1 /dev/rdsk/c4t0d1

disk 4 1/0/10/0/0.1.1.39.0.0.1 sdisk CLAIMED DEVICE HP A6189A
/dev/dsk/c6t0d1 /dev/rdsk/c6t0d1

pvcreate -f /dev/rdsk/c4t0d1
pvcreate -f /dev/rdsk/c6t0d1
insf -C disk
mkdir /dev/vgora
mknod /dev/vgora/group c 64 0x010000
insf -C disk
vgimport -v -m /dev/lvmconf/vgora.mapfile /dev/vgora /dev/dsk/c4t0d1 /dev/dsk/c6t0d1
Physical Volume "/dev/dsk/c4t0d1" is not part of a Volume Group
Physical Volume "/dev/dsk/c6t0d1" is not part of a Volume Group
I get this error if I didn't do a vgcreate before, so...
vgcreate -g pvora /dev/vgora /dev/dsk/c4t0d1
vgchange -a y vgora
vgextend /dev/vgora /dev/dsk/c4t0d1
lvcreate -L 3000 -n lvora vgora
lvcreate -L 7000 -n lvnetfs vgora
# fsck /dev/vgora/lvora
file system is larger than device
vxfs fsck: cannot initialize aggregate
file system check failure, aborting ...
so........
newfs -F vxfs -o largefiles /dev/vgora/rlvora
newfs -F vxfs -o largefiles /dev/vgora/rlvnetfs
mount /dev/vgora/lvora /oracle
mount /dev/vgora/lvnetfs /netfiles

.... and this will show a new file system on host 1 sitting on the same lun as host 2, with all the liabilities previously mentioned. What am I doing wrong?

[Brian M Rawlings · Accepted solution]

Randy: as mentioned, this is emminently possible, but there are major issues. Actually, this is routinely done with clustering SW.

As has been mentioned, you can get the VGs to be recognized on the 2nd host (vgimport, vgscan, etc), but the issue is in keeping both systems from walking on each other's data, since there is no global authority as to who can write, locking of files, etc.

This is resolved with MC/Serviceguard, which defines shared VGs as part of one or more "packages", which can run on one and ONLY one node in the cluster at any given time. Now, MCSG is the global authority, and makes sure that multiple hosts can't write to the VG.

It is possible to mount a FS R/W to one host, and to then mount it R/O to other hosts. Per my LVM class, this is even supported by HP, although I've never needed to test this or try it, except in class. Multiple hosts can mount the FS as R/O, no limit specified. I can't think of any cases (in my sphere of responsibility) where doing this would be useful.

While it's possible to share LUNs, VGs, FSs, etc, between hosts, I would suggest that maybe you are attempting SAN functions that you really don't need to do... perhaps you are just trying this for your own knowledge, in which case, good luck and get back to us when the smoke clears...

In reality, for SAN-connected arrays, LUNs get assigned to one and only one host, and other LUNs get assigned to other hosts, as needed for your storage requirements. When you do this right, you accrue great benefits of performance, scalability, and security. When you do it in unusual ways, you accrue great pain and aggravation, but sometimes you have to do it to meet your goals and requirments.

If you can tell us what you are trying to accomplish, maybe we can suggest useful alternatives that will work better for you.

Best Regards, --bmr

[Ian Dennison_1]

2 quick points without detracting from Clays impending answer,...

1. 'pvcreate' is trashing the contents of the disks. No chance of seeing the data afterwards.

2. mknod - make sure the minor number is the same on both Servers, or use the 'vgchgid' command.

Share and Enjoy! Ian

[Eugeny Brychkov]

This is not related to your problem, but as I see from ioscan output: Why VA is working in public loop with 3xxx brocade switch? It should be in fabric login. Use arm* VA command to change topology.
Note: after changing topology device files will change!
Eugeny

[Brian M Rawlings]

Sorry, missed your last post, which adds lots of detail and clears up what you are trying to do. Sort of ignore my previous post.

It sounds like you are having the more or less traditional problem of mounting up a business copy LUN. I have seen this handled several ways, either with 'vgscan' to read the VGID of the BC LUN volumes, and allowing this 'new' VG to be imported, or via vgexport/vgimport as you are suggesting, with map files and so forth.

The same problem exists with 'snapshot' LUNs, which once again have the same VGID as the original. How to mount the snapshot up to the same host for testing, decision support, backups, and so forth? How to mount it up to a different host for same? It looks like you are very close, I can't see an obvious problem with your approach. I've made it work before, but it took several itterations of try/reconsider/change/repeat. EMC actually has a command to change the VGID for their Timefinder operations, everyone has this problem as they try to effectively use their SAN arrays.

Regards, --bmr

[Randy Tarrier]

To all:
Thanks for the explanations; I understand the SAN-NAS analogy, and I can live with the limitations on sharing luns between hosts.

My goal is to automate disaster recovery using one host(2) as backup/restore, the other (1) as a db server. The backup/restore host can pinch-hit for db-server host, if needed.
My strategy then is:
(1) create a lun #3, vg, lvol, fs for reload of the db on host 2
(2) Restore from tape.
(3) Unmount fs, deactivate vg on host 2 after restore.
(4) On host 1, (1) create a vg, lvol, fs for the same lun (#3) we reloaded the db on host 2.
(5) cp db files back to original fs (on separate lun (#1)) on host 1.
(6) Unmount fs, deactivate vg on lun #3 on host 1 after copy.

I tried this (my reply to A Clay) I still can't get the fs mounted on the reload lun(#3) without newfs'ing that vg. I'm doing something wrong, I don't know whether it's conceptual or Homer Simpsonesque!