Activating vg on two systems (rw and ro)

[Mark van Hassel]

Hi people,

On two systems we want to share the vg's. On one system we activate it read/write and the other read only, same accounts for the file systems (mounted read/write and read only resp.).
When system A fails, the idea is to umount the file system on the node B, vgchange it read/write and mount the the fs read/write.
I know I have to edit the /etc/fstab and /etc/lvmrc files to make sure the vg and fs are in read only mode during boot. (and the danger when system A boots again and mounts the fs read/write or performs a file system check !)

Has anyone experience with a setup like this, are there any drawbacks, problems we should expect, suggestions ?

I know something we should use MCSG, but that is not an issue here.

Thanks,

Mark

[James R. Ferguson]

Hi Mark:

MC/ServiceGuard is designed/intended for high-availablity requirements. You can do automatic or manual failovers. I would urge you to look further at that solution. You do not have to have a configuration that maintains a server (node) doing nothing but waiting for failover (active/passive). Instead, all nodes can be active, and if one fails you can load-shed packages if necessary to yield more resources to your critical ones.

Regards!

...JRF...

[Carlos Fernandez Riera]

You maust activate as shared ( see man vgchange). Also you must mount as readonly.
Not checked by me.

But, what about sharing by NFS?

[Mark van Hassel]

Hi,

I know the MCSG options, but again that is not an option here. ALso the shared option of vgchange is for MCSG only, I believe.

NFS is no issue since it requires both systems to be up.

[Carlos Fernandez Riera]

Make your own SG ( Service Warning - Service Guarring in Spain-).

Add a floating IP address on lan0:1

Check this IP from other node. When fail, activate vg, and add lan0:1 in backup box.

Good Luck.

[Animesh Chakraborty]

hi,
Go for nfs mount.
It will be safe.
mount :/mount directory /local mount directory.
before that export the directory.
make entry in /etc/exports
then exportfs -a
start nfs server and nfs client.

best of luck
Animesh

[Mark van Hassel]

It is not my intention to rebuild MCSG only to make a file system visible on more than one system, where only one system is allowed to write. The command options (vgchange -a r, mount -r) are there. I was just wondering if someone has experience with this.

NFS is not an option: if the server crashes, the client has no access to the exported fs.

[Deshpande Prashant]

Hi
Some time back, I had done similar to this.
I had same VGs seen by two servers. On second server we mounted the file systems, to do nightly backups only.
I think you can mount file system as readonly on second machine while it's read-write on first machine.
You can keep small script handy, which can unmount readonly file systems and mount them as read-write on second server, when needed.


Thanks.
Prashant.

[Carlos Fernandez Riera]

MC Service Guard is required for this. As well as for mirror, MirrorDisk/UX is required.

[Sanjay_6]

Hi Mark,

Do you want to activate the Vg at the same time on both the servers. It is not possible, whether you do it in rw or ro mode. Even with plain SG it is not possible. You cannot activate the Vg at the same time on both node. You can think of something like that with SG-OPS.

Hoep this helps.

thanks.

[Carlos Fernandez Riera]

If you can not activate VG, then bypass LV Manager.

Go backward to partitoned disks...

[G. Vrijhoeven]

Hi Mark,

Thare are a lot of answers and i like to add one. For your sollution you need to have the disk attached to two servers at the same time, you do not want to depend on one server so running nfs is out. What you can do is install an SAN box in the network.

[Mark van Hassel]

Yes, we do have a SAN. That's the whole idea. The setup is for contingency purposes. In case one system fails we have the possibility to import the vg's and mount the volumes on another system. To make the procedure more simple, I want the vg's and file systems to be there all the time (read only) zo in case of a crash we only have to vgchange them read/write and mount them read/write. This way we don't have to look up the device files and import the vg's first.

It is possible to activate vg's on more than one node and mount file systems (even read/write on both systems), I already tested that. I was just wondering whether someone has experience with a setup like this.

And again, MCSG is not an option because the application is not suited for it, i.e. it requires "manual" checks after a system crash. It does not have transcation logs like Oracle for instance.

[Santosh Nair_1 · Accepted solution]

Mark,

LIke you said, it is possible to share the VG on two machines (one as rw and other ro), but this in mind.

Suppose the VG is mounted rw on systemA and that machine crashes, now you have a possibility of corrupt filesystems. So, to do what you'd want to do, you'd have to 1. umount the filesystems on systemB, 2. vgchange VG to rw, 3. fsck all the filesystems and 4. remount the filesystems rw.

Your best bet, given your requirements would be to write a small mount script to do the remounting. Also, I would change the /etc/lvmrc to make AUTO_VG_ACTIVATE=0 and then change the custom_vg_active script (also in this file) to active all the VGs except the shared one, i.e.:

/sbin/vgchange -a y -s
parallel_vg_sync "/dev/vg00 /dev/vg01"
parallel_vg_sync "/dev/vg02 /dev/vg03"
return 0

Finally, write a small startup script to activate and mount these filesystems ro.

This way, if the machine crashes or reboots, the VG is automatically activated ro. Then manual intervention would be required, i.e. run the mount script, to mount the VGs rw on the appropriate system.

Hope this helps.

-Santosh

[Mark van Hassel]

Thanks Santosh,

Do you know if its possible to specify different vgchange options per vg in /etc/lvmrc ? For example:

/sbin/vgchange -a y -s /dev/vg00 /dev/vg01
parallel_vg_sync "/dev/vg00 /dev/vg01"

/sbin/vgchange -a r /dev/vg02 /dev/vg03
parallel_vg_sync "/dev/vg02 /dev/vg03"
return 0

[Santosh Nair_1]

Mark,

To be honest I've never had to set up something like this, but I would image you could what you said.
Not sure if you have the luxury, but I would test it out somewhere first, just to be sure.

-Santosh

[Mikula Oliver]

Mark,
you don`t need mount fs ro on system B,
simply create vg on system B, update fstab
and if your system A crashes activate vg on
system B and mount...
What you must do on system B
-update /etc/lvmrc add list of all vg`s
without vg from system A
(default is rw for all vg`s in lvmtab)
-update /etc/fstab with disks from system A

Some notices :
-I suppose you have disk array or disks with two host connections
- you have vxfs fs not hfs because sometimes fsck is needed and hfs spend a lot of time
- read only fs don`t synchronize fs and somtimes content of disk is not correct.

[Steve Bonds]

Mark:

I've done this, and it doesn't work like you'd hope. The problem is the aggressive UNIX filesystem caching. If you create a file on the r/w filesystem you may not see the changes on the r/o filesystem for a long time. (In theory, this should be 30s as fsflush syncs the memory cache with disk, but I saw things like inode changes held off the filesystem even longer.) Running manual "sync" commands did not seem to help. My guess is the r/o system doesn't check against its inode cache very often so changes are slow to appear. Unmounting/remounting the r/o filesystem will sync things up but it not all that convenient.

The solution, as others have noted, is to use NFS which has a cache flush/locking mechanism already built in. If you *really* want to use fibre channel, then you can always use IP over FC for this. I'd suggest gigabit ethernet instead as the drivers seem to be a lot more reliable.

There are a number of people working on third-party products to allow simultaneous access (even simultaneous r/w access) over a fibre channel SAN. These include SGI (http://www.sgi.com/developers/feature/2000/cxfs_jun00.html), Sistina (http://www.sistina.com/products_gfs.htm), VERITAS (http://www.veritas.com/products/category/ProductDetail.jhtml?productId=sanpointfoundationsuiteha) and others. Note that not all of these are yet available for HP-UX.

In theory if you can completely disable file caching (i.e. through QuickIO or use of raw logical volumes) you can have data immediately visible on the r/o node. Using QuickIO would be dangerous unless you had outside locking since two processes could easily stomp each others' data on disk. Using the raw logical volume is not real handy since there's no file information available.

[Mark van Hassel]

Hi,

I am aware of the caching "problems" . However it is not the intention to use the data on the ro file systems when running normally. I want to set it up like this simply to make procedures easier in case of an emergency. When something happens, the only thing we have to do is umount the fs, vgchange the vg to rw and mount the fs rw. Any other way, we have to look up the devices and impoort the vg and mount the fs. I think the procedure is easier when the fs are already there.

Thanks for the responses.

[Alan Riggs]

Mark,
If you do not intend on having the read-inly filesystems actively used then I would suggest not messing with the simultaneous access and its incumbent management headaaches. Simple run a vgexport -s -p on the active server and transfer that file to the "backup" server. This will allow you to vgimport without worrying about specifying devices. You can then either replicate the fstab structures in /etc/fstab or use a separate fstab.failover file (I usae a separate file) which specifies mountpoints and mounting options for the new lv's.

The advantages of this method, as I see them, are:
minimal maintenance "overhead".
instantly obvious when "backup" server is "active".
*NO* risk from simultaneous disk access.
Easily scriptable (mknod; vgimport -s; fsck; mount)

[Steve Bonds]

Heck, if you just want the volume to be ready, but not in use, it's easy!

Just do a "preview" export of the volume group on the system using the filesystems (vgexport -p -m /tmp/.map) to create the logical volume map file (you'll get an error, but it should still create the map file).

On the other system, figure out which disk devices map to the same disks in use on the original. Then use "vgimport" with the above-created map file and the new device names (include alt paths). Run a "strings /etc/lvmtab" after the import to verify that the VG is imported. Add the LV entries into /etc/fstab (commented out for now)and make sure all your mount points are present.

You won't be able to activate the VG until the other system is no longer using it. When you can, arrange a test and do the following:

original system:
+ umount all the filesystems on that VG
+ run "vgchange -a n "
+ comment out entries in /etc/fstab

other system:
+ run "vgchange -a y " (you might get an "incorrect CPU ID" error, which you can ignore.)
+ uncomment the mounts in /etc/fstab
+ mount the filesystems and go!

This is much simpler than I thought your original question was. ;-)

[Mark van Hassel]

Hi,

I am aware of all the vgexport/vgimport options. But I still think the procedureI have chosen is the best. The problem is that LUNs will be added or removed and people will forget to add or remove LUNs on the ro vg's. Offcourse this is still the case in my option, but the big advantage is that the vg and file systems are active so people will notice them.
When there are just a few systems to administrate its not a problem, but when there are hundreds you just want to choose for the easier procedure.

I have run into problems before when not specifying the PV's when using vgimport, especially on systems with a large amount of LUNS.

[Manel Ventura]

Hi,

Once I thought to use this way to get high-availability but there's a problem. Certainly, you can have the same filesystem on two hosts, but it's inconsistent because that filesystem is managed separately from those hosts.

You can do a little test. Create an empty file on that filesystem from one host and next try to see that file from the other host. The second host won't can see that file.

If you're afraid to make changes on one host and forget to refresh the other one, just do a script to remove and recreate the vg's every day. I think that's all you need.

Manel.

[Steve Bonds]

I don't see how trying to keep the volume group active on two systems would help with the "forgot to add a new PV (LUN)" issue. You'll still need to add the new PVs on both systems since the LVM VG information is stored locally in /etc/lvmtab, not on the disks in the VG itself.

(I.e. both systems would need to have the vgextend command run, which doesn't seem significantly different from running a vgimport command with new disks.)

It would be possible to work out your own system for detecting missing LUN/PV problems on the host where the VG is not active. (I.e. have a script check /etc/lvmtab on both hosts and compare the number of devices in each VG to make sure they're in sync.) If you want to get fancy, you would even have the script include a mapping table so that it can determine the appropriate "cXtYdZ" values on the standby system.

[Alan Riggs]

"The problem is that LUNs will be added or removed and people will forget to add or remove LUNs on the ro vg's. Offcourse this is still the case in my option, but the big advantage is that the vg and file systems are active so people will notice them."

Well, using the -s flag will explicitely correct for this problem. As others have noted, having the VG active on another system will not "alert" when the VG is changed on system 1. In fact, unless you drop below the quorum threshhold, you can even activate/deactivate the VG without generating an error (I believe, haven't tested it).

I don't know what problems you have encountered with the -s flag and a large number of LUNs. I use it regularly on a backup system with more than 100 LUNS presented. Other than a bit of lag in response time, I have seen no problems. You do need to exercise care when addressing split BCV volumes, of course.