Share 1 rw LV and 1 ro LV between two systems

[tech1214]

I've already got the following setup working and tested under very light loading,so I am primarily looking for a sanity check on the basic assumptions.

The goal:

Two identical systems share simultaneous read-only access to one filesystem and exclusive read-write access to a second filesystem. So if one machine crashes or hangs, the other machine can be made to "manually" fsck and mount the read-write filesystem.

The implementation:
Two machines, each with two fibre controllers connected to two JBODs such that each fibre controller can see both JBODs (thus enabling each machine two paths to get to each disk in the system).

Each machine has a boot/swap/etc disk, mirrored across JBODs. Then there is a volume group of 3 disks, mirrored across JBODs (for a total of 6 disks) and extent-striped. This VG has two LV's both with vxfs.

The first LV is mounted read-only by both machines simulatenously. The second LV is mounted read-write by the first machine and should it crash or otherwise become unavailable, we manually fsck it and mount it on the second machine.

Right now, BOTH machines have the shared volume group imported and activated as read-write. The idea is to reduce complexity of handling the "move" of the read-write filesystem from one machine to the other. If I were to activate the volume group as read-only on one machine and read-write on the other, that quickly results in a bunch of scripting to keep who has what mounted where with what permissions straight - particularly if one machine gets rebooted.

My assumption is that as long as I am not changing the LVM configuration (like reallocating extents) that LVM itself will not write anything to the volume groups. Instead, the only writing that could occur is directly in the filesystem that is mounted read-write. So, as long as I make the effort not to allow a simulatenous mounting of the read-write filesystem, everything will just work.

So, is this assumption about LVM correct? Also, if I need to resynchronize the mirror, do I need to force the other machine to read-only on the volume group during syncronization (or worse, no-access until synchronization completes)?

[A. Clay Stephenson]

You are an accident waiting to happen. The only way than more than one system can SAFELY mount a given filesystem is if ALL systems mount the filesystem readonly. I know you think everything is fine but the fundamental problem is buffer-cache consistancy. If system A updates a block in it's buffer cache, system B won't have a clue that this has been done. The problems are compounded if some of these writes are to filesystem data structures (e.g. directories, super-blocks, ...). Remember, the buffer cache is updated and then at some INDETERMINATE later time the data are written to disk.

I know your tests have given you confidence but I assure you that it is false confidence. The ONLY way to safely do read-write I/O to a shared LVOL is raw/io (which uses no buffer cache or filesystem) and even then there must be some sort of synchronization mechanism.

[Sundar_7]

First of all, I am sure you are already aware of this - consider purchasing service-guard - makes the life lot easier and less complex :-)

If you have already considered this option and didnt go for it for whatever reasons, then I have the following suggestions for you.

1) Consider seperating the RO LV and RW LV in to two seperate volume groups. I understand this may not be possible in all the cases.

2) I would not go with the above solution if I am really serious about the data in the RW filesystem :-)

As far as mirroing goes, Once the mirror is established, you should not need to resync manually.

[tech1214]

Clay - I appreciate your concern, but it is misdirected.

As I said, the read-only filesystem is the only filesystem ever mounted simulatenously. I am fully aware of the way buffercache works. What I am not fully aware of is how LVM works and that's why my question is about the integrity of the LVM data-structures when the VOLUME GROUP is actived read-write on two machines.

Sundar - When you say: "I would not go with the above solution if I am really serious about the data in the RW filesystem." Are you referring to my current plan of one volume group, or your suggestion of two volume groups?

Thanks

[A. Clay Stephenson]

I noted what you said in your text but I also saw your thread title. You are actually making this much harder than it it. As long as you have already done a vgimport on all the boxes for all the VG's that you wish to be able to mount and modified /etc/lvmrc setting AUTO_VG_ACTIVATE=0 so that you must
explicitly do the vgchange -a y, the actual volume group activation takes only seconds. Anytime that you add or remove an LVOL or add or remove a PV from the volume group, you must vgexport and vgimport again. Routine changes (like expanding LVOL's) need no further changes on other hosts and will be recogized when the VG is mounted on the other host. The resyncing of mirrors should not be required except when a disk fails -- and that is no reason to down a box. I've replaced tens of disks and have not shutdown to do this in well over 5 years -- nor did any application miss a beat while this was being done.

As mentioned, MC/ServiceGuard is tailor-made to do what you are trying to do and much more.

[Sundar_7]

Sundar - When you say: "I would not go with the above solution if I am really serious about the data in the RW filesystem." Are you referring to my current plan of one volume group, or your suggestion of two volume groups?


I am referring to your current plan. HP-UX Kernels cannot really communicate volume group changes with a kernel running in other HP-UX box.

If you are really want to achieve your gal without MC-serviceguard, I would suggest creating two seperate volume groups - one for RO and one for RW logical volume.

Edit /etc/lvmrc and set AUTO_VG_ACTIVATE=0 and activate only vg00 and the RO volume group.

whenever the node that currently has the filesystem in RW mode crashes, activate the volume group in this node and mount the LVOL in RW mode.

[Dietmar Konermann]

OK, the limitations are actually on 2 different layers. Concerning the upper layer(the FS layer) all questions are answered already.

Your question is about the level below that (the LVM layer). And since you're using LVM mirroring I would strongly recommend *not* to do what you want. LVM is consistently updating mirror conistency information on the disks. This information is only read at activation time for crash recovery... this may get dangerous in your setup, when the R/W-System crashes and the other one takes over without a fresh activation.

You should have a look at the WARNING section of vhchange(1M), which explains it also:

---
WARNINGS
Ordinary Operation

In ordinary operation (i.e., without the optional high availability software), it is possible to activate a volume group for read-write access from more than one physically connected system, leading to a high potential for data corruption. Therefore, if access is desired from more than one system to a single volume group, it is important that only one system activate the volume group for read-write access; the other systems can use read-only access. There is no problem if all systems activate the volume group for read-only access.

Furthermore, volume group information is only read from the disks during volume group activation. Dynamic changes to the volume group such as the following are not propagated to other systems sharing the volume group:

Logical volume configuration changes.

Changes to the status of the mirrored extents.

Bad-block relocation that occurs during write operations.

Because of these limitations, when sharing volume groups between systems it is recommended that logical volumes be accessed only by one system at a time. If logical volumes need to be accessed simultaneously, the logical volumes should not be mirrored and should not have bad-block relocation turned on, or all systems should use read-only access to the logical volumes.
---

Best regards...
Dietmar.

[Dave Wherry]

You already have several very good answers that tell you not to do it, and I agree. For me the biggest issue in the plan is the simultaneous mounting of the file systems and activation of the volume groups.

Part of what you are trying to accomplish is what I've always called Poor Man's ServiceGuard. Devices shared between systems, active on one system and a script ready to activate on the other system.

I did this with an SAP environment. Production on a V2500, QA on a K570 and an XP256 for storage. If the V2500 was down for what would be an extended time, within minutes I could disable my QA environment and bring up production on the K570.

In this environment a little downtime was acceptable. In that time we could determine if the V2500 would be back up quickly or it was going to be an extended period. We did not need immediate failover to the 2nd system.

You seem to be wanting to limit the downtime by being almost in a warm standby mode. If your requirements are such that the system can not be down I agree that you need to look at ServiceGuard.

[tech1214]

Thanks guys. For this application, Serviceguard is not appropriate - mainly because of all the other stuff it does. We need lean and mean for what is effectively an embedded control system, not a commercial database type app.

At this point, we have decided to go with mounting both LVs on only one machine at a time. In the future, we may decide to throw more disks at it to allow a unique VG for each LV, so that we can simulatenously mount the read-only filesystem.