vgchange headache

[Chris Watson]

I have have a two node cluster conencting with two va7410. The va luns are pv's in vg's and are used to mirror all lvols.

Yesterday I depowered one of the VA's (deliberately), and the L3000 in the same rack (which was running our only package), and thus simulating a building power failure (the failover node and other va7410 being in another building).

When the package came up on the other node (automatically), the package came up clean (I had set 'no quorum on vgchange'). I thought 'this okay'.

THEN, I brought back the depowered va, and l3000 (the l3000 is not further relevant to this problem) and a short time later an ioscan on my failover node running the the active package showed the links were available. SO, I shutdown the package, and started it again. It was a painful 80 mins until the package came backup up.

The question is, if I read the man page for vgchange correctly, that if I use the s switch 'no resync' it will be ignored due to NOMWC being relevant to this case.

Want I really want to happen is that on vgchange it becomes active immediately for use, and stale syncing goes on the background. I don't want certain scenarios putting an hours delay into package startup!

Further, the syncing that did occur in the foreground of package startup seemed to involve almost as much time as the creation of a mirror copy would, even though there were only a maximum of 10MB's of data being change during the depower!

Finally, if I lose sitewide power, the va's will go down at diffent times, due to differnt ups capacities. Thus, when node come back up, its seems I will have to go through all this again.

Basically, the problem is that I WILL have quorum if the package is restarted, and will thus get a foreground resync, thus hours of waiting.

HELP!

PS, Can I get my cluster nodes to tell the va's to shutdown if the ups sends a signal to the node?

[Steven E. Protter]

I don't suppose re-starting the entire infrastructure is an option right now. If production is already down, I'd consider it.

There may be problems with your test methodology and/or the setup.

Restart should resolve the short term problem and let you deal with the quorum issue at your leisure.

SEP

[Sanjay_6]

Hi Chris,

This won't happen. When the mirror is unavilale the other vg would still get activated if it is set to activate without quorum. but once the mirror copy is online, the vg has to be activated manually to recognize the stale disk and then you have to manually run vgsync to sync up the mirror.

In your case once the other VA was brought up, you should have done,

On the node where the package / VG was active,

vgchange -a e /dev/vg_name for all the VG's and then,

vgsync /dev/vg_name for all the VGs.

Otherwise once the vg is deativated and then reactivated, it will first sync up the mirror before it does any mounting and all.

Hope this helps.

Regds

[Jean-Louis Phelix · Accepted solution]

Hi,

You should use this vgchange option in the package control script :

# Uncomment the second line (VGCHANGE="vgchange -a e -q n -s"), and comment
# out the default, if your disks are mirrored on separate physical paths,
# and you want the mirror resynchronization to ocurr in parallel with
# the package startup.
#

Service Guard checks if -s option has been used in the VGCHANGE variable (no synchronization), but instead of doing nothing, it adds a 'background' vgsync. Service Guard ALWAYS synchronize vg at startup, either in foreground, either in background if -s is used, which is different from the usage of this command at the hp-ux command level.

Regards.

[James Murtagh]

Hi Chris,

I think I can see where the headache is coming from. :-)

Am I correct in thinking you are refering to the entry in the man page for vgchange where is says (regarding the -s option) :

"However, for those logical volumes that have Mirror Write Cache turned off, mirror synchronization is done independently of whether the -s option appears on the command line."

I am not 100% sure of exactly what this means but I don't _think_ it is how you are reading it. Uisng the -s option should not resync the mirrors, even with NOMWC specified. I thinks its more to do with how lvm can track changes when the MWC is off.

However, you can get round all your problems by turning MWC on? You are seeing the long sync times as without MWC on all extents of all but one mirror are marked stale so in effect it is like mirroring the whole volume again. With MWC on it will only write those tracks that it knows are out of sync between the volumes. An analogy may be the difference between vxfs and hfs fsck procedures (and times!).

I'll have a play with my boxes at home later and see what I find.

cheers,

James.

[James Murtagh]

I'm glad I put those underscores around the "think". :-) Using NOMWC and -s a nomwcsyncd process gets forked for each volume group, which syncs the logical volumes in the background, even if they have been deactivated (lvchange -a n). There is actually reference to it in the vgchange manpage - see the -P option which controls the number of these nomwcsyncd's. With MWC enabled you have to sync the volumes manually after (at least outside SG?).

-- James.

[Chris Watson]

Jean,

Thanks for your comment. I took a look in /sbin/lvmrc and can't see how it will be invoked as a background test. Please clarify.

James,

Thanks for your assistance. I do have MWC on for all volumes, but as you say I got the impression from the man page that a -s would be ignored. However, after another read I interpret it diffently. I have now turned on the -s switch. However, a commented above by others, I will avoid stopping the package in future.

To you both and the rest of the team, a big thanks!