Question on LVM vgchange quorum issues

[Brian M Rawlings]

I recently migrated data from an old Nike Model 20 array onto an Autoraid 12H. I used mirrordisk to do it, extending the VGs to include the new LUNs, extending the LVs to have a mirror on the new LUNs, and then using lvreduce to eliminate the mirror extents which were on the old LUNs, with a final step of vgreduce to eliminate the old LUNs from the volume groups entirely. This all went well, no major hitches and only a couple of late-night brain fades that caused no real grief.

Once the old LUNs were all completely eliminated from LVM, as seen with vgdisplay -v for each VG, and after using 'strings' to check the /etc/lvmtab file to make sure no remnant of the old devices remained anywhere, I wanted to shutdown and reboot, to prove that we would come up only on the new array with no problems. It ended up being good that we did that last, final, bedrock test.

Upon rebooting, the volume groups all refused to activate, with the message "Quorum not present, or some physical volume(s) are missing". I ended up manually activating them without quorum checking, using 'vgchange -a y -q n /dev/ora_vg' (etc), and each activated without any further fuss.

Mystified, I looked again through vgdisplay, lvmtab, lvmconf, and any other place I could think of that might have given the VG a reason to think that they should have the old LUNs, but all references to the old LUNs was gone (unless one of you can think of a sneaky place they might be referenced out of).

I deactivated each VG (vgchange -a n /dev/ora_vg...), and then attempted to activate one again, with the standard command (vgchange -a y /dev...) This time, it worked without error or complaint. I then tried our standard script, which activates all with "-a y" and mounts our file systems... no problems. The VGs all appear to work normally, no further complaint about quorum not beeing met. Life is good, right? Except for that little nagging feeling that, in our business, what you don't know can eat you for lunch. And I don't know why the VGs should have one itteration which required a "no quorum" activation, never to be needed again.

So, other than amusing you all with my magnificent tale of daring-do, I am trying to determine if this was just a last gasp of spite from the old array, that was eradicated by the no-quorum activation, never to return, or does activating a VG with quorum-checking turned off set a flag that quits checking for quorums during subsequent activations? Bottom line, should I be worried about this, is there anything I should look into, or can I sit back, put my feet up, and bask in the joy of a job well done?

I've looked through the 'vgchange' man page, and it does not state one way or the other about just what the "no quorum" option does, or if it is persistent, or is only in place for that itteration of the command. Generally, I'd say that if the man page doesn't mention something, go with the simple explanation, but I've been burned before by such assumptions.

If any of you has seen this, or has any suggestions to contribute about where else I might look for the old LUN devices, I'd be grateful.

Please, no shoving, there are points enough for ALL serious responses...

Thanx --bmr

[Ashwani Kashyap]

Hmm . Very interesting . I can't think of anything else .
Did you have PVG's earlier . If yes then were the PVG's in /etc/lvmpvg file with the old LUN's in there .

If yes they might be the reason .

I can't think of anything else at this point .

[Ashwani Kashyap]

You can also trying rebuilding your /etc/lvmtab file by doing vgscan after saving the original copy first . But I guess you might have tried this already .

[Animesh Chakraborty]

Hi,
Have you tried rebooting the server second time ??
Is is the same at every reboot time?

[Animesh Chakraborty]

Hi,
I also suggest you to clean the old device files of model 20 from /dev/dsk directory.
Following link will guide you how to delete the unwanted device files from your systems.
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x8abdc8ecad09d6118ff40090279cd0f9,00.html

[Brian M Rawlings]

Thanks for your thoughts on this. I didn't set the initial volumes up, so PVGs might exist. I'll check on that. It is probably worthwhile to recreate the lvmtab file as well, since who knows what is hidden in a binary file?

Lastly, I was only able to reboot once, so I can't be positive about whether this will reoccur at next reboot... but I'll be sure to keep it in mind, when we next find time to drop production. I don't want to remove the special files for the old gear, since the developers will be using it as a sandbox play area, but if the problem returns and persists, I may do that and recreate them.

I guess if this never shows up again, it will just be one of those things we all love about this job... the kind that didn't make it into a twilight zone episode, but were contenders.

Thanks again. --bmr

[Jeff Schussele]

Hi Brian,

Not exactly sure what happened, but I suspect it had to do with the fact that there was still VG info on the old PVs. These are VGDAs (Volume Group Descriptor Areas) & they contain the vg_id data that vgexport/vgimport uses with the -s option.

I think I would have done one extra step AFTER I knew the data was safely over on the new array & BEFORE I rebooted.

I would have

pvcreate -f /dev/rdsk/cxtydz

on the OLD PVs to insure no remnant of the old VG descriptor data remained. I *think* LVM will perform some scans at boot to determine just what VGs all PVs belong. This may have caused it to complain about quorum - it may have *thought* they still belonged to that VG.

I believe if you delete/rename the lvmtab file & run a vgscan -v right now w/o running pvcreate -f on the old PVs they'll be put BACK in the VG as the VG descriptor data will still reside on them.

Rgds,
Jeff

[Brian M Rawlings]

Good thought, Jeff. I thought that removing the old PVs via 'vgreduce' would eliminate the VGRA data from them, but that is rather undefined, I suppose. I can easily do that, and will, at my earliest opportunity. I've had old LVM disks mess with me before, so I should have thought of this one.

Thanks to you all for your consideration of this matter. Unless some further jem of wisdom is forthcoming in the next 24 hours, I'm going to close this thread out. Thanx Again!! --bmr