Quorum not Present, or Physical volume missing

[Dave Robinet]

CHAOS.

The machine in question is a C3600, with an external tape drive and external 73gb hard disk on the SCSI 68pin chain.

Today, I removed the tape drive, and the machine wouldn't detect the presence of the HD anymore. I worked on it, and ended up recreating the kernel via steps elsewhere on this forum (mismatching symbol table) and all was well.

NOW, the problem is this:

The external drive contained a volume group with a single, really large logical volume on it. I can't, for the life of me, get that volume group back online:

Quorum not present, or some physical volume(s) are missing.

So, I do a vgchange -a y -q n vg02, and I get the same thing.

In desperation, I've tried several things. Mainly:

After exporting the vg:

1) mv /etc/lvmtab /etc/lvmtab.old;vgscan -v

2) mkdir /dev/vg02

3) mknod /dev/vg02/group c 64 0x050000

4) vgimport /dev/vg02 /dev/dsk/c3t3d0
(Warning: A backup of this VG may not exist on this machine, please remember to take a backup using vgcfgbackup)

5) vgchange -a y vg02

(Couldn't activate volume group 'vg02': Quorum not present, or some physical volume(s) are missing).

!!!

There's only 1 disk in the VG.

Another hint, possibly: vgcfgbackup vg02 gives:

vgcfgbackup: Warning: couldn't query physical volume "/dev/dsk/c3t3d0":
The specified path does not correspond to physical volume attached to
this volume group
vgcfgbackup: Couldn't query the list of physical volumes.
vgcfgbackup: Volume group not activated.

I'd *LOVE* some help on this one. It looks like it thinks the disk is a different disk or something. The disk itself seems to have the LVOL on it, but I can't bring the volume group online.

Thanks a bunch - you guys are my heroes. ;)

[Dave Robinet]

Oh yes:

ioscan -funC disk

disk 3 10/0/15/1.3.0 sdisk CLAIMED DEVICE SEAGATE ST373405LW
/dev/dsk/c3t3d0 /dev/rdsk/c3t3d0

...so it can speak to the device properly, far as I can tell.

[Patrick Wallek]

Does the disk show up in an ioscan?

# ioscan -fnC disk

What about a pvdisplay?

# pvdisplay -v /dev/dsk/c?t?d0

Is the SCSI chain terminiated correctly? Is the drive powered up? Has anything else changed other than removing the tape drive?

[A. Clay Stephenson]

First, did you accidently change the SCSI ID of the drive? Second, is this bus now terminated in exactly two places - on both ends of the bus? Third, have you done an ioscan to see if the disk is claimed? Fourth, is the device node /dev/dsk/c3t3d0 present?

[Dave Robinet]

pvdisplay -v /dev/dsk/c3t3d0 returns:

pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t3d0":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Couldn't query the list of physical volumes.
pvdisplay: Couldn't retrieve the names of the physical volumes
belonging to volume group "/dev/vg02".
pvdisplay: Cannot display physical volume "/dev/dsk/c3t3d0".

..Yes, I forgot to mention: I *DID* change the SCSI ID of the drive while trying to get it to detect, but it is now (I believe) changed back to where it originally was.

Thanks for the replies - appreciate the help!

[A. Clay Stephenson]

On most drives and hosts, changing the SCSI ID has no effect after the drive is powered up. I would power off your UNIX box and the drive. Set the SCSI ID back to the original value and see if you are back to normal.

[Dave Robinet]

I did do that, a few times. The drive is set to SCSI 3, and the system detects it as SCSI 3.

Again, it's talking to the drive - I'm able to see it as a "ready to import" drive in SAM - but when I try and import it, it gives me the error that vgchange failed because of the quorum/missing disk thing.

I guess what I can't figure out is where it decides to "remember" that it's looking for a particular disk?!

Dave

[MANOJ SRIVASTAVA]

what is o/p of diskino /dev/dsk/c3t3d0 , I have a wrong feeling that the disk has conked off.


Manoj Srivastava

[Dave Robinet]

diskinfo /dev/rdsk/c3t3d0:

SCSI describe of /dev/rdsk/c3t3d0:
vendor: SEAGATE
product id: ST373405LW
type: direct access
size: 71687370 Kbytes
bytes per sector: 512

...that looks right to me...?

[A. Clay Stephenson]

This could all be explained if the bus is no longer properly terminated. In that mode, the drive will almost work well - about as bad a state as you can be in. If you know the bus is properly terminated then I would do this to make certain that you can read the drive.

dd if=/dev/rdsk/c3t3d0 bs=64k of=/dev/null

If that reads without errors then the drive should be physically ok. That does not mean that the LVM data structures haven't been clobbered.

Since you have done a vgexport, do a strings /etc/lvmtab and see if it looks ok. None of your vg02 data should be displayed. The other thing to make certain is that the minor device of your group file (0x050000) is unique and not in use by some other VG. By the way, 0x020000 would be more standard for vg02 but that is up to you.

[Dave Robinet]

The dd was running, but slowly and sporadically (2 seconds on, 2 off - and rhyhmically).

So, I cancelled it and went into my syslog. Whoa:

Aug 1 17:38:09 dino vmunix: SCSI: Resetting SCSI -- lbolt: 370301, bus: 3
Aug 1 17:38:09 dino vmunix: SCSI: Reset detected -- lbolt: 370301, bus: 3
Aug 1 17:38:09 dino vmunix: lv_readvgdats: Could not read VGDA 1 header & trailer f
rom disk H/W path 10/0/15/1.6.0 (error = 5)

...and others. Now, it's talking about H/W path 1.6.0, which is the internal (OS) disk - but above, it says lbolt: 370301, bus: 3...

Are you still thinking termination, here?

I'm going to unplug everything and plug it back in and check. Visually, it looks fine (and the light on the terminator IS on)... but let's have a look. :)

Thanks for the advice so far!

[Dave Robinet]

Still no go.

I actually decided to move the drive to another machine (identical config, except external devices), and am getting the EXACT same error using the SAM import.

I purposely put the drive in the middle of the chain to ensure that there are no SCSI bus issues. I also had to re-address the drive to 4, as 3 was already in use on the chain. (now c3t4d0).

dd command runs successfully, as far as I can tell.. or, I should say, is runNING successfully as far as I can tell (it's still rollin' along). It's got around 60 gigs of data to crunch through - how long should that take?

I'd love some other suggestions. It's entirely possible that I messed something up on my own (in terms of something on the disk) when I was trying to bring it back online - is there a way to check and see what a VG (that's sitting on a disk) expects in terms of PVs?

Dave

[Dave Robinet]

Update:

dd finished successfully - 1120115+1 records in, 1120115+1 records out.

Anyone have any thoughts?

[Dave Robinet]

*Sigh* - I always get the tough ones. ;)

Anyhow, since DD runs.. is there a way to get the data from the drive? (I'm thinking of a copy of the LV structure, or just the data contained within the LV, or something, here) I have a tar backup of the drive, but a) it's LARGE, and b) the user did it. I'd only like to lean on that if I *HAVE* to. Since the drive looks okay, and it looks like something in the VG header has gotten hosed instead (are there no tools to look at what the VG *thinks* it should be having in terms of PVs?), I'm going on the assumption that the drive will be usable if I simply make a new VG (but that will obviously render the data on the drive unusable).

Thanks again, guys. -Dave

[A. Clay Stephenson]

Hi Dave:

My best guess is that you have somehow (maybe by having duplicate SCSI ID's on the same bus briefly) overwritten a portion of either the PVRA (Physical Volume Reserved Area) or the VGRA (Volume Group Reserved Area) of this disk. If either of these has been clobbered, you are almost certainly not going to be able to recover.

It's probably time to pull out the backup but certainly read the backup's table of contents without restoring before you do a new pvcreate/vgcreate.

[Dave Robinet]

Unfortunately, if it's "almost certainly", it doesn't make a difference if the tape is good or bad, no? If I have no accessible data on the drive and there are no ways to try and get it back, the only data I have is the backup tape the user has made.

I guess what I'm asking is if there's no way you can think of to attempt to get this back online, or should I just go ahead? I'll entertain *anything* at this point.

What I've been sort of surprised by is that on HP-UX (apparently), there's no way of listing the contents of that area in a readable (or otherwise?) manner.

[Sajid_1]

Hi,

what I would do is shutdown the system, check all SCSI IDs physically, make sure nothing conflicts, check all termination, boot the system, interrupt it at starting, give "search" command at Boot menu, make sure all devices are detected properly and boot the system in single user mode (hpux -is)

Once booted, mv /etc/lvmtab to lvmtab.old and do a:
# vgscan -a
Check the output and find out which HDDs are on same VG. Check output of the:
#strings /etc/lvmtab
If the VG is still missing, try doing a vgimport manually:
Create a new VG name, mknod group, vgimport and vgchange

If you do very carefully and if you still have the data on the disk, you will defenitely able to configure it !

rgds

[A. Clay Stephenson]

If you haven't clobbered the vgcfgbackup files (and you did try to run vgcfgbackup so that's not good), you might be able to do a vgcfgrestore.

Try this to see what the backup files look like:
ls -l /etc/lvmconf/vg02*
This will list any possible backup files assuming you did not specify a backup file when you ran vgcfgbackup or any of the standard LVM commands that automatically backup the LVM data structures.



vgcfgrestore -f /etc/lvmconf/vg02.conf -l

Try that with any backup files that you find.

[Dave Robinet]

Oooh! Oooh! (Excited "2 year old kid at Disneyland" dance) Okay, there's a backup file for the original volume group name (teamext) that's 1181696 bytes in length. The modified date is July 17/02 - the date when the VG was created.

When I was trying to import the drive after problems began happening, I was using other volume group names - just in case there was a backup out there somewhere under the original name. ;)

I need to move the drive back over to the original machine (again, I moved it to remove any potential SCSI bus errors on there) and try that command. So, I'll be putting the drive back on there, and issuing the

vgcfgrestore -f /etc/lvmconf/teamext.conf -l

...command unless I hear otherwise in the next 30 mins.. I'll be doing it while the user takes lunch (the joys of sysadmin duty), and will report back here!

Thanks so much for the suggestions, regardless as to how it works out. What a great forum! -ave

[Thomas Schler_1]

A lot of things happened, since you detected malfunctions in getting back your vg....

But, here are my thoughts:

. If you're running HP-UX 10.20 be sure having PHKL_25439 installed and not one of its predecessors.

or
. Try to go back to the original state when all was running fine. I.e. put back your tape drive and use the old kernel to boot from. Use vmunix.prev or another kernel from a backup (if you have one).

or
. As you told you tried to import the vg on another machine without success, the same error message. Install PHKL_25439 on that machine and try it again.

or
. Use the tar backup (maybe, you should be lucky that at least the user made a backup instead of you, thanks to the user!!)

Unfortunately, no software is really secure. So, if your problem rised due to the missing PHKL_25439, any HP program reading VGRA would give you wrong or out of date information. If not, try to contact HP support. Maybe, they have a solution for you. If not and if your data on the disk is really of lot of worth, try to contact companies specialised in restoring data from nearly destroyed disks. But this would cost some money.

[Sajid_1]

hello,

the '-l' option with vgcfgrestore will only list the content of the config file. If you want to do the actual task, then apply it without -l and with VGname and PV-path:
# vgcfgrestore -f filename pv_path

[Dave Robinet]

Ok, status update:

I tried the vgcfgrestore command, and it did restore the configuration (so it says), but the thing still won't import.

Same message - Quorum not present, or physical volumes missing.

Whatever the case, my thinking is that the volume group config (backed up July 17, when the VG was created) should be current on the drive. The vgcfgrestore -l gives:
--
Volume Group Configuration information in "/etc/lvmconf/teamext.conf"
VG Name /dev/teamext
---- Physical volumes : 1 ----
/dev/rdsk/c3t3d0 (Non-bootable)
--

...and that's what restored to the disk. The deviceid IS c3t3d0 right now, it lists only 1 physical volume in the backup file... therefore, I can't see why it's looking for an "additional" physical volume. How is that possible?!

I'm running HP-UX 11 (sorry I forgot to mention that).

I believe I tried the unix kernel trick as well - though I can't say for sure. I'll give that a try later on.

[Sajid_1]

hello,

can you try this:
# vgexport /dev/teamext
# mkdir /dev/vg05 (if vg05 is not already there)
# mknod group c 64 0x050000
# vgimport /dev/vg05 /dev/dsk/c3t3d0
# vgdisplay -v /dev/vg05
# vgchange -a y /dev/vg05

gl,

[Dave Robinet]

The vgdisplay -v doesn't work - "Volume group not activated, cannot display volume group".

Then, the vgchange -a y godvg fails as before.

[Thomas Schler_1 · Accepted solution]

Dave,

first, your style in using this forum is very good. I hope you will have success, here.

Have a look on patch description of PHKL_27178 (HP-UX 11.00), search for "quorum". There appears the same error message you are getting (description of PHKL_25775). Check the text, compare with your situation, and install PHKL_27178, if necessary.