quorum not present after reboot (LVM has 2 fault disks)

[Frank Sauer]

Mirroring is useless, when both disks disappear.
We tried to reboot our system after the volumegroup with oracle went away (finally did the reboot with GSP RS).
Now the reboot (init 1) hangs, when trying to mount this volumegroup.
How to manage the reboot (probably single user mode..)?
There are some patches around (quorum not present..), should I first apply some patches to have a chance to bring the volumegroup up?

regards frank

[Dietmar Konermann]

Frank,

boot to single user mode, then comment out the entries of that VG from /etc/fstab. Continue bootup using "init 3".

Best regards...
Dietmar.

[Pete Randall]

You need to set the quorum flag in the AUTO file so that it will boot without quorum:

mkboot -a "hpux -lq" /dev/rdsk/c0t6d0

changing the address to match your primary boot disk.


Pete

[Pete Randall]

Or are you saying that 2 out of 3 disks are missing?

[Frank Sauer]

thanks for the quick replies.
Dieter, which editor is available on single user mode?

frank

[Dietmar Konermann]

Frank,

Well... cat (1) and echo(1) :-)
Since your vg00 has no problems I would mount /usr and /var to use vi(1) then.

Best regards...
Dietmar. (*not* Dieter)

[Frank Sauer]

I'm sorry Dietmar.
mount /dev/vg00/lvol8 /usr (also mount ..../var)doesn't work.
vxfs mount: /dev/vg00/lvol9 is corrupted. needs checking
Probably a result of shutdown by GSP RS.

frank

[A. Clay Stephenson]

You need to run fsck before mounting the filesystems.

Something like fsck -F vxfs -y /dev/vg00/rlvol8

You may need to add the -o full and/or nolog options as well.

Man fsck_vxfs for details.

[Frank Sauer]

fsck works. thanks.
removed the fstab entries.
But after init 3 the system still tries to mount the missing volumegroup (vgchange) and ends up with "quorum not present...".

frank

[A. Clay Stephenson]

That's because one of your problems has nothing to do with the other. You had corrupt filesystems in vg00; you have at least one failed or cuppupt disk ion your other VG. I suspect that you actually have two failed disks or cabling/controller/termination problems with this VG. I also suspect that syslog was screaming at you for a long time when the first disk failed and then when a second disk failed you were toast. There was no reason to reboot.

Try manually activating the VG (I'll call it /dev/vg01 here) and overriding quorum.

vgchange -a y -q n /dev/vg01

If that works then you should be able to mount your filesystems after fscking each one. After that, imediately start the normal disk replacement procedure for the failed disk.

[Dietmar Konermann]

Frank,

LVM's /sbin/lvmrc script automatically tries to activate all VGs in /etc/lvmtab by default. If you don't want that to happen, then you could set AUTO_VG_ACTIVATE=0 in /etc/lvmrc. You could also vgexport the VG in question... or -- even better -- fix the hardware problem.

Best regards...
Dietmar.

[Frank Sauer]

Clay, I know the difference between fsck on vg00 and my stale VG ...
You are right, there were serveral errors logged in syslog THIS weekend. But there was no chance to kill the running processes which used the "stale" VG, that's why reboot.
Strange, that a disk and its mirror went down (ioscan-->no_hw)same time (actually one disk --> no_hw).
Clay, please give a short hint what is behind -q n (ignore quorum requirement) option (vgchance).
Dietmar, actually I can't replace the hardware.

regards frank

[A. Clay Stephenson]

The -q n option of vgchange simply overrides the default quorum requirement of 50% of all physical volumes which comprise a given VG plus 1. With 2 PV's in a VG, you will never satify 50% + 1 with a failed disk and thus you must override quorum with -q n. If you are able to activate the VG, you should be home free. If only 1 disk is really failede, recovery should be trivially simple. Because this VG became "unmounted" while running there is a strong indication that you have more than two disks failing. If properly mirrored (and assuming the disks were on separate buses) virtually any failure of one mirror should not have affected the other. The machine should have kept right on running. However, if warning messages were ignored and both disks failed then you are toast and probably deservedly so.

[Frank Sauer]

Finally the vgchange -a y -q n command didn't work because it missed two physical disks.

Solution: shortly unplugged the two disks, they turned from "no_hw" to "claimed". VG was mounted successfully. We will move the logical volumes to other disks.

Thanks to all how tried to help me, it's allways a pleasure ...

regards Frank

[Mohanasundaram_1]

Hi Frank,

In case the quorum is lost in your root VG and you want to boot the system then you have to interact with IPL and at the ISL prompt give hpux -lq

e.g.
ISL>hpux -lq

This would have booted your system in multi-user mode without checking the quorum. The other suggestions given earlier can be now implemented once the system is in multi-user mode.

This should have taken care of your filesystem corrupt scenario too. You need to check your disks which gave problem. They may fail again.

Cheers,
Mohan.