vgreduce crashed disk

[Ruan_3]

Hi all

Recently we powered down a K570 server running HP-UX 11i in order to add some new hardware. When the server rebooted, the primary boot disk failed, but it was still able to boot from the mirror on the alternate boot disk. The faulty disk was replaced and now I need to mirror the alternate boot disk onto this new primary boot disk.

The problem is that I cannot seem to remove /dev/dsk/c0t6d0 from vg00. I will list the output of the commands which I have used in an attempt to remove it


sudo vgreduce -f vg00 | more

vgreduce: Couldn't query physical volume "/dev/dsk/c0t6d0":
The specified path does not correspond to physical volume attached to
this volume group
Not all extents are free. i.e. Out of 4340 PEs, only 271 are free.
You must free all PEs using lvreduce/lvremove before the PV can be removed.
Example: lvreduce -A n -m 0 /dev/vg01/lvol1.
lvremove -A n /dev/vg01/lvol1
Here's the map of used PEs.......

sudo lvreduce -m 0 /dev/vg00/lvol1 /dev/dsk/c0t6d0

lvreduce: Warning: couldn't query physical volume "/dev/dsk/c0t6d0":
The specified path does not correspond to physical volume attached to
this volume group
lvreduce: Warning: couldn't query all of the physical volumes.
lvreduce: Physical volume "/dev/dsk/c0t6d0" does not belong
to volume group "/dev/vg00".
Logical volume "/dev/vg00/lvol1" is not reduced.


pvdisplay /dev/dsk/c0t6d0

pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t6d0":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
pvdisplay: Couldn't retrieve the names of the physical volumes
belonging to volume group "/dev/vg00".
pvdisplay: Cannot display physical volume "/dev/dsk/c0t6d0".

I also moved /etc/lvmtab to /etc/lvmtab.old
and ran vgscan:

sudo vgscan -a

Creating "/etc/lvmtab".
Couldn't stat physical volume "/dev/dsk/c1t2d0":
Invalid argument

Following Physical Volumes belong to one Volume Group.
Unable to match these Physical Volumes to a Volume Group.
Use the vgimport command to complete the process.
/dev/dsk/c0t1d0
/dev/dsk/c0t4d0

But strings /etc/lvmtab still yields c0t6d0:

sudo strings /etc/lvmtab
/dev/vg00
/dev/dsk/c0t5d0
/dev/dsk/c0t6d0

sudo diskinfo /dev/rdsk/c0t6d0

SCSI describe of /dev/rdsk/c0t6d0:
vendor: SEAGATE
product id: ST318275LC
type: direct access
size: 17783240 Kbytes
bytes per sector: 512

Should I unplug the SCSI cable from the new disk and run ioscan and vgscan? Must one power the box down when unplugging HDD's?

[melvyn burnard]

Why are you trying to vgreduce the disk out of the vg?
all you need to do is do a vgcfgrestore to hte new disk, this will write the correct lvm info back to hte disk, an dthen you just do a vgsync which will mirror up the lv's
man vgcfgrestore

[Massimo Bianchi]

You must use the lvreduce with the "-k" option, because you have some missing phisilca disks.

Once this is done, go with the vgreduce -f and you are done.

Massimo

[Robert-Jan Goossens]

Hi Ruan,

Why do you wan't to remove /dev/dsk/c0t6d0, you just have to restore the vg information to the disk (man vgcfgrestore) and resync the disk.

Hope it helps,

Robert-Jan.

[Ruan_3]

I have already tried vgcfgrestore and vgsync:

vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t6d0

vgcfgrestore: Couldn't query physical volume "/dev/dsk/c0t6d0":
The specified path does not correspond to physical volume attached to
this volume group
Volume Group configuration has been restored to /dev/rdsk/c0t6d0

vgsync vg00

vgsync: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgsync: Couldn't resynchronize logical volume "/dev/vg00/lvol1".
vgsync: Couldn't resynchronize volume group "vg00".

lvreduce -k also does not seem to do the job

lvreduce -k -m 0 /dev/vg00/lvol1

Physical extents on remaining physical volumes are stale or
Remaining physical volumes are not responding.
lvreduce: The LVM device driver failed to reduce mirrors on
the logical volume "/dev/vg00/lvol1"

[Geoff Wild]

I had the same issue about 6 months ago - had to put the "bad" disk back in, then I was able to lvreduce, vgreduce it out.

You may also have to boot into lvm maintenance mode to perform the tasks...

ISL> hpux -lm


Rgds...Geoff

[Geoff Wild]

Actually, you need to boot with:

ISL>hpux -lq

hpux(1M) hpux(1M)

-lq Boot the system with quorum override option. This option is used in a scenario where a disk is removed from the system or is otherwise unavailable, but the corresponding entry for the physical volume has not yet been removed from the volume group using vgreduce.

Rgds...Geoff

[Helen French]

You need to boot in LVM maintenance mode (hpux -lm) and then recreate LVM defenitions with 'vgcfgrestore'. You need to reactivate the vg and have to use mkboot and lvlnboot commands. Read this document which lists all steps (TKB #LVMKBRC00006116):

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000068955762

[Massimo Bianchi]

Specify the missing disk:


lvreduce -k -m 0 /dev/vg00/lvol1 /dev/dsk/BAD_DISK

otherwise it will try to reduce the PE from the existing mirror.

Massimo

[Vincent DANTEN]

On my opinion the right way is:

DO not TRY to remove your disk from vg00 !

(all on line)

vgcfgrestore -n .... : OK !
As it is a boot disk : mkboot /dev/rdsk/c0t6d0
recover boot data : lvlnboot -R
integrate your new disk in vg00 :
vgchange -a y vg00
restynchronize the vg : vgsync vg00

many links deal with this issue in ITRC

hope this help

Vince

[Artyom Voronchihin]

Replacing a Boot Disk with Mirror step by step (from LVM student workbook):
1. Load system in single user mode
2. mount filesystems
mount -a
3. Restore LVM configuration structures
vgcfgrestore -n vg00 /dev/drsk/cItTd0
4. Rebuild the boot area
mkboot /dev/rdsk/cItTd0
mkboot -a "hpux -lq" /dev/rdsk/cItTd0
5.Reactivate the volume group an check your vg and lv status
vgchange -a y vg00
vgdisplay -v vg00
lvdisplay -v /dev/vg00/lvol1
lvdisplay -v /dev/vg00/lvol2
lvdisplay -v /dev/vg00/lvol3
6. Resync physical extents
vgsync vg00
7. Verify that both mirrors are current again
vgdisplay -v vg00
lvdisplay -v /dev/vg00/lvol1
lvdisplay -v /dev/vg00/lvol2
lvdisplay -v /dev/vg00/lvol3
8. Reboot the system ensure the recovery was sucessful
shutdown -ry 0

[Ruan_3]

vgcfgrestore and vgscan was not doing the trick. I'm not sure, but I think it is because /etc/lvmconf/vg00.conf still contained the replaced disk.

I used "vgcfgrestore -n vg00 -l" o view the contents of this file. So I RTFM'd and found the -l option for vgreduce.

Extract from man:

-l pv_path This option removes the specified pv_path/s from
the lvmtab file. This task will only be performed
if the pv_path is currently marked as missing from
the volume group.

Did the vgreduce and then used my mirroring cook book

# vgreduce -l /dev/vg00 /dev/dsk/c0t6d0
# pvcreate -f -B /dev/rdsk/c0t6d0
# vgextend -g PVG2 vg00 /dev/dsk/c0t6d0
# rmboot /dev/rdsk/c0t6d0
# mkboot -l /dev/rdsk/c0t6d0
# lifls ???l /dev/rdsk/c0t6d0
# mkboot -a ???hpux ???lq??? /dev/rdsk/c0t6d0
# mkboot -a ???hpux ???lq??? /dev/rdsk/c0t5d0
# lifcp /dev/rdsk/c0t6d0:AUTO -
# mkboot -b /usr/sbin/diag/lif/updatediaglif2 -p ISL -p AUTO -p HPUX -p LABEL -p PAD /dev/rdsk/c0t5d0
# lvlnboot -b /dev/vg00/lvol1
# lvlnboot -r /dev/vg00/lvol3
# lvlnboot -s /dev/vg00/lvol2
# lvlnboot -d /dev/vg00/lvol2
# setboot ???a 10/0.6.0
# for N in 1 2 3 4 6 6 7 8
> do
> lvextend -m /dev/vg00/lvol$N /dev/dsk/c0t6d0
> done

But then I got

vgcfgbackup: Invalid LVMREC on Physical Volume /dev/rdsk/c0t5d0

while vgextend'ing

So I had to run vgcfgrestore on /dev/rdsk/c0t5d0

#vgcfgrestore -n vg00 /dev/rdsk/c0t5d0

Thanks a lot for all your help guys!!

[Todd McDaniel_1]

NO POINTS for me... just an observation...

Okay, you say primary boot path failed and system booted to alternate and came up fine...Logic seems to say that you can do an online replacement and mkboot and such vgcfgrestore/vgsync etc... commands to get it back in...

but for some reason it failed to recog the disk when you replacd it at single user mode...


_________________________
This is curious to me why the replacement disk with vgcfgrestore failed...

It is almost as if the OS refused to recognize the disk... You usually only see this with a disk that has been removed and it wasnt reduced beforehand...

Im curious if the disk was ever recognized by ioscan? Did you run an ioscan?

I agree that you should have reinstalled the bad disk and tried to reduce it out...

also weird is that the trick I have used occasionally is to lvremove -0 the lvols from that disk or lvremove -m 0...

_______________________________________
*light bulb goes off*
Actually i would have tried using vgscan... wiht the new disk in place or maybe with NO disk in there

(IE...

mv /etc/lvmtab
vgscan

This should restore the lvmtab and then you can vgcfgrestore and read the disk to the vg00 and reestablish the mirror...

[Glen Trevino]

My situation and the advice I got from a HP CE that was on site:

I've seen something similar to this twice in one week on two totally different systems that we lost mirrored disks on.

HP-UX 11.00 on a K450 with the bad disk in a Jamaica.

HP-UX 11i on an L3000 with the bad disk in a DS2300.

In both cases the system did not recognize that the bad drive was removed when we went to replace it. You could run ioscan, pvdisplay, and lvdisplay on the disk and it still thought the disk was there. The only way I could think to make it recognize that the the disk was removed (with out jepardizing data) was to put the bad disk back in the server, shutdown, remove the disk and boot up. I didn't like this at all because I wasn't planning on having to boot the system, why should I've had to, the disks are hot swapable...

I then got all the error reponses that I expected to see before (entire disk stale/unavailable, etc). I then continued and replaced the disk as you normally would.

We then had an HP CE in to replace a bad SCSI controller on another system and I asked him about it. He told me that he ALWAYS does a vgreduce of the disk, regardless of it's role as a boot or not before removing the bad drive. He says that he's seen LVM not recognize removed disks when pulled and even LVM just freezing up a system when pulling a hot swappable drive and has never had a problem at all if he vgreduced the disk out of the vg first. Any comments on this?