topic Re: Broken primary mirror in Operating System - HP-UX

Broken primary mirror

Ian Killer_1 — Mon, 09 Sep 2002 12:48:38 GMT

Hi experts...
Another curly one.
Two disks in vg00. At some stage (before I was contracted) the secondary mirror went stale and wasn't maintained. THe system was rebooted last week and didn't boot with a LIF area problem. I attempted a lif recovery with the recovery cd (10.20) and failed with bad magic number. Tried this with a number of different super blocks without joy. Found the old mirror, changed the boot path and booted using the old mirror. So I tried pvcreate -B the old primary hoping to re-add it as the mirror.

Here's my status...
two disks... c0t5d0 and its original mirror c1t5d0
I've booted from c1t5d0
pvcreate -B c0t5d0
dd'd c0t5d0 without error.
vg00 looks bad with Cur PV = 2 and Act PV =1 All LV's are available stale and only c1t5d0 appears in PV's section..

vgdisplay -v /dev/vg00
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 10
Open LV 10
Max PV 16
Cur PV 2
Act PV 1
Max PE per PV 2000
VGDA 2
PE Size (Mbytes) 4
Total PE 1023
Alloc PE 714
Free PE 309
Total PVG 0

LV's are all available/stale

--- Physical volumes ---
PV Name /dev/dsk/c1t5d0
PV Status available
Total PE 1023
Free PE 309

strings /etc/lvmtab
/dev/vg00
/dev/dsk/c0t5d0
/dev/dsk/c1t5d0
/dev/vg01
/dev/dsk/c0t3d0
/dev/dsk/c0t4d0
/dev/dsk/c1t3d0
/dev/dsk/c1t4d0
/dev/dsk/c1t2d0
/dev/dsk/c0t2d0

lvdisplay -v /dev/vg00/lvol1 | more
lvdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t5d0":
The specified path does not correspond to physical volume attached to
this volume group
lvdisplay: Warning: couldn't query all of the physical volumes.
--- Logical volumes ---
LV Name /dev/vg00/lvol1
VG Name /dev/vg00
LV Permission read/write
LV Status available/stale
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 100
Current LE 25
Allocated PE 50
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict/contiguous
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c1t5d0 25 25

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
0000 ??? 0000 stale /dev/dsk/c1t5d0 0000 current
0001 ??? 0001 stale /dev/dsk/c1t5d0 0001 current
: :

The -k flag in lvreduce was designed specifically for this reason but fails also as follows...

lvreduce -m 0 -k /dev/vg00/lvol1 1
Physical extents on remaining physical volumes are stale or
Remaining physical volumes are not responding.
lvreduce: The LVM device driver failed to reduce mirrors on
the logical volume "/dev/vg00/lvol1".

Please help.

Ian

Re: Broken primary mirror

John Palmer — Mon, 09 Sep 2002 13:08:07 GMT

Try restoring the VG information to c0t5d0:-

vgcfgrestore /dev/vg00 /dev/rdsk/c0t5d0

You should then be able to sync the LV's.

Regards,
John

Re: Broken primary mirror

Ian Killer_1 — Mon, 09 Sep 2002 13:29:01 GMT

vgcfgrestore -n /dev/vg00 /dev/rdsk/c0t5d0
vgcfgrestore: Couldn't query physical volume "/dev/dsk/c0t5d0":
The specified path does not correspond to physical volume attached to
this volume group
Volume Group configuration has been restored to /dev/rdsk/c0t5d0

and vgdisplay -v still reports as above with 2cur pv's, 1 act pv and only c1t5d0 showing in the PV's section.

Re: Broken primary mirror

Ian Killer_1 — Mon, 09 Sep 2002 13:34:33 GMT

Also I feel it important to note that at one point I vgextend /dev/vg00 /dev/dsk/c0t5d0 and vgdisplay -v reported 3 current pv's and 2 active.

Is there a way I can clean the vg and the lv's so that they no longer think there is a mirror?

That way I would be able to recreate the mirror pv from scratch.

Ian

Re: Broken primary mirror

steven Burgess_2 — Mon, 09 Sep 2002 13:49:54 GMT

Hi Ian

I think you need to remove the disk from the volume group with a vgreduce -f

http://support4.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000063208748

I know your not replacing the disk but have a look at

http://support4.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062910934

HTH

Steve

Re: Broken primary mirror

S.K. Chan — Mon, 09 Sep 2002 13:52:09 GMT

The "lvreduce" with the key value usually will do the trick. Are you using the correct key value ? Did you ..
# lvdisplay -v -k /dev/vg00/lvol1 | more
.. so that the "PV1" and "PV2" column display the key value instead of the "???" or "disk-path". So far the 1st PV (ie PV1) usually has key value of 0 (zero) meaning you should ..
# lvreduce -m 0 -k /dev/vg00/lvol1 0
instead of using value "1". Again run the "lvdisplay" above to confirm.

Re: Broken primary mirror

James R. Ferguson — Mon, 09 Sep 2002 13:53:12 GMT

Hi Ian:

Identify the missing extents looking for "stale" extents by logical volume:

# vgdisplay -v vgXY | grep stale

Now look for the "disk key" in the PV column of every logical volume involved, in this output:

# lvdisplay -v -k /dev/vgXX/lvolX

Now, remove that mirror:

# lvreduce -m 0 -k /dev/vgXX/lvolX [diskkey]

Finally, recreate your 'etc/lvmtab':

# mv /etc/lvmtab /etc/lvmtab.old
# vgscan -av
# vgchange -a y vgXX
# lvlnboot -R

Then, remove the missing physical device:

# vgreduce -f vgXX

Regards!

...JRF...

Re: Broken primary mirror

Sridhar Bhaskarla — Mon, 09 Sep 2002 13:57:10 GMT

Hi Ian,

You should not be doing a vgextend of c0t5d0. All you would do was to "vgcfgrestore c0t5d0;vgchange -a y vg00" and a vgsync to restore your mirrors.

Try a "vgreduce -f vg00" and see if there is any luck.

Post your latest status.

-Sri

Re: Broken primary mirror

Ian Killer_1 — Mon, 09 Sep 2002 13:59:15 GMT

Yep I was using the wrong key value. It looks like it might work for me. I'll get back to you with results and points shortly (30 min).

Ian

Re: Broken primary mirror

Ian Killer_1 — Mon, 09 Sep 2002 14:14:54 GMT

OK.. the lvreduce worked a treat reducing all lv's to a single copy.
Unfortunately...
vgreduce -f /dev/vg00
vgreduce: Couldn't query physical volume "/dev/dsk/c0t5d0":
The specified path does not correspond to physical volume attached to
this volume group
I removed lvmtab and recreated (vgscan -v), and it still contains c0t5d0.

Re: Broken primary mirror

Ian Killer_1 — Mon, 09 Sep 2002 14:19:02 GMT

Oh.. vgchange seems to have cleared up this issue. More soon...

Ian

Re: Broken primary mirror

S.K. Chan — Mon, 09 Sep 2002 14:29:56 GMT

If everything else failed (vgreduce -f and vgscan failed), the last resort is to recreate vg00. First take note of /dev/vg00/group minor number. Then you would have to boot in LVM maintenance mode (hpux -lm), then do ..
# vgexport -m /mapfile /dev/vg00
# mkdir /dev/vg00
# mknod /dev/vg00/group c 64 0x0#0000
=> Use the same minor number that you have noted down earlier.
# vgimport -m /mapfile /dev/vg00 /dev/dsk/c1t5d0
# vgchange -a y /dev/vg00
Now check vg00 to make sure the offending PV is gone for good.

Re: Broken primary mirror

Ian Killer_1 — Tue, 10 Sep 2002 07:26:10 GMT

Yep it was the vgchange that corrected the cur/act PV discrepancey in vgdisplay. Thankyou all for your help. It's fabulous to know I can depend this forum, when my situation gets ugly, and all in the space of an hour or so.
Thanks..

ian