Stale PE on good? disk.

[imrichar]

I have a system where the mirror disk has gone bad. All PEs on the good disk are "current", with the exception of 1 PE on my /stand lvol.

(Sample from lvdisplay -v /dev/vg00/lvol1)

00007 /dev/dsk/c1t2d0 00007 current /dev/dsk/c2t2d0 00007 current
00008 /dev/dsk/c1t2d0 00008 current /dev/dsk/c2t2d0 00008 stale
00009 /dev/dsk/c1t2d0 00009 stale /dev/dsk/c2t2d0 00009 current
00010 /dev/dsk/c1t2d0 00010 current /dev/dsk/c2t2d0 00010 current
00011 /dev/dsk/c1t2d0 00011 current /dev/dsk/c2t2d0 00011 current

Any attempts at touching /stand (ll, etc.) result in a hung shell so I'm concerned that there is either an issue with the primary physical disk, or some level of corruption has occurred. System load is high (60), processes have stacked up over time that can no longer by terminated via kill -9, etc.

I've seen a couple instances where /stand was able to be re-created while the system was up. Anyone have thoughts on how to proceed, additional tests I could perform, etc.? My concern is that any kernel tweaks, etc. that were made would be lost if rebuilt. I have a sister host with the same/similar configuration I can get an ignite off of for a full restore of the OS drives, but I hope it doesn't come to that.

Thanks everyone!

[Mark McDonald_2 · Accepted solution]

Hi

Have you tried vgsync?

There is at least 1 good copy of the data there, I would tar up stand in to another dir before playing around too much.

Mark

[imrichar]

Hi Mark,

I've used vgsync before, but that would require the mirror drive be alive? Despite lvdisplay showing the mirror drive PEs as "current", that drive is toast, NO_HW in ioscan.

And I would love to copy out of /stand, but its like a black hole, any attempt to touch it results in a hung shell.

Additional info:

This is a 9000 box running HP-UX 11.0

[Patrick Wallek]

Both drives have problems!!!!!

This is a VERY BAD situation to be in. Hopefully you have an Ignite tape that you can rebuild from because I have a strange feeling it will come to that!

Processes are hanging, and unable to be killed via 'kill -9', likely because they are waiting on an I/O from the "dead" disk and will likely NEVER get it, so they can never die.

I do not foresee a good end to this......

[Mark McDonald_2]

Sorry, I missed that you had 1 definitely bad disk.

When I saw this:
00007 /dev/dsk/c1t2d0 00007 current /dev/dsk/c2t2d0 00007 current
00008 /dev/dsk/c1t2d0 00008 current /dev/dsk/c2t2d0 00008 stale
00009 /dev/dsk/c1t2d0 00009 stale /dev/dsk/c2t2d0 00009 current
00010 /dev/dsk/c1t2d0 00010 current /dev/dsk/c2t2d0 00010 current
00011 /dev/dsk/c1t2d0 00011 current /dev/dsk/c2t2d0 00011 current

I thought you had 1 stale extent on each disk.

Yes if you have 2 bad disks this is very bad. it is unlikely that you can replace the mirror and sync up, although may be worth a try?

[Patrick Wallek]

If there are stale extents on both disks, and one is showing NO_HW in ioscan output, then you are more than likely well and truly **hosed**.

I recently had both disks in a mirrored pair, housing Oracle data files, have stale extents and vgsync would not work to resync them. I would up copying the data I could from the mirror to another mount point and got all but 1 Oracle data file. From there I wound recreating the VG and LV for that pair of disks, recreated the mount point, copied the data from its temp location and then restored the 1 missing datafile from the previous Oracle backup. All in all, no data was lost.

The lesson here --- Be prepared to recover your system. Unfortunately for you this is on /stand and you may have to rebuild from your Ignite image (hopefully you have one!).

I cannot think of any way that this can end any other way at the moment.

[imrichar]

Yeah, you're all confirming my worst case scenario. :) Luckily I can produce an ignite from the sister box, seems to be the only way to go. Hopefully I can get most of the data outside of /stand copied over somewhere.

[Ganesan R]

Hi,

Have you tried taking ignite backup of the same box?

you can even try with dd to read the stand lv.