Invalid LVMREC on replaced disk

[Matt Harrell]

We had a disk fail in vg02 on an HP 9000/785 workstation running HP-UX 11.00 last week. The new disk (18 GB) arrived Friday, and I *thought* I added it in to the VG correctly, following the HP "Procedure for replacing an LVM disk in HP-UX 10.x and 11.x" document (Chapter 2, "Replacing a NON-Boot disk WITH Mirroring). The disk had both mirrored and non-mirrored extents on it. The two filesystems that were not mirrored what had extents on it were /opt2 (no longer needed and has been removed) and /depot (was going to be recreated and restored from tape backup).

The steps all seemed to go OK. Now, however, I'm trying to add /depot to vg02. It seems to work, but I get these errors:

vgcfgbackup: Unable to read the physical volume: I/O error
vgcfgbackup: Invalid LVMREC on Physical Volume /dev/rdsk/c3t15d0

c3t15d0 is the new disk.

There are 3 logical volumes that are supposed to have extents on c3t15d0 (not including opt2 and depot): lvol1, lvol2, and lvol3.

I've run string on /etc/lvmtab and it looks fine, as does /etc/lvmconf/vg02.conf. I ran

vgdisplay -v /dev/vg02, and lvol1 and lvol2 are fine. lvol3, however, shows as stale. There are indeed stale extents on c3t15d0 for lvol3. The PV Status on c3t15d0 shows "unavailable", and I *cannot* get a dd copy (to /dev/null) to work; I get this:

dd read error: I/O error
0+0 records in
0+0 records out

That would seem to indicate the disk is physically not working right, but it shows up correctly in "ioscan -fnC disk" and in "lssf /dev/dsk/c3t15d0"

I checked syslog.log, and there is a bad message from 1:55 this morning:

SCSI: unrecoverred deferred error (dev = 0x1f03f000, lba = 0x84d).

I don't know what this error means, but it certainly doesn't sound good.

I'm baffled how the disk can look fine in some commands, but then fail in dd, and how some extents can be "current" (for lvol1 and lvol2) but for lvol3, all the extents be "stale".

What should I do?

[steven Burgess_2 · Accepted solution]

Hi Matt

How did you add the disk to the new volume group ?

It looks as though there is either 'no' or 'corrupt' header information on the disk

If you can see the disk in lvmtab then vgreduce the volume

vgreduce /dev/vg02 /dev/dsk/c3t15d0

Then

pvcreate -f /dev/rdsk/c3t15d0

Then extend back into the volume group

vgextend /dev/vg02 /dev/dsk/c3t15d0

De activate the volume group

vgchange -a n /dev/vg02

Restore the lvm information back to the disk

Then vgcfgrestore -n /dev/vg02 /dev/dsk/c3t15d0

Re activate the volume group

vgchange -a y /dev/vg02

HTH

Steve

[Matt Harrell]

The vgreduce says that the disk cannot be removed because some of the extents are in use. Does this mean they're in use by a currently running process (i.e., I should go to single-user mode), or does it mean that some of the extents are in use in a mirror? I know that the second situation is true--lvol2 is definitely making active use of extents on c3t15d0. How do I get the disk to no longer be in use? Thanks for the assistance.

[steven Burgess_2]

Hi

unmount the mirrored lvol for eg lvol4

Split the mirror of lvol14 to create lvol4 and lvol4b1.

lvsplit /dev/vg02/lvol4

Verify the disks you want to reduce out of the VG are associated with lvol1b by doing a:

lvdisplay -v /dev/vg02/lvol4b.

Remove the old mirror of lvol14:

lvremove -f /dev/vg02/lvol4b

Reduce out the disks from the VG:

vgreduce /dev/vg02 /dev/dsk/c3t15d0

Then follow procedure to pvcreate and re add to volume group and extend back onto mirror

HTH

Steve

[Matt Harrell]

Crap. It won't let me do the lvsplit because c3t15d0 cannot be accessed. It seems I'm stuck in a "chicken or the egg" problem here.

What other choices do I have but to shut down, pull the drive, turn it on (making it fail and look like a bad disk), and then start over?

[Matt Harrell]

But isn't that exactly the same scenerio as a disk failing? Anyway, here's the message I get when I run lvsplit:

Physical Volume /dev/dsk/c3t15d0 cannot be accessed - Aborting ...

[steven Burgess_2]

Matt

You can't do that as you will effectively have a ghost disk in your lvol that will have to be reduced.

Whats the message when you attempt to split the lvol ?

Steve

[Matt Harrell]

Ah! I got it. I had to run lvreduce for the two lvols on c3t15d0 first. Excellent! Now I will start from scratch. I think I might have forgotten to run the pvcreate command the first time, as it is not explicitly listed in the HP tech doc I have. Thanks for your help!

[steven Burgess_2]

Hi

Yes I suppose it is

Remove it then follow this doc

http://www5.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000065011474

HTH

Steve