topic Re: Failed physical volume causing lvmtab weirdness in Operating System - HP-UX

Failed physical volume causing lvmtab weirdness

Erwin Mascardo — Wed, 01 May 2002 17:57:00 GMT

Yes, "weirdness" is the best technical term I can come up with to explain what's going on here...

I've got a K box running 11.0 with several disk arrays attached, with the disks arranged in several volume groups. One of the disks in the VG /dev/foo is in the process of failing. However, we didn't realize this was happening before our last software update, and we rebooted the machine (necessary step in our in-house update process).

Here's where the weirdness starts. I'm trying to figure out exactly which disk (one of 4) in the VG is the one that's dying, since you can't tell by looking at the drives themselves (no amber lights to be seen, and there's no logical correlation between the PVs and which VG they belong to -- they all kind of got put where they were needed at the moment (not my doing, the system's been in place far longer than I've been in the job)). Here's what the system tells me:

# vgdisplay -v /dev/foo
vgdisplay: Volume group "/dev/foo" does not exist in the "/etc/lvmtab" file
vgdisplay: Cannot display volume group "/dev/foo"

Okay, not an insurmountable problem. However, oddly enough, the other three LVs within the volume group are mounted and are accessible -- /dev/foo/lvx, /dev/foo/lvy, and /dev/foo/lvz are mounted where they need to be, and we've got programs using directories under the mount points. /dev/foo/lva is missing as it should be, since it's on the dying disk.

Following standard procedure, I back up /etc/lvmtab, delete the existing copy, and do a vgscan. And I seem to find the three disks which correspond to /dev/foo/lvx, /dev/foo/lvy, and /dev/foo/lvz. However...they're listed as being in /dev/vg01.

We don't have a /dev/vg01. And /dev/foo doesn't show up.

Using vgdisplay on /dev/vg01 lists the three PVs, all right. Using pvdisplay -v on those PVs shows that they're not associated with any LV (it gives "???" for the LV for each logical extent).

I can't use lvdisplay on /dev/foo/lvx -- it complains that /dev/foo isn't in lvmtab (which we knew). However, it's mounted, and nothing changed in /etc/fstab -- it's still listed as /dev/foo/lvx in there.

Any ideas on (1) what's going on, and (2) how I can fix it? Thanks.

Re: Failed physical volume causing lvmtab weirdness

David Burgess — Wed, 01 May 2002 18:04:08 GMT

Has someone kindly renamed your /dev/foo devices files to /dev/vg01?

What are the timestamps on /dev/vg01?

Regards,

Dave.

Re: Failed physical volume causing lvmtab weirdness

James R. Ferguson — Wed, 01 May 2002 18:05:30 GMT

Hi Erwin:

One way to identify a failing disk is:

# dd if=/dev/rdsk/cXtYdz of=/dev/null

Look for errors...

Regards!

...JRF...

Re: Failed physical volume causing lvmtab weirdness

David Burgess — Wed, 01 May 2002 18:11:51 GMT

Also look for "lbolt" in /var/adm/syslog/syslog.log and dmesg

Regards,

Dave.

Re: Failed physical volume causing lvmtab weirdness

Jeff Schussele — Wed, 01 May 2002 18:18:03 GMT

Hi Erwin,

Sounds like David may be on to something here.

If the /dev/foo dir exists & /dev/vg01 doesn't.

Why don't you create a link from /dev/foo -> /dev/vg01?

vgscan looks at the info on the disk & doesn't care about the /dev dirs - so it's possible someone renamed the /dev/vg01 to /dev/foo.

This may get you past the current problem until you can take it down & fix it the way it's supposed to be.

HTH,
Jeff

Re: Failed physical volume causing lvmtab weirdness

David Burgess — Wed, 01 May 2002 18:24:27 GMT

If it is the case that it has been renamed then you can simply rename it back again. Alternatively just edit the /etc/fstab and change it from /dev/foo to /dev/vg01 for each logical volume that it affects.

Then see if the problem goes away. Before you do make sure you check for the lbolt

grep -i lbolt /var/adm/syslog/syslog.log

and

dmesg | grep -i lbolt

dmesg will be the same as syslog, but syslog had more history and timestamps.

HTH

Dave.

Re: Failed physical volume causing lvmtab weirdness

ashish nanjiani — Wed, 01 May 2002 18:29:23 GMT

Hi

If your mount points still show as /dev/foo/lvx than there has to be a directory structure under /dev as /dev/foo. I think renaming as suggested should do the work.

Re: Failed physical volume causing lvmtab weirdness

Erwin Mascardo — Wed, 01 May 2002 18:36:16 GMT

Thanks for the help, folks. /dev/vg01 did exist, but nothing was underneath it (except for the group file). I renamed it and then symlinked it to /dev/foo. Lo and behold, I can now use lvdisplay on the LVs and vgdisplay on the whole VG.

This will work until I can do a real fix. Thanks again.