Failed physical volume causing lvmtab weirdness

[Erwin Mascardo]

Yes, "weirdness" is the best technical term I can come up with to explain what's going on here...

I've got a K box running 11.0 with several disk arrays attached, with the disks arranged in several volume groups. One of the disks in the VG /dev/foo is in the process of failing. However, we didn't realize this was happening before our last software update, and we rebooted the machine (necessary step in our in-house update process).

Here's where the weirdness starts. I'm trying to figure out exactly which disk (one of 4) in the VG is the one that's dying, since you can't tell by looking at the drives themselves (no amber lights to be seen, and there's no logical correlation between the PVs and which VG they belong to -- they all kind of got put where they were needed at the moment (not my doing, the system's been in place far longer than I've been in the job)). Here's what the system tells me:

# vgdisplay -v /dev/foo
vgdisplay: Volume group "/dev/foo" does not exist in the "/etc/lvmtab" file
vgdisplay: Cannot display volume group "/dev/foo"

Okay, not an insurmountable problem. However, oddly enough, the other three LVs within the volume group are mounted and are accessible -- /dev/foo/lvx, /dev/foo/lvy, and /dev/foo/lvz are mounted where they need to be, and we've got programs using directories under the mount points. /dev/foo/lva is missing as it should be, since it's on the dying disk.

Following standard procedure, I back up /etc/lvmtab, delete the existing copy, and do a vgscan. And I seem to find the three disks which correspond to /dev/foo/lvx, /dev/foo/lvy, and /dev/foo/lvz. However...they're listed as being in /dev/vg01.

We don't have a /dev/vg01. And /dev/foo doesn't show up.

Using vgdisplay on /dev/vg01 lists the three PVs, all right. Using pvdisplay -v on those PVs shows that they're not associated with any LV (it gives "???" for the LV for each logical extent).

I can't use lvdisplay on /dev/foo/lvx -- it complains that /dev/foo isn't in lvmtab (which we knew). However, it's mounted, and nothing changed in /etc/fstab -- it's still listed as /dev/foo/lvx in there.

Any ideas on (1) what's going on, and (2) how I can fix it? Thanks.

[David Burgess]

Has someone kindly renamed your /dev/foo devices files to /dev/vg01?

What are the timestamps on /dev/vg01?

Regards,

Dave.

[James R. Ferguson]

Hi Erwin:

One way to identify a failing disk is:

# dd if=/dev/rdsk/cXtYdz of=/dev/null

Look for errors...

Regards!

...JRF...

[David Burgess]

Also look for "lbolt" in /var/adm/syslog/syslog.log and dmesg

Regards,

Dave.

[Jeff Schussele · Accepted solution]

Hi Erwin,

Sounds like David may be on to something here.

If the /dev/foo dir exists & /dev/vg01 doesn't.

Why don't you create a link from /dev/foo -> /dev/vg01?

vgscan looks at the info on the disk & doesn't care about the /dev dirs - so it's possible someone renamed the /dev/vg01 to /dev/foo.

This may get you past the current problem until you can take it down & fix it the way it's supposed to be.

HTH,
Jeff

[David Burgess]

If it is the case that it has been renamed then you can simply rename it back again. Alternatively just edit the /etc/fstab and change it from /dev/foo to /dev/vg01 for each logical volume that it affects.

Then see if the problem goes away. Before you do make sure you check for the lbolt

grep -i lbolt /var/adm/syslog/syslog.log

and

dmesg | grep -i lbolt

dmesg will be the same as syslog, but syslog had more history and timestamps.

HTH

Dave.

[ashish nanjiani]

Hi

If your mount points still show as /dev/foo/lvx than there has to be a directory structure under /dev as /dev/foo. I think renaming as suggested should do the work.

[Erwin Mascardo]

Thanks for the help, folks. /dev/vg01 did exist, but nothing was underneath it (except for the group file). I renamed it and then symlinked it to /dev/foo. Lo and behold, I can now use lvdisplay on the LVs and vgdisplay on the whole VG.

This will work until I can do a real fix. Thanks again.