vgscan doesn't update lvmtab correctly

[Tim Leong_1]

When I remove the current lvmtab and run vgscan -av, the new lvmtab doesn't show any information about vg00, which contains my root filesystem.

Is there anyway that I can update the lvmtab file to contain data about vg00?

[Sanjay_6]

Hi Tim,

Do you have Service Guard on the system ?

Regds

[Tim Leong_1]

No, I don't.

[Jeff Schussele]

Hi Tim,

Could it be possible that the system was brought up in LVM maintenance mode & then not rebooted but instead init 3 was run?
If that's the case /dev/vg00 could be designated as /dev/root instead.
Is /dev/root listed in lvmtab now?

Rgds,
Jeff

[Sanjay_6 · Accepted solution]

Do a ll in /dev and see if there is a device file with the same minor no as /dev/vg00/group. Also do a ll /dev/*/group to see that there are no duplicate groups with the same minor number,

ll /dev |grep "0x000000"
ll /dev/*/group

Hope this helps.

Regds

[Tim Leong_1]

Jeff,
There is no /dev/root entry in lvmtab. But root is mounted from /dev/root.

Sanjay,
Lots of the same minor numbers in /dev, no same major numbers though. All of the group files have different minor numbers.

[Sanjay_6]

Hi Tim,

Try this,

cd /dev
ll |grep "0x000000" |grep "64"

Hope this helps.

Regds

[Tim Leong_1]

Returns nothing...

[Sanjay_6]

Hi,

Can you post your bdf output ?.

Regds

[Tim Leong_1]

/dev/root 143360 92469 48118 66% /
/dev/vg00/lvol1 83733 43404 31955 58% /stand
/dev/vg00/lvol7 1048576 169928 831122 17% /var
/dev/vg00/lvol6 1048576 954265 90692 91% /usr
/dev/vg00/lvol5 524288 16965 476092 3% /tmp
/dev/vg00/lvol4 1548288 1338397 196780 87% /opt
/dev/vg04/lvol 87947160 56113656 23038784 71% /disk3
/dev/vg03/lvol 87947160 59156088 19996352 75% /disk2
/dev/vg02/lvol 8847360 7304150 1496146 83% /apps

[Jeff Schussele]

Hi (again) Tim,

Well that bdf output looks to me like the system was not rebooted properly from LVM maintenance mode. / is typically mounted from /dev/vg00/lvol3.
If that's the case you'll have to verify that /etc/fstab & /dev/vg00/* are correct & then schedule a reboot to correct the situation. You're going to have trouble because what's defined to the system & what's actually mounted don't match.

Rgds,
Jeff

[KapilRaj]

rm /etc/mnttab
mount -a
vgscan

Can you try this way please

Kaps

[SureshKumar_2]

Hi

Before u removing /etc/mnttab, check whether u have entry /dev/root. If it exists then only u can proceed further.

[Tim Leong_1]

Thanks to everyone for your replies. This has been really helpful.

After removing mnttab and running mount -a, / is now mounted from vg00/lvol3. But vgscan -av still doesn't update the lvmtab correctly. Two of my volume groups are giving these errors:

Couldn't stat physical volume "/dev/dsk/c1t2d0":
Invalid argument
Couldn't stat physical volume "/dev/dsk/c2t0d0":
Invalid argument
Couldn't stat physical volume "/dev/dsk/c3t2d0":
Invalid argument

vgscan: has no correspoding valid raw device file under /dev/rdsk.
Verification of unique LVM disk id on each disk in the volume group
/dev/vg00 failed.

Do I need to boot into LVM maintenance mode to fix this? If I reboot, will / even mount?

[Elmar P. Kolkman]

Depends on the output of lvlnboot -v.

Also make sure you're disks are still working (those messages about none-responding disks are alarming). Check the output of ioscan -fnC disk. If you see a lot of NO_HW entries, check your disk(cabinet)s for broaken power supplies or loose SCSI cables before doing the reboot.

[Tim Leong_1]

I think that's the problem. I'm getting NO_HW on 2 of the disks in vg00. AFter running "cat /dev/dsk/c1t2d0 > /dev/null", I'm getting I/O errors. Ugh. Sounds like total system rebuild time.

[Bill Hassell]

The key is the missing physical disks mentioned in vgscan. The disks are dead (electronically) so you have some serious hardware problems. If multiple disks are involved, check for a common feature such as the same cable string (controller). Sounds like you might have an Ignite/UX backup. That would make restoring the system to a complete set of new disks about a 1/2 day job.

[Bill Hassell]

The key is the missing physical disks mentioned in vgscan. The disks are dead (electronically) so you have some serious hardware problems. If multiple disks are involved, check for a common feature such as the same cable string (controller). Sounds like you might not have an Ignite/UX backup. That would make restoring the system to a complete set of new disks about a 1/2 day job.

[Tim Leong_1]

Unfortunately, these are internal disks attached through the SCA interface. Through pvdisplay and lvdisplay, I've found that that one of the disks is on one filesystem and the other is distributed amongst many FS's and they both show STALE extents. I guess that means they are pretty much dead.

I've also determined that the reason for the failure is the temperature around the system. Air flow isn't too good where the machine is, so I'm looking to change its location.

[Bill Hassell]

Wow, I hope these are cheap disks. Air conditioning and temperature control is crucial to reliable disk operation. Electronics (and especially mechanical devices like disks) degrdae with temperature so the damage is already done and will only get worse on other disks. IF this is a very expensive disk array, run, don't walk to the CEO of your company and state that total loss of the disk farm and most of the computers is a possibility because of the poor air conditioning design.