Problems with vg00 and lvmtab

[Christian Schulze]

Hi

I have a strange problem with one of my systems.

When I do an
moskau:/# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 65008 73509 47% /
/dev/vg00/lvol1 83733 26764 48595 36% /stand
/dev/vg00/lvol8 512000 254258 241813 51% /var
/dev/vg00/lvol6 770048 595554 163627 78% /usr
/dev/vg00/lvol5 131072 8753 114783 7% /tmp
/dev/vg00/lvol4 819200 774485 42725 95% /opt
/dev/vg00/lvol7 40960 1463 37084 4% /home

I see al my LV's in vg00.

But if I do an
moskau:/# vgdisplay /dev/vg00
vgdisplay: Volume group "/dev/vg00" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00".

When I do a
moskau:/# strings /etc/lvmtab
/dev/
/dev/dsk/c0t5d0
/dev/dsk/c0t9d0
/dev/dsk/c0t10d0
/dev/vgtest
/dev/dsk/c0t11d0

there is no vg00 (so the message from vgdisplay is ok) but what about the three disks c0t5d0,c0t9d0,c0t10d0, they are members of vg00.

I tried to reconstruct the lvmtab-File but it did'nt help

System is still (??!!) runnning fine

Any Ideas ?

Christian

[Vincenzo Restuccia]

#rm /etc/lvmtab
#vgscan -v

[James R. Ferguson]

Hi Christian:

Try recreating your /etc/lvmtab as follows:

# mv /etc/lvmtab /etc/lvmtab.old
# vgscan -v

Make sure you *read and follow* the guide in the 'vgscan' man pages!

...JRF...

[Christian Schulze]

Hi
thanxs for your quick reply

but did'nt help.
there is a strange part in the output of vgscan -v:

moskau:/# vgscan -v
Creating "/etc/lvmtab".
vgscan: Couldn't access the list of physical volumes for volume group "/dev". <-- ???
vgscan: Couldn't access the list of physical volumes for volume group "/dev/vgtest".
Physical Volume "/dev/dsk/c0t8d0" is not part of a Volume Group <-- that is OK
Couldn't stat physical volume "/dev/dsk/c2t2d0": <-- the cdrom
Invalid argument


/dev/ <-- this should be my vg00
/dev/dsk/c0t5d0
/dev/dsk/c0t9d0
/dev/dsk/c0t10d0



/dev/vgtest
/dev/dsk/c0t11d0



????

[Christian Schulze]

found something more

moskau:/# lvlnboot -v
lvlnboot: Cannot open the control file "/dev//group":
No such file or directory


Probably better to contact the support-guys ??

[Vincenzo Restuccia]

#ioscan -nfC disk is OK?

[Christian Schulze]

yes, seems ok to me

moskau:/# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
disk 0 8/4.5.0 sdisk CLAIMED DEVICE SEAGATE ST34573WC
/dev/dsk/c0t5d0 /dev/rdsk/c0t5d0
disk 1 8/4.8.0 sdisk CLAIMED DEVICE SEAGATE ST34573WC
/dev/dsk/c0t8d0 /dev/rdsk/c0t8d0
disk 2 8/4.9.0 sdisk CLAIMED DEVICE SEAGATE ST34573WC
/dev/dsk/c0t9d0 /dev/rdsk/c0t9d0
disk 20 8/4.10.0 sdisk CLAIMED DEVICE SEAGATE ST34572WC
/dev/dsk/c0t10d0 /dev/rdsk/c0t10d0
disk 21 8/4.11.0 sdisk CLAIMED DEVICE SEAGATE ST318436LC


as I said, system is (still) up and running ...

[Vincenzo Restuccia]

#ll /dev/vg*
#ll /dev/group/*
output is OK?

[MANOJ SRIVASTAVA]

Hi Christian

Try this

vgcfgrestore -R n /dev/vg00 /dev/rdsk/c0t8d0.

and then do

vgscan -v

this should solve your problem

Manoj Srivastava

[Christian Schulze]

moskau:/# ll /dev/vg*

/dev/vg00:
total 0
crw-r----- 1 root sys 64 0x000000 Jan 3 08:05 group
brw-r----- 1 root sys 64 0x000001 Mar 30 13:08 lvol1
brw-r----- 1 root sys 64 0x000002 Jan 3 08:05 lvol2
brw-r----- 1 root sys 64 0x000003 Jan 3 08:05 lvol3
brw-r----- 1 root sys 64 0x000004 Jan 3 08:05 lvol4
brw-r----- 1 root sys 64 0x000005 Jan 3 08:05 lvol5
brw-r----- 1 root sys 64 0x000006 Jan 3 08:05 lvol6
brw-r----- 1 root sys 64 0x000007 Jan 3 08:05 lvol7
brw-r----- 1 root sys 64 0x000008 Jan 3 08:05 lvol8
brw-r----- 1 root sys 64 0x000009 Jan 3 08:05 lvol9
crw-r----- 1 root sys 64 0x000001 Jan 3 08:05 rlvol1
crw-r----- 1 root sys 64 0x000002 Jan 3 08:05 rlvol2
crw-r----- 1 root sys 64 0x000003 Jan 3 08:05 rlvol3
crw-r----- 1 root sys 64 0x000004 Jan 3 08:05 rlvol4
crw-r----- 1 root sys 64 0x000005 Jan 3 08:05 rlvol5
crw-r----- 1 root sys 64 0x000006 Jan 3 08:05 rlvol6
crw-r----- 1 root sys 64 0x000007 Jan 3 08:05 rlvol7
crw-r----- 1 root sys 64 0x000008 Jan 3 08:05 rlvol8
crw-r----- 1 root sys 64 0x000009 Jan 3 08:05 rlvol9

looks the same on the other node in the cluster (identical System), esp. 0x000000


/dev/group is not existent

[Vincenzo Restuccia]

Scan the primary controller device paths for all the physical volumes on the system, but do not update the /etc/lvmtab file:

vgscan -p -v


Scan all controller device paths to all physical volumes on the system:

vgscan -a -v



Scan the primary controller device paths for all the physical volumes on the system and re-create /etc/lvmtab file:

vgscan -v



The following messages will appear after /etc/lvmtab file is re-created.

*** LVMTAB has been created successfully.
*** If PV links are configured in the system
*** Do the following to resync information on disk.
*** #1. vgchange -a y
*** #2. lvlnboot -R

[Vincenzo Restuccia]

Sorry,ignore ll /dev/group/*
group is in /dev/vg*

[Christian Schulze]

Thanks again, but Problem is still there

everything I do (vgscan ..., vgcfgrestore ...)
doesnt help, when I do an vgscan -p -v

I see
moskau:/# vgscan -v
Creating "/etc/lvmtab".
vgscan: Couldn't access the list of physical volumes for volume group "/dev". <-- ???

a little bit lower

/dev/ <-- this should be my vg00
/dev/dsk/c0t5d0
/dev/dsk/c0t9d0
/dev/dsk/c0t10d0

It looks as if the Kernel doesnt know about vg00, but it is still mounted ...

[Vincenzo Restuccia]

Use vgextend to include any missing alternate paths in the VG.

[Christian Schulze]

Well, vgextend says that vg00 does'nt exists ...

moskau:/# vgextend /dev/vg00 /dev/dsk/c0t8d0
vgextend: Volume group "/dev/vg00" does not exist in the "/etc/lvmtab" file.

c0t8d0 is still an empty disk, doesnt belong to any vgXX


I think, that the missing vg00-Entry in lvmtab is the problem.
strings /etc/lvmtab says
moskau:/# strings /etc/lvmtab
/dev/
/dev/dsk/c0t5d0
/dev/dsk/c0t9d0
/dev/dsk/c0t10d0

On my other node it says
passau:/# strings /etc/lvmtab
/dev/vg00
-9v'
/dev/dsk/c0t5d0
/dev/dsk/c0t9d0
/dev/dsk/c0t10d0

Hmm ...

[Peggy Fong]

Hi
When you followed the steps to re-create the lvmtab (mv lvmtab lvmtab.old) then did vgscan -v you would receive all those errors that you listed. Did you immediately run vgscan -p -v after that to see if it changed or did you put the old lvmtab file back in place?

Also, check to see if you have another group file with the same minor node (doubt it but it doesn't hurt to check ll -d /dev/vg*/group.

I'd try re-creating your lvmtab again as before but after you get all the error, run vgscan -p -v (preview) and see if vg00 appears. If not, then I'd do a vcfgrestore on all three disks and then re-do the lvmtab again. I believe as well the lvmtab is corrupt.

do lvlnboot -R vg00 after the vgcfgrestore and prior to redoing the lvmtab.

Peggy

[Christian Schulze]

Hi, thanks to all
the problem ist not fixed yet, but before I do anything else, I take a backup-tape first, as long the system is running ...
Then I'll start again tomorrow morning. I think its not a good Idea to restore the System if anything fails while I fall asleep.


The next step will be that I try Peggy's tips.


Christian

[Christian Schulze]

Last update for today

even ignite won't create a recovery-tape because of "problems executing the vgdisplay command on vg00" ...

but it looks as if a tar works.

[Kandaswamy Natarajan]

have you tried just running vgdisplay ???

DO u get any errors??? when u run just vgdisplay -v ( without giving any volume group name)??
check for device files under /dev with major no 64 and minor no 0x000000, if exists remove them.
after removing try the vgscan commands to recreate your lvmtab.

if all this does not work, then
boot in maintanence mode
ISL> hpux -lm

rm /dev/vg00/lvol*
rm /dev/vg00/rlvol*
vgimport vg00
vgchange -a y vg00
check all the logical volumes are recreated in /dev/vg00
check for swap and dump device
run lvlnboot -R
reboot the box for a complete and clean reboot..

[Christian Schulze]

Well ,I tried all the suggestions made (but not rebooted, because System is productive), but nothing helped.

I opened a call at HP-Support.

I'll let you know what happened.
Christian

[Wieslaw Krajewski]

Hi,

One more thing you could try is to vgexport and vgimport vg00. This helps in such difficult cases. Having all information about your vg00 you could try the following steps:
1.Boot your system in the maintenance mode, i.e.
stop automatic booting and fter reaching ISL prompt run
ISL> hpux -lm
2. after reaching prompt
vgexport vg00
3. mkdir /dev/vg00
mknod /dev/vg00/group c 64 0x000000
vgimport vg00 /dev/dsk/c#t3d# ....

where /dev/dsk/c#t3d# ....denotes your disks.

This procedure can be used as the last resort even in productive systems.

Rgds.

[Christian Schulze]

Hi to all

after some E-Mail and Phone-calls with HP-Support I got th thing running again (without reboot !!!) Of course I'll test booting from disk an mirror this weekend, but I feel good that I will work.

what happened ?
short Document from HP:

vgscan will create /etc/lvmtab with /dev/ instead of /dev/vg00 DocId: KBRC00002565 Updated: 4/24/01 9:17:00 AM

PROBLEM
when trying to recreate the /etc/lvmtab using the vgscan, vgscan will
create it without any error or warning but instead of /dev/vg00 it is
/dev/ but all the physical drives part of vg00 are listed.
CONFIGURATION
HP-UX 10.20 s800
RESOLUTION
The problem is with the /dev/slvmvg file (only with MC/ServiceGuard)
having same major/minor number as /dev/vg00/group file, to verify it you can
type:

# ll /dev | grep 64

To fix it you can do:

rm /dev/slvmvg
# vgscan
# strings /etc/lvmtab -> now it should be ok

do I need te recreate the /dev/slvmvg ?

When cmlvmd runs, it checks if /dev/slvmvg exists. If not, it creates one so
this file is recreated manually.


But here is the way te re-create it manually if you want:

Recreate the special file:

# mknod /dev/slvmvg c 64 0x0

Change the file mode:

# chmod 660


This is a known problem that has no permanent fix available yet.






Thanks to all

Christian,

feeling much better now ...

[James R. Ferguson]

Hi Christian:

Thanks for posting the solution to an unusual one! Glad your happy again ;-)

...JRF...

[Peggy Fong]

Christian
I too, thank you very much for posting the solution to a strange problem. I'm glad you were able to get a fix without rebooting.
Peggy