Volume Groups are not activating

[Venkat_11]

Hi,
i am having a weird problem. when ever i reboot the server . The volume groups are not activating. I need to mually run commands vgchange to activate the volume group and then mounting the filesystems on volumes. Can any body please let me know what the problem is exactly..? and how to make the volume groups activate .

Thanks in advance
Venkat

[James R. Ferguson]

Hi:

Look at '/etc/lvmrc'. You probably need to set 'AUTO_VG_ACTIVATE=1'. Read the script commentary for more information.

Regards!

...JRF...

[A. Clay Stephenson]

Check your /etc/lvmrc file. I'll bet that your AUTO_VG_ACTIVATE=0. If this is part of an MC/SG cluster then leave this valiue at 0 and modify the custom_vg_activation function to activate any VG's not under control of MC/SG.

[Venkat_11]

I checked that file the

AUTO_VG_ACTIVATE=1

its seems ok but they are not activating any advice please...Thanks for your quick responce

Venkat

[A. Clay Stephenson]

In that case, do you have quorum? Are any disks in the VG not responding?

[Geoff Wild]

Check your /etc/rc.log or /var/adm/syslog/syslog.log for messages related to the vg's.

Rgds...Geoff

[Venkat_11]

Sorry for the late and thanks for your patience.

Here is the output from syslog file

usxsl003:root# grep vg syslog.log
Nov 1 15:04:35 usxsl003 LVM[5960]: vgchange -a y /dev/vg01
Nov 1 15:04:43 usxsl003 LVM[6137]: vgchange -a y /dev/vg02
Nov 1 15:04:48 usxsl003 LVM[6141]: vgchange -a y /dev/vg03
Nov 1 15:04:56 usxsl003 LVM[6183]: vgchange -a y /dev/vgtmp

seems its trying to activatethe volume groups...


And the rc.log file output

usxsl003:root# grep vg rc.log
mountall: cannot mount /dev/vg03/lvol2
vxfs mount: Cannot open /dev/vg03/lvol2: No such device or address
mountall: cannot mount /dev/vg01/lvol1
vxfs mount: Cannot open /dev/vg01/lvol1: No such device or address
mountall: cannot mount /dev/vg01/lvol2
vxfs mount: Cannot open /dev/vg01/lvol2: No such device or address
mountall: cannot mount /dev/vg02/lvol4
vxfs mount: Cannot open /dev/vg02/lvol4: No such device or address
mountall: cannot mount /dev/vg01/lvol13
vxfs mount: Cannot open /dev/vg01/lvol13: No such device or address
mountall: cannot mount /dev/vg01/lvol3
vxfs mount: Cannot open /dev/vg01/lvol3: No such device or address
mountall: cannot mount /dev/vg01/lvol4
vxfs mount: Cannot open /dev/vg01/lvol4: No such device or address
mountall: cannot mount /dev/vg01/lvol5
vxfs mount: Cannot open /dev/vg01/lvol5: No such device or address
mountall: cannot mount /dev/vg02/lvol1
vxfs mount: Cannot open /dev/vg02/lvol1: No such device or address
mountall: cannot mount /dev/vg02/lvol2
vxfs mount: Cannot open /dev/vg02/lvol2: No such device or address
mountall: cannot mount /dev/vgtmp/lvol1
vxfs mount: Cannot open /dev/vgtmp/lvol1: No such device or address
mountall: cannot mount /dev/vg01/lvol6
mountall: cannot mount /dev/vg03/lvol1
vxfs mount: Cannot open /dev/vg03/lvol1: No such device or address
mountall: cannot mount /dev/vg01/lvol7
vxfs mount: Cannot open /dev/vg01/lvol7: No such device or address
mountall: cannot mount /dev/vg01/lvol8
vxfs mount: Cannot open /dev/vg01/lvol8: No such device or address
mountall: cannot mount /dev/vg01/lvol10
vxfs mount: Cannot open /dev/vg01/lvol10: No such device or address
mountall: cannot mount /dev/vg01/lvol11
vxfs mount: Cannot open /dev/vg01/lvol11: No such device or address
mountall: cannot mount /dev/vg01/lvol12
vxfs mount: Cannot open /dev/vg01/lvol12: No such device or address
Output from "/sbin/rc1.d/S400set_prvgrp start":
"/sbin/rc1.d/S400set_prvgrp start" SKIPPED


Please advice me

Thanks
Venkat

[Raj D.]

HI Venkat ,

seems to be some problem with activating the vg.
What is the OS version and patch level.
Is there any recent changes done,

Thx.
Raj.

[Venkat_11]

Hi,

Thanks for the quick responce

usxsl003:root# uname -a
HP-UX usxsl003 B.11.11 U 9000/800 1127164808 unlimited-user license

This is a newly build box.

Venkat

[Mel Burslan]

please provide the output of the following 2 commands:

strings /etc/lvmtab
ls -lR /dev/vg*

looks like your problem is not only with vg activation but from a corrupt lvmtab or deleted device files

[Venkat_11]

Here is the out put

usxsl003:root# strings /etc/lvmtab
/dev/vg00
/dev/dsk/c2t1d0
/dev/dsk/c2t0d0
/dev/vg01
/dev/dsk/c10t1d4
/dev/dsk/c10t1d5
/dev/dsk/c10t1d6
/dev/dsk/c10t1d7
/dev/vg02
/dev/dsk/c10t2d1
/dev/dsk/c10t2d2
/dev/dsk/c10t2d3
/dev/dsk/c10t2d6
/dev/vg03
/dev/dsk/c10t2d5
/dev/dsk/c10t2d0
/dev/dsk/c10t2d4
/dev/vgtmp
/dev/dsk/c10t2d7
usxsl003:root# ls -lR /dev/vg*

/dev/vg00:
total 0
crw-r----- 1 root sys 64 0x000000 May 12 21:51 group
brw-r----- 1 root sys 64 0x000001 May 12 21:51 lvol1
brw-r----- 1 root sys 64 0x000009 Nov 1 11:58 lvol11
brw-r----- 1 root sys 64 0x00000a Nov 1 11:58 lvol12
brw-r----- 1 root sys 64 0x00000b Nov 1 11:59 lvol13
brw-r----- 1 root sys 64 0x000002 May 12 21:51 lvol2
brw-r----- 1 root sys 64 0x000003 May 12 21:51 lvol3
brw-r----- 1 root sys 64 0x000004 May 12 21:51 lvol4
brw-r----- 1 root sys 64 0x000005 May 12 21:51 lvol5
brw-r----- 1 root sys 64 0x000006 May 12 21:51 lvol6
brw-r----- 1 root sys 64 0x000007 May 12 21:51 lvol7
brw-r----- 1 root sys 64 0x000008 May 12 21:51 lvol8
crw-r----- 1 root sys 64 0x000001 May 12 21:51 rlvol1
crw-r----- 1 root sys 64 0x000009 Nov 1 11:58 rlvol11
crw-r----- 1 root sys 64 0x00000a Nov 1 11:58 rlvol12
crw-r----- 1 root sys 64 0x00000b Nov 1 11:59 rlvol13
crw-r----- 1 root sys 64 0x000002 May 12 21:51 rlvol2
crw-r----- 1 root sys 64 0x000003 May 12 21:51 rlvol3
crw-r----- 1 root sys 64 0x000004 May 12 21:51 rlvol4
crw-r----- 1 root sys 64 0x000005 May 12 21:51 rlvol5
crw-r----- 1 root sys 64 0x000006 May 12 21:51 rlvol6
crw-r----- 1 root sys 64 0x000007 May 12 21:51 rlvol7
crw-r----- 1 root sys 64 0x000008 May 12 21:51 rlvol8

/dev/vg01:
total 0
crw-r--r-- 1 root sys 64 0x010000 Oct 12 18:32 group
brw-r----- 1 root sys 64 0x010001 Oct 12 19:10 lvol1
brw-r----- 1 root sys 64 0x01000a Oct 12 19:32 lvol10
brw-r----- 1 root sys 64 0x01000b Oct 12 19:33 lvol11
brw-r----- 1 root sys 64 0x01000c Oct 12 19:35 lvol12
brw-r----- 1 root sys 64 0x01000d Oct 12 19:36 lvol13
brw-r----- 1 root sys 64 0x010002 Oct 12 19:13 lvol2
brw-r----- 1 root sys 64 0x010003 Oct 12 19:15 lvol3
brw-r----- 1 root sys 64 0x010004 Oct 12 19:16 lvol4
brw-r----- 1 root sys 64 0x010005 Oct 12 19:22 lvol5
brw-r----- 1 root sys 64 0x010006 Oct 12 19:23 lvol6
brw-r----- 1 root sys 64 0x010007 Oct 12 19:26 lvol7
brw-r----- 1 root sys 64 0x010008 Oct 12 19:29 lvol8
brw-r----- 1 root sys 64 0x010009 Oct 12 19:30 lvol9
crw-r----- 1 root sys 64 0x010001 Oct 12 19:10 rlvol1
crw-r----- 1 root sys 64 0x01000a Oct 12 19:32 rlvol10
crw-r----- 1 root sys 64 0x01000b Oct 12 19:33 rlvol11
crw-r----- 1 root sys 64 0x01000c Oct 12 19:35 rlvol12
crw-r----- 1 root sys 64 0x01000d Oct 12 19:36 rlvol13
crw-r----- 1 root sys 64 0x010002 Oct 12 19:13 rlvol2
crw-r----- 1 root sys 64 0x010003 Oct 12 19:15 rlvol3
crw-r----- 1 root sys 64 0x010004 Oct 12 19:16 rlvol4
crw-r----- 1 root sys 64 0x010005 Oct 12 19:22 rlvol5
crw-r----- 1 root sys 64 0x010006 Oct 12 19:23 rlvol6
crw-r----- 1 root sys 64 0x010007 Oct 12 19:26 rlvol7
crw-r----- 1 root sys 64 0x010008 Oct 12 19:29 rlvol8
crw-r----- 1 root sys 64 0x010009 Oct 12 19:30 rlvol9

/dev/vg02:
total 0
crw-r--r-- 1 root sys 64 0x020000 Oct 12 18:35 group
brw-r----- 1 root sys 64 0x020001 Oct 12 19:52 lvol1
brw-r----- 1 root sys 64 0x020002 Oct 12 19:40 lvol2
brw-r----- 1 root sys 64 0x020003 Oct 12 19:54 lvol3
brw-r----- 1 root sys 64 0x020004 Oct 12 19:55 lvol4
crw-r----- 1 root sys 64 0x020001 Oct 12 19:52 rlvol1
crw-r----- 1 root sys 64 0x020002 Oct 12 19:40 rlvol2
crw-r----- 1 root sys 64 0x020003 Oct 12 19:54 rlvol3
crw-r----- 1 root sys 64 0x020004 Oct 12 19:55 rlvol4

/dev/vg02a:
total 0
crw-r--r-- 1 root sys 64 0x050000 Feb 13 2005 group
brw-r----- 1 root sys 64 0x050004 Apr 9 2005 lv9eb
brw-r----- 1 root sys 64 0x050002 Feb 13 2005 lvai4
brw-r----- 1 root sys 64 0x050003 Feb 13 2005 lvbi4
brw-r----- 1 root sys 64 0x050001 Feb 13 2005 lvdb4
crw-r----- 1 root sys 64 0x050004 Apr 9 2005 rlv9eb
crw-r----- 1 root sys 64 0x050002 Feb 13 2005 rlvai4
crw-r----- 1 root sys 64 0x050003 Feb 13 2005 rlvbi4
crw-r----- 1 root sys 64 0x050001 Feb 13 2005 rlvdb4

/dev/vg03:
total 0
crw-r--r-- 1 root sys 64 0x030000 Oct 12 18:38 group
brw-r----- 1 root sys 64 0x030003 Oct 12 19:06 lv_temp
brw-r----- 1 root sys 64 0x030001 Oct 12 19:01 lvol1
brw-r----- 1 root sys 64 0x030002 Oct 12 19:03 lvol2
crw-r----- 1 root sys 64 0x030003 Oct 12 19:06 rlv_temp
crw-r----- 1 root sys 64 0x030001 Oct 12 19:01 rlvol1
crw-r----- 1 root sys 64 0x030002 Oct 12 19:03 rlvol2

/dev/vgtmp:
total 0
crw-r--r-- 1 root sys 64 0x040000 Oct 12 18:40 group
brw-r----- 1 root sys 64 0x040001 Oct 12 18:49 lvol1
crw-r----- 1 root sys 64 0x040001 Oct 12 18:49 rlvol1
usxsl003:root#


Thanks
Venkat

[Mel Burslan]

immediately after the boot, if you run

vgdisplay /dev/vg01 (or 02, or 03 )

do you get a "volume not activated message" ? If yes, can you activate them manually as follows

vgchange -a y vg01 (or 02, or 03) ?

if not, please provide the error message to the last command or if vg activation happens, please attach the full /etc/rc.log from the last unsuccessful boot, where the activation at boot time failed.

[Venkat_11]

Hi,

Once after the server is rebooted if i try the command
vgdisplay -v vg01 or vg02 or vg03
i am getting error volume group is not activated and when i ran the comamnd vgchange -a y then its getting activating.

And here i am attaching the rc.log file. PLeae let e know if you need any thing else.
Thanks for your quick responces

Venkat

[A. Clay Stephenson]

This looks as though /sbin/lvmrc is never being invoked. Check to see that /sbin/bcheckrc is present and that it in turn calls /sbin/lvmrc. Check the modes on these files and look for any possible corruption or "improvement" of these files.

If you could copy these files from anther machine, that would be a good idea.

[Raj D.]

Hi Venkat ,

Have you checked the files , mentioned above.
The /sbin/bcheckrc and lvmrc file . Any luck !

Please post further.

cheers,
Raj.

[Venkat_11]

Hi,

I compared both the files on this server with another server i didnt find any difference at all

usxsl003:root# ls -l | grep lvmrc
-r-xr--r-- 1 bin bin 1410 Nov 14 2000 lvmrc
usxsl003:root# ls -l | grep check
-r-xr--r-- 1 bin bin 9169 Nov 14 2000 bcheckrc

Suggest still if do you want me to copy accross..?


Thanks

Venkat

[Marco Santerre]

Have you checked to see if bcheckrc gets invoked in your /etc/inittab?

[Alzhy]

Venkat,

Are these volume groups that are not activating at server startup composed of disks from a particular array? And what kind of array? Possible that the disks from your array/SAN is slow in being recognized by the system? And do you have all the requirements for whatever storage array you're using -- i.e. pathing software, etc.


Truly weird situation you're in sir. If you've a spare disk - have you tried going through another install - following exactly what you've done - to see if you've the same issues?

[Raj D.]

Hi Venkat ,

Check /etc/inittab entry for bcheckrc is there:

# cat /etc/inittab | grep bcheck

brc1::bootwait:/sbin/bcheckrc /dev/console 2>&1

hth,
Raj.

[A. Clay Stephenson]

Check the inittab and make sure that there is an entry with runstate set to "bootwait" that does start /sbin/bcheckrc. Look at this line very carefully for errors and possibly add some probes to the bcheckrc script. It certainly seems that this script is never being called so that only vg00 is ever activated.

[Venkat_11]

Hi,

Thanks for every body who are trying to solve my problem.

I checked the /etc/inittab
usxsl003:root# cat /etc/inittab | grep bcheck
brc1::bootwait:/sbin/bcheckrc /dev/console 2>&1 # fsck, etc.

And this system is ignited by using ignite tape of another HP server.
After that we deleted the old volume groups and created new volume groups with new luns which we got from hitachi storage.
I think it doesnt impact the problem we are facing. We followed the same procedure for lot other systems which are working fine.

Please let me know if you need any more details

Thanks
Venkat

[A. Clay Stephenson]

At this point, we need to activate the VG's exactly as lvmrc does. Bring the system up in single-user mode. Only vg00 should be active. Execute "/sbin/vgchange -a y -s". If this is okay then manually loop through the vgsync in /sbin/lvmrc.

I am now leaning towards a corrupt lvmtab.

[Alzhy]

I agree with A. Clay.

Try re-creating your lvmtab to see if it makes any difference.