VG could not be deactivated in MC/SG

[Elif Gius]

Hi,
we have following problem.
We extended a VG with a disk. Then we switched the package. But it didn't come up.
So we forgot to do a vgimport on the other node. After a vgimport the package still didn't come up.

In the logfile following error message appeared.
+++
Feb 5 18:54:23 - "estragon": Activating volume group /dev/vgPCVWEB with exclusive option.
Volume group "/dev/vgPCVWEB" has been successfully changed.
/dev/vgPCVWEB/lvPCVWEB: No such device or address
/dev/vgPCVWEB/lvHOME: No such device or address
/dev/vgPCVWEB/lvTIVOLIep: No such device or address
/dev/vgPCVWEB/lvMAESTRO: No such device or address
+++
Then we checked /dev/vg*/group for duplicates.

crw-r----- 1 root sys 64 0x000000 Apr 30 2002 /dev/vg00/group
crw-r--r-- 1 root sys 64 0x010000 May 5 2004 /dev/vgORACLE/group
crw-r--r-- 1 root sys 64 0x030000 Jan 31 15:33 /dev/vgP16_old/group
crw-r--r-- 1 root sys 64 0x040000 Jan 28 08:54 /dev/vgP11/group
crw-r--r-- 1 root sys 64 0x050000 May 19 2004 /dev/vgP09/group
crw-r--r-- 1 root sys 64 0x060000 May 18 2003 /dev/vgWP1/group
crw-r--r-- 1 root sys 64 0x070000 Sep 2 15:08 /dev/vgP15_neu/group
crw-r--r-- 1 root sys 64 0x080000 Feb 23 2004 /dev/vgsapput/group
crw-r--r-- 1 root sys 64 0x090000 Jul 12 2002 /dev/vgASEPWEB/group
crw-r--r-- 1 root sys 64 0x0a0000 Sep 30 11:40 /dev/vgP13/group
crw-r--r-- 1 root sys 64 0x0b0000 Jan 16 13:01 /dev/vgPMOL/group
crw-r--r-- 1 root sys 64 0x0c0000 Jan 8 2003 /dev/vgD32P13/group
crw-r--r-- 1 root sys 64 0x0d0000 Jan 8 2003 /dev/vgD33P13/group
crw-r--r-- 1 root sys 64 0x0e0000 Jan 4 2004 /dev/vgD12P11_XP/group
crw-r--r-- 1 root sys 64 0x0f0000 Jan 31 15:29 /dev/vgP16/group
crw-r--r-- 1 root sys 64 0x100000 Jul 30 2004 /dev/vgQ2RS/group
crw-r--r-- 1 root sys 64 0x110000 Jan 4 2004 /dev/vgD13P11_XP/group
crw-r--r-- 1 root sys 64 0x120000 Mar 6 2004 /dev/vgENFINITY/group
crw-r--r-- 1 root sys 64 0x130000 Jan 27 15:50 /dev/vgP12/group
crw-r--r-- 1 root sys 64 0x140000 Feb 19 2003 /dev/vgP08/group
crw-r--r-- 1 root sys 64 0x150000 Feb 27 2003 /dev/vgSBAD/group
crw-r--r-- 1 root sys 64 0x160000 Jan 1 2004 /dev/vgPCV/group
crw-r--r-- 1 root sys 64 0x170000 Jul 30 2004 /dev/vgDBQ2OF/group
crw-r--r-- 1 root sys 64 0x180000 Mar 29 2003 /dev/vgRLN/group
crw-r--r-- 1 root sys 64 0x1a0000 Jan 4 2004 /dev/vgD14P11_XP/group
crw-r--r-- 1 root sys 64 0x1b0000 Jul 12 2002 /dev/vgPCVstripe/group
crw-r--r-- 1 root sys 64 0x1c0000 Jul 12 2002 /dev/vgPCVarch/group
crw-r--r-- 1 root sys 64 0x1d0000 Nov 14 2002 /dev/vgSVSarch/group
crw-r--r-- 1 root sys 64 0x1e0000 Jan 19 16:41 /dev/vgDEVWEB/group
crw-r--r-- 1 root sys 64 0x1f0000 Jul 12 2002 /dev/vgSVSWEB/group
crw-r--r-- 1 root sys 64 0x200000 Jul 12 2002 /dev/vgQDPS/group
crw-r--r-- 1 root sys 64 0x210000 Jan 26 20:45 /dev/vgQDIS/group
crw-r--r-- 1 root sys 64 0x220000 Jan 23 15:59 /dev/vgBWP/group
crw-r--r-- 1 root sys 64 0x230000 Feb 3 17:06 /dev/vgSAP_ROUTER/group
crw-r--r-- 1 root sys 64 0x260000 Jul 12 2002 /dev/vgMAESTRO/group
crw-r--r-- 1 root sys 64 0x270000 Jul 12 2002 /dev/vgRSBHP/group
crw-r--r-- 1 root sys 64 0x280000 Jan 2 2004 /dev/vgQ2ISWEB/group
crw-r--r-- 1 root sys 64 0x290000 Dec 19 2002 /dev/vgOTP2/group
crw-r--r-- 1 root sys 64 0x2a0000 Jul 12 2002 /dev/vgOTP2WEB/group
crw-r--r-- 1 root sys 64 0x2b0000 Feb 7 10:42 /dev/vgPCVWEB/group
crw-r--r-- 1 root sys 64 0x2c0000 Jul 12 2002 /dev/vgPCVbase/group
crw-r--r-- 1 root sys 64 0x2d0000 Jul 12 2002 /dev/vgPMIS/group
crw-r--r-- 1 root sys 64 0x2e0000 Jan 4 2004 /dev/vgWMV/group
crw-r--r-- 1 root sys 64 0x2f0000 Nov 14 2002 /dev/vgSVSstripe/group
crw-r--r-- 1 root sys 64 0x300000 Jul 26 2002 /dev/vgSVSstatspack/group
crw-r--r-- 1 root sys 64 0x320000 Aug 13 2002 /dev/vgQKUSEM/group
crw-r--r-- 1 root sys 64 0x350000 Oct 25 15:41 /dev/vgQKU/group
crw-r--r-- 1 root sys 64 0x360000 Feb 27 2003 /dev/vgSBAD2/group
crw-r--r-- 1 root sys 64 0x370000 Jan 19 16:33 /dev/vgGODOTNFS/group
crw-r--r-- 1 root sys 64 0x380000 Jul 11 2004 /dev/vgSVS/group
crw-r--r-- 1 root sys 64 0x390000 Jun 23 2004 /dev/vgASS/group
crw-r--r-- 1 root sys 64 0x3a0000 Jun 11 2003 /dev/vgGAS/group
crw-r--r-- 1 root sys 64 0x400000 Oct 31 2002 /dev/vgFLATFILE/group
crw-r--r-- 1 root sys 64 0x410000 Mar 10 2003 /dev/vgSBADredo/group
crw-r--r-- 1 root sys 64 0x420000 Mar 10 2003 /dev/vgSBADproduct/group
crw-r--r-- 1 root sys 64 0x430000 Mar 10 2003 /dev/vgSBADoradata/group
crw-r--r-- 1 root sys 64 0x440000 Mar 10 2003 /dev/vgSBADarch/group
crw-r--r-- 1 root sys 64 0x450000 Feb 17 2003 /dev/vgFVZ/group
crw-r--r-- 1 root sys 64 0x460000 Jun 29 2004 /dev/vgP14/group
crw-r--r-- 1 root sys 64 0x470000 Mar 12 2004 /dev/vgP20/group
crw-r--r-- 1 root sys 64 0x480000 Jul 4 2003 /dev/vgQ20/group
crw-r--r-- 1 root sys 64 0x490000 Jul 22 2003 /dev/vgTRACK/group
crw-r--r-- 1 root sys 64 0x4a0000 Nov 21 13:05 /dev/vgP19/group
crw-r--r-- 1 root sys 64 0x4b0000 Sep 19 2003 /dev/vgD02P19/group
crw-r--r-- 1 root sys 64 0x4c0000 Sep 19 2003 /dev/vgD03P19/group
crw-r--r-- 1 root sys 64 0x4d0000 Sep 19 2003 /dev/vgD04P19/group
crw-r--r-- 1 root sys 64 0x4e0000 Mar 10 2004 /dev/vgMARSPX/group
crw-r--r-- 1 root sys 64 0x4f0000 Nov 2 17:22 /dev/vgSBYMARSPX/group
crw-r--r-- 1 root sys 64 0x500000 Mar 10 2004 /dev/vgREGMARSPX/group
crw-r--r-- 1 root sys 64 0x510000 Jan 9 2004 /dev/vgEASYWEB/group
crw-r--r-- 1 root sys 64 0x520000 Dec 22 08:26 /dev/vgEKXML/group
crw-r--r-- 1 root sys 64 0x550000 Jan 15 2004 /dev/vgQDPS_NEU/group
crw-r--r-- 1 root sys 64 0x560000 Jun 9 2004 /dev/vgP07/group
crw-r--r-- 1 root sys 64 0x570000 Jul 22 2004 /dev/vgP06/group
crw-r--r-- 1 root sys 64 0x590000 Nov 17 10:24 /dev/vgASGPX/group

But now the vg is still activated on the node on which we wanted to start the package.
We tried to deactive the vg.
# vgchange -a n /dev/vgPCVWEB
vgchange: Failed to notify clvm daemon about volume group deactivation - No such file or directory
Couldn't deactivate volume group "/dev/vgPCVWEB":
No such file or directory

Any ideas?
Since this package is in a productional environment, we can't reboot the server.

Thanky in advance for your help.

Bye
Elif

[Mark Nieuwboer]

Hi Elif,

Try
vgchange -c n /dev/vgPCVWEB
vgchange -a n /dev/vgPCVWEB

maybe something stupid to ask but did you make the mount point on the second server.
for example /appl/PCVWEB. if not that gives that kind of problem

grtz Mark

[Elif Gius]

Hi,

I've tried # vgchange -c n /dev/vgPCVWEB
vgchange: The volume group "/dev/vgPCVWEB" is active on this system.
Cannot perform requested change.

The Mountpoints are available.
As you can see the message didn't occured as the cluster wanted to mount the FS.
We the cluster activated the vg and then wanted to do a fsck on the Lvols, then follwing error message was displayed "/dev/vgPCVWEB/lvMAESTRO: No such device or address
Feb 5 18:54:23 - Node "estragon": Checking filesystems:
/dev/vgPCVWEB/lvPCVWEB
/dev/vgPCVWEB/lvHOME
/dev/vgPCVWEB/lvTIVOLIep
/dev/vgPCVWEB/lvMAESTRO
fsck: /etc/default/fs is used for determining the file system type
fsck: /etc/default/fs is used for determining the file system type
fsck: /etc/default/fs is used for determining the file system type
fsck: /etc/default/fs is used for determining the file system type
vxfs fsck: Cannot open /dev/vgPCVWEB/rlvPCVWEB: No such device or address
file system check failure, aborting ...
"

[Duncan Edmonstone]

Thats a lot of volume groups... you aren't at or near the kernel parameter maxvgs on this host are you?

HTH

Duncan

[Elif Gius]

No. no kernel limit is reached.
# kmtune |grep maxvg
maxvgs 100 - 100

The package was able to run on this node two weeks ago.

[Henk Geurts]

hi Elif
did you make sure the added disks can be seen from both servers ?
and did the import gave any errors.

you can use
insf -e to make sure device files were created on both nodes.

regards.
Henk

[Elif Gius]

No,
vgimport didn't have any problems.
And I can see the disks from all nodes.

[Duncan Edmonstone]

Did you use a mapfile with the import? What is the output of :

ll /dev/vgPCVWEB

HTH

Duncan

[Elif Gius]

Here is the output:
crw-r--r-- 1 root sys 64 0x2b0000 Feb 7 10:42 group
brw-r----- 1 root sys 64 0x2b0002 Feb 5 18:39 lvHOME
brw-r----- 1 root sys 64 0x2b0004 Feb 5 18:39 lvMAESTRO
brw-r----- 1 root sys 64 0x2b0001 Feb 5 18:39 lvPCVWEB
brw-r----- 1 root sys 64 0x2b0003 Feb 5 18:39 lvTIVOLIep
crw-r----- 1 root sys 64 0x2b0002 Feb 5 18:39 rlvHOME
crw-r----- 1 root sys 64 0x2b0004 Feb 5 18:39 rlvMAESTRO
crw-r----- 1 root sys 64 0x2b0001 Feb 5 18:39 rlvPCVWEB
crw-r----- 1 root sys 64 0x2b0003 Feb 5 18:39 rlvTIVOLIep
Yes, we used a mapfile at the import.

[Duncan Edmonstone]

I assmume you have ITRC access for knowledgebase info - see document UMCSGKBRC00013823 - basically this lists about 6 causes for this message and how to solve them...

Cheers,

Duncan

[Mark Nieuwboer]

Hi Elif,

in order to change the volumegroup first you have to umount the filesystems that make use of the volumegroup. After the umount it must be easy the deactived the volume group with vgchange -a n /dev/vgPCVWEB.

[Elif Gius]

There are no FS mounted on the node I want to deactivate the volume group.

[Geoff Wild]

You can't vgchange if it is held by a process - my guess is fsck has hung - do a ps -ef |grep fsck.

If yes - your only option (sigh) will be to reboot that box...

Rgds...Geoff

[Elif Gius]

there is no fsck in the process list

[Henk Geurts]

Elif

you wrote that vgchange -c n vgname
went wrong ...
did you also try
vgchange -a n vgname ?

[Geoff Wild]

Okay - try this on the node you want to deactivate the vg:

cmhaltnode -f -v NODENAME

WARNING - that will shut down all packages running on NODENAME!!!

Rgds..Geoff

[Stephen Doud]

This document discusses various causes and fixes for "Failed to notify clvm daemon about volume group deactivation "

UMCSGKBRC00013823

If you have a support contract, login to the ITRC, search Technical knowledge documents, using the document ID method.

-sd

[Steve Lewis]

It looks like you forgot to copy your volume group directory over again.

As well as vgexport/vgimport with a map file, I always find that it really helps to do this. Starting on the primary node:

tar cvf /tmp/vgNNNNN.tar /dev/vgNNNNN
rcp /tmp/vgNNNNN.tar othernode:/tmp
rlogin othernode
tar xvf /tmp/vgNNNNN.tar

untar it BEFORE you do the vgimport.

by default vgimport creates default logical volume names lv01, lv02...
You may have lvNN... instead of /dev/vgPCVWEB/lvPCVWEB etc in your volume group directory.