unable to halt cluster

[Samer Dahr]

Hello,
when trying to halt my service guard cluster installed on HP-UX 11.0 , im getting the following error:

root@ab-dwh1:/etc/cmcluster# cmhaltcl
Disabling all packages from starting on nodes to be halted.
Package clstwrpkg is already disabled on node ab-dwh1
Package racinstpkg is already disabled on node ab-dwh1
Package datawhpkg is already disabled on node ab-dwh1
Package clstwrpkg is already disabled on node ab-dwh2
Package racinstpkg is already disabled on node ab-dwh2
Package datawhpkg is already disabled on node ab-dwh2
Warning: Do not modify or enable packages until the halt operation is completed.
The group member is still running on ab-dwh1.
Use the -f option to forcefully halt the cluster/node including halting group members.


and when trying to use the -f option , im getting the following:

root@ab-dwh1:/etc/cmcluster# cmhaltcl -f
Disabling all packages from starting on nodes to be halted.
Package clstwrpkg is already disabled on node ab-dwh1
Package racinstpkg is already disabled on node ab-dwh1
Package datawhpkg is already disabled on node ab-dwh1
Package clstwrpkg is already disabled on node ab-dwh2
Package racinstpkg is already disabled on node ab-dwh2
Package datawhpkg is already disabled on node ab-dwh2
Warning: Do not modify or enable packages until the halt operation is completed.
This operation may take some time.
Unable to halt the LVM daemon on ab-dwh2.
Clustered Volume Groups must be deactivated before the cluster can be halted.
Check the syslog file for more detailed information.

does anyone have any idea on how to solve such a problem ?

thanks in advance

Samer

[Steven E. Protter]

Shalom,

Key:
Clustered Volume Groups must be deactivated before the cluster can be halted.

Your cluster configuration is not properly deactivating shared volume groups.

Go get the cluster to stop you will need to do this manually.

vgchange -y n vg##

use a real volume group name not vg##

You may need to look at the man page for vgchange and run other commands relavent to shared volume groups.

Then you will need to check your cluster configuration and make sure the vg activate and deactivate commands are of course correct in the package control scripts.

A common problem is when two packages need a volume group in a certain status and they run in reverse order. There is no way you can control the order packages are run so you may need to combine to packages to 1 to avoid this scenario.

See cluster logs and /var/adm/syslog/syslog.log for further information.

SEP

[Samer Dahr]

Hi Steven,

thanks for your answer, but when trying to halt a node from the cluster i am getting the same error, even if it is not the node that running the shared volume groups.
please note that the cluster configuration is fine, because everything was working fine before.

best regards
Samer

[Stephen Doud]

Use cmviewcl to determine the status of packages - and note if any are in NO_RESTART state.
Use a simple 'vgdisplay' on both nodes to see if any clustered VGs are still active. If so, this would explain why the cmlvmd daemon cannot shut down. Manually attempt to deactivate the VG using
# vgchange -a n
and report the result here.
If a VG cannot be activated, use 'fuser' to check the mount points of each volume in that VG. They must be umounted before deactivating the VG.
If file systems cannot be umounted, it may be necessary to remove /etc/mnttab and recreate it using 'mount'.

Please report your findings in more detail here.

[Sajjad Sahir]

DearSamer

first u have to deactivate the volume group

vgchange -a n
then if is not halting
use

cmhaltcl -v -f

if it is againg not halting properly
u reboot the system
I think then everything will work fine

use cmviewcl command again

[Mridul Shrivastava]

thanks for your answer, but when trying to halt a node from the cluster i am getting the same error, even if it is not the node that running the shared volume groups.

----> Does it mean that shareg VGs are active on other node and you are executing cmhaltcl on different node ??

Let me clear one thing here , In cluster environemtn it hardly matters on which node those shared vgs are active but if they are active then they should be deactivated before halting the cluster else you will get the error messages... which is normal.
So please deactivate all the VGs manually which are part of the cluster and still active then execute cmhaltcl.

[Michael Steele_2]

Hi Samer:

There are some MC/ServiceGuard utilities that can analyze your package.

cmcheckconf -v -C /etc/cmcluster/####.ascii -P /etc/cmcluster/****/###.conf -P /etc/cmcluster/***/###.cntl

# cmgetconf -C /tmp/temp.ascii
-or-
# cmgetconf -v -c mycluster temp.ascii

cmscancl -n node1 -n node2 -o output.file

cmreadlog /var/opt/cmon/cmomd.g -or- /var/opt/sqmgr/#####.sqmgr.log

Also check OLDsyslog.log and syslog.log