clustering in HP-UX

[Sridhar R]

Hi,

I have a 3 node HP-UX (11.23) cluster having service guard installed.

I have a bunch of questions for which i expect good answers from clustering experts:

1. If i need to down all the 3 nodes, should i execute cmhaltcl, which will stop the package running, before i execute the shutdown command?

2. If my requirement is to down one single node out of the 3, then should i execute the cmhaltnode before shutdown command?

3. What would be the impact if i bring down the server without halting the package?

4. If i bring down the server without halting the cluster, i have to issue cmruncl or cmrunnode to make the package up and running, after booting up the server.

My question here is, can anything be done so that the cluster packages comes up automatically (after booting up the server) without any necessity to run these commands?



Thanks in Advance!!

Sridhar

[Duncan Edmonstone]

Sridhar,

1. To halt all 3 nodes at the same time, yes you should run cmhaltcl before executing the shutdown command. However if you haven't stopped all the packages already a straight 'cmhaltcl' would fail, you would either need to run 'cmhaltpkg' on each package first, or run 'cmhaltcl -f'.

2,3,4. To answer all these questions together - note that all this behaviour depends on whether you have AUTOSTART_CMCLD set to 1 in the file /etc/rc.config.d/cmcluster. If this parameter is set to 1 then cmhaltnode/cmrunnode is called on shutdown/startup of a node. So you can choose to just execute a shutdown and let the shutdown script handle this, or you can manage it yourself with commands first - its your choice. . The one thing to note is that to bring up the cluster when all 3 nodes have been rebooted all nodes must be present, and cmrunnode will only wait 10 minutes when trying to form a cluster, so if there's more than 10 minutes difference in the startup time of the 3 nodes the cluster can fail to form. You can of course just run cmruncl once all 3 nodes are up though:

http://docs.hp.com/en/B3936-90117/ch03s02.html#d0e2572

Packages will start automatically when a cluster forms if the packages in it have the auto_run flag set to yes:

http://docs.hp.com/en/B3936-90117/ch03s04.html#d0e3695

HTH

Duncan

[Sajjad Sahir]

Dear Sridhar

before shutdown u have to halt the cluster
if u want to halt in all three nodes
u can use cmhaltcl command
if u want to halt nodes one by one
cmhaltnode
ok.
after that only do shutdown
ok

[Sajjad Sahir]

My question here is, can anything be done so that the cluster packages comes up automatically (after booting up the server) without any necessity to run these commands

yes
u have to wait for that

u can use cmviewcl command
then u can see all nodes are up or not

[Steven E. Protter]

Shalom Sridar,

1) cmhaltcl will halt the packages anyway.
2) Two or more ways to do manyu things. cmhaltnode will shut down the package as will the package stop script.
3) SG is designed to deal with TOC, so even if you pull power, a properly configured SG cluster should continue to run and fail pakages to other nodes.
4) You should not need to. See configuration of autostart in /etc/rc.config.d/cmcluster

SEP

[Sridhar R]

Hi Duncan,

Thanks for the answer!

I found that AUTOSTART_CMCLD has been set to 0 in the /etc/rc.config.d/cmcluster file.

So, changing it to 1 should make the cluster up automatically after reboot, right??

Yesterday, we had a issue in one of the nodes, we wanted to replace a HBA Card and henceforth i issued a cmhaltnode -f which failed as follows:

# cmhaltnode -f
Disabling all packages from starting on nodes to be halted.
Warning: Do not modify or enable packages until the halt operation is completed.
Halting package cbadbpkg on node CBADB101
The package script for cbadbpkg failed with no restart. cbadbpkg should not be restarted
Unable to halt package cbadbpkg on node CBADB101
Check the syslog and pkg log files for more detailed information

I didn't issue a cmhaltpkg before running this! Is it the reason why this has failed??

Thanks!

[Duncan Edmonstone]

Sridhar,

Nope thats not the reason - doing a cmhaltnode -f causes all packages on the current node to be halted before taking the node out of the cluster. The package halt failed for some other reason - you'll have to look in the packages log file to determine what reason that was. Determine the name of the package log script by running:

cmgetconf -p cbadbpkg | grep HALT_SCRIPT

Append ".log" to the name of the halt script and that will give you the name of the logfile to look in.


HTH

Duncan

[Sridhar R]

Hi Duncan,

i can find the pkg log files existing in /etc/cmcluster/pkg directory as:

CBADB103$pwd
/etc/cmcluster/pkg

CBADB103$ls -ltr
total 544
-rwxrwxrwx 1 root sys 56762 Jul 18 2007 cbadbpkg.cntl
-rwxrwxrwx 1 root sys 16371 Jul 19 2007 cbadbpkg.conf
-rw-r--r-- 1 root root 202277 Mar 3 13:10 cbadbpkg.cntl.log


I can find in the cbadbpkg.cntl file as follows:

# The volume group activation method is defined above. The filesystems
# associated with these volume groups are specified below.
#
#VG[0]="votevg01"
#VG[1]="ocrvg"
VG[0]="votevg01"
VG[1]="ocrvg"
VG[2]="erpdatavg01"
VG[3]="erpdatavg02"
VG[4]="erpdatavg03"
VG[5]="erpdatavg04"
VG[6]="crmdatavg01"
VG[7]="crmdatavg02"
VG[8]="crmdatavg03"
VG[9]="crmdatavg04"
VG[10]="dmsdatavg01"
VG[11]="dmsdatavg02"
VG[12]="bscdatavg01"
VG[13]="hrdatavg01"
VG[14]="hrdatavg02"
VG[15]="hrdatavg03"
VG[16]="hrdatavg04"
VG[17]="hrdatavg05"

I found that all these were not having the # mark infront of them, and also surprised to get the below message while rebooting:



vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/votevg01".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/ocrvg".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/erpdatavg01".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/erpdatavg02".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/erpdatavg03".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/erpdatavg04".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/crmdatavg01".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/crmdatavg02".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/crmdatavg03".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/crmdatavg04".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/dmsdatavg01".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/dmsdatavg02".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/bscdatavg01".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/hrdatavg01".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/hrdatavg02".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/hrdatavg03".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/hrdatavg04".
vgchange: Cluster Services not available or unreachable on this system, for volume group "/dev/hrdatavg05".



(Note: The above message came up in the MP -> CO (console) , while we rebooted the server)

This is the same case with all the 3 servers.

I am afraid whether there should be a # mark in front of them??

Is that what making the problem with cmhaltnode -f as well.

Suggest me if you have any clue!!

Thanks!!

[Mridul Shrivastava]

As far as those # marks are concerned they are not required in front of all VGs, # mark is required if that VG is no more part of this package so whenever this package starts it will not try to activate the VG.

Second, you need to chcke the cntl.log for the cause of package halt failing not the console log.

[Duncan Edmonstone]

Sridhar,

This is the file you should be looking at:

cbadbpkg.cntl.log

The messages you are seeing on the console are a result of the package not halting correctly.

Here's what happened:

1. You called cmhaltnode -f

2. The cmhaltnode attempted to shut the package down - that didn't work for some reason. The volume groups in the package weren't deactivated (maybe the filesystems were bust and couldn't be unmounted?)

3. cmhaltnode continued and stopped cluster services on the system

4. Now when you tray and shutdown the server, as part of the shutdown all the active volume groups are closed. However the volume groups listed are cluster volume groups and can only be deactivated when the cluster daemons are up, hence the messages.

So everything else you are seeing is a symptom of your initial problem - which is that the package isn't halting cleanly. You need to look in the log file I mentioned to determine why this is.

HTH

Duncan

[Siju Vadakkan]

To analyse the package failure, please provide the following log file.

cbadbpkg.cntl.log

1.If i need to down all the 3 nodes, should i run cmhaltcl, which will stop the package running, before i run the shutdown command?
Yes. Cmhaltcl â f - will stop all packages and then it will halt the cluster as well.

2.If my requirement is to down one single node out of the 3, then should i run the cmhaltnode before shutdown command?
cmhaltnode , command will halt the node and the packages with switching enabled will switch to the adoptive node and the packages with switching disbled will not switch to the adoptive


3. What would be the impact if i bring down the server without halting the package?
If u bring down the server without halting the cluster, , packages with switching enabled will switch to the adoptive node and the packages with switching disbled will not switch to the adoptive

4. If i bring down the server without halting the cluster, i have to issue cmruncl or cmrunnode to make the package up and running, after booting up the server.
a) If /etc/rc.config.d/cmcluster is updated with AUTOSTART_CMCLD=1 , then the node will join the cluster automatically as soon as the server comes up.

Packages with AUTO_RUN set to YES in the package configuration file will start automaticall when u start the cluster
And the packages with AUTO_RUN set to NO, u have to start manually.

b) if AUTOSTART_CMCLD =0 , To join the node to the cluster , you have to issue cmrunnode.
Packages with AUTO_RUN set to YES in the package configuration file will start automaticall when u start the cluster
And the packages with AUTO_RUN set to NO, u have to start manually.


My question here is, can anything be done so that the cluster packages comes up automatically (after booting up the server) without any necessity to run these commands?
U have to update the cluster package configuration file with AUTO_RUN parameter to YES to start the package automatically during the cluster startup.
If donâ t want to set AUTO_RUN to YES, but still u want to enable switching during the cluster node crash , you can enable using â cmmodpkg â e . this will enable packaging switching whenever cluster node crashes.

[Sridhar R]

Hi,

Thanks for the replies.

I have attached the requested file. Kindly get back to me in case you get any clue!!

Thanks!