MCSG package not starting after reboot

[Leoanrdo Bowens]

I have a 2 node cluster with a package on each node. The primary node is set to failover to secondary. The secondary node is not set to failover to any node. My problem is that when the secondary node reboots the package will not rejoin the cluster when the server is back online. I modified and checked all pkg.conf settings and all appears to be ok. What else could prevent the pkg from starting up or have I overlooked something in the conf file or another file? The pkg log file does not log what happened.

[Bernhard Mueller]

As long as the cluster remains running on at least one node and a package is down it will remain down until you run cmrunpkg or cmmodpkg -e on a suitable node.

Only if both nodes reboot at the same time and the cluster itself restarts then all packages will attempt to start on their default nodes (assuming AUTO_RUN is set).

Regards,
Bernhard

[Sundar_7]

Is it just the package that is not starting up or the node itself is not joining the cluster ?

You should have AUTOSTART_CMCLD=1 in /etc/rc.config.d/cmcluster file for the node to automatically join the cluster at startup.

Once the server is back online, execute cmviewcl and confirm the node is part of the cluster.

The package will not start automatically if AUTO_RUN is set to false

# cd /etc/cmcluster/

# cmgetconf -v -p .conf

# grep AUTO_RUN .conf

AUTO_RUN should be YES.

If not, you need to modify the package configuration file and apply again.

Let us know if this helps.

[Jose Mosquera]

Hi,

Pls check with "cmviewcl -v" your AUTO_RUN status, this must be enabled. Also check the SWITCHING nodes status, these must be enabled too. Pls take that AUTO_RUN status always override the SWITCHING values.

To change them on-line:
#cmmodpkg -e (enable AUTO_RUN)
#cmmodpkg -e -n (enable SWITCHING nodes)

To make a permanent change to AUTO_RUN parameter you need stop the package, go to /etc/cmcluster// directory and modify the .conf file, after this create the binary file:
#cmcheckconf -v -P .conf
#cmapplycong -v -P .conf

If the compilation is free of errors start the pkg again:
#cmrunpkg

Rgds.

[Leoanrdo Bowens]

I had already done all that has been suggested and the package will not join the cluster automatically after a reboot of the one server. One technician has said the package will not startup unless both servers reboot. It that a true statement?

[Geoff Wild]

As others have said, check AUTO_RUN...

Now my 2 cents - I do NOT set AUTO_RUN to yes nor have the cluster start automatically on a reboot...Reason, what it there was something terribly wrong - with disks or network or what have you - and you start your cluster - maybe you make things worse - like corrupt data - cause the other node is already running - yet the 2 servers can't communicate - a race for the cluster lock - and wammo - the box you rebooted wins - oracle comes crashing down as the other node TOC's....

I always, always set my clusters to manual startup....just in case....

Rgds...Geoff

[Sundar_7]

Nope, that is not true. You dont have to reboot both the servers for the package to start.

Look for messages from the cluster in /var/adm/syslog/syslgo.log file

Post the output of cmviewcl -v command

Once the node back is online, are you able to successfully start the package using cmrunpkg command ?

[Leoanrdo Bowens]

Sundar, I know I can start the package from command line but, what I want to happen is for the package to startup automatically after each reboot on the one node. Remember, this node is not set to failover. Should a just create a startup script in /sbin/rc3.d to run (cmrunpkg pkgname)?

[Sundar_7]

There should be no need to create a script in the sequencer directories.

Looks like you are not ready to post the output from the system. It is difficult to troubleshoot the problem only based on your interpretation of the output :-).

[John Poff]

One other thought is to check the NODE_NAME setting in the package config file, and make sure it is set to the secondary node name. I'm guessing that if that value was wrong you wouldn't even be able to start the package manually, but I haven't tried it before [not yet, anyway :) ].

I agree with Sundar. If you can post the output of your 'cmviewcl -v' it might give us a clue about what isn't working.

JP

[Leoanrdo Bowens]

Here is the output of cmviewcl -v:


CLUSTER STATUS
NorcastTM1 up

NODE STATUS STATE
zrtph0s0 up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
PRIMARY up 0/3/1/0 lan1
PRIMARY up 0/4/1/0 lan2

PACKAGE STATUS STATE AUTO_RUN NODE
norprod up running enabled zrtph0s0

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Subnet up 47.140.144.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled zrtph0s0 (current)
Alternate up enabled zrtph0s1

NODE STATUS STATE
zrtph0s1 up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
PRIMARY up 0/3/1/0 lan1
PRIMARY up 0/4/1/0 lan2

PACKAGE STATUS STATE AUTO_RUN NODE
nortest up running enabled zrtph0s1

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback automatic

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Subnet up 47.140.144.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled zrtph0s1 (current)

[John Poff]

Thanks for posting the output. I've looked at it and I don't see anything strange.

Let me rehash your problem to make sure I understand it. Your cluster is up and running, with both nodes in the cluster and the correct package running on each node. Then, the second node (zrtph0s1) crashes/reboots. This node rejoins the cluster, but the 'nortest' package doesn't make any attempt to startup, and there is no evidence of it trying to startup in either the package log file or syslog. Is that it?

JP

[Leoanrdo Bowens]

That is it. The node will rejoin the cluster but the package will not. I can start it from command line without a problem.

[Sridhar Bhaskarla]

I was too busy today to participate in forums today. I just had a glance and this thread attracted me. So, I can't help myself posting.

I have always believed that the packages won't start (not failback) automatically unless it is a cluster startup. Your case is not a cluster startup. So, I am not surprized to see this behaviour.

Four years back on a three node cluster with three packages on each node, I used to halt the packages on two nodes but keep the third node up and running, do the maintenance and then start the both nodes, halt the third node and do the maintenance on it. I did have PKG_SWITCHING_ENABLED (now AUTO_RUN) on those nodes. It never happened to me that the packages would start automatically when the node came up. This was just to avoid automatic startup of all packages when I brought up the cluster after maintenance as there were too many dependencies. Later, I changed the configuration so I didn't have to do that way ofcourse.

-Sri

[Sundar_7]

hmm..interesting post Sri :-).

I believe the author is talking about regular reboot (when the package was not manually halted before shutting down the system).

I have few 2 node clusters and some 1 week back I had to reboot one node when the package was running, the package got started automatically when the machine came back.

# who -b
. system boot Sep 7 14:22
#

# grep cmcld syslog.log.2004-09-07-dyTue
.....
Sep 7 14:25:49 machinname cmcld: Attempting to form a new cluster
Sep 7 14:25:52 machinname cmcld: Turning on safety time protection
Sep 7 14:25:52 machinname cmcld: 2 nodes have formed a new cluster, sequence #17
Sep 7 14:25:52 machinname cmcld: The new active cluster membership is: primary(id=1), machinname(id=2)
Sep 7 14:25:52 machinname cmcld: Executing '/etc/cmcluster/oracle_ocnux/oracle_ocnux.cntl start' for package
oracle_ocnux, as service PKG*39173.
Sep 7 14:25:55 machinname cmcld: Service PKG*39173 terminated due to an exit(0).
Sep 7 14:25:55 machinname cmcld: Started package oracle_ocnux on node machinname.
...
#

[John Poff]

Hi Sri,

Good point. I've been digging in the manuals, and here is the definition for the AUTO_RUN parameter [from the Managing MC/ServiceGuard manual]:

http://docs.hp.com/hpux/onlinedocs/B3936-90065/B3936-90065.html


"The AUTO_RUN parameter (known in earlier versions of ServiceGuard as the PKG_SWITCHING_ENABLED parameter) defines the default global switching attribute for the package at cluster startup, that is, whether the package should be restarted automatically on a new node in response to a failure, and whether it should be started automatically when the cluster is started."

So, according to the manual the AUTO_RUN parameter only starts the package at cluster startup, which is what Sri has observed before also.

On the other hand, the FAILBACK_POLICY parameter is set to 'automatic' for the 'nortest' package, so it seems like it should start the package, but maybe I'm misunderstanding how that parameter works. Here is the definition from the manual:

"The use of the FAILBACK_POLICY parameter allows you to decide whether a package will return to its primary node if the primary node becomes available and the package is not currently running on the primary node."

When the second node joins the cluster, it would seem like the FAILBACK_POLICY parameter setting for 'automatic' would start the package for that node, but maybe I've got it wrong.

Where is a real MC/SG wizard like Melvyn when we need him? :)

JP

[Sundar_7]

John,

The package never FAILED OVER to some other node to FAIL BACK to the primary node :-).

There is no adoptive node defined for the package.

I have a 2 node cluster, with each node running a package. I had to reboot one node in the cluster. I rebooted the node when the package was running.This package doesnt have any adoptive nodes defined.

When the node came back online, the package got automatically started.

Even if AUTO_RUN is set, I believe the package will not be started, ONLY when the package was halted manually or failed by the package manager.

It is highly possible that I got it wrong too :-)

- Sundar.

[Jose Mosquera]

Hi,

Please analyze your /etc/cmcluster//.cntl.log file. May be an important hint is showed inside.

Rgds.

[Bernhard Mueller]

I suggest you re-read my first post and then
take a look at /sbin/init.d/cmcluster.

If you have a package configured to run on a single node only and want to start this package upon reboot of the node irrespectively of what the other nodes state is, you may use a conventional run script as you mentioned or put something in /etc/cmcluster/rc. See above.

Regards
Bernhard

[Leoanrdo Bowens]

Thanks for all the suggestions. The client has decided to have the package removed from the HA environment and have it mounted as a standalone volume group. I simply deleted the package with cmdeleteconf, vgchange -c n the volume group, updated fstab and etc.