Package does not restart after reboot

[Michael Elleby III]

I have a 2 node cluster, where one of the nodes rebooted and looks like it rejoined the cluster, but the package did not restart. I do not see any notes in the control log, nor the syslog to tell me what actually did or did not happen.

The second node had its cron entry for reboot commented out.

Any suggestions?

[melvyn burnard]

what does cmviewcl show?
and which syslog.log did you check?
and the pkg logs?

Is the pkg configured to autostart? and switch between nodes?
also do a cmscancl and check for the settings in teh binary file.
finally, did the node rejoin the cluster?

[Sridhar Bhaskarla]

If you bring the entire cluster down and then start it, you will get the package up and running if the AUTO_RUN variable (for Service Guard version 11.09 above) is set to YES in your package configuration file.

I don't think your package will automatically start if a node fails and rejoins the cluster that is already up and running.

-Sri

[Michael Elleby III]

Melvyn-

A. cmviewcl shows both nodes up and both packages started, but pkg_switch is disabled.

B. I checked the syslog on both nodes in the cluster and I do not see anything outside of the halt when the server was coming down on the reboot and when I manually issued the cmrunpkg to start the halted package.

C. I checked the pkg log on the node where the pkg was halted. nothing to note...

D. I did not note the AUTO_RUN variable in either the .conf or the .cntl files. Currently, pkg_switch is disabled on this node. we are running version A.11.09

Mike E.

[Michael Elleby III]

Sri-

Well, I know we are unable to bring down the cluster at this point in time, but I see what you are saying. It's funny, I cannot locate the AUTO_RUN variable in the pkg conf. file.

You're right on your second note... Since it was a node failure, but since pkg_switch was disabled on both nodes, the package had nowhere to go..

Mike

[melvyn burnard]

Well without knowing the configuration of teh cluster/pkgs it ios not easy to give an answer here, but hte pkg should have swiched to the standby node if configured todo so and it was enabled for switching.
the fact that you state that switching is disabled for both nodes makesme think that is the reason your pkg did not start again.
The AUTO_RUN parameter is new as of 11.09, but will ONLY be in your pkg config files if you have created new version and merged the old ones into it.
The variable you should be looking for in old parlance is
PKG_SWITCHING_ENABLED

[Michael Elleby III]

Melvyn, I believe you are in the arena where I was thinking.. Since PKG_Switching is disabled, there was no node to fail the first pkg to since this primary node, and this was the key, since normally, when the first node went down, it should have 'failed over' to the other node.., and to add that the package won't restart once it re-joins the cluster, since we do not have the 'failback policy' set to automatic.

[melvyn burnard]

sounds about right then ;-}

You should make some checks to confirm all is ok for the pkg on both nodes, and then enable the switching by using the cmmodpkg command.
don't forget to ensure thaht bot GLOBAL and LOCAL switching are enabled , i.e. cmmodpkg -e pkgname for GLOBAL,
and cmmodpkg -e -n hostname pkgname for LOCAL.

[Michael Elleby III]

You would be correct, but, in our environment, we reboot our servers every Sunday. The point here is, only one rebooted and the other didn't (cron entry commented out). I could go ahead and re-enable pkg_switch, but then, if one of the nodes goes down for non-critical reasons (i.e. scheduled reboot), then the other node would pick up the pkg and that's not what we want.

Any suggestions?

[melvyn burnard]

Other than to ask why you reboot every Sunday, not much can be suggested, exceptto perhaps write a shutdown script that first does a cmmodpkg -d pkgname, before shutting down.
But again, as SG is for High Availability, why reboot every Sunday??

[Ricardo Bassoi]

1- Please check if the Autostart is enable:
# cd /etc/rc.config.d/
change the file cmcluster:
# vi cmcluster
change:
AUTOSTART_CMCLD=1 ( Must be 1 to autostart )

Please do this on Both Nodes in your system.

2 - Please check if your package script is correct.
Located on the path:
# cd /etc/cmcluster
Check your package script.