clusternode won't join cluster anymore

[Paul Ettema]

After reboot linux cluster node won't join cluster anymore.

CLUSTER STATUS
emxteamspreprodcl up

NODE STATUS STATE
ln2004 down unknown
ln2005 up running


ln2004:~ # cmruncl -v -n ln2004
cmruncl: Validating network configuration...
cmruncl: Network validation complete

WARNING:
Performing this task overrides the data integrity protection normally
provided by Serviceguard. You must be certain that no package applications
or resources are running on the other nodes in the cluster:
ln2005

To ensure this, these nodes should be rebooted (i.e. /usr/sbin/shutdown -r)
before proceeding.

Are you sure you want to continue (y/[n])? y

Waiting for nodes to join .............. timed out
Check the syslog files for information.
cmruncl failed: timed out waiting for cluster to form


found next in /var/log/messages:

Jun 30 08:50:12 ln2004 cmruncl: cmruncl -v -n ln2004
Jun 30 08:50:15 ln2004 cmclconfd[14842]: Request from root on node ln2004 to start the cluster on this node
Jun 30 08:50:15 ln2004 cmclconfd[14842]: The Serviceguard daemon, cmcld[14877], exited with a status of 127.

What is going wrong ?

Paul.

[AnthonySN]

give the command
cmrunnode on ln2004

[Paul Ettema]

@SASJ

:-(

ln2004:/opt/cmcluster/conf # cmrunnode
cmrunnode: Unable to communicate with a running cluster or with all nodes in the cluster.
cmrunnode: In order to use cmrunnode, the cluster must already be running on a subset of reachable nodes or else all cluster nodes must be reachable.
cmrunnode: Issuing cmrunnode again may succeed.
ln2004:/opt/cmcluster/conf # cmrunnode
cmrunnode: Unable to communicate with a running cluster or with all nodes in the cluster.
cmrunnode: In order to use cmrunnode, the cluster must already be running on a subset of reachable nodes or else all cluster nodes must be reachable.
cmrunnode: Issuing cmrunnode again may succeed.
ln2004:/opt/cmcluster/conf # cmviewcl

CLUSTER STATUS
emxteamspreprodcl unknown

NODE STATUS STATE
ln2004 down unknown
ln2005 unknown unknown

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
emxteamspreprod down halted enabled unowned
emxteamstraining down halted enabled unowned
mxrouterprep down halted enabled unowned
ln2004:/opt/cmcluster/conf #

[Modris Bremze]

Looks like the node cannot communicate with the cluster. Try cmquerycl -v.
Did you apply any updates/patches or changed configuration files (e.g. hosts or inetd) before the reboot?

[AnthonySN]

in ur previos post cluster was up
CLUSTER STATUS
emxteamspreprodcl up

NODE STATUS STATE
ln2004 down unknown
ln2005 up running

but now cluster is down hence node is not joining
run cmruncl on node ln2005
else
cmruncl -v -n node ln2005
and then on node ln2004
give
cmrunnode

[Paul Ettema]

@ Modris Bremze

there were no updates, between last good working and failing working, because:

I rebooted the server and all was ok, 20 min later (no changes) I rebooted again and then failing (still failing)

@ SASJ
Yes, first post cluster up second post cluster down.
a several reboots and test between posts.

server ln2005 is now up (cmruncl -n ln2005)



So @ all the status and "try" of Moris
---------------------------------------

ln2004:/opt/cmcluster/conf # cmviewcl

CLUSTER STATUS
emxteamspreprodcl up

NODE STATUS STATE
ln2004 down unknown
ln2005 up running

PACKAGE STATUS STATE AUTO_RUN NODE
emxteamspreprod up running enabled ln2005

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
emxteamstraining down halted disabled unowned
mxrouterprep down halted disabled unowned


ln2004:/opt/cmcluster/conf # cmquerycl -v
Looking for Serviceguard nodes ... Done

Cluster Name Node Name Version Status
emxteamspreprodcl
ln2004 out of date
ln2005 out of date

[AnthonySN]

you can give cmruncl instead of
cmruncl -n ln2004

also post your cluster log from /etc/cmcluster/(pkg name)/(pkgname.log)
and syslog.

[Paul Ettema]

@ SASJ

Sorry, SASJ, I have already try "cmruncl" but no solution.

And package log are not useful now, while the running on other node now, so no problem with packages.
Only a problem with 1 node

Paul.

[AnthonySN]

what is the error you get when you give the cmrunnode command on ln2004 when the cluster is up and running on node ln2005.

post ur messages log from ln2004.

[Steven E. Protter]

Shalom Paul,

Some essential part of clustering may have gone off line. Perhaps the heartbeat network, or issues with the interface of hub.

I recommend rebooting the recalcitrant mode and taking a look at all the logs to get a cue as to what the error is.

It is in there, somewhere.

Also, check lights on networking for the machine and any hardware the cluster depends on.

SEP

[Paul Ettema]

@ SASJ


error messages/output is the same as in my 1st and 2nd post. No differences when ln2005 is run or not run cluster.

[S.N.S]

Hi Paul,

May be time to contact HP Response Center - the last time I had a problem with cnruncl -n...it got solved by itself - ie after reboot...

Thankfully, it started.....But that Prod Server wasn't patched in 3yrs....

Just consider all the basic cluster trouble shooting steps -something fundamental you might have missed out -before you contact the HP biggies.

SNS.