topic Re: clusternode won't join cluster anymore in Operating System - Linux

clusternode won't join cluster anymore

Paul Ettema — Wed, 30 Jun 2010 06:48:09 GMT

After reboot linux cluster node won't join cluster anymore.

CLUSTER STATUS
emxteamspreprodcl up

NODE STATUS STATE
ln2004 down unknown
ln2005 up running

ln2004:~ # cmruncl -v -n ln2004
cmruncl: Validating network configuration...
cmruncl: Network validation complete

WARNING:
Performing this task overrides the data integrity protection normally
provided by Serviceguard. You must be certain that no package applications
or resources are running on the other nodes in the cluster:
ln2005

To ensure this, these nodes should be rebooted (i.e. /usr/sbin/shutdown -r)
before proceeding.

Are you sure you want to continue (y/[n])? y

Waiting for nodes to join .............. timed out
Check the syslog files for information.
cmruncl failed: timed out waiting for cluster to form

found next in /var/log/messages:

Jun 30 08:50:12 ln2004 cmruncl: cmruncl -v -n ln2004
Jun 30 08:50:15 ln2004 cmclconfd[14842]: Request from root on node ln2004 to start the cluster on this node
Jun 30 08:50:15 ln2004 cmclconfd[14842]: The Serviceguard daemon, cmcld[14877], exited with a status of 127.

What is going wrong ?

Paul.

Re: clusternode won't join cluster anymore

AnthonySN — Wed, 30 Jun 2010 09:36:09 GMT

give the command
cmrunnode on ln2004

Re: clusternode won't join cluster anymore

Paul Ettema — Wed, 30 Jun 2010 10:02:25 GMT

@SASJ

:-(

ln2004:/opt/cmcluster/conf # cmrunnode
cmrunnode: Unable to communicate with a running cluster or with all nodes in the cluster.
cmrunnode: In order to use cmrunnode, the cluster must already be running on a subset of reachable nodes or else all cluster nodes must be reachable.
cmrunnode: Issuing cmrunnode again may succeed.
ln2004:/opt/cmcluster/conf # cmrunnode
cmrunnode: Unable to communicate with a running cluster or with all nodes in the cluster.
cmrunnode: In order to use cmrunnode, the cluster must already be running on a subset of reachable nodes or else all cluster nodes must be reachable.
cmrunnode: Issuing cmrunnode again may succeed.
ln2004:/opt/cmcluster/conf # cmviewcl

CLUSTER STATUS
emxteamspreprodcl unknown

NODE STATUS STATE
ln2004 down unknown
ln2005 unknown unknown

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
emxteamspreprod down halted enabled unowned
emxteamstraining down halted enabled unowned
mxrouterprep down halted enabled unowned
ln2004:/opt/cmcluster/conf #

Re: clusternode won't join cluster anymore

Modris Bremze — Wed, 30 Jun 2010 12:55:05 GMT

Looks like the node cannot communicate with the cluster. Try cmquerycl -v.
Did you apply any updates/patches or changed configuration files (e.g. hosts or inetd) before the reboot?

Re: clusternode won't join cluster anymore

AnthonySN — Wed, 30 Jun 2010 13:06:48 GMT

in ur previos post cluster was up
CLUSTER STATUS
emxteamspreprodcl up

NODE STATUS STATE
ln2004 down unknown
ln2005 up running

but now cluster is down hence node is not joining
run cmruncl on node ln2005
else
cmruncl -v -n node ln2005
and then on node ln2004
give
cmrunnode

Re: clusternode won't join cluster anymore

Paul Ettema — Fri, 02 Jul 2010 06:19:18 GMT

@ Modris Bremze

there were no updates, between last good working and failing working, because:

I rebooted the server and all was ok, 20 min later (no changes) I rebooted again and then failing (still failing)

@ SASJ
Yes, first post cluster up second post cluster down.
a several reboots and test between posts.

server ln2005 is now up (cmruncl -n ln2005)

So @ all the status and "try" of Moris
---------------------------------------

ln2004:/opt/cmcluster/conf # cmviewcl

CLUSTER STATUS
emxteamspreprodcl up

NODE STATUS STATE
ln2004 down unknown
ln2005 up running

PACKAGE STATUS STATE AUTO_RUN NODE
emxteamspreprod up running enabled ln2005

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
emxteamstraining down halted disabled unowned
mxrouterprep down halted disabled unowned

ln2004:/opt/cmcluster/conf # cmquerycl -v
Looking for Serviceguard nodes ... Done

Cluster Name Node Name Version Status
emxteamspreprodcl
ln2004 out of date
ln2005 out of date

Re: clusternode won't join cluster anymore

AnthonySN — Mon, 05 Jul 2010 05:41:26 GMT

you can give cmruncl instead of
cmruncl -n ln2004

also post your cluster log from /etc/cmcluster/(pkg name)/(pkgname.log)
and syslog.

Re: clusternode won't join cluster anymore

Paul Ettema — Mon, 05 Jul 2010 09:23:18 GMT

@ SASJ

Sorry, SASJ, I have already try "cmruncl" but no solution.

And package log are not useful now, while the running on other node now, so no problem with packages.
Only a problem with 1 node

Paul.

Re: clusternode won't join cluster anymore

AnthonySN — Tue, 06 Jul 2010 09:30:06 GMT

what is the error you get when you give the cmrunnode command on ln2004 when the cluster is up and running on node ln2005.

post ur messages log from ln2004.

Re: clusternode won't join cluster anymore

Steven E. Protter — Tue, 06 Jul 2010 12:46:42 GMT

Shalom Paul,

Some essential part of clustering may have gone off line. Perhaps the heartbeat network, or issues with the interface of hub.

I recommend rebooting the recalcitrant mode and taking a look at all the logs to get a cue as to what the error is.

It is in there, somewhere.

Also, check lights on networking for the machine and any hardware the cluster depends on.

SEP

Re: clusternode won't join cluster anymore

Paul Ettema — Tue, 06 Jul 2010 13:33:15 GMT

@ SASJ

error messages/output is the same as in my 1st and 2nd post. No differences when ln2005 is run or not run cluster.

Re: clusternode won't join cluster anymore

S.N.S — Tue, 06 Jul 2010 17:59:28 GMT

Hi Paul,

May be time to contact HP Response Center - the last time I had a problem with cnruncl -n...it got solved by itself - ie after reboot...

Thankfully, it started.....But that Prod Server wasn't patched in 3yrs....

Just consider all the basic cluster trouble shooting steps -something fundamental you might have missed out -before you contact the HP biggies.

SNS.