topic Re: node didnt automatically join cluster in Operating System - HP-UX

node didnt automatically join cluster

Richard Woolley — Mon, 17 Feb 2003 10:42:00 GMT

yesterday a node in the cluster was rebooted however it did not appear to come automatically back into the cluster.

below is so far the only clue and is taken from the syslog

saturn cmclconfd[7152]: Permission denied for user root on node saturn (/etc/cmcl
uster/cmclnodelist)

an output of cmclnodelist is

-rwxrwxr-x 1 root sys 219 Feb 19 2002 /etc/cmcluster/cmclnodelist

saturn root is an entry in this file.

any ideas where else i could look?

Re: node didnt automatically join cluster

melvyn burnard — Mon, 17 Feb 2003 10:46:51 GMT

and on the other cluster nodes?
has it joined the cluster yet?
what version of OS and SG, and what patch is on for SG?

Re: node didnt automatically join cluster

Richard Woolley — Mon, 17 Feb 2003 10:52:09 GMT

it joined the cluster immediately after a cmrunode was manually executed. So joining wasnt the problem. Just did not do it automatically and there is a 1 set in the /etc/rc.config.d/cmcluster

what should i look for on the other nodes?

Re: node didnt automatically join cluster

T G Manikandan — Mon, 17 Feb 2003 10:52:40 GMT

check this doc

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000063208976

Re: node didnt automatically join cluster

T G Manikandan — Mon, 17 Feb 2003 10:55:25 GMT

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000065774292

Re: node didnt automatically join cluster

Richard Woolley — Mon, 17 Feb 2003 11:13:55 GMT

O/S version 11.00
S/G version 11.13
PHSS_25915 1.0 MC/ServiceGuard and SG-OPS Edition A.11.13
PHSS_26928 1.0 MC/ServiceGuard and SG-OPS Edition A.11.13

have reviewed docs and checked what they said is correct on the machine.

cmclnodelist exists as mentioned earlier with itself in. So cannot understand why that error message appears.

saturn root - is an entry in this file.

Re: node didnt automatically join cluster

Kent Ostby — Mon, 17 Feb 2003 16:43:28 GMT

Mark --

Can you post the complete entries in the cmclnodelist files.

Also, please post the output of "hostname".

Also, which machine did you run the "cmrunnode" command from ?

If you ran it from a machine other then "saturn", please post that cmclnodelist as well.

Finally please post what you are using for name resolution as that can play a part in a "permission denied" situation.

Re: node didnt automatically join cluster

Richard Woolley — Mon, 17 Feb 2003 16:52:31 GMT

i have attached the cmclnodelist!

output of hostname is "saturn"

the cmrunnode command was run from the node itself

name resolution is determined by the /etc/hosts file.

cheers.

Re: node didnt automatically join cluster

monasingh_1 — Wed, 19 Feb 2003 02:21:06 GMT

do you have AUTOSTART_CMCLD=1 in /etc/rc.config.d/cmcluster ?.

This may be a culprit..
hope this works..

Re: node didnt automatically join cluster

Jeff Schussele — Wed, 19 Feb 2003 02:58:31 GMT

Hi Mark,

Monasingh has probably pegged it. IF you don't have
AUTOSTART_CMCLD=1 in /etc/rc.config.d/cmcluster
then the node will NOT auto start the cmcl daemons & hence won't join the cluster until cmrunnode is executed.

Rgds,
Jeff

Re: node didnt automatically join cluster

Richard Woolley — Wed, 19 Feb 2003 15:54:16 GMT

AUTOSTART_CMCLD=1

is what is in the /etc/rc.config.d/cmcluster file :(

Re: node didnt automatically join cluster

Kent Ostby — Wed, 19 Feb 2003 20:14:17 GMT

Here's a few other questions.

#1) Were there other nodes in the cluster that were rebooting at the same time or was this the only one ?

Since CMRUNNODE joins an existing cluster, if all of your nodes are down, then CMRUNNODE may not cause the node to join unless there multiple nodes are running cmrunnodes at the same time.

#2) Given the "permission denied" error, it leads back to the question of hostname resolution.

So please post the output of:

grep saturn /etc/hosts
and
nslookup saturn

Also, from the OTHER nodes in the cluster, please do:

grep saturn /etc/cmnodelist

To compare to the cmnodelist on this system.

Re: node didnt automatically join cluster

Richard Woolley — Thu, 20 Feb 2003 13:16:14 GMT

1. afraid not, this is a 5 node cluster and this was the only 1 booting at the time.
(though just found out from hp predict the machine crashed yesterday with "init spawning too rapidly" - also another forum thread).

here's the info requested

Re: node didnt automatically join cluster

Stephen Doud — Thu, 20 Feb 2003 13:55:06 GMT

Hi Mark,

Having dealt with a similar failure-to-join-cluster case some months ago, the resolution took patience to discover through experimentation. In that case, Tivoli was altering /etc/syslog.conf and having started just before /sbin/init.d/cmcluster, it held off cmclconfd long enough to prevent it from working.

In that situation, we closely inspected the boot-time messages in /etc/rc.log and the console, looking for abnormal termination or error messages.

Since the error condition in this issue is "permission denied", we have to assume that at the time the cluster is performing cmrunnode, either the local host, or the cluster manager node is not performing hostname resolution properly or not permitting root to operate at that time. What is "hosts" set to in /etc/nsswitch.conf

Does the condition repeat itself?
Are there any clues in syslog.log?
Does inetd start properly (syslog.log)?

I suggest you open a case with the response center to continue to narrow down the root cause. As I said before, the cause is not obvious.

-S.