topic Re: cluster restart problem in Operating System - HP-UX

cluster restart problem

jpcast_real — Fri, 17 Sep 2004 11:00:30 GMT

Hello ,

I am making tests with my two node cluster . I have a A.11.14 over hp-ux 11.11 . If both nodes are running cluster works ok . If I shutdown the server 2 one node the cluster remain in the node 1 but if I halt the node 1 the cluster remains in an unknow state and I can not restart it again:

Athos:/var/adm/syslog> cmviewcl
cmviewcl : Cannot talk to all the nodes.
Cluster does not appear to be up

CLUSTER STATUS
BHS unknown

NODE STATUS STATE
Athos down unknown
Porthos unknown unknown

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
pkg-oracle unknown unknown
pkg-bhs unknown unknown
Athos:/var/adm/syslog> cmruncl -v
cmruncl : Validating network configuration...
Gathering configuration information ......... Done
Not probing node Porthos as it is currently unreachable.
cmruncl : Network validation complete
The configuration file on node Porthos will not be checked.
Cannot reach node "Porthos".
Athos:/var/adm/syslog> cmvie3wcl
sh: cmvie3wcl: not found.
Athos:/var/adm/syslog> cmviewcl
cmviewcl : Cannot talk to all the nodes.
Cluster does not appear to be up

CLUSTER STATUS
BHS unknown

NODE STATUS STATE
Athos down unknown
Porthos unknown unknown

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
pkg-oracle unknown unknown
pkg-bhs unknown unknown

For me is impossible in this state to start the cluster just with one node , why ?????

Is it possible to migrate from SG A.11.14 to A.11.15 ????

Re: cluster restart problem

Sanjay_6 — Fri, 17 Sep 2004 11:16:50 GMT

Hi,

You can force the cluster startup using only one node.

cmruncl -f -v -n node1

here node1 is the node on which you want to start the cluster when the other node "say node2" which is part of a 2 node cluster is unavailble for some reason.

Hope this helps.

Regds

Re: cluster restart problem

bhavin asokan — Fri, 17 Sep 2004 23:08:50 GMT

hi,
is the second server is up and you can login from first server.
check the inetd.conf for the hacl-cfg entries on the servers.

regds,

Re: cluster restart problem

David Child_1 — Sun, 19 Sep 2004 00:42:29 GMT

The single node won't come up because it cannot be sure what the state of the other node is. You can force it however using the -f flag as Sanjay mentioned.

Re: cluster restart problem

Jannik — Sun, 19 Sep 2004 04:21:17 GMT

Make sure yor heart can run before you force the cluster up linkloop is your command and the it is time to make a forced run :-)

Re: cluster restart problem

jpcast_real — Mon, 20 Sep 2004 05:21:15 GMT

Thanks to everybody for the help . I just have a couple of questions about your comments:

- what is the purpose of the hacl entries in the inetd.conf file ???? Is the inetd daemon the one who controls the cmcld heartbeat???Which port uses the heartbeat communication??

- how can I test if the heartbeat is running using linklook??? I have alwais used netstat and lsof to test it ..

Thanks again

Re: cluster restart problem

Wed, 22 Sep 2004 09:38:51 GMT

Most commands such as cmruncl talk to cmclconfd rather than doing everything themselves. e.g. when you issue a cmruncl if sends message to cmclconfd which actually starts cmcld.

The hacl entries in inetd.conf are to start the cmclconfd daemons when the commands are run.

The hacl-hb port defined in /etc/services is used for cluster HB communications, i.e. port 5300. You can see if this is being used with a netstat command as you state.

P.S. Be very very careful when using cmruncl -f -n to run a 1 node cluster. If the other node is already running as a 1 node cluster but is isolated due to networking trouble you can form a split brain cluster and mount filesystems on 2 systems at the same time leading to data corruption. You should only ever use cmruncl -f -n to start a cluster when you are 100% certain the cluster is down on all other nodes.