topic Re: CM Cluster Error in Operating System - HP-UX

CM Cluster Error

Alberto Hurtado — Fri, 12 Sep 2003 15:20:37 GMT

please send me information about this errors i have two hp-9000 k570 in a Cluster of MC-Service Guard. The system did not go away down and either was shutdown or reboot.
Your aid is thanked
______________________________________________
Sep 11 13:47:08 mvi902 cmcld[2438]: Communication to node mvi903 has been interrupted

Sep 11 13:47:08 mvi902 cmcld[2438]: Node mvi903 may have died

Sep 11 13:47:08 mvi902 cmcld[2438]: Attempting to form a new cluster

Sep 11 13:47:13 mvi902 cmcld[2438]: 2 nodes have formed a new cluster, sequence#101

Sep 11 13:47:13 mvi902 cmcld[2438]: The new active cluster membership is: mvi903(id=2), mvi902(id=1)

Sep 11 13:47:15 mvi902 vmunix: mpc_bindlwp: Overriding conflicting mandatory binding!

Sep 11 13:47:15 mvi902 vmunix: mpc_bindlwp: Migrating process 491 from processor 1 to processor 0!

Sep 11 13:47:39 mvi902 vmunix: mpc_bindlwp: Migrating process 499 from processor 2 to processor 0!

Sep 11 13:47:39 mvi902 vmunix: mpc_bindlwp: Overriding conflicting mandatory binding!

Sep 11 13:47:40 mvi902 above message repeats 2 times

Sep 11 13:47:39 mvi902 vmunix: mpc_bindlwp: Migrating process 499 from processor 2 to processor 0!

Sep 11 13:47:49 mvi902 vmunix: mpc_bindlwp: Overriding conflicting mandatory binding!

Sep 11 13:47:49 mvi902 vmunix: mpc_bindlwp: Migrating process 567 from processor 3 to processor 0!

Sep 11 16:20:25 mvi902 cmcld[2438]: Communication to node mvi903 has been interrupted

Sep 11 16:20:25 mvi902 cmcld[2438]: Node mvi903 may have died

Sep 11 16:20:25 mvi902 cmcld[2438]: Attempting to form a new cluster

Sep 11 16:20:29 mvi902 cmcld[2438]: Obtaining Cluster Lock

Sep 11 16:20:30 mvi902 cmcld[2438]: Turning off safety time protection since the cluster

Sep 11 16:20:30 mvi902 cmcld[2438]: now consists of a single node. If ServiceGuard

Sep 11 16:20:30 mvi902 cmcld[2438]: fails, this node will not automatically halt

Sep 11 16:20:32 mvi902 cmcld[2438]: Attempting to adjust cluster membership

Sep 11 16:20:35 mvi902 cmcld[2438]: Enabling safety time protection

Sep 11 16:20:35 mvi902 cmcld[2438]: Clearing Cluster Lock

Sep 11 16:20:37 mvi902 cmcld[2438]: Timed out node mvi903.

Sep 11 16:20:37 mvi902 cmcld[2438]: Attempting to adjust cluster membership

Sep 11 16:20:41 mvi902 cmcld[2438]: Clearing Cluster Lock

Sep 11 16:20:46 mvi902 cmcld[2438]: 2 nodes have formed a new cluster, sequence #104

Sep 11 16:20:46 mvi902 cmcld[2438]: The new active cluster membership is: mvi902 (id=1), mvi903(id=2)

Re: CM Cluster Error

Geoff Wild — Fri, 12 Sep 2003 15:32:28 GMT

Looks like you lost your heartbeat....

Rgds...Geoff

Re: CM Cluster Error

melvyn burnard — Fri, 12 Sep 2003 16:02:02 GMT

Your nodes are losing contact wit each other through their heartbeat networks, possibly due to network overload or other networking issues.
You would be advised to locate htese network problems and fix them.
As an aside, what are the configuration ssettings for HEARTBEAT_INTERVAL
and
NODE_TIMEOUT

These may have to be increased.

Re: CM Cluster Error

Zigor Buruaga — Fri, 12 Sep 2003 16:14:17 GMT

Hi,

Maybe this link could help:
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x9dfc6c96588ad4118fef0090279cd0f9,00.html

Kind regards,
Zigor

Re: CM Cluster Error

Alberto Hurtado — Fri, 12 Sep 2003 20:59:09 GMT

I have a question... why service guard make the cluster reformation when a LAN card failed if ia had two lan card:
mvi902> lanscan
10/4/8.1 0x080009BA3C7C 4 UP lan4 snap4 2 ETHER Yes 119

10/12/6 0x0060B0838AEB 1 UP lan1 snap1 4 ETHER Yes 119

I guess shuold be go for the secondary card

Re: CM Cluster Error

Geoff Wild — Sat, 13 Sep 2003 04:54:15 GMT

You may have a second card - but if it isn't configured as a STANDBY - then it won't be used.

Can you post your /etc/cmcluster/"cluster".conf file?

Rgds...Geoff

Re: CM Cluster Error

Zigor Buruaga — Sat, 13 Sep 2003 06:09:24 GMT

Hi,

In your cluster config file you define which networks are "critical". We have 4 different IP's for each node and only two of them ( used by clients in different sites ) are "critical". When one of this critical networks becomes unavailable ( HEARTBEAT fails ), then the package "jumps" to the other node, in order to be available to all clients.

Hope this helps, and sorry for my initial confusion.
Kind regards,
Zigor

Re: CM Cluster Error

Sritharan — Mon, 15 Sep 2003 03:09:10 GMT

Hi,

Please send us your cmcluster configuation file,to check whether the heartbeat have been configured or not.

Please attach the cmscancl -v output also

Thanks & Regards
Sri

Re: CM Cluster Error

Sridhar Bhaskarla — Mon, 15 Sep 2003 03:56:33 GMT

Hi,

Couple of years back, we experienced the same issue where "identd",as used by sendmail seemed to have been causing the network cards to freeze for a short time intermittently. We remedied the issue by commenting out ident in /etc/inetd.conf (inetd -c) and increasing the HEARTBEAT_INTERVAL and NODE_TIMEOUT values in the cluster configuration file to 12 seconds.

-Sri

Re: CM Cluster Error

Alberto Hurtado — Tue, 16 Sep 2003 15:24:49 GMT

here the document with the command:
mvi902 /-> cmscancl -n mvi902

Re: CM Cluster Error

Sridhar Bhaskarla — Tue, 16 Sep 2003 16:50:24 GMT

Hi Alberto,

You already have .2 defined as the "default" gateway on the box. Do you know why you have so many static routes using the same gateway?.

You may want to get rid of them though it may not necessarily be the problem.

-Sri