MC/SG packages do not start after reboot node

[G. Vrijhoeven]

Hi all,

We have a 3 nodes in an MC/SG cluster. All the nodes are rebooted for kernel adjustments. All the servers start up OK and try starting the packages. And this fails for all the packages on one node. In the package log file we get the following failure:

cmmodnet : Unable to verify IP address 145.78.160.142.
cmmodnet : 145.78.160.142 might already be configured as a heartbeat IP address.
ERROR: Function add_ip_address
ERROR: Failed to add IP address to subnet
Dec 17 19:30:21 - Node "prc60b01": Remove IP address 145.78.160.142 from subnet 145.78.160.128
cmmodnet : Subnet 145.78.160.128 is not a configured subnet.
cmmodnet : Use the "netstat -in" command to list the configured subnets.
ERROR: Function remove_ip_address
ERROR: Failed to remove 145.78.160.142

If we do an netstat -i we get:

prc60b01:/etc/cmcluster # netstat -in
Name Mtu Network Address Ipkts Opkt
lan5:9 1500 145.78.160.128 145.78.160.153 637 6
lan8 1500 145.78.191.64 145.78.191.72 103598 25166
lo0 4136 127.0.0.0 127.0.0.1 68381 6838
lan7 1500 145.78.191.32 145.78.191.46 229 17
lan6 1500 145.78.191.0 145.78.191.14 1114 15
lan5 1500 145.78.160.128 145.78.160.132 43906 4201

If we try to start the packages on other nodes they startup OK.
If we disable the networkpart of the controle file and add the ipadress manually it works fine.

I found some other related problem on the forum:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=11613

We also have 145.78.160.132 lan5 configured as an Hartbeat lan in the ASCII file, but we have the same configuration on the other two nodes who do not have this problem.


Thanks in advance, anwsers will be assigned points.

Gideon

PS. this is an production cluster and all packages are currently in production.

[Ashwani Kashyap]

Check if that package IP is configured in the /etc/rc.config.d/netconf file . It should not be .

The subnet is available and it should add the floating IP to the subnet .

Also check for service guard patches .

[Jeff Schussele · Accepted solution]

Hi Gideon,

You really need to run cmcheckconf on this cluster. Appears to me you have multiple errors in the cluster binary file on this node - including:

cmmodnet : 145.78.160.142 might already be configured as a heartbeat IP address

This might indicate you have the same IP configured for a heartbeat as you're trying to assert as a virtual IP. Heartbeats must be a different IP than a virtual. They can be the same as a stationary IP = but not the same as a virtual

cmmodnet : Subnet 145.78.160.128 is not a configured subnet

This indicates you have a non-zero digit in the 4th octet entry for the subnet field when that should always be zero to indicate a subnet.

Should look like:
# For example, if this package uses an IP of 192.10.25.12 and a subnet of
# 192.10.25.0 enter:
# IP[0]=192.10.25.12
# SUBNET[0]=192.10.25.0

I would compare file size/date of the cmclconfig file on all nodes. I suspect you'll see a difference on this one.

Rgds,
Jeff

[Kent Ostby]

Gideon -- I would also take a look at your /etc/rc.config.d/netconf file and talk to co-workers to see if any changes were made there since last reboot (or check the timestamps).


The other thing you can do is to do a cmquerycl on the box and see if one of your subnets has changed or the subnet mask has changed. This could also cause your problem.

Best regards,

Kent M. Ostby

[G. Vrijhoeven]

Thanks Ashwani Kent and Jeff,

We found the cause of the problem. On the prc60b01 lan1 was configured as userlan and lan5 was failoverlan. In the /etc/rc.config.d/netconf file lan5 was configured as the default with the ipaddress for the userlan. This netconf file was altered in August after the last reboot and never reactivated. The reboot activated this settings and caused a problem for the cmmodnet command.

Jeff we never had a problem with the subnet 128 as long as the package ipadresses are above this subnet. This indicates that there are issues with subnetting in a MC/SG environment. But i never found documentation that say's it is not allowed.



Regards,

Gideon