cluster IP address not routed after switchover

[Mihails Nikitins]

Hi,

I run two node MC/SG cluster with one package only.

After a HW crash (system reboot) on node A, cluster software migrated the package to node B. Package address also migrated to node B, but it was available only from the same subnet. Nodes were available from other networks by their unique addresses, cluster address was fully available by its subnet neighbours.

After restarting both nodes, package bacame up on node B with the same problem. I made switchover back to node A and now it's OK.

I compared files under /etc/cmluster in binary mode and they are equal. Netconf is also the same. netstat -r looked OK.
I run the following version of MC/SG

B3935DA A.11.15.00 MC / ServiceGuard

Many thanks and points in advance for any help!

BR,
Mihails

[G. Vrijhoeven]

Hi Mihails,

Normally when a package starts its subnet must allready be active for the cmmodnet command to succeed. Could you provide the syslog logging from node B at the time the package was started.
How did you test the access, ping or telnet or application services?
If it is application services are these started, reserved in services ( /etc/services or NIS).
HTH,

Gideon

[melvyn burnard]

Sounds to me like something out on your network did not respond to the arp rebroadcast that would have been sent by Serviecguard to advertise the fact that the MAC/IP combination had changed.
I would start by looking at what hardware is out on the network,.
I do know some older network hardware did NOT react to these re-arps. and would eventually respond after 10 minutes (default arp cache flush timer).

[Dietmar Konermann]

Mihails,

when an interface gets configured, in your case during package failover, HP-UX sends out an unsolicited ARP to notify remote systems of address mapping between MAC (link level) addresses and IP level addresses... and I'm not aware of any known problems in that specific area.

Maybe your switches/routers or other components have problems updating their arp caches and continue to use node A's MAC address for communication with the package's IP address? During the next downtime you should reproduce the problem and have a close look at the arp caches. An network trace could be of use also!

Best regards...
Dietmar.

[Jeff Schussele]

Hi Mihails,

As noted you should already have an IP in that subnet *before* the virtual is asserted on that I/F.
Was that traffic routing OK prior to the pkg switch?
If not then I'd suspect your default gateway is not setup properly OR the route is being dropped - possibly due to dead gateway detection. That can be turned off in ndd.conf.

Rgds,
Jeff

[Francois LAURENT]

Hi Mihails,

Certainly you have routing problem and not McSG problem.
Routers are they able to route this subnet to nodeB ?
Perhaps if you have vlan, vlan are configured on nodeA 's switch and NOT on nodeB 's switch ?

[Bharat Katkar]

Hi,
I think laurent is suggesting correctly and i doubt the same. Check the physical connectivity of NIC's to the switches and there VLAN configuration's ...

Regards,

[Mihails Nikitins]

Hi,

Thanks for all the replies.

1. There are no known problems in our switches/routers. We'll take care about arp tables on our routers next time. There were no network problems during switchovers with routing before the case. However, I never tried to pull power plug off to simulate _a very nasty switchover_. :-)

2. There were no problems with any routing to the IPs of my nodes. Cluster address was totally unavailable from other subnets (no ping, no telnet, no ftp). Cluster address was fully available from its subnet.

3. I lost crash syslog because of HP-UX style of syslog rotating.

4. This is a package log of Node B after Node A crash


########### Node "node1": Starting package at Mon Jun 28 18:45:56 EEST 2004 Jun 28 18:45:56 - "node1": Activating volume group vgdb with exclusive option.
Activated volume group in Exclusive Mode.
Volume group "vgdb" has been successfully changed.
Jun 28 18:46:47 - Node "node1": Checking filesystems:
/dev/vgdb/lvol1
fsck: /etc/default/fs is used for determining the file system type
log replay in progress
replay complete - marking super-block as CLEAN
Jun 28 18:46:48 - Node "node1": Mounting /dev/vgdb/lvol1 at /MySQL_1
Jun 28 18:46:50 - Node "node1": Adding IP address 192.168.1.190 to subnet 192.168.1.0

5.
Routing on node B looked like


mypackage.mydomain.com mypackage.mydomain.com UH 0 lan0:1 4136
node1.mydomain.com node1.mydomain.com UH 0 lan0 4136
10.20.2.0 node1.mydomain.com U 3 lan0 1500
10.20.2.0 mypackage.mydomain.com U 3 lan0:1 1500
127.0.0.0 localhost.0.0.127.in-addr.arpa
U 0 lo0 0
default 192.168.1.254 UG 0 lan0 0

Note the sequence of the routes.

6. I found a patch PHSS_30769. Is it a similar problem?

BR,
Mihails

[Francois LAURENT]

hi Mihails,

your node and your package are on 10.20.2.0 network isn't it ?
What is 192.168.1.190 appears in pkg's log at startup ? This @ don't appears in routes table.
Is it a correct @

Is your pkg have many ip@ ?
what show cmscancl ?
What about traceroute ?




You can test PHSS_30769 but i don't think it solves this pb (hang, memory leak, reformation pb).

[Mihails Nikitins]

Please read 192.168.1.0 instaed of 10.20.2.0.
I just replaced real names/addresses and made a mistake. :-)