monitoring failover lan interface in MC/SG environment

[Dan Copeland]

Admins,

We are trying to setup some sort of monitoring of nodes in a MC/SG cluster. Particularly, we want to know when the failover interface loses connection to the network it needs to be connected to. For example, the primary interface (lan0) and the failover interface (lan1) are connected to the same network through different switches--they can linkloop to each other---they are in the same VLAN. This allows them to be setup in a lan card failover situation in MC/SG.

The package ip address is brought up on lan0, but if lan0 were to fail, the ip could be switched to lan1. But if lan1 happens to have lost connection to it's VLAN some time between when the node joined the cluster and when the failure to lan0 happens, the ip won't failover and the package will halt.

lan1 may have had it's cable pulled inadvertently or someone switched it's VLAN by mistake. So we won't to know when this disconnect (physical or virtual) occurs.

One possible solution we thought of was having a script run linkloops to each and every interface a couple times a day. This will work but it's tedious to setup and maintain. HP gave us nothing. They can only monitor whether the card is UP or not (output of lanscan). This doesn't tell us if the network is reachable. Their other suggestion was to monitor the nettl log file for disconnects. This requires running the nettl format command and dumping it to ascii every hour or so--again, not so clean. Monitoring the syslog.log file is also an option since the standby interface disconnect is logged there but it's a bit vague. Best case scenario would be an OVO trap for the situation---HP said it doesn't exist.

How are you monitoring this (if you are)?

tia,
Frank

[David Griffiths · Accepted solution]

Hello Frank,

We got caught out with this one when the 'live' interface failed and the secondary card was dead. Our answer was to use the 'cmcheckconf' command in a script. This will check all interfaces within a specified cluster. If the cluster is fine the words 'no errors' will appear in the output. Our script issues the command, greps the output for 'no errors' and e-mails the person on-call if there is a problem. We run the script once a day at 10:00am. It could be run more often.

Hope this is of use to you.

[melvyn burnard]

You could look at using EMS to monitor the lan status, but as you say, it only checks to see whether there is lan connectivity or not.
You could also do as you thought, writing hte script and using that as a monitor or service process.
My question is, if the standby has failed, and it is recorded in syslog, if you lose the primary lan I would have thought the package would switch to the alternat ebode, or am I missing something here?

[John Palmer]

How about writing a script to check the output from cmviewcl -v. This displays the status of the network interfaces as far as the cluster is concerned.

Regards,
John

[Dan Copeland]

Hi Melvyn,

Yes you're right about the package failover, but we would be able to avoid a failover (which is a application outage--takes about 15 min for our oltp instance to switch nodes) if the secondary lan is always up.

Thanks to all the other replies. I have some good ideas now.

Frank