Losing network connectivity. (A Class and switch)

[Ramon Irizarry_1]

I have a couple A Class servers running HPUX 11i and remedy connected to a cisco catalyst 3900 xl switch, which every week for no apparent reason loose connectivity to the switch. The servers and the switch are both set to 100mbs, full duplex and mtu = 1500. I'm using the internal 10/100 lan card that come with these servers.

Symptoms:

1) cannot ping the machine from anywhere in its own network or outside it.

2) cannot ping anywhere in the network even the default gateway from the A-Class servers.

3) can ping my own ip

4) all services are up and running normally.

I'll like to know ideas of where to start looking for solution to this problem.

[Jaikishan_1]

Dear ,
# netstat -rn
see if default route entry is there or not if not then please add the default route entry
# route add default gateway-ip 1
and also chk the physical connectivity.

Rgds
JAI

[Ravi_8]

Hi,

is the switch connected to power directly or thru spike burster. i am not seeing any apparent reason other than power supply.

[Duncan Edmonstone]

Ramon,

Some extra questions for you...

1. You say this happens every week? Do you mean just once a week at a specific time or at least once a week but in a random fashion?

2. What do you do to remedy the problem?

3. When the problem occurrs do you still have a link beat light on your network connections (both ends)?

4. Do you have auto-negotiation DISABLED on both the A class and the switch? (Assuming that the internal LAN card on your A class is lan0 you can check this by issuing the command 'lanadmin -x 0'. Not sure how you check this on your switch though)

Thx

[Ramon Irizarry_1]

#netstat -rn ==> yes the default route is there.

1. You say this happens every week? Do you mean just once a week at a specific time or at least once a week but in a random fashion?

Answer: It happens every week but in a random fashion.

2. What do you do to remedy the problem?

Answer: After some troubleshooting using lanadmin, netstat, lanscan, another commands the only solution is to do a shutdown -ry 0. Which is not the best solution right now.


3. When the problem occurrs do you still have a link beat light on your network connections (both ends)?

Answer: Yes, I still have the link beat light on both the server and the switch. Which, amazes me because there seems to be no communication at all.



4. Do you have auto-negotiation DISABLED on both the A class and the switch? (Assuming that the internal LAN card on your A class is lan0 you can check this by issuing the command 'lanadmin -x 0'. Not sure how you check this on your switch though)

Answer: Yes, I have disabled auto-negotiation on both sides.

Thank you, any more suggestions will be greatly appreciated.

Ramon

[rick jones]

if the switch were a router, and it was happening more often, i'd ask if the router was sometimes not responding to icmp echo requests, which are used by HP-UX 11x to see if the gateway is still alive (ip_ire_gw_probe in ndd).

the old standby of being on the latest driver patches might apply.

if other devices on the swtich retain connectivity, perhaps you have a cable that is ever so slightly out of spec? perhaps once a week, someone runs a vacuum cleaner near it and messes-up the electricals?

[Ramon Irizarry_1]

The gateway is a nokia firewall.

About the cable a switch the cable from another server and I still have the problem in the original server. So, I discarded the cable as a problem.

[Duncan Edmonstone · Accepted solution]

Two (and a half) extra things to try...

1. You say your default gateway is a firewall - is this set up to respond to pings during normal operation? ( a lot of firewalls aren't ) Also are there any other devices on your subnet, or are they ALL the other side of your firewall? HP-UX 11+ uses something called 'Dead Gateway Detection', and it can get screwed up when the default gateway is a firewall. What happens is that periodically the HP box will send out ICMP echo requests to the gateway - if it doesn't respond after a certain amount of time the gateway is marked as DEAD, and no more traffic is sent to it. You can check whether your have dead gateway detection enabled using the following command:

ndd -get /dev/ip ip_ire_gw_probe

If this returns 1 you have Dead Gateway Detection enabled.

Next time the problem occurrs, you can see if dead gateway detection has kicked in by running:

ndd -get /dev/ip ip_ire_status | grep GATEWAY

if the gateway comes back marked as DEAD then dead gateway detection has kicked in... Of course this could be a symptom of your problem rather than the answer.

2. As you still have a link-beat light, it's worth checking whether the problem is definately occurring at Link level as well, the following test should work:

i)Get the MAC address for your default gateway (or any other device on the same subnet as the server) - You can do this using 'arp -a | grep ' where is the IP address for your default gateway.

ii) While everythings working, check that you can do a link level ping to the default gateway (again this depends on the firewall configuration, not all firewalls will respond to these) using linkloop as follows:

linkloop -i 0 -n 3 -v

iii) Next time you have the failure, try the linkloop again - if it still works when pings don't then the problem is somewhere in the TCP/IP level, rather than further down the protocol stack.

2.5 We've seen a situation with a 100BaseT card with similar symtoms to yours, and this was fixed by applying the latest driver patches - This was on HP-UX11.00 though and from the quick search I've done I can't see any patches for 11i which relate to 100BaseT. But I would still agree with Rick - check with HP support that you are on the latest patch level for these drivers.

[Kevin Johnson_9]

My network engineer turned off ICMP echo request on the gateway I use. If your Server is set up for dead gateway detection, then it will stop talking to the gateway completely. This is a big problem if you want to get to any other subnet. Since I only have one gateway out I really don't need to have dead gateway connection enabled. So I disabled it on my server, since the network guys needed ICMP requests turned off at the router. Here is how you do it. Edit "/etc/rc.config.d/nddconf" and add an entry such as:

TRANSPORT_NAME[0]=ip
NDD_NAME[0]=ip_ire_gw_probe
NDD_VALUE[0]=0

Make sure the array is sequential.The ndd parameter ip_ire_gw_probe defaults to the value of 1 which is enabled. The 0 value for NDD_VALUE disables dead gateway connection detection. Reboot after updating the entry.