High latency to procurve switch management IPs

[MWjag]

Many of my Procurve (2650, 2524) switches' mgmt IPs show returns pings in the 100ms range. Strangely the return pings from devices connected to the switch are consistently 1ms.
Is there some reason the switch is so slow to return a ping to it's own mgmt ip?
I wouldn't care about it, but it makes it harder to monitor network health when the switches themselves show high latency.

any help would be appreciated...

[Mohammed Faiz]

Are the switch IP addresses on the same VLAN as the connected devices or are you running a separate management VLAN?

[Michael_Breuer]

Hello,

measuring ICMP round trip times is not a reliable indicator about switch health, because ICMP packets will be answered with a low priority. Did you check CPU load and free packet buffer for IP management to ensure that the switch has enough system resources to answer the ICMP packets?

Cheers,

Michael

[MWjag]

Yes the devices are on the same vlan as the managment vlan. There are 8 vlans and both devices have management on vlan 2.

the swtich proc is hovering near 6-10% and there is 24MB of free ram. 2500 packet buffers free.

These are mosly edge swtiches connected via fiber 1000sx. They only have about 6 access points connected to each of them and not much else. Performance is not my worry -- Link health is my big concern.

But my monitoring software shows my switches with the highest latency -- that's misleading since the devices behind the swtich have very low latency. I don't understand why the swtich would be so slow to return pings.

[rick jones]

Perhaps the management processor of the switch(es) has set a very high interrupt coalescing setting on its network interface.

While my email ends in .hp.com, the switches are almost as much a black box to me as anyone else. One experiment you could try is see if something like linkloop (on HP-UX, and a port for Linux exists), which is a Layer-2 "ping" to which ProCurves will respond shows the same high RTT. You might have to get an idea of the rtt by using the time command against the linkloop command on UX - linkloop doesn't (iirc) calculate/return an RTT.

Also, you could find a ping program that would let you send two (or more, but bounded to a small number) of ICMP Echo Requests at the same time. If the RT latency of the second (or Nth) is much lower than the latency of the first, that suggests aggressive interrupt coalescing.

[Sean O'Leary]

Since everything seems to be on the same VLAN, your network is all running at layer 2 regardless of IP settings on the HOST devices. I would break the management IPs for your switches into their own VLAN so that the ICMP traffic you generate only hits the infrastructure links on your network and isn't broadcast to every port. You will see that response time drop.

[MWjag]

The vlan they are on IS solely a managment vlan. The AP's behind it have that same vlan as well for management only. All production traffic is on different VLANs.

Basically the switch is able to forward and return a packet from a connected device faster than it's own IP stack can respond to a ping.
Granted that ping replies from a switch should be at a lower priority than fowarding traffic, but it really seems excessively slows. It'll go from 1ms to 30ms to 64ms etc.

Note this doesn't happen on all of my switches either - mainly just the newer ones. The older 800m's seems to be fine.