Device does not respiond to ping packets

[FoxtrotOff]

Hi all

Weird one for you today.

Over the past few weeks, we've been getting 2 switches in particular that now and again IMC seems to think it can't see. It throws a critical and we obviously spring into action to find that the switches are running perfectly fine?

While IMC is in a critical state saying a switch cant be reached, i can ping the switches no problem, and even SSH to them, checking the logs on the switches themselves reveals no issues whatsoever..

For some reason, this only happens on 2 of our 43 devices, both these devices are connected at 10g, one ethernet, one fibre.

We have updated to 7.2 last week, and i have also changed the uplink on the switch side to see if that would resolve the issue, unfortunately it has not.

Has anyone else had the issues i am describing or could point me in the direction of a fix?

Thanks!

[LindsayHill]

There probably is some connectivity issue between IMC & those switches. The challenge is in tracking down exactly where.

We had a thread discussing something similar here http://community.hpe.com/t5/IMC/Polling-false-negatives-on-C7000-modules/td-p/6844948

To isolate the problem, you could start with a Wireshark capture on the IMC server. Look for ping packets to/from the devices in question. When you get an alarm in IMC, check the packet capture. If the ping response was never received by the IMC server, then the problem lies outside IMC, and IMC is alerting you to a genuine issue somewhere in the network. IMC has multiple retries for ping, so it's not sending an alarm because one single ping got dropped. 

That packet capture will at least help in telling you which direction to go next with your troubleshooting.

[FoxtrotOff]

Thanks for the suggestion.

I checked the firmware on the devices as there were newer versions released on the 6th.

Ive deployed these firmwares yesterday and this seems to have resolved the issue, we didnt have a single down alarm yesterday on one switch, the second switch i deployed this morning and so far, no down alarms either.

I think this is going to be blamed on bad firmware to be honest but ill update this in a few days with the results of the firmware deployment.

Switches we're using are 2920-24G-PoE, Suspected offending firmware level: WB_16_01_0004

[LindsayHill]

Hmmm, interesting. I wonder if there was something going on with that firmware that was causing the control plane CPU to be too busy, and unable to respond to ping requests?

Hopefully it settles down now.

[RogerKaram]

Hello,

When the devices are critical, can you ping them from IMC itself? It could be a span-tree or balancing issue going to a link where a specific VLAN is not available.

RK

[FoxtrotOff]

Hi guys

Unfortunately my hunch with the firmware hasn't panned out. Here's what i have found over the last few days of messing with IMC and the switches:

Wireshark does show an issue where pings are showing port unreachable, i am not very good with wireshark so ill post a screenshot of the errors on my next capture.

We're getting many more switches going down with does not respond to pings now. I have increased the timeout from 2 seconds to 15 seconds, the alarms usually persist for about 1 minute.

Having looked at the alarms in the past, some of them are happening within seconds of each other, right now, glancing at IMC i can see six critical alarms all starting within 10 seconds of each other and persisting for around 1 minute and 2 seconds approximately.

I have verified that i can speak with the switches from another machine, switch event logs show no disruptions and no warnings whatsoever about an issue.

I rebuilt the IMC server yesterday with a new VM and that continued the same behaviour so there is an issue on that network somewhere.

Some info about the network we have:

All our switches are on a management vlan, IMC has 2 NICs to allow monitoring from the curriculum network we are on. We havent had this kind of issue before and it started around a month ago with the original two switches going offline during that period but is now getting worse.

One final thing, im seeing a few response time of device minors, getting upwards of 100ms.

As mentioned, my next wireshark capture i will post to imgur and get your opinions on it.

Thanks

[FoxtrotOff]

As promised, here is a wireshark capture

http://imgur.com/xvjWRgZ

I did ping the switch from IMC and indeed it couldnt be reached, when i pinged from my machine, i was able to receive replies but the latency was in the range of 1100ms

[LindsayHill]

Seems like you've got a few network issues. Ping times of 1100ms is extremely long. Usually you only see that sort of latency with satellite links. Time to do some investigating into your network. 

Start with the usual things. Map out your network. Understand what the interconnections are, and what path your packets take. Look for congestion, errors, duplex mismatches, etc. 

[FoxtrotOff]

Hi again

We're pretty up on our infrastructure as we have network maps, trunk lists, all of the good stuff. The issue we're experiencing just does not make any sense

Here is a switch experiencing an issue of high latency, CPU and memory util is low, i've also provided an interface list of TX and RX bandwidth and as you can see, there is little to no utilization happening of the switch itself.

http://imgur.com/2yqrTEz

This switch is 1 hop from our core.

Also, no errors or drops are being TX'd on ANY of the ports..

Drawing a loss here sadly and its getting more frustrating by the day

[FoxtrotOff]

Oh and one final thing, all the switches that are having this latency issue, are ALL 2510s