High ping response times on stacked 2920s with firmware 16.01.0004

[rcho]

I just implemented a stack of four 2920-48 port switches (ring topology).  They are all running firmware 16.01.0004 with ROM WB.15.05.  I use WhatsUp Gold to monitor its response time and I'm seeing it fluctuate.  In the past three hours, it has spiked up to 400ms fours times.  I checked its trunk ports and I don't see any send or receive errors.  I also checked the Cisco 3750 switch that the stack is connected to, and I don't see any errors there as well.  I have other 2920 stacks in my environment and they are not exhibiting this high response time.  They are all running firmware 15.16.XX.  This leads me to believe that it's a firmware issue.  Can anyone confirm this to be the case, or provide any other potential explanations/solutions?  Thanks for your time.

[Marcus J]

Hi,

Yes i have seen the exact same thing, i have a case ongoing at HP for the last 3months with no resolve.

And its not just high ping times, the reason for high ping times is CPU load, the CPU load hits 70-80% during these high ping times.

Downgrading to WB.15.16.0012m will solve the problem for you, this is the last firmware with no issues, going to any newer ones like WB.15.18.xxx or newer introduces the problem.

I suggest you also setup a case with HP to have them resolve this!

[rcho]

Thanks for your reply Marcus.  I assumed that it was the firmware but couldn't rule out fiber patch cabling and SFP transceivers as the culprits. 

I did create a support ticket with HP yesterday as well.  I haven't received a response yet.  I noticed the higher than usual CPU utilization and how it spikes randomly.  For example at 3am when there's no load on the network/stack.    For the time being, I will downgrade my stacks to 15.16.0012m until I hear that a fix has been issued.

[Vince-Whirlwind]

What about pinging *through* the switch?

It's normal for network devices to give very low priority to generating ICMP responses. On the other hand, *real* network traffic should receive very high priority. The lack of errors would indicate no frames are going stale in the buffers, so it is probably passing traffic no problem?

[Marcus J]

That is correct, traffic passing through the switch seeems not to be affected.

But its bad enough that the switch is going 100% cpu and responsetimes of 200-400ms every now and then.

As you see this problem only on newer releases, there must be some kind of bug.

[Vince-Whirlwind]

You said 70-80% earlier.

So long as the switch is continuing to switch properly, what it uses its spare CPU cycles for shouldn't be too concerning?

[Marcus J]

This is a new behavior was introduced after upgrades, so obvisouly not normal situation.

And this high cpu usage is really affecting the switch, since managing the switch is extremely slow during the high cpu usage. Another problem is that our monitoring software triggers alarms when ping and CPU spikes. 

[BOSsysadmin]

Hi, Do you have any updates on how this issue get fixed?

I deployed stack of 3 switches and I have the same issues with response time.

[Marcus J]

hi,

Still no solution to the problem, im using WB.15.16.0012m on all my 2920 installations.

Pls setup a case with HP regarding this, i think that if we all complain about these things they will eventually get the point.

/Marcus

[KroFIt]

Hi,

I vave exactly the same problem!

Since I updated my 2920 Switches to WB.16.01.0004, my Monitoring is reporting cyclic high ping latency but thats only the effect of a high CPU-Load!

The attached File is a screenshot of the corresponding Icinga message history.

At the moment i am testing some config options on a test switch (disable some services eg. web management, telnet an so on)

When i find something interesting i will post it here!