HPE Ethernet 10Gb 562SFP+ Adapter run 100X slower after firmware update to 10.4.3?

[Dennis_Lee]

Hi experts,

  We run our mission critical SAP system on HPE DL580 with WS Failover Cluster and we found the system sometimes reboot unexpectedly and our SI(system integrated) partner told that it is a possible bug occurred when system ROM is lower than 1.46. They updated system ROM, ilo firmware and SPP to 2019.03.1(including 10 GbE NIC firmare from 1.1752.0 to 10.4.3 and drivers).

  The wired thing happened. Sometimes the 10GbE became extremely slow(ping response time up to 1000ms, that's right, 1000ms and copy a 500MB file took more than 5 minutes. if it is is normal status, it finished in zero seconds. We complaint to our SI because HPE support did not support DL580 server directly and to level 2 support but we did not get any good response. Please help.

  Is this a bug? Can sometimes send me a KB(knowledge base article) to prove it is not my fault? Thank you very much.

[Dennis_Lee]

By the way, there is a workaround to fix the issue temporarily.

1. You can disable NIC/enable NIC from WS Control Panel.

2. You can perform a warm reboot.

However, if there is a cold reboot, the problem probably will reoccur(because firmware is reloaded?)

[ftepe]

Hi,

I have got a similar Situation:

6 ESXi Hosts with 2x 562SFP+ NICs running 10.4.3

1 Veeam Backup Host with 1x 562SFP+ NIC running 10.4.3

Throughput sometimes falls down to 1MB/s and backups never complete. Will try to disable and enable the NIC to see if Performance stays stable. Hopefully NIC down / up on ESXi works the same way?

Any Update from HPE on this issue? This Intel 710 chipset sucks!

[Paul_J_K]

Please ensure that the matching drivers for the firmware from 2019.03.1 are also installed, as I only see reference to firmware updates above. 

[ftepe]

Drivers were also deployed using the same SPP Version 2019.03.01 so that should match quite well….

Opened also a Case with HPE support, lets see what will happen...

[Dennis_Lee]

Hi ftepe,
Our SI(HPE partner) have opened a case to HPE support, too. When they took the "failed 10Gb 562 SFP+ adapter"
to HPE's lab and tested it. They could reproduce it and the throughput is about 10~20Mbps.

They downgrade firmware to 5.60 and the transfer speed became normal. They upgrade firmware to 10.4.3 and this time
the transfer speed did not become weird. The problem just disappears.

HPE still try more tests.

Here is a few temporarily workaround from our SI.
1. downgrade firmware(just do not use 10.4.3)
2. disable/enable NIC from OS
3. (update NIC driver and) warm reboot

Hope this help. Good luck.

[ftepe]

Hi,

thanks for the Update! I will try the workarround with enable / disable the nic. Also it's hard to find out which one or multiple servers of the 7 hosts involved causes the issue

Downgrade is not really an Option as these Intel NICs cause a lot of issues when you Google for VMware Intel 710

And when you open an HPE Case... you have to first update Firmware and Drivers!

Seems to be a very poor engeniered NIC