VERY slow network after many HW updates on DL370 G6

[mdreinders]

During our routine MS server patching last night, we wanted to update the firmware on the NICs on 2 DL370 G6s.  HP NCC552SFP cards.  During the attempt, it said we needed a newer BIOS.  So, we ran the whole gamut from SPP.  I don't have all the details, but the BIOS got updated to around 12/02/12. (P63).  NIC Firmware is now at 4.0.590.

Driver version in Server 2008 R2 SP1 is at 4.7.17.926.  The RAID Array controller was also updated (6.00).  We have the HP Network Configuration Utility saying we're running at 10GbE/Full (single port).

Anyway, after all the reboots,  running as part of an Exchange 2010 DAG, it's barely keeping up with maintaining replication between the other DAG member.  Copying files from my workstation to its C: drive shows VERY slow, and intermittent throughput.  Before last night, anywhere from 50MB/sec to 110MB/sec, in general.  Now, as low as 200kb/sec.  Ouch.

The team that worked overnight with Microsoft and HP tried downgrading the NIC firmware to older versions, downgrading the Windows driver to older and varying versions.  This morning, we actually re-IP'd the IP in use on the 10GbE adapter to one of the ports on the HP Quad 1Gb adapter.  We noticed exactly the same speed results.  Each NIC is plugged into a separate network switch.

We have 10 physical MB servers (Exchange), we did these HW/firmware upgrades on 2 boxes, and only these 2 boxes are seeing these results.  HP support is NOT being helpful.  They suggested clearing the NVRAM, which we did, no go.  We re-iterated Max Performance in the profile config in the BIOS, no help.

If anyone has ANY similar issues, please let me know what else we can try or what other diagnostic info you need to assist.  

Thank you!

Michael R.

[Jeroen_Kleen]

Hi Michael, I'm sorry to hear your trouble.

In regards to this issue: please do not hesitate to put your caseID here so we can follow your case at HP support.

In regards to this issue: are you able to test any local performance speed on the storage subsystem?  I would assume this is OK and performing as expected?  What if you copy local data against the \\localhost  to the same server; is that fast or slow?
Did you run already a full power-removeal on the server? As this could help? Please combine this as well with settting switch 6 on the systemboard so that NVRAM is clread in a forced way to factory defaults.

Did you try to install any other (intel?) NIC card in the server to see if this would behave equally? This would rule out a PCI-subsystem issue.  Please keep us posted with progress or that some of the above actions could be run .

Cheers, jeroen

[Abhishek Bangalore]

Hello,

Please try to perform the steps provided below and check for the occurence of the issue:

To prevent performance issues with NICs located in other slots, change the Receive Side Scaling (RSS) Rings from the default of 2 to 4. The number of RSS rings can be changed via Windows Device Manager. Select "Properties" for the appropriate device and then change the "Receive Side Scaling Rings" to 4.

Abhishek Sharma

[SteveA52]

Same problem with slowness on HP NC375I, through put could hardly exceed 1 mbps coping a file from the local fileserver. Rolled back driver under device manager from V4.7.17.926 to 4.6.17.502 and through put backup to 50 MBps on coping test. If you are using teaming then you will have to down grade NCU from 10.70.0.0 to 10.65.0.0 also. Seems there a bug in the latest driver update!

[mdreinders]

At this point, all local filecopy tests on the server itself are up to speed.  No issue there.

When you say full power-removal, I assume you mean shut the server down completely, and remove all AC power.  That has been done.  Also, the NVRAM has been cleared.

We are waiting for some downtime to put in another Quad GB NIC (HP NC364T).  The server is a passive node in Exchange, but dropping the NIC (switching to another NIC) causes issues with the Exchange DAG.  Waiting for MS to call back to walk us through retaining the replication level when switching NICs.

Thank you for your assistance.

Michael R.

[mdreinders]

As that would likely 'reset' the NIC, we need to wait to hear back from MS so that the Exchange DAG stays up and healthy while making setting modifications to the NIC.

Thank you for the suggestion.

Michael R.

[mdreinders]

The guys said that they were troubleshooting with HP and trying different versions of NIC firmware and Windows drivers, but, I definitely can roll back to the 4.6.xx driver.  As I've replied, I need to hear back from MS before breaking the NIC connection so they can walk me through maintaining our Exchange DAG healthy status.

Thank you for the help.

Michael R.

[ivan_D]

Hello,

I just wanted to confirm that i have the issue reported from some customers on diff. servers. there is workaronds in order to prevent poor networking performance:

Workaround by configuration :

This issue can be worked around by disabling LRO functionality by setting either of the following settings in the NIC properties under the Advanced tab:

• Large Receive Offload: Disabled

• TCP Checksum Offload (IPv4): Tx Only OR

Workaround by using alternate NIC driver version :

Alternately the previous version of the NIC driver (version 4.6.17.502) can be downloaded and installed from this HP website. This has been seen to resolve the issue in some cases.

If it is not effective then the alternate workaround action is to use Windows Device Manager to uninstall all instances of network adapters, and then rescan the system to reinstall the network adapters using the in-box NIC driver.

next release from the driver should fix the issue

The workarounds are confirmed as working from my customers!

p.s  this should affect the Qlogic chipset NICs.

I am HP employee

[KimO_1]

This is a known problem with the driver for the following devices:

• HP CN1000Q Dual Port Converged Network Adapter
• HP NC375i Integrated Quad Port Multifunction Gigabit Server Adapter
• HP NC375i 1G w/NC524SFP 10G Module
• HP NC375T PCI Express Quad Port Gigabit Server Adapter
• HP NC522SFP Dual Port 10GbE Server Adapter
• HP NC523SFP 10Gb 2-port Flex-10 Server Adapter

The issue is corrected in an updated driver: a workaround is to disable LRO in the driver properties.  The fix is included in versions 4.7.18.131 (Windows 2008/R2), and 4.7.20.131 (Windows 2012) and later:

• ftp://ftp.hp.com/pub/softlib2/software1/sc-windows/p2040148688/v85399 Windows Server 2012
• ftp://ftp.hp.com/pub/softlib2/software1/sc-windows/p1398775720/v85398  Windows 2008 R2
• ftp://ftp.hp.com/pub/softlib2/software1/sc-windows/p389205072/v85397  Win2K8 x64
• ftp://ftp.hp.com/pub/softlib2/software1/sc-windows/p677931535/v85396  Win2K8 x32

[Pete Cousins]

We had this exact problem with a DL580 G7 after a firmware and network driver update (to try and stop NICs resetting).

This latest driver fixed the problem.

Shame it's not the latest driver listed for DL580 G7 servers:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=4142792&prodTypeId=15351&prodSeriesId=4142916&swLang=8&taskId=135&swEnvOID=4064#11395