VERY slow network after many HW updates on DL370 G6

[mdreinders]

During our routine MS server patching last night, we wanted to update the firmware on the NICs on 2 DL370 G6s.  HP NCC552SFP cards.  During the attempt, it said we needed a newer BIOS.  So, we ran the whole gamut from SPP.  I don't have all the details, but the BIOS got updated to around 12/02/12. (P63).  NIC Firmware is now at 4.0.590.

Driver version in Server 2008 R2 SP1 is at 4.7.17.926.  The RAID Array controller was also updated (6.00).  We have the HP Network Configuration Utility saying we're running at 10GbE/Full (single port).

Anyway, after all the reboots,  running as part of an Exchange 2010 DAG, it's barely keeping up with maintaining replication between the other DAG member.  Copying files from my workstation to its C: drive shows VERY slow, and intermittent throughput.  Before last night, anywhere from 50MB/sec to 110MB/sec, in general.  Now, as low as 200kb/sec.  Ouch.

The team that worked overnight with Microsoft and HP tried downgrading the NIC firmware to older versions, downgrading the Windows driver to older and varying versions.  This morning, we actually re-IP'd the IP in use on the 10GbE adapter to one of the ports on the HP Quad 1Gb adapter.  We noticed exactly the same speed results.  Each NIC is plugged into a separate network switch.

We have 10 physical MB servers (Exchange), we did these HW/firmware upgrades on 2 boxes, and only these 2 boxes are seeing these results.  HP support is NOT being helpful.  They suggested clearing the NVRAM, which we did, no go.  We re-iterated Max Performance in the profile config in the BIOS, no help.

If anyone has ANY similar issues, please let me know what else we can try or what other diagnostic info you need to assist.  

Thank you!

Michael R.

[Jeroen_Kleen]

Hi Michael, I'm sorry to hear your trouble.

In regards to this issue: please do not hesitate to put your caseID here so we can follow your case at HP support.

In regards to this issue: are you able to test any local performance speed on the storage subsystem?  I would assume this is OK and performing as expected?  What if you copy local data against the \\localhost  to the same server; is that fast or slow?
Did you run already a full power-removeal on the server? As this could help? Please combine this as well with settting switch 6 on the systemboard so that NVRAM is clread in a forced way to factory defaults.

Did you try to install any other (intel?) NIC card in the server to see if this would behave equally? This would rule out a PCI-subsystem issue.  Please keep us posted with progress or that some of the above actions could be run .

Cheers, jeroen

[Abhishek Bangalore]

Hello,

Please try to perform the steps provided below and check for the occurence of the issue:

To prevent performance issues with NICs located in other slots, change the Receive Side Scaling (RSS) Rings from the default of 2 to 4. The number of RSS rings can be changed via Windows Device Manager. Select "Properties" for the appropriate device and then change the "Receive Side Scaling Rings" to 4.

Abhishek Sharma

[SteveA52]

Same problem with slowness on HP NC375I, through put could hardly exceed 1 mbps coping a file from the local fileserver. Rolled back driver under device manager from V4.7.17.926 to 4.6.17.502 and through put backup to 50 MBps on coping test. If you are using teaming then you will have to down grade NCU from 10.70.0.0 to 10.65.0.0 also. Seems there a bug in the latest driver update!

[mdreinders]

At this point, all local filecopy tests on the server itself are up to speed.  No issue there.

When you say full power-removal, I assume you mean shut the server down completely, and remove all AC power.  That has been done.  Also, the NVRAM has been cleared.

We are waiting for some downtime to put in another Quad GB NIC (HP NC364T).  The server is a passive node in Exchange, but dropping the NIC (switching to another NIC) causes issues with the Exchange DAG.  Waiting for MS to call back to walk us through retaining the replication level when switching NICs.

Thank you for your assistance.

Michael R.

[mdreinders]

As that would likely 'reset' the NIC, we need to wait to hear back from MS so that the Exchange DAG stays up and healthy while making setting modifications to the NIC.

Thank you for the suggestion.

Michael R.

[mdreinders]

The guys said that they were troubleshooting with HP and trying different versions of NIC firmware and Windows drivers, but, I definitely can roll back to the 4.6.xx driver.  As I've replied, I need to hear back from MS before breaking the NIC connection so they can walk me through maintaining our Exchange DAG healthy status.

Thank you for the help.

Michael R.

[ivan_D]

Hello,

I just wanted to confirm that i have the issue reported from some customers on diff. servers. there is workaronds in order to prevent poor networking performance:

Workaround by configuration :

This issue can be worked around by disabling LRO functionality by setting either of the following settings in the NIC properties under the Advanced tab:

• Large Receive Offload: Disabled

• TCP Checksum Offload (IPv4): Tx Only OR

Workaround by using alternate NIC driver version :

Alternately the previous version of the NIC driver (version 4.6.17.502) can be downloaded and installed from this HP website. This has been seen to resolve the issue in some cases.

If it is not effective then the alternate workaround action is to use Windows Device Manager to uninstall all instances of network adapters, and then rescan the system to reinstall the network adapters using the in-box NIC driver.

next release from the driver should fix the issue

The workarounds are confirmed as working from my customers!

p.s  this should affect the Qlogic chipset NICs.

I am HP employee

[KimO_1]

This is a known problem with the driver for the following devices:

• HP CN1000Q Dual Port Converged Network Adapter
• HP NC375i Integrated Quad Port Multifunction Gigabit Server Adapter
• HP NC375i 1G w/NC524SFP 10G Module
• HP NC375T PCI Express Quad Port Gigabit Server Adapter
• HP NC522SFP Dual Port 10GbE Server Adapter
• HP NC523SFP 10Gb 2-port Flex-10 Server Adapter

The issue is corrected in an updated driver: a workaround is to disable LRO in the driver properties.  The fix is included in versions 4.7.18.131 (Windows 2008/R2), and 4.7.20.131 (Windows 2012) and later:

• ftp://ftp.hp.com/pub/softlib2/software1/sc-windows/p2040148688/v85399 Windows Server 2012
• ftp://ftp.hp.com/pub/softlib2/software1/sc-windows/p1398775720/v85398  Windows 2008 R2
• ftp://ftp.hp.com/pub/softlib2/software1/sc-windows/p389205072/v85397  Win2K8 x64
• ftp://ftp.hp.com/pub/softlib2/software1/sc-windows/p677931535/v85396  Win2K8 x32

[Pete Cousins]

We had this exact problem with a DL580 G7 after a firmware and network driver update (to try and stop NICs resetting).

This latest driver fixed the problem.

Shame it's not the latest driver listed for DL580 G7 servers:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=4142792&prodTypeId=15351&prodSeriesId=4142916&swLang=8&taskId=135&swEnvOID=4064#11395

[sfora]

The just released SPP 2013.02 version B still does not solve this broken driver issue.

HP, please unlist the broken driver from the SPP or add the new one.

This advisory dates from April.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c03734205

[Crestj]

4 months after this driver appeared on their FTP site it is still not listed on any Proliant G7  driver page.

This driver fixed a ajor issue I had and instead of 4mb/s I now get 400mb/s throughput.

Why is this driver not public yet?

[BASF_FMA]

Hi,

we have very many G6 machines with Qlogic based NIC. The problem just occurs, if the backbone is set to 100 Mbit/s. 1 Gbit/s solves all your problems. Alternatively we added on Intel based NIC, 360T, I think. In VCenter it shows up as 82571EB.

Hope this helps,

BASF_FMA

[sfora]

SPP 2013.09.0B released yesterday finally includes an updated driver.

Thank you HP !

ftp://ftp.hp.com/pub/softlib2/software1/doc/p1822529277/v91225/SPP2013.09.0B-ReleaseNotes.pdf

HP QLogic P3P Multifunction Driver for Windows Server 2008 x64 Editions
Version: 4.7.28.523 (B) (Optional)

Fixes
This component addresses an issue which can result in installed adapters remaining in a disabled state when they should be enabled after installation is complete.

Supported Devices and Features

This driver supports the following network adapters:
o HP NC375i Integrated Quad Port Multifunction Gigabit Server Adapter
o HP NC375i 1G w/NC524SFP 10G Module
o HP NC375T PCI Express Quad Port Gigabit Server Adapter
o HP NC522m Dual Port 10GbE Multifunction BL-c Adapter
o HP NC522SFP Dual Port 10GbE Server Adapter
o HP NC523SFP 10Gb 2-port Flex-10 Server Adapter
o HP FlexFabric 10Gb 2-port 526FLR-SFP+ Adapter
o HP CN1000Q Dual Port Converged Network Adapter

[d3vnul]

Thanks for the update , i was looking for the release notes pdf and couldn't find it until i found your post :)

[waaronb]

FYI, this isn't entirely related since I'm dealing with a different network card, but the 2013.09 SPP that came out the other day has updated NIC drivers for the NC382i on my DL360 G7 servers (and many other models... it's the "HP Broadcom 1Gb Multifunction Driver) for Win64.

It's version 7.8.6.0 and it does something very strange... we're running Server 2012 with Hyper-V, and after installing this NIC driver, our virtual machines running on this box (2 different servers in fact, have the same issue) are having problems.

The network connection between the virtuals and any other system are sporadic.  I haven't taken the time to track it specifically to see if it's just dropping packets or what.

For instance, we have a virtualized Ubuntu 13.04 server running Couchbase.  Attempts to connect to the default port 8091 that Couchbase uses are useless, it just times out.  But if I connect from another virtual machine on the same physical host, it's okay.  It's only when it goes on the actual network that it freaks out.

It may be all communications on the server, even to the host OS, but a lot of things like RDP sessions are pretty tolerant of network issues.

The solution for me was rolling back to the previous drivers, version 7.4.14.0.  That's all it took, and all of a sudden, all the software running on those virtual machines like Couchbase, MSSQL, IIS, etc. started working just fine.

I'm glad this was on our test environment... if I hadn't caught that before deploying to our production servers, it would have been a very bad day.  It was bad enough as it is, dealing with developers and QA who couldn't do their work for most of a morning.

[raeberli]

Thanks waaronb for your post. We upgraded some 2012 Hyper-V Servers on Proliant 360/380 G7 hardware with SPP 2013.09 and had the same problems as you. Reverting back to 7.4 drivers resolved it too on our systems. 

[jason835]

Thanks waaronb for your post.

I installed the 7.8.6.0 driver on my dl380 g7 servers and it caused the handful of Windows XP machines that I have on my network to go from booting up in 2-3 minutes to taking ~30 minutes to boot up (including login).  They would get stuck at "Applying Computer Settings".  It also cause problems on Windows XP and Windows 7 machine with MS Outlook, they would get a "The server is unavailable" message.

I rolled back the driver to 7.4.14.0 and everything is ok now.

[SurendraKamath]

Hi All,

I have the similar problem in the HP NC364T PCIe Quad port card. Intially we had created LACP in the brocade switch level and configured the HP Teaming. It worked for two years without having any issues noticed. My collegue had updated the driver three months ago and he has not noticed the running driver version.

Now the teaming is slows down the switch and connected machines. We are getting request timout to the teamed server. We thought this problem is related with the switch and changed to single port trunk in the brocade switch and configured the SLB in the HP networking utility. It worked very well for two days and we got in to the similar problem. We have noticed the  below error in the windows event log.

Event ID 27 :HP NC364T PCIe Quad Port Gigabit Server Adapter #12 Link has been disconnected.  

Please let me know if any one know the fix for this issue.

Best, Surendra