New 1Gb Broadcom Multifunction Win64 driver (7.8.6.0) is buggy

[waaronb]

The 2013.09 SPP that came out the other day has updated NIC drivers for the NC382i on my DL360 G7 servers (and many other models... it's the "HP Broadcom 1Gb Multifunction Driver) for Win64.

It's version 7.8.6.0 and it does something very strange... we're running Server 2012 with Hyper-V, and after installing this NIC driver, our virtual machines running on this box (2 different servers in fact, have the same issue) are having problems.

The network connection between the virtuals and any other system are sporadic.  I haven't taken the time to track it specifically to see if it's just dropping packets or what.

For instance, we have a virtualized Ubuntu 13.04 server running Couchbase.  Attempts to connect to the default port 8091 that Couchbase uses are useless, it just times out.  But if I connect from another virtual machine on the same physical host, it's okay.  It's only when it goes on the actual network that it freaks out.

It may be all communications on the server, even to the host OS, but a lot of things like RDP sessions are pretty tolerant of network issues.

The solution for me was rolling back to the previous drivers, version 7.4.14.0.  That's all it took, and all of a sudden, all the software running on those virtual machines like Couchbase, MSSQL, IIS, etc. started working just fine.

I'm glad this was on our test environment... if I hadn't caught that before deploying to our production servers, it would have been a very bad day.  It was bad enough as it is, dealing with developers and QA who couldn't do their work for most of a morning.

[broca002]

I can confirm I have have seen the exact same strange buggy behaviour with Broadcom 7.8.6.0 driver on 2012 Hyper-V. As with you rolling back to the previous driver resolved the issue.

[waaronb]

Whew, I'm glad it's not just me then.  Hopefully HP will find this thread and fix it, and other people who may run into issues might stumble on this thread.

For me, since rolling back to the old driver fixed it, and nothing in the new version was important to me, I didn't think it was worth opening a ticket with HP.

Also, just to cover my bases, I rolled back the NIC driver for my other Hyper-V hosts that have different network cards besides that 382i... just in case.

 also rolled back the driver for my systems that aren't running Hyper-V.  I hadn't seen any issues with them in particular, but why take a chance?  I don't know for sure that the issue ONLY affects virtuals... maybe it's just easier to spot.  I had installed the new driver on 2 or 3 of my production IIS servers when I noticed the Hyper-V problem, so I didn't really check back to see if those servers had been having any communications issues, but I'll just wait until HP addresses the issue before installing it on ANY systems.

[PZel]

Does anybody has  seen problems with this driver under W2K8 R2 ??

(or its it only buggy under 2012 ???)

[waaronb]

I had it installed on a couple of 2008 R2 servers that we use as IIS servers.  I didn't notice any trouble, but when we saw the issues with Server 2012 and Hyper-V, I decided to play it safe and roll back the driver version on those 2008 R2 servers also.

I didn't see anything in the release notes for the new driver version that I cared about, so I didn't want to take a chance that our live web traffic could have problems.

[ThomasTrain]

Glad to see its not just me.  I perform an annual update of all my systems to the latest SPP, in this case to 2013.09 as the rest of you.

I'm running Windows 2008 R2 with Hyper-V and also immediately noticed major problems after the update affecting network traffic on my guest VM systems (all running Windows 2008 R2).  My issues were massive packet loss, particularly to/from anything outside of the Hyper-V virtual switch but most noticably affecting HTTP/HTTPS traffic.

I ended up rolling back the NIC Firmware and drivers to resolve my issues (before I found this thread). 

I definately recommend avoiding the 7.8.6.0 broadcom drivers if you run Hyper-V though there are likely additional issues that I didn't see [yet] so I'm removing it from everything.

[zeroagemain]

We have just had exactly the same issue (huge network issues after upgrade to SPP 2013-09-0B) but this was on a DL380 G6 with NC382i but simply running Windows Server 2008 x64, no R2, no 2012 and no Hyper-V.

We also had to roll back to 7.4.14.0 to fix it.

[Stephan G]

It broke our NPS on Windows 2k8 and our Lotus Domino replication on Windows 2k8. Both physical.

Stay away from this driver and HP .. second network problem with one of your updates in the last months .. so fix this as fast as possible or just remove it from the homepage.

[robbied31]

I am having the same exact problems with a new Hyper-V Cluster on 2-DL360e gen8 servers running Server 2012 r2.  They have the 4-port 331 FLR network adapters and I am using drivers from the 9-30-13 SPP.  I am not using the multifunction driver, It won't allow me to install it. The problem is the only driver version available for the HP
Broadcom 1Gb Driver for Windows Server x64 Editions (American,
International) is the 16.0.0.17.  There is no previous version available for me to try.  I have only one option. 

does anyone know where I can find a previous version of drivers?

Thanks.

Rob

[waaronb]

I have a few DL380p Gen8 with the 331FLR adapters, and I'm using Hyper-V on Server 2012 with those okay.

However, I just realized that apparently I'm using the drivers that came with Windows Server, not the HP drivers.  Which is odd, since I used the SPP to install.  Well, it doesn't show up in my Version Control Agent... the driver version says it's 15.4.0.19 which looks like the previous one you're looking for.

On the link you have to that latest version, if you click on revision history you can find links to the older versions and download those.  Here's the link to the previous version (15.4.0.19b):

HP Broadcom 1Gb Driver for Windows Server 2008 x64 Editions 15.4.0.19 (B)

[robbied31]

Thank You,

I had actually found the previous version to the one you linked to. 15.4.0.17 which has fixed my issues for now.  I will try 15.4.0.19 next.

Rob

[mpiana70]

do you manage to get it working at 1 gb ?

[stryqx]

Wish I found this two days ago...

Exact same problem  - VM to off-host traffic was all FUBAR.

After much trial and error of turning one thing off at a time and testing (all those frakking offload settings),  I've found that disabling VMQ on the Hyper-V host NICs that are physically connected and also disabling VMQ in each VM's configuration has restored performance for VM to off-host traffic.

Why HP persist with putting such cr*ppy NICs onboard their servers is beyond me.

[waaronb]

Thanks for putting in some time troubleshooting exactly where the problem is. I was too lazy to bother going through all of the (many) settings, especially knowing that it still might not work.

I guess the VMQ settings were an obvious place to look, in hindsight, since at least for me, the problems were all related to virtuals. My connection to the host machine seemed fine, but I wasn't testing that specifically. My remote desktop connection didn't seem to be suffering at any rate.

Well, this should be a good option for anyone who needed a fix from this version, so they can work around the HUGE bug it introduced. Or someone who installed this and can't roll back for some reason. It's sure a lot of work though. Still probably easier to roll back. But I'm sure HP and/or Broadcom appreciates you doing the legwork for them. :)

[ZackInMA]

I also had this issue and can confirm that rolling back the driver to version 7.4.14 did correct the issue. Thanks for posting so we could share and learn from each other's experiences. Sad but true is that we learn more from each other than the manufacturer.... 

[Tuur]

Other drivers from broadcom Multifunction driver (7.8.21.0) seems to have the same issue. Downgrading works. We had problems with Altiris PXE and SCCM PXE.

This page discripes the SCCM isssue

http://translate.google.nl/translate?hl=en&sl=pl&u=http://sebastiangrugel.blogspot.com/2013/12/downgrade-sterownikow-broadcom-hp.html&prev=/search%3Fq%3D7.8.21.0%2Bbroadcom%2Bmtu%2B1486%26biw%3D1920%26bih%3D1053

On all servers we checked had a MTU size of 1486, downgrading the driver changed de MTU size back to 1500.

Fixing the MTU size with the new driver also solved the PXE issue for us :

netsh interface ipv4 set subinterface interface=<interfacename> MTU=1500

[waaronb]

I'm a little shocked that HP hasn't addressed this issue yet, considering it must be affecting a lot of customers who installed the latest driver.

So if you did (for some reason) want the latest driver installed, make sure the MTU is still 1500, and disable VMQ.  I wonder why they changed the MTU... that seems like a really strange thing to do.

Between this and their new policy that, sometime this month, you'll need an active contract or warranty to access driver downloads, I'm not enjoying the HP support experience as much lately.

Although when I need to fix some server item under warranty, they do handle that really fast.  Overnight cross shipments, and as long as I do the proper diagnostics first, they're happy to send the requested part without bothering me with questions.

So it's not all bad, but it is pretty bad when a new driver version can knock a server offline and HP doesn't even put out a notice, warning people and advising to downgrade if they run into issues?

[Oscar A. Perez]

Have you tried the new driver 7.8.50.0?
Fixes:

This driver corrects an issue that could result in halted traffic when configuring jumbo frame size to 9000.

This driver corrects an issue that could result in a Windows Stop Error (BSOD) when a large number of ports are repeatedly disabled.

http://www.hp.com/swpublishing/MTX-e28e4668ad3f4f028fdf4e802e

[waaronb]

Weird, that must have just come out?  "7.8.50.0 (B) (18 Feb 2014)"

In other news, HP invents time travel.  This version still isn't showing up, at least on the DL360 G7, Server 2012 list of software.  Which product page and OS version did you find this hidden gem? :)

The description of the fix doesn't quite sound like the problems noted in this thread, but maybe I can give it a shot over the weekend on a test system, a DL360 G7 running Server 2012, as a Hyper-V host.  This is one where I saw access to the virtual guest machines dropping network traffic left and right with the 7.8.6.0 driver.

[Oscar A. Perez]

I have a Flux Capacitor on my DeLorean :)

Let me know if it works.

[waaronb]

Well, I haven't tried the 7.8.50.0 driver yet, but over the weekend I updated one of my dev environment DL360 G7 systems from Server 2012 to Server 2012 R2.

I had the working 7.4.14.0 driver on there before, but apparently during the upgrade to Server 2012 R2, it upgraded the network driver to 7.8.6.0.

I didn't notice, and my initial testing was just on the server itself, not any of the guest machines.

So I come in Monday morning and things are all out of whack on the test systems and I get a flurry of complaints from the developers.

I tried disabling the VMQ option on all of the adapters (they're all teamed together, so I did it on all 4 of the physical NICs). But that didn't do anything.

I also checked the MTU but since they're all teamed using the Server 2012 built-in teaming thing, there was just that one team adapter with IP4 bound to it, and it's MTU was 1500 just like it should be.

In other words, I checked the couple of things people mentioned might get this newer driver working, but with one or both of them, we still had a lot of problems where the virtual guest machines were having connection issues. Particularly when the packet of data is over a certain size. It seems like that points to an MTU issue, but I'm not sure.

For instance, I could remote desktop onto a virtual Windows machine okay, and maybe I was doing network retries I wasn't aware of, but it worked.

From that virtual guest, I was trying to access a web page that might return 10K of data or more. That was *always* failing. If I tried getting HTML that was small, like a few hundred bytes, then it worked.

It almost acted like the MTU was weird and it wasn't fragmenting properly. Like when you try a ping larger than the MTU and set the do-not-fragment bit.

But none of the IPv4 advanced settings I scanned over seemed strange, and simply installing the old version again fixed it, and I doubt downgrading the driver would change any network settings.

By that point I was not feeling brave enough to try that 7.8.50.0 version, especially since it doesn't mention anything in the fixes that seem relevant. It works on the older version and I'm just going to leave it there until I see the problem and solution mentioned in the release notes. :)

[Herr Franzen]

Hi there, we have had problems with NC382i on some HP Servers with Windows 2008 R2.

Check "ping -l 5000 <servername>", there may be no reply, means your server network card does not fragment the network data.

This may help for windows:

Start a cmd box with ping -t - l5000 <servername>

Now open Windows System Control and Open HP Network Control, the following windows shows HP Network Control Option, now look for then ping window and press in the HP Network Control the OK Button, maybe one or two seconds later the ping will start to respond.

At the Moment you click o.k. the Network Connection will be lost for a little moment, when it is back the problem is away too. Means the network card fragments the data. We have tested with server reboot, network card will work fine.

Please reply to me if this works for your configuration.

Greetings from Germany by Peter

[waaronb]

I'm not declaring success quite yet, but I did just update a couple of my servers to the 7.8.50.0 drivers, and so far it seems like it's working without doing any other modifications, fixes, workarounds, etc.

This was on a couple of DL360 G7's running Server 2012 (not R2).  Both the host OS and the guest Hyper-V systems seem to be doing okay.  No dropped packets or any of the other weird behavior I saw with 7.8.6.0.

I'll have a better idea tomorrow once these systems get a little more real world traffic under their belt, but the initial tests look promising.

[waaronb]

It's been about a week now running my Hyper-V hosts on the 7.8.50.0 version of the driver, and it does seem like the issue is gone.  Funny that the release notes didn't seem to say anything related to this problem.

The notes for 7.8.50.0 say this:

Fixes
- This driver corrects an issue that could result in halted traffic when configuring jumbo frame size to 9000.

- This driver corrects an issue that could result in a Windows Stop Error (BSOD) when a large number of ports are repeatedly disabled.

As a side note, on my Hyper-V hosts, these are Server 2012 systems and I use the native Microsoft teaming service, with all of the NIC ports set to Hyper-V Port load balancing (not aggregated or anything since they're connected to redundant switches on separate power feeds).  I don't know if that would make a difference or not.

We're not using jumbo frames, so that wouldn't apply.  I guess it's just some undocumented thing they slipped in or fixed silently.

[SebastianGrugel]

Hi waaronb can you tell me if after install version  7.8.50 value of MTU was still 1500 or change ?

In what exactly HP server and system this works ? Only Hyper-V Server 2012 ?

Grettings