ProLiant Servers (ML,DL,SL)
1748003 Members
4754 Online
108757 Solutions
New Discussion юеВ

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

 
SOLVED
Go to solution
Ken Krubsack
Trusted Contributor

SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Folks,

 

Is there some known flakiness in the combination of SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1?  I've tried this install on two different servers and both have done the same thing: go into a boot loop just after the W2K8 splash screen comes up.  Both are upgrades from SPP 2013.09.  It is hosed enough I have to rebuild the server.  Fortunately, one was pre-production and the other was a purpose-built test server, so it hasn't cost me anything except time and aggravation. 

 

We've run 2014.02 against other hardware and win2K12 (with and without R2) with no problems.

 

Thanks

Ken

19 REPLIES 19
Torsten.
Acclaimed Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

SPP 2014.02 or SPP 2014.02(B)?

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Torsten,

 

Looks like the original release.  I'm downloading 2014.02 B now.

 

The original had some problems, I take it?

 

Thanks

Ken

waaronb
Respected Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

The Broadcom NIC firmware update in 2014.02 had a nasty issue where it would try to flash a CCM update onto NICs that don't actually have that feature (I forget what CCM stands for).

The re-released 2014.02(B) has an udpated NIC firmware that fixes that (it won't try flashing CCM onto unsupported NICs).

There are also a few other updated packages in there... I think (B) might have the newer SMH and VCA versions to fix the Heartbleed vulnerability.

If you have a server where the Broadcom flash messed up your network cards, HP says to call them for help fixing it, or you can find the thread in this section of the HP forums where some people have devised a method to fix it by flashing it back using some other tools.

I don't think that would cause a bootloop though... people said it made their NICs disappear in Windows or show errors in device manager, but their machines still managed to boot. You might have a different problem if you're bootlooping. I'd try hitting F8 during boot and tell Windows not to auto-reboot on an error so you can see what kind of stop error you're getting and what driver is was choking on.
Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Well,

 

2014.02 (B) did the exact same thing.  I don't think it is this particular server - I've had two of them with the same problem.

 

What I've done for the testing is break our system drive mirror pair and only screw up one side of it.  This is why I don't think the problem is that flash from 2014.02 (original) which did get run on this box - when I reload the good side of the mirror pair (with the 2013.09 driver pack) life is good.  This tells me the NICs are still healthy and I don't need to be calling HP to unscrew physical hardware.

 

Ken

 

waaronb
Respected Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Were you able to nail the reboot down to a particular driver causing a BSOD or anything?

 

I've never fully trusted the HPSUM automated updating.  Maybe it's the perfectionist in me, but when new drivers or mgmt software comes out, I give the cp*.exe a friendlier name and then install the updated parts manually on my servers.

 

If you have hundreds of servers then I can understand the desire for the automated updates, but if you only have a handful of servers, it seems safer to do them as needed.

 

Right now I'm managing 18 physical Proliant servers (down from 30, but we virtualized a bunch over the past year), and updating each machine doesn't really take too long... I try to do it at the same time I'm doing Microsoft patches, and I'll start with my QA servers first in case things go bad.

 

My last job, I managed 50 or 60 Proliants (G1 through G4 systems) and I still did the updates manually, although I admit for a few that didn't require reboots like SMH or something I would script it by remotely execuing the cp*.exe file with the unattended install options.

 

Firmware updates in particular scare me if they're automated... those require reboots anyway and since almost all of my physical servers are virtual hosts, I have to coordinate when each one reboots so I can rebalance the loads... we're a 24/7 organization with essentially no maintenance window, so everything is highly available/fully redundant... it's a delicate procedure to reboot a machine... I just don't trust the HP  SUM to do it right, as timesaving as it might be.

 

Sometimes I see HPSUM auto-selecting some updates that don't really apply to the server (those tape driver updates get annoying for example... we don't use tape drives).  And sometimes if it can't determine the version of some firmware, it selects the latest one to be applied anyway, even if the server already has that version installed (maybe the ilo management driver wasn't installed yet so it couldn't read the ILO version).

Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

No, I haven't been able to nail down the driver.  Right now my test box is booting into Windows Recovery Mode.  It's more lack of time to dig into it that has kept me from possibly figuring out which driver hosed me up.

 

What I do remember from the first time this happened with 2014.02 (not B) was that I was able to do a "Last Known Good" boot, but the network stack was hosed enough that I was unable to configure IPv4 and no matter what I did to try to get it back (ripping out all the NICs and letting Windows re-find them, backing down the driver revs, etc) it didn't work.  Only option was to rebuild the server.  That tends to make me think it might be the NIC driver that screwed things up, but I can't prove that right now.

 

Unfortunately, manual installation/reboot isn't an option for us.  We have almost 700 physical servers, and 320 of those are DL385 G7's.  We do have a 5 hour maintenance window once a week, and it takes us a couple of months to get them all rolled out.  The previous versions of HP SUM worked quite well - we stood up a few dedicated HP SUM servers and just pushed the updates out about 10-12 servers at a time.  Worked fine through 2013.09.  This 2014.02 interface needs some help - it works fine as a local upgrade, but trying to get the across the network pushes going is a challenge to say the least.  Clear, concise, documentation would be wonderful (HINT HINT HINT).

 

Ken

sfora
Frequent Advisor
Solution

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Ken,

 

It is a problem with the NIC driver update package and was not fixed in 2014.02(B)

It affects G6 and G7 servers.

driver: HP Broadcom 1Gb Multifunction Drivers for Windows Server x64 Editions

package: cp021772.exe

 

read this post for more info, the easiest is to disable that driver update when running the SPP2014.02, or stay with SPP2013.09.

 

http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-SL/SPP-2014-02-on-DL380-G6-problem/m-p/6435816

waaronb
Respected Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Oh wow, that's interesting... Fortunately I didn't have that problem with my G5/G6/G7 boxes or I would have been pulling my hair out.

I wonder if it's because I was running the older unbuggy version of the driver from 2 revisions ago, and then installed that new one when it was finally available. Maybe the install problem was only if you had that 7.8.6.0 buggy version already installed.

Whatever the case, it's definitely good info.

And for the original poster, Ken: Whenever I feel like I have too many servers to manage, I'll remember you and your 700 physical servers to maintain. :)
Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

sfora,

 

Thanks for that update - kinda bears out what I've been suspecting.  Also explains why we haven't seen it on the Gen8 servers we have.

 

Doesn't look like that driver was replaced in the 2014.04 supplement, either.  I've dug through that and found nothing talking about Broadcom or NIC driver updates - just a bunch of stuff for array controllers (unless I've missed something, which is entirely possible).

 

Ken