ProLiant Servers (ML,DL,SL)
cancel
Showing results for 
Search instead for 
Did you mean: 

SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

SOLVED
Go to solution
Ken Krubsack
Trusted Contributor

SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Folks,

 

Is there some known flakiness in the combination of SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1?  I've tried this install on two different servers and both have done the same thing: go into a boot loop just after the W2K8 splash screen comes up.  Both are upgrades from SPP 2013.09.  It is hosed enough I have to rebuild the server.  Fortunately, one was pre-production and the other was a purpose-built test server, so it hasn't cost me anything except time and aggravation. 

 

We've run 2014.02 against other hardware and win2K12 (with and without R2) with no problems.

 

Thanks

Ken

19 REPLIES
Torsten.
Acclaimed Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

SPP 2014.02 or SPP 2014.02(B)?

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Torsten,

 

Looks like the original release.  I'm downloading 2014.02 B now.

 

The original had some problems, I take it?

 

Thanks

Ken

waaronb
Respected Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

The Broadcom NIC firmware update in 2014.02 had a nasty issue where it would try to flash a CCM update onto NICs that don't actually have that feature (I forget what CCM stands for).

The re-released 2014.02(B) has an udpated NIC firmware that fixes that (it won't try flashing CCM onto unsupported NICs).

There are also a few other updated packages in there... I think (B) might have the newer SMH and VCA versions to fix the Heartbleed vulnerability.

If you have a server where the Broadcom flash messed up your network cards, HP says to call them for help fixing it, or you can find the thread in this section of the HP forums where some people have devised a method to fix it by flashing it back using some other tools.

I don't think that would cause a bootloop though... people said it made their NICs disappear in Windows or show errors in device manager, but their machines still managed to boot. You might have a different problem if you're bootlooping. I'd try hitting F8 during boot and tell Windows not to auto-reboot on an error so you can see what kind of stop error you're getting and what driver is was choking on.
Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Well,

 

2014.02 (B) did the exact same thing.  I don't think it is this particular server - I've had two of them with the same problem.

 

What I've done for the testing is break our system drive mirror pair and only screw up one side of it.  This is why I don't think the problem is that flash from 2014.02 (original) which did get run on this box - when I reload the good side of the mirror pair (with the 2013.09 driver pack) life is good.  This tells me the NICs are still healthy and I don't need to be calling HP to unscrew physical hardware.

 

Ken

 

waaronb
Respected Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Were you able to nail the reboot down to a particular driver causing a BSOD or anything?

 

I've never fully trusted the HPSUM automated updating.  Maybe it's the perfectionist in me, but when new drivers or mgmt software comes out, I give the cp*.exe a friendlier name and then install the updated parts manually on my servers.

 

If you have hundreds of servers then I can understand the desire for the automated updates, but if you only have a handful of servers, it seems safer to do them as needed.

 

Right now I'm managing 18 physical Proliant servers (down from 30, but we virtualized a bunch over the past year), and updating each machine doesn't really take too long... I try to do it at the same time I'm doing Microsoft patches, and I'll start with my QA servers first in case things go bad.

 

My last job, I managed 50 or 60 Proliants (G1 through G4 systems) and I still did the updates manually, although I admit for a few that didn't require reboots like SMH or something I would script it by remotely execuing the cp*.exe file with the unattended install options.

 

Firmware updates in particular scare me if they're automated... those require reboots anyway and since almost all of my physical servers are virtual hosts, I have to coordinate when each one reboots so I can rebalance the loads... we're a 24/7 organization with essentially no maintenance window, so everything is highly available/fully redundant... it's a delicate procedure to reboot a machine... I just don't trust the HP  SUM to do it right, as timesaving as it might be.

 

Sometimes I see HPSUM auto-selecting some updates that don't really apply to the server (those tape driver updates get annoying for example... we don't use tape drives).  And sometimes if it can't determine the version of some firmware, it selects the latest one to be applied anyway, even if the server already has that version installed (maybe the ilo management driver wasn't installed yet so it couldn't read the ILO version).

Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

No, I haven't been able to nail down the driver.  Right now my test box is booting into Windows Recovery Mode.  It's more lack of time to dig into it that has kept me from possibly figuring out which driver hosed me up.

 

What I do remember from the first time this happened with 2014.02 (not B) was that I was able to do a "Last Known Good" boot, but the network stack was hosed enough that I was unable to configure IPv4 and no matter what I did to try to get it back (ripping out all the NICs and letting Windows re-find them, backing down the driver revs, etc) it didn't work.  Only option was to rebuild the server.  That tends to make me think it might be the NIC driver that screwed things up, but I can't prove that right now.

 

Unfortunately, manual installation/reboot isn't an option for us.  We have almost 700 physical servers, and 320 of those are DL385 G7's.  We do have a 5 hour maintenance window once a week, and it takes us a couple of months to get them all rolled out.  The previous versions of HP SUM worked quite well - we stood up a few dedicated HP SUM servers and just pushed the updates out about 10-12 servers at a time.  Worked fine through 2013.09.  This 2014.02 interface needs some help - it works fine as a local upgrade, but trying to get the across the network pushes going is a challenge to say the least.  Clear, concise, documentation would be wonderful (HINT HINT HINT).

 

Ken

sfora
Frequent Advisor
Solution

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Ken,

 

It is a problem with the NIC driver update package and was not fixed in 2014.02(B)

It affects G6 and G7 servers.

driver: HP Broadcom 1Gb Multifunction Drivers for Windows Server x64 Editions

package: cp021772.exe

 

read this post for more info, the easiest is to disable that driver update when running the SPP2014.02, or stay with SPP2013.09.

 

http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-SL/SPP-2014-02-on-DL380-G6-problem/m-p/6435816

waaronb
Respected Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Oh wow, that's interesting... Fortunately I didn't have that problem with my G5/G6/G7 boxes or I would have been pulling my hair out.

I wonder if it's because I was running the older unbuggy version of the driver from 2 revisions ago, and then installed that new one when it was finally available. Maybe the install problem was only if you had that 7.8.6.0 buggy version already installed.

Whatever the case, it's definitely good info.

And for the original poster, Ken: Whenever I feel like I have too many servers to manage, I'll remember you and your 700 physical servers to maintain. :)
Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

sfora,

 

Thanks for that update - kinda bears out what I've been suspecting.  Also explains why we haven't seen it on the Gen8 servers we have.

 

Doesn't look like that driver was replaced in the 2014.04 supplement, either.  I've dug through that and found nothing talking about Broadcom or NIC driver updates - just a bunch of stuff for array controllers (unless I've missed something, which is entirely possible).

 

Ken

waaronb
Respected Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

I've noticed a bunch of 7.8.50.0 mini-updates... the ones where they keep the version the same but give it a (B) or (C) add-on.  The release notes for 7.8.50.0 (D) don't mention anything, but usually when HP does that, they're fixing an installation issue, not the driver itself.

 

For HP to go through 4 different revisions for this one driver seems to say they know there's some problem and have been trying to fix it (unsuccessfully a couple times?).

 

The most recent one, revision (D), is out now with a release date of May 16: cp023518.exe

 

I'd test that out on a few machines and then modify the SPP/HPSUM to include it instead of that older one, which I think was 7.8.50.0 (B).

 

Here's the typical super long link to that driver from the DL380 G7 model:

http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?sp4ts.oid=4091432&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3Didx%253D%257CswItem%253DMTX_c7164df4c48141cea2a44818f3%257CswEnvOID%253D4168%257CitemLocale%253D%257CswLang%253D%257Cmode%253D%257Caction%253DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vig...

Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Waaronb,

 

I have been through modifying the driver packs inside of the PSP/SPP, so I know how to do that.

 

To answer your one other point from earlier in this thread, I quit trusting the automatic mode on the PSP back when version 9.00 came out.  Reason being, that one did automatic DOWNGRADES without prompting.  I had just fought through some ILO FW/driver issues that had servers rebooting on us, and did HP-recommended manual upgrades to stabilize things, and then that turkey came out and automatically down-revved the firmware on a few ILOs on us before I caught it.

 

I suggest that anyone experiencing these current issues that is going to HP Discover stop by the appropriate part of the display area, find the right people, and growl about the problems in SPP 2014.02.  Also, if there is a session of the future of these things, that would be another good place to drive these issues home.  If I were going, I'd be doing just that.

 

Ken

Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

waaronb,

 

Regarding the Broadcom driver versions - the version from 2013.09 is 7.8.6.0, which is fine.  The version in 2014.02 (B, anyway) that is flaking off is 7.8.50.0.

 

Ken

Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Waaronb,

 

Just ran a test of 2014.02 with eliminating the Broadcom driver update.  The server booted to Windows and is talking on the network properly.

 

The only thing I saw that is less than perfect is that the HP Network Config Utility v10.85 (from 2014.02) didn't like the 7.8.6.0 (from 2013.09) Broadcom drivers - to the point that it won't start, so you lose all that functionality.  Once I backrevved the utility to v10.80 (from 2013.09), life is again good.

 

So, if you want to maintain full functionality,  there are two pieces you don't want to upgrade in 2014.02: the Broadcom driver and the network config utility.

 

Ken

waaronb
Respected Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Oh yeah, I forgot about the latest NCU not working with older NIC drivers.

 

Warning on the 7.8.6.0 Broadcom multifunction driver though... that was the version that caused a lot of problems for me and some other folks on Server 2012, specifically with Hyper-V.  It was a real disaster... the symptoms were flaky network connections, almost like it forgot how to fragment packets.

 

Downgrading to 7.4.14.0 fixed it.

 

I have some 2008 R2 and 2012 R2 servers with the latest 7.8.50.0 driver (I don't remember what revision we used to install), but thankfully none of them had problems.  But I installed that manually, not through SPP, so that may have made all the difference.

sfora
Frequent Advisor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

If I remember correctly SPP 2013.02 also had a problem with network drivers (slow network issues). Quality control is really lacking...

Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

The only place we've run into issues with 2014.02 and the NICs is the 385 G7/W2K8 R2 combo.  I haven't seen this flakiness on any other combo of hardware and windows.

 

Yet.

 

Ken

Ken Krubsack
Trusted Contributor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

We have a workaround, or so it seems.

 

If I take the CP023518.EXE Broadcom driver [7.8.50.0 (D) from the link above] and replace the CP021772.EXE version of the driver that is included in SPP 2014.02, the SPP installation will run successfully and reboot.  I have one successful test under my belt and will check a couple of others.

 

No other driver replacements seem to be necessary.

 

Thanks for the help, guys.

sfora
Frequent Advisor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

SPP 2014.06 includes this fixed installer cp023518.exe Version:7.8.50.0 (D) (16 May 2014)

Rod Coleman
Occasional Visitor

Re: SPP 2014.02 + DL385 G7 + Win2008 R2 SP1 U1 = bad news

Anyone else experienced the error below?

 

I have applied SPP 2014.06, but cp023518 failed. I tried cp023518 manually, but got the same error.

 

ProLiant DL380 G7. Windows Server 2008 R2 + SP1.

 

error.jpg

 

Cheers

Rod