Absolute nightmare of a DL380 G7

[Greg Rice]

Our whole shop is HP so it's hard to say we will never be going to them again. But, I've been building hp servers for 10 years (compaq for 2 of those :-)) and I just got 2 dl380 g7's in and never had so much issue. BSOD's everywhere. I also just got some BL460 g7's in. 2 for a SQL cluster and 6 to add to an ESX/VM farm. More than half had issues. Not impressed with the G7's HP. And calling support...I've never had such issues. If I tell you I can't get an OS on the hardware, and I've reseated everything, tried new iso's, tried new hd's...don't tell me to build on 1 CPU. That won't do anything but waste my time. Just admit the g7's are POS's.

[Martijn Goudkamp]

Replaced the 1GB flash today with HP's module they gave me last week. Let's see how long it stays alive this time.

For reference: the only difference between the old module is a little label with this version number: V. C05 (the old one has V. C03).

[BrightMinds]

HP's Level 2 Support has asked me to install the latest SP (did it ~6 weeks ago) and to rearrage the RAM modules.

I'll be beyond startled if rearranging the RAM modules makes any difference to the storage controller...

[Simon.H]

Well, my crashing server has been absolutely fine for the last week with the FBWC removed. Today I reinstalled the cache, and within four hours it had crashed twice.
I've now raised a case with HP to get the FBWC replaced.

[马明]

To Martijn Goudkamp


I THINK IT'S A FALUT MOTHERBOARD.i've experienced the same problem,after chaning a board,it's solved.and i think the automatic reboot or shutdown is caused by the ASR,i recommend you disable the ASR in the BIOS.
HP's dl380 g6 has this kind of problem with vmware ESX server.

[BrightMinds]

It's not the motherboard. Ours has already been replaced and it's on the latest BIOS.

It doesn't exhibit the ML350 G6 random reboot exactly, instead it becomes unresponsive and BSODs with error 0xF4. Then it (can) reboot and reports an error on the storage controller (error 1709).

[Simon.H]

Josh, get HP to replace the FBWC. My dodgy server has now been stable for 24 hours with a replacement FBWC, whereas before it would have crashed with the same symptons you are seeing 2 or 3 times !

[BrightMinds]

Mine seems to stay stable for around 5-7 days at the moment.

I'll reply to the support ticket and ask for a replacement FBWC.

[BrightMinds]

HP are sending out a replacement FBWC and interestingly enough they think it's a problem with the 1GB sticks. So they're sending me a 512MB stick to test.

I'll update you all with what happens.

Just to check with you Simon - are there any issues to be aware of when changing it? I've known of a busted controller & raid array in the past...

[Simon.H]

The FBWC card is tight on the right hand side of the motherboard, underneath the PCI cage, and is a little bit tricky to extract and re-insert if you don't remove the cage, but is possible. It has an external capacitor pack joined to it with a three wire pluggable lead which I found easier to connect to the card before plugging it back into the motherboard (But I didn't remove the PCI cage). Shouldn't be any problem with the raid array as that is all integrated on the motherboard.
Don't let them palm you off with a 512MB one as a permanent solution though, unless they are going to give you a discount on the cost difference !
The replacement card we received was a different revision than the dodgy one. I wonder if they are having to replace so many they have run out of stock !

Martijn, has your server been ok since swapping the FBWC ?

[Martijn Goudkamp]

Hi all, the new 1GB BBWC that I've replace last week Monday has NOT been blocking my server; the machine has been running ever since without any problems. Knock wood, also because I'm away on holiday :-)

Indeed, also my module had a different revision, see my previous post. So, all could be good!

[Jiad Elkadri]

we bought 5 DL380 G7's with the 1 GB FBWC. Out of the 3 servers that we've used so far, 2 of them have had these issues. the first server the motherboard and FBWC were replaced and that fixed the issue. the second one the tech had us update the bios and we're just waiting to see if it reboots again. I'm going to put operating systems on the other two and see if they reboot. we're an hp shop and have never had so many issues with servers as we have with these. they shouldn't be giving us a hard time with this. it seems like it's a global issue. if a customer calls in with these problems and with the specific hardware that we all have, They should just send out a new FBWC instead of wasting our time with these troubleshooting steps.

[Simon.H]

Hi Jiad,

the v3.52 P410i firmware did sort the issue for us on four of our servers that were exhibiting the problem. We've only had to swap the FBWC on one, so perhaps there are a couple of different issues, and HP now that, so that's why they are getting you to update the firmware.

[BrightMinds]

I'd love to be able to tell you how replacing the FBWC fixed everything, sadly though it looks as though they shipped the wrong part (see photo).

[Jiad Elkadri]

wow, it looks like updating the bios seems to be working for us so far. it's been one day with no reboots. we were already at the latest firmware for the p410i.

they had us go to the latest bios: 2010.10.19 (15 Nov 2010).

here's hoping that it doesn't reboot anymore.

[Enrique Olmos Jr]

I have been experiencing the same issue with 4 of my BL460 G6 servers. All get a BSOD that says "Hardware error Call your hardware vendor for support. IN the IML log i get these errors -1719 - A controller failure event occurred prior to this power-up
Uncorrectable PCI Express Error (Embedded device. I have upgraded firmware to newest, have swapped out Mother board with no luck. I have been working with HP for about a month and no resolution. They have told me to upgrade firmware, we chave changed bios settings for PCI and CPU and issue is still the same. All our servers have a p410i controller with 512 cache card. Can anyone shed any light to this issue??????
Thanks

[Martijn Goudkamp]

OK, it has been over 2 weeks since the 1GB BBWC has been replaced and: still up and running, without any problem! So, the combo of the right P410 firmware v3.52 (iLO 1.10) and a newer BBWC revision did the trick!

[BrightMinds]

I've just put in the 512mb replacement that HP have sent to test in our DL380. If it's still unstable it will fall over in less than 1 week.

The 1GB FMBC that I've pulled out has HP 4K1045 on the back and is labelled as REV.0C.

[Simon.H]

Enrique, have you got Battery Backed Write Cache or Flash Backed Write Cache connected to your P410i ?

If it's FBWC, you might have the same issue we have been suffering on the DL380 G7's. Suggest you make sure your P410i firmware is v3.52 and if you still have an issue, get HP to swap out the FBWC.

[Enrique Olmos Jr]

Simon.H
Thanks for your Reply. MY BL460 servers have a Battery Backed Write Cache connected to the Smart Array P410i.I have replaced the 512 mb cache card and still having same issue. The firmware is current on all our servers v3.52.I have yet to replace the battery. We received a batch of about 21 BL460's about 3 months ago and only 4 have the BSOD issue all have same Hardware config, all have firmware up to date.
I read on one of your comments you removed the cahce card, so what i have done is reomoved the cache card on one of the servers and will wait to see what happens. You also mentioned that when they replaced one of the cards the revision was different??

[Sanjeev Malhotra]

Hi all, so am I getting the gist that one should NOT buy DL380 G7's? I'm looking to purchase a few for a client; and myself have used DL380 G5's without issue for many years. Would any of you have any recommendations as for what (hp) server to get in it's place?

Thanks in advance, and I hope you've all resolved your G7-related issues!

[Simon.H]

Enrique, apologies for not replying sooner. Yes, the FBWC cards that HP have been sending as replacements are revision C.05, whereas the original ones we had were revision C.03, but this will be irelevant to you as you have got BBWC not FBWC.

Sanjeev, the problems we have been having have been tied down to some FBWC cards and the P410i array controller firmware, so as long as you know that if you have this issue, you need to get HP to swap the FBWC card, then personally, I would say that the DL380 G7 is fine to buy.

[BrightMinds]

I agree, it's been a real nightmare for us to have an unstable, business critical SQL server.

However, I replaced the FBWC last week and (touch wood) it's been fine since. So if you're happy knowing that you may need to order and wait for a replacement FBWC then fine.

Personally I'm not happy with the support HP have given. Clearly I'm not alone with this issue - the post on my blog gets around 100 hits a day! http://www.geakeit.co.uk/2010/11/11/review-avoid-the-hp-dl380-g7/

They should have made their support staff more aware of this issue and I hope they will soon.

Other than that the server seems fine, it won't stop me going for a Dell/IBM next time though.

[Sanjeev Malhotra]

Thanks very much for your advice Simon, much apreciated.

Best of luck to you all!

[Ali]

Hi Brightminds,

I agree with you on no of posts for the similar issue and this needs to be brought to HP Engineering team so that they work out a permanent solution.

MY experience for the similar issue on G7 server is :
Controller lock-up, malfuction or error on the post could be related to devices/components connected to the controller or controller itself.

BBWC/FBWC is first point of suspection, then controller then other devices like hdd, backplane etc.

cheers,
Aftab