Help! BIOS update failed to reboot DL165 G5

[Kevin Heatwole]

I'm going to upgrade the processors in my DL165 G5 and my dealer recommended that before I do anything I should upgrade all firmware to the latest.

The BMC was running version 3.01 and the BIOS was O13 series version 2008.08.15. When I looked at the update BIOS page, I saw that the latest version requires 3.11 BMC firmware so I made a USB Key and upgraded to 3.11. The system rebooted and all was fine.

Next, I returned to the HP download page and saw that the latest BIOS was 2010.02.26, BUT DID NOT NOTICE THE IMPORTANT NOTE THAT I HAD TO ALREADY BE RUNNING A MORE RECENT BIOS TO UPGRADE.

So, I upgraded to 2010.02.26. All looked good for the upgrade, but when I got the screen to power cycle to reboot, the reboot sent nothing to the console (no nice HP graphic while it POSTs). I rechecked the web and then noticed the important note and I should have upgraded to 2009.11.03 first (if the note is to be believed).

So, I am stuck with a system that the green power light goes on (and stays on), but it won't even start to boot (no output on the concole).

I looked at the documentation and it says I can recovery the firmware, but putting the 2009.11.03 boot rom on a USB Floppy and calling the file AMIBOOT.ROM. I went out and bought a floppy drive and put the file on it, but the system doesn't even begin to access the USB Floppy (just the green power light on and some other lights on the motherboard/P400 controller light up).

I then saw that there is some "crisis recovery jumper" somewhere on the motherboard that the Maintenance and Service manual is supposed to tell you where it is, but doesn't. I do see a 2-pin jumper labeled P26 on the motherboard that is supposed to set the Chasis ID1 or ID2. It says to open up the server and jump pins 2 and 3, attach the floppy, and connect power. Then, it should access the floppy. But, nothing happens when I change this P26 jumper. Maybe this isn't the "crisis recovery jumper"?

So, I am stuck. Can anyone help? The system is really a D2D2503i/EH945A Backup server but is just a DL165 G5 with some extra parts (including the P400 controller with BBWC).

I've tried all sorts of other things, but so far, nothing is responsive. I can get to the BMC through the web interface, but nothing looks odd and there doesn't seem to be anyway to flash the BIOS through the web interface. I can telnet to the BMC but I don't know any commands to this. I am new to HP servers, and just want to get the BIOS reflashed and working again.

[Kevin Heatwole]

BTW, I did search the web for advice how to recover AMI BIOS. It said that if the floppy isn't accessed with the AMIBOOT.ROM file on it, then I should press CTRL-HOME as soon as possible after power is applied to the server.

Nothing I do seems to get a response from the floppy. So, I'm still thinking that maybe the "crisis recovery jumper" is somewhere I'm not seeing and when I find it, I can reflash this BIOS.

[Kevin Heatwole]

I'm about to give up on this. Nothing I try seems to work.

The documentation for the DL165 G5 says there is a 2 pin jumper at P9 on the motherboard. But, P9 is just a couple soldiers. I found jumpers at P20, P26, P27, and P28. None of them changed to get any activity on the USB Floppy.

The documentation also mentions an HP Crisis Recovery Floppy, but I couldn't find any such animal in all the web (there was reference to Phoenix BIOS recovery floppy, but never for the AMI BIOS.

The BIOS chip itself is soldiered to the motherboard (why would they do that to a part that might need to be replaced?).

Anyway, I've tried everything I can think of. The documentation gives hope that you can just put the ROM on a floppy and it will pick it up. I guess my BIOS isn't messed up enough to get it to look for the floppy. Indeed all lights are green except that the system status light on the motherboard is blinking green (like it would during power up, but it never stops blinking green).

Any ideas at all?

[Popadic]

Same for me

[Matt Palmer_2]

Hi,

I dont have a DL165 G5, but on the 360s and 380s there is a redundant ROM on the board(not sure if the 165 offers the same?)

Maybe you could enable the 'other' ROM if there is one and try a recovery from there?

HTH

Matt

[Matt Palmer_2]

Hi,

just another thought. I know you said you get no console. - Have you tried to hit the console using the iLO port? might be you can reset through there?

thanks

Matt

[Popadic]

The problem is that there is no documentation on where is this jumper.
The Server Maintenance and Service Guide document is wrong.
:(

[ServerSoft]

If you cannot find any solution to this, I've used aqstech.com for server bios repair in the past. They're good.

[Gene Fox]

Did you ever find a solution to this problem? It seems I am now in the same boat.

[Kevin Heatwole]

No solution yet. My DL165 G5 is just a big paper weight.

I'm sure there probably is some way to trigger the ROM to reload the BIOS, but I couldn't figure it out and the documentation is inconsistent at best on how to recover from disaster.

[David Philander]

Did you manage to resolve this issue with your DL165 G5. I'm having the same problem after a firmware update. The fans running at high speed and nothing happens during startup.

[PaulCh]

Hi colleagues,
I am in the same position with my ProLiant DL185 G5. It is a cactus after updating ROM with recommended update - cp013066.exe. Update finished with message Successful and asked to restart server. Now:
- fans are running on high-speed,
- screen is black,
- NumLock is working on keyboard first 5 second after restart,
- Power-button is not working (can only unplug the power cables),
- Heartbeat is blinking on motherboard,
- SW4 to reset the BIOS doesnâ t help,
- no attempts to rear from USB-FDD or USB-key with BIOS updates,
- BIOS chip is soldered in.
Did anyone have any luck with this?

[Ed Andrade]

I'm having the same problem and have gone through the same steps. So if anyone has any resolutions to this I would love to hear them as well.

[PaulCh]

Hi colleagues,

I have no luck so far with this problem. I spent about 30 hours all together. There are some interesting links I managed to find but cannot implement. Probably you will have more luck/knowledge (or find something else):

1) General BIOS flashing facts. I learned several interesting things from here.
http://miro.pair.com/BIOS_Information.php

2) Flashing soldered BIOS connecting directly to by serial cable
http://mondotech.blogspot.com/2009/05/asus-p5b-deluxe-bios-recovery-spi-flash.html

3) How to fix boot-block and force BIOS to read from FDD or USB-stick
http://www.ehow.com/how_2185633_fix-boot-block.html

4) Direct micro-chip programming documentation, general information and etc for SST 49LF080A flash-memory chip.
http://www.sst.com/products/?inode=41352

Good luck!

HP official phone helpdesk is useless. Actually they are worse than useless - they are wasting my time - they put call on hold and never pick it up! I also sent an email more than one week ago and still receive no answer beside auto-reply. This is the last year my company pays for HP "unlimited support".

[Nitin100]

Any update?

[Pascal Normandin]

Has anyone been able to resolve this issue? I'm having the exact same problem.

Thanks

[Fabrice Bacchella]

Just get bitten by this trap too. How many servers have been destroyed by this â upgradeâ ?

[Kim Gregersen]

Why has HP not come back with a solution to this mess yet?

Now i'm stuck with a server, that I cant do anything with.

Has anyone found a solution for this ?

[Thomas Clodfelter]

Me as well on a DL185G5. HP Update breaks server. May be cause for legal action...anyone? HP needs to address this!

[Michael A. McKenney]

I had firmware 9.2 break two servers in December. HP will say you did not follow the procedures to upgrade. Try suing them.

Get a Carepaq and have them replace the hardware.

[Travis Truax]

I have the same issue with a DL185 G5. Aside from desoldering the BIOS chip (which I'd be seriously impressed if you were able to do by hand) there is nothing you can do for these servers now. They are 100% hosed.

The AMIBIOS "recovery" functions are only possible when the boot block of the bios is intact. That's what tells the system to read from the floppy in an attempt to allow for recovery. The specific problem with these HP BIOS updates are that they are mangling the boot block.

As some others have mentioned in this thread and others, the BIOS revision notes DO INDICATE that a certain version of BIOS is required to upgrade to the later versions. That's fine, right? But it doesn't mention that what they mean by that is that HP's update utility will just do it anyway, and DESTROY THE SERVER! How hard is it to check the old version before writing the new one?

So there are about 10-12 versions of this server's BIOS, and it looks like HP fiddled with the Boot Block 2 or 3 times over the years, so anyone wanting to update these machines needs to perform them sequentially. There are probably a few you could skip, but by now you should be very scared to even attempt a BIOS flash on one of these clunkers.

By the way, whichever user pointed out that some servers have a backup BIOS - not these. You'll have to buy a higher quality model to get that feature.

Beyond the monumental stupidity of writing the update utility to just blindly perform the flash, I see 3 things that make this screw-up so much worse than it could have been:
1. HP's BIOS revision notes only mention the critical information about being on a certain previous version in *past* revision notes. For example: Current version "D" says nothing. But notes for version "C" state that you need to be on "B" before upgrading to "C". Anyone quickly skimming the latest notes sees no special requirements.
2. HP's downloads page lists ONLY the last version or two. (depending on where you look) Who's going to think they need to search the site for old previous versions? They ARE still on the site, but you have to search for them.
3. No warnings of any kind. Sorry HP, but a one liner buried in the release notes about only being able to upgrade if you are on a previous version is *not* a warning.

Most of ours are mid-line DL3XX, and those are rock solid, and well engineered. The 100 series blows.
Oh, and the replacement mobo is $750 from HP parts. Ha! We got robbed....

[Kim Gregersen]

@Travis:

I've called the Danish HP service office several times, and all they do is to fend this off, by claiming that the product is "out of warranty" - well - DUH!
My product isn't broke, YOUR HP SoftwareUPDATER broke MY hardware... Would you think that HP has a responsibility here?

I hope that somewhere in HP marketing division there is a person who realizes how much damage this (and other) threads about this problem is causing for them, and corrects this.

[Travis Truax]

@Kim:

>>Would you think that HP has a responsibility >>here?

It definitely IS their responsibility from a ethical standpoint, but I doubt much will come of it. They could at least correct the BIOS updater to prevent the bug from occurring. As for those of us that have already fallen into the trap (as others put it) - HP couldn't care less if they loose a few hundred small business customers - compared with admitting their mistake and being on the hook replacing these boards for years out of warranty.

[Travis Truax]

The ASUS-related BIOS repair info Paul posted is definitely interesting, but I'm still of the opinion that these servers are permanently hosed. I spent a few hours reviewing it and looking at the board, and the conclusions I've come to are:

There is no SPI connector on the board. Without that, you're back to soldering some leads to the chip itself. I tried using my meter to trace some of the pins back to some of the mystery connectors and pads where a connector could be if they had installed one, but I got no love.

If someone understood the pin-out and comm protocol better than I (which would not be difficult) I still think it would be technically possible to fix it, but it looks like it would be extremely difficult.

[PaulCh]

Hi All,

It looks like HP doesn't have a support department at all I don't know about your areas guys, but here in Brisbane, Australia DELL provides free 3 years 24x7 services with new servers and replaces a failed part within 4 hours!!!! And I've got testimonials from my friends here that they do provide the replacement in 4 hours! In comparison, we waited for 3 weeks while HP shipped the motherboard to replace the one with screwed up BIOS.

By the way, I've sent the link to this forum to my boss ;-)