DL360 G5, Can't get into RBSU at all

[iggdawg]

I was remote during a power blip recently and noted I couldnt get into any of my gear.  Finally got VPN'd into and noticed all my VMs were down.  So I logged into iLO and checked out the console since this unit appeared to have remain powered on.  I saw the VMWare console screen up but noted I couldnt interact with it.  

Upon rebooting I noted it didn't get past a pair of PXE boot attempts and that broadcomm netboot dealie.  Just a blinking underscore cursor right below the netboot attemp.  I rebooted again and hit F9, noting it said "F9 pressed".  The server stopped at the same spot with the blinking cursor.

I've done quite a lot of testing, sorry for the wall of text.  TL;DR at bottom.

 - Storage controller works fine.  I can get into the controller menu and view/change arrays.  I even removed a disk to make sure it would mention the failure.  booted again after reinserting it, confirmed it came up with the "repair or skip" dialog.  So the controller sees the drives, but they never get looked at by the server to boot off them.

 - Won't boot off any combination of local or iLO-loaded CD-ROM or USB images.

 - I can't get into RBSU to even check boot options

 - On booting, the "Redundant ROM Detected - This system contains a valid backup system ROM" line shows up, indicating the primary is valid.  I suspect this is not the case and that something is wrong with the primary, but if the primary is bad the system doesn't think so.

 - tried flipping dipswitch #6, booting till it hangs, powering off, flipping the switch back, and booting again.  I didn't note if it said it noticed switch 6 being flipped, nothing specific jumped out at me.  but my suspcion is that the boot ROM doesn't get loaded to the point it would even notice this

 - Tried booting off a single pair of RAM sticks, then swapped to another pair to make sure that first pair wasn't bad.  No change

 - Tried booting off both processors initially, then each processor individually

 - No alarm lights in the device or on the front panel popout thingy, no strange BIOS beeps

TL;DR - Basically the system thinks all is well right up until the point it hangs on me.  I can't get into RBSU at all to change anything.  I think the primary ROM is bad, but the system doesn't think so.  I tried searching the googles like a crazy man, but nobody had a scenario that quite fit mine.  If anyone has any experience with a scenario like this, that'd be great.  Failing that, if anyone could point me at any general or model specific info on utilizing the backup boot ROM I would be happy to RTFM.

EDIT - for what it's worth, I have done some digging into using the backup ROM, and all accounts seem to indicate it either autodetects a bad primary ROM or one uses the "ROMPaq Utility" to deal with ROMs.  But not being able to boot the smartstart disk sort of obviates that option.

[iggdawg]

I did manage to find one very similar situation over at serverfault:

"HP Proliant 360 G4 hangs on Ethernet Boot Agent"

"My server hangs on Ethernet Boot Agent. I can't access the BIOS.

It used to work fine. I can access the RAID controller, but it stops booting and fails to respond on key inputs once it gets to the Ethernet Boot Agent stage."

The OP accepted answer was to try removing the CMOS battry, waiting a bit, then reinstalling it.   This is something I haven't tried yet.  I figured it would yield similar results to flipping maint. switch position 6 to on.  I'll give this a shot when I get home today.

[Jimmy Vance]

Check page 73 of the users guide, it explains what switches need to be set to enable the backup ROM

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c00710435/c00710435.pdf

[iggdawg]

Thank you for the info. I did see that part earlier, but sadly I can't access RBSU. The system does not move forward after PXE/netboot options even when it has registered "F9 pressed"

[Jimmy Vance]

You didn't read down far enough. You don't need to acces RBSU.

To access the redundant ROM manually:

1. Power down the server (on page 18).

2. Remove the access panel (on page 20).

3. Set positions 1, 5, and 6 of the system maintenance switch to On.

4. Install the access panel (on page 20).

5. Power up the server (on page 18).

6. Wait for the server to emit two beeps.

7. Repeat steps 1 and 2.

8. Set positions 1, 5, and 6 of the system maintenance switch to Off.

9. Repeat steps 4 and 5.

When the server boots, the system identifies whether the current ROM bank is corrupt. If a corrupt ROM is

detected, the system boots from the backup ROM and alerts you through POST or IML that the ROM bank

is corrupt.

If both the current and backup versions of the ROM are corrupt, the server automatically enters ROMPaq

disaster recovery mode.

[iggdawg]

You are very correct. I've been spread thin today, and you are indeed correct. I stopped reading early. Thank you for putting up with my haste and providing the info in spite of it. That's actually exactly what I was looking for. I'll try that in a couple when I'm with the machine.

[iggdawg]

Sadly, neither the maint switch procedure nor removing the CMOS battery had any affect. I think I've got a parts box here unless anyone else has any magic I can try.

Jimmy, Thank you for your help in any case.

[TarunJain]

Replace the motherboard

[iggdawg · Accepted solution]

Solved the problem.

As the previos post suggested, ebay'd myself a motherboard to replace the one I thought was bad.  I had already swapped out the RAM, CPUs, and disks into a new chassis to get back up and running.  I had a pile of spare RAM, a spare CPU, and a new set of disks to throw in this chassis with the old motherboard.  Booted it up, hung at the same exact place.  I figured the most likely culprit at this point was that the chassis was cursed.   No KB articles for exorcisms, so I dig further.

Ironically, it did end up being something similar to a cursed chassis.  Turns out the optical/multibay hardware had turned sour somehow.  I had unplugged absolutely everything I could, including the multibay and front panel control cables.  Had to plug in the front cable controls to turn it on, but I left the other connector out.  Sure enough, the device booted up.

Nobody ever blames the optical drive.  For the record, I had tried booting with disk in and disk out with no change previously, and having the bay empty with USB boot media in didn't help.  The multibay hardware itself was damaged somehow and the unit would not boot with it plugged into the motherboard.

Hopefully this thread/story keeps someone from tearing out too much hair.

[TarunJain]

Hi Iggdawg,
I can understand your situation.
We all learnt by experience the rule of "Try to boot with minimum hardware"
We follow this for desktops but sometimes trust on Server hardware and skip it.
Thanks for reminding.
Tarun Jain