DL360p gen8 ilo 4 not available/responding

[Torsten.]

Don't care about the tools now, we can run the format command from a booted SPP or just from another PC.

[clusterman]

It gets to about 40% upload, then the session expires. I'm going to try to find a way to increase timeout of the web interface.

[Torsten.]

try a putty ssh session.
but if the ILO resets during the update for error, there is probably no way to update it.

[clusterman]

I was able to increase web interface time out to infinite, which should help, because changing windows takes 10-30s each time.

Ok, tried putty again. I'm using the given ipv6 address in brackets and port 22, then connecting. Putty gives me a black box, but nothing else, so I don't think it's connecting. Maybe I'm not using putty right?

[Torsten.]

i never usedipv6 with ILO so far, try ipv4.

[clusterman]

Excellent news. I was able to update ilo 4 from v2.20 to v2.55, though it's still slow. I used the web interface to upload the .bin (which was extracted from the windows ilo 4 firmware download). I set the timeout setting to infinite. The upload took about 2 hours, and then the "firmware flash" message hung at 2%. I reset ilo 4 after awhile, then it started progressing. When it got to 100%, it logged me out, then I could log back in to the new ilo 4 version. I'm not sure why the web interface suddenly decided to start working. I'm guessing that something with SPP attempts ticked something into working again, albeit very poorly.

I think a nand flash might still be a good idea. My guess is that there's something bad in it still.

I switched to ipv4, and was able to get putty to log in. I executed that command Torsten, and it worked. What's next?

[Torsten.]

Well done!

by the way, putty was another attempt to update the firmware.

Next steps follow.

- logon to ILO, go to the Diagnostic page. the yellow exclamation mark will be another indicator for the NAND issue

- take a SPP of your choice, extract the USB directory and use this tool to create the USB stick

- create a file called "format-nand.xml" with this content

<!-- RIBCL Sample Script for HP Lights-Out Products -->
<!--Copyright (c) 2016 Hewlett-Packard Enterprise Development Company,L.P. -->

<!-- Description: This is a sample XML script to force format ll -->
<!-- the iLO partitions. -->
<!-- iLO resets automatically for this operation to take effect -->

<!-- Warning: This command erases all data on the partition(s) -->
<!-- External providers will need to be re-configured if -->
<!-- partition is formatted -->

<!-- Input: VALUE tag: all - format all available partitions -->

<!-- NOTE:You will need to replace the USER_LOGIN and PASSWORD values -->
<!-- with values that are appropriate for your environment -->

<!-- See "HP Integrated Lights-Out Management Processor Scripting -->
<!-- and Command Line Resource Guide" for more information on -->
<!-- scripting and the syntax of the RIBCL XML -->

<!-- Firmware support information for this script: -->
<!-- iLO 4 - Version 2.42 or later. -->
<!-- iLO 3 - None. -->
<!-- iLO 2 - None. -->

<RIBCL VERSION="2.0">
<LOGIN USER_LOGIN="Administrator" PASSWORD="password">
<RIB_INFO MODE="write">
<FORCE_FORMAT VALUE="all" />
</RIB_INFO>
</LOGIN>
</RIBCL> 

- place this file onto the USB stick XML directory

- boot SPP in interactive mode and accept the EULA

- hold down CTRL and ALT keys and type   d  b  x

- you get a shell now

- type "cd /mnt/bootdevice/xml"

- type "hponcfg -f format-nand.xml"

- shutdown and pull the power cords

- plug in power, connect to ILO

- check the ILO log "devices have been formatted" message

- check if the Diagnostic page is green

- re-install Intelligent Provisioning

Does this work?

[clusterman]

Ok, I made sure to change the password in the script to my Administrator password.

The SPP interactive mode loads completely now! It must hang when ilo 4 isn't working properly.  Ran the script successfully.

Logged into web interface to check log and diagnostics: looks ok. Seems faster now.

Downloaded intelligent provisioning 1.7 (latest for gen8) iso from this server's HP Enterprise Support Center page, and put it on a usb using the hp key creator. Booted server to that. "Running Flash Process" hangs at 80% for a minute or so, then completes. Auto-reboots.

Selected F10, grub loads (is that weird?), selected intelligent provisioning from the list, and it loads!! But it freezes on step 1 page and mouse won't work. Trying to reboot it.

Oh! I forgot to flip the security switch back. I don't know if that had anything to do with it. Flipped switch, rebooted, hit F10, selected Int.Prov., and same problem. I can't move mouse, so I can't get past step 1. I'll keep trying later today.

Thanks for your help so far, it's been really useful. 

[clusterman]

Tried reflashing with the older IntProv 1.63. It loaded AND allowed me to use the mouse! YES. DONE.

Maybe there's a bug in Intelligent Provisioning 1.7 that prevents people from using the mouse?

I'll write a summary guide next for noobs like me.

[clusterman · Accepted solution]

How you know you have the problem I had:

Intelligent provisioning won’t load when F10 is pressed during boot up, it isn’t responding, and attempts to install it don’t work.

iLO 4 web interface is unresponsive ("Connection with iLO cannot be established.”) or you get the "1500: iLO4 configuration is temporarily unavailable" error during boot up, and pulling power cables and rebooting doesn’t help.

Anything else on this link (advisory document id c04996097).

From what I’ve read online, this is a common problem.

The problem: I believe something in the iLO 4 firmware and NAND memory got corrupted. These have to be re-flashed.

What this means for you: Plan A. Follow these steps and hope they work. Plan B. Replace system board (~$300 on eBay) because the iLO chip is corrupted beyond recovery and send HP angry emails. The good news is that you can’t really make the problem worse since your plan B is replacing the system board.

Note: there are tools that allow you to reflash the iLO 4 from an OS (HP scripting toolkit – See Jimmy Vance post earlier). I didn’t have an OS, so I couldn’t do that. Feel free to try that method if you have an OS installed. UPDATE: Turns out that the scripting tool kit is a bootable image, so you don't need an OS or the SPP. See Jimmy Vance's post below. I haven't tried this method, but I probably will if I come across another corrupted ilo 4.

Prerequisites:

1. Download the HP USB Key Utility, v2.0 or greater. This can be downloaded from the hp support center, or it can be obtained from a recent HP SPP (service pack for proliant) by mounting the iso (various free software online to do this) and extracting the usb key utility, then installing on your computer.
2. Obtain the iLO 4 firmware v2.4 or greater. Do this via the HP Enterprise Support Center (enter your model number and search for “integrated lights out flash”). I used v2.55. There are multiple file types. HP has switched to rpm package distributions for linux, but they’ve kept the older .scexe format files up. Either should work. Also grab the Windows .exe file while you’re there. You might need it (explained below).
3. Obtain an SPP iso. The latest for gen 8 ProLiant servers is 2017.04.0. This is what I used. See one of my earlier posts for why this is the last and suggestions for how to obtain it. If you have a warranty or service contract, you can download it (even a custom trimmed down version for your server) from HP for free. If you don’t have a contract, a 1 year post warranty was quoted at $650 for me, which is more than the system board. Put this iso on a USB key with the usb utility.
4. If you have a VERY old system (not updated at all in last ~5 years), you may need to download the firmware update DVD (search the HP Enterprise Support Center for that, it’s still free) and do that automatic update first.
5. Download the Intelligent provisioning 1.63 iso from the HP Enterprise Support Center or from here. Version 1.X are for gen 8, versions 2.x are for gen 9’s. Note: I tried v1.7, but it wouldn’t let my mouse work. Maybe it or a newer version will work for you. Put this on another usb key using the usb utility.
6. If you don’t know your username and password, find the info pull tab on your server. On that should be the default user name and password.

Solution procedure:

1. Turn off server. Open cover and flip security switch. You’ll have to look in your user guide for where and which switch this is. Mine was tiny; I used a paper clip.
2. Plug the SPP USB key you made in to your computer and copy the .scexe OR the rpm of the iLO 4 firmware to the /hp/swpackages folder
3. Eject the USB key and plug it into the server.
4. Boot server, select F11, boot to USB key.

Option 1: Try SPP interactive mode. This never fully loaded for me. It got stuck, but I was able to open a shell using the ctrl + alt + d + b + x keys. I navigated to /mnt/boot/hp/swpackages, typed su to become a super user, then tried executing the ilo4 firmware update with “./<your version here>.scexe –direct”, where <your version here> depends on your ilo 4 version…should be something like CP0XXXXX. If you loaded the rpm instead of the .scexe, you could try the rpm installation commands. This didn’t work for me because the scripts necessary for stopping hpilo (it will prompt you to do that first, try what it suggests) were not included in that SPP.

Option 2: Try SPP in automatic mode. This sort of worked for me. It updated some components' firmware, but it said that the ilo 4 update failed "ERROR: Unable to start flash. [ilo4_255.bin]". However, I think it tricked ilo 4 into sort of working.

If one of the above worked completely and installed ilo 4 successfully, then you’re golden. Shut down and continue with step 8 below.

1. Try the web interface or putty/SSH (if you have ipv4 configured…putty didn’t work with the ipv6 address for me). You need the IP address that shows up when the server boots. You also need to be on the same network as server. Then in a browser, type “https://<your ip here>/” and press enter. It should pull up the ilo 4 log in screen. Put in the user name and password and try logging in. If it doesn’t work the first time, try refreshing or just waiting. It was extremely slow for me.
2. Now, if you’re in the web interface, navigate to Administration->access settings and set the timeout to infinite. There’s probably some command for this with the scripting tools via putty/SSH, too, or it may just not have a timeout.
3. In the web interface, navigate to Administration->firmware.
4. On your windows computer, find the windows .exe of the ilo 4 firmware you download, and run it. Do not install. Hit “extract all” and pick a folder. In that folder there should be a .bin file. Note its location.
5. Back in the web interface, click choose file, select the .bin file, and hit upload. This took over 2 hours for me.
6. When it starts installing, you’ll see “firmware flash” message near the bottom. This hung at 2% for me for a long time. I reset ilo 4 on the Information->Diagnostics menu, and it started progressing. When it got to 100%, it logged me out. I waited about 30s, refreshed the page, and got a new log in screen. I logged in and had the new ilo 4 interface.
7. Torsten said there’s a way to do the firmware update via putty/SSH from another computer without the web interface. I believe it uses the scripting utilities, but I have not tried it. UPDATE: See the tested method in Torsten's post below. Very clean way to do this, and doesn't require SPP.
8. Check if there’s something wrong with your NAND. Go to this link. If any of that is true, then you need to reflash the NAND. I still couldn’t access Intelligent Provisioning, so I reflashed the NAND.
9. Create the xml file in that link (Torsten copied it to one of his posts in this thread). Make sure you change the username and password.
10. Copy that file to your SPP usb in the XML directory. Eject the USB drive and plug into the shutdown server.
11. Boot up the server to the SPP. Hold down ctrl + alt + d + b + x to get a shell.
12. Type "cd /mnt/bootdevice/xml"
13. Type "hponcfg -f format-nand.xml". It should display a successful message.
14. Shutdown servers, pull power cords.
15. Plug in power cords, connect to ilo 4 (via web interface or putty). Check iLO log to see any “devices have been formatted” messages. Make sure diagnostics page has green status for the embedded flash.
16. Flip the security switch back
17. Remove SPP usb and plug in Intelligent Provisioning USB. Boot server into Intelligent Provisioning USB and install.
18. Shutdown server.
19. Boot server. During boot, if you can access iLO 4 configuration with F8 and Intelligent Provisioning with F10, then you’re done! YAY

Potentially useful links:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04996097

https://support.hpe.com/hpsc/doc/public/display?sp4ts.oid=5219994&docId=emr_na-c03334058&docLocale=en_US

https://community.hpe.com/t5/Server-Management-Remote-Server/1500-iLO-4-Configuration-is-temporarily-unavailable-5/m-p/6840733

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c03458406&docLocale=en_US

[Moderator edit: Removed the broken link. Please refer to https://support.hpe.com/]