DL360p gen8 ilo 4 not available/responding

[Torsten.]

What OS is running on your servers in question?

[clusterman]

No OS at the moment. I haven't installed it yet, but planning to use CentOS. I figured it would be a good idea to get most of the onboard firmware updated before installing the OS.

[Torsten.]

There are 2 utilities you can use.

hpeoncfg  - can run on the local server OS and can manage ILO

hpqlocfg - can manage ILO remotely

First you need ILO make work again, otherwise you cannot do anything except replacing the system board.

Shut down the server and pull the plugs.

Connect the power cords again and check if ILO is alive. If yes, capture the ILO diagnostics web page and post.

Once ILO is alaive, update ILO firmware to 2.55, if older than 2.42.

Then use hpqlocfg to execute the following XML file

<!-- RIBCL Sample Script for HP Lights-Out Products -->
<!--Copyright (c) 2016 Hewlett-Packard Enterprise Development Company,L.P. -->

<!-- Description: This is a sample XML script to force format ll -->
<!-- the iLO partitions. -->
<!-- iLO resets automatically for this operation to take effect -->

<!-- Warning: This command erases all data on the partition(s) -->
<!-- External providers will need to be re-configured if -->
<!-- partition is formatted -->

<!-- Input: VALUE tag: all - format all available partitions -->

<!-- NOTE:You will need to replace the USER_LOGIN and PASSWORD values -->
<!-- with values that are appropriate for your environment -->

<!-- See "HP Integrated Lights-Out Management Processor Scripting -->
<!-- and Command Line Resource Guide" for more information on -->
<!-- scripting and the syntax of the RIBCL XML -->

<!-- Firmware support information for this script: -->
<!-- iLO 4 - Version 2.42 or later. -->
<!-- iLO 3 - None. -->
<!-- iLO 2 - None. -->

<RIBCL VERSION="2.0">
<LOGIN USER_LOGIN="Administrator" PASSWORD="password">
<RIB_INFO MODE="write">
<FORCE_FORMAT VALUE="all" />
</RIB_INFO>
</LOGIN>
</RIBCL> 

If you can the "success" message, pull the plugs again. Re-insert the power cords and check ILO log if the format was done.

[clusterman]

iLO is not currently available, even after pulling cords. During boot, instead of giving F8 option, it says " Warning - iLO 4 not responding." and "1500: iLO4 configuration is temporarily unavailable". It does give the IP address, and I can access the login page using the ilo web interface, but it won't let me log in due to a communication error (see above). 

I'm confused...if iLO is working, why do I need to do the format?

[Jimmy Vance]

---

@clusterman wrote:

Sorry, I'm a noob at all of this. 

It looks like the scripting toolkit for linux and windows is not free. I think I managed to find the scripting toolkit for linux download here. I think the ilo firmware .scexe can be downloaded separate from an rpm, so I shouldn't have to extract it. How do I use the toolkit to make the usb bootable such that the .scexe should run? What is hpsetup?

---

The scripting toolkit is free, The main page is here https://www.hpe.com/us/en/product-catalog/detail/pip.5219389.html

Click the download link, you should then see a link for the Latest Linux and Windows versions

Linux
 https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=1008917055&swItemId=MTX_914b2313ebe849708ccc005eb8&swEnvOid=4184

Windows
https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=1008917055&swItemId=MTX_229ba1a8c74d4c6286c41e4a8d&swEnvOid=4184

You shouldn't need to extract the scexe, it is basically a tarball wrapped in a shell script. 

hpsetup is the utility used to install the firmware whan using the rpm file

There is a customer advisory that explains how to use the toolkit to reformat the NAND, but not sure that is the issue here. 

[clusterman]

Thanks, I think you gave me enough info to track down the required documents for learning how to do all that.

[clusterman]

Update attempting this method (direct flash of ilo). I was able to obtain SPP 2017.04.0 and put the iso on a usb key using the usb key creator utility thing. 

The server will boot into this, I select interactive mode, but it hangs while trying to find the system hardware. I'm able to open the script like in that link, but I'm unable to run the .scexe because hpilo is in use. The prompt gives directions on stopping/unloading it:

Hp Lights-Out Driver “hpilo” must be unloaded (it can be unloaded with commands “/etc/init.d/hp-snmp-agents stop”, “/etc/init.d/hp-health stop”, “/etc/init.d/hp-ams stop”, “rmmod hpilo”)

But those 3 scripts are nowhere to be found, so I have no way of stopping those processes, and no way of unloading hpilo. So this is currently a dead end. Does anyone know why those scripts wouldn't be there?

Moving forward, I'm going to try an older SPP, old firmware dvd iso, and then the other method(s) suggested here. 

[clusterman]

I couldn't find any download links for 2016.04 or 2016.10. I tried the dvd iso, and that worked (SUM started and it analyzed the system), though it is quite old (ilo 4 v1.05), so I didn't install. I may go back to that eventually because any ilo 4 is better than no ilo 4. 

I then tried SPP 2017.04.0 again, but in automatic mode. It installed a couple things, and attempted to update ilo 4 from 2.2 to 2.55, but failed: "ERROR: Unable to start flash. [ilo4_255.bin]". I rebooted, and it's currently hung on "please wait, analyzing the system...". I might try putting the ilo4 rpm on there instead of the scexe and see what happens. 

[Torsten.]

Review this advisory again:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04996097

The ILO may have the evil NAND problem or it is really broken.

In order to format the NAND, we need to update the firmware to 2.55 first.

If the driver cannot be loaded, we cannot update the firmware from the server side.

It's complicated.

Try this:

use putty and point to the IP address of the ILO.

Can you logon?

Try, if a command works:

</>hpiLO-> show /map1/firmware1

If it works, we can proceed.

[clusterman]

Thanks for the link. It seems that in order to run the RIBCL XML script Force_Format I need to use those tools, which require an OS to be installed. I don't currently have an OS installed.

Putty didn't seem to connect. What settings do I use? SSH, port 22? 

I tried the web interface again. It initially gave me a "could not connect to ilo" (or something like that) error, which was what I was getting before. However, I was actually able to login this time (maybe trying to install ilo 4 with the SPP reset something?), though it is extremely slow. I was able to select the new ilo bin file (extracted from the windows ilo 4 firmware download), and hit upload. It's going to take about half an hour to upload at this rate, so we'll see what happens.