DL360p gen8 ilo 4 not available/responding

[clusterman]

Hi all,

I purchased a used dl360p gen8 (v1, older version). Intelligent Provisioning and iLO 4 aren't working.

For Int.Prov., I could hit F10 during boot, but the system would beep twice and then reboot. Int.Prov. was/is enabled in BIOS. I tried updating the firmware using a usb drive and the usb iso utility (I tried 1.63, 1.62b, and 1.61). It seemed to finish for 1.63, but then F10 still had same behavior. The process wouldn't complete for 1.62b and 1.61 (said something about ilo4 was unreachable?). So I figured I would try to use iLO 4. 

However, instead of the line that says to hit F8 during boot up to configure iLO4, it says that it's unavailable or not responding. It does give the ilo version as 2.20. I tried to connect to the IP address that shows up during boot; the web interface pops up, but I can't log in because it says "Connection with iLO cannot be established. If you recently made changes to the network configurations, you may need to refresh this page to re-negotiate an SSL connection.". I don't know the network settings other than the IP address because I can't access the ilo4 configurator. I tried flipping the security switch on the motherboard, but it doesn't seem to have done anything. Also tried unplugging for awhile and rebooting: nothing.

My guess is that a file on the flash is corrupted. Any idea how I can get iLO4 and Int.Prov. to work? Is there someway to reflash ilo4?

Thanks

[clusterman]

I saw this: https://community.hpe.com/t5/Server-Management-Remote-Server/1500-iLO-4-Configuration-is-temporarily-unavailable-5/m-p/6840733

But I don't want to pay for SPP. Any idea how to do this without SPP?

Seems like this is a common problem for many ilo4 versions: https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/iLO-4-issues/m-p/6710352

[clusterman]

More info: I can print the IP address successfully. But I cannot log in to the web interface.

[clusterman]

I found this: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c00719376

But it appears to be aimed at older generations. It seems the smart update manager requires an OS to be installed. Any one know if I should try that or not? I'd rather get all of the firmware sorted out before installing an OS. Supposedly I can install new firmware using SUM. 

I say that, then I find this: https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=5194969&swItemId=MTX_32179cba6b174fb4a43c47fd45&swEnvOid=4184#tab3

Installation:

To update firmware from the Linux operating system on target server:
Download the SCEXE file to the target server.
Execute: sh CP0xxxxx.scexe

 To obtain firmware image for updating via iLO user interface, utilities, or scripting interface:
Download the SCEXE file to a client running a Linux operating system.  Execute: sh  CP0xxxxx.scexe --unpack=directory
This command will unpack the ilo4_1XX.bin into a user specified "directory". If the directory does not exist, the unpacker will attempt to create it.

To use HP Smart Update Manager on the Firmware Maintenance CD:
Place the Firmware Maintenance CD on a USB key using the USB Key Creator Utility. 
Copy  CP0xxxxx.scexe to /compaq/swpackages directory on the USB Key.
Follow HP Smart Update Manager steps to complete firmware update.

I couldn't find anything more recent than 2010 on the firmware maintenance CD/DVD... my guess is that SPP replaced it. The SUM 7.? manual says that "in offline mode, the SPP boost a small linux kernal", but again, I don't want to buy SPP because it probably costs more than I paid for the server. A newer SUM 8.? manual says:

Running from an ISO If your ISO contains updates, you can copy the ISO to a DVD or USB key. From the DVD or USB key, you can run SUM in an offline interactive or automatic mode. • Automatic mode — Firmware components update without interaction. To launch automatic mode: 1. Install the DVD or USB key into the server, boot the server to the DVD or USB key. 2. SUM launches and deploys updates after a short period. • Interactive offline mode — Allows you to use the GUI mode to deploy updates. To launch offline interactive mode: ◦ Install the DVD or USB key into the server, boot the server to the DVD or USB key.

In offline mode, SUM boots a small SuSE Linux Enterprise Server 12 boot kernel and enables updates to occur on a single server

So does this strategy make sense?

1. Download latest ilo4 online rom flash component for linux
2. Download SUM iso and put it on a usb with the HP USB Key Creator Utility
3. Copy the ilo4 .scexe file to the usb key in X folder
4. Flip security switch on server mobo (done)
5. Boot server into SUM interactive mode
6. somehow install the new firmware
7. Shutdown server, flip security switch back, and reboot.

I really don't want to try this without an expert's input. 

I'm guess I can use the same method to update other firmware, right? What else should I update?

Thanks!

[clusterman]

Another update: I can't find the iso for SUM 8.0.0. SUM 7.6.0 supposedly has an iso listed (https://support.hpe.com/hpesc/public/km/search#q=service%20pack%20for%20proliant&t=All&sort=relevancy&numberOfResults=25), but I can't find where to download it. I've only found windows (zip) or linux (rpm) downloads in the enterprise support center.

The iso is required for the usb key utility. I suppose I could create one from the files, but I'm not sure if that will work.

[Moderator edit: Updated the broken link.]

[clusterman]

A few more comments:

It seems that SPP is the only "right" way to do this now. 

I attempted to chat with technical experts, but I can't because I don't have a warranty or service agreement. 

The sales rep said a 1 year post warranty would cost $650, which is what I paid for the server and is about 2x what a new system board is worth. She gave me the number to call for out of warranty support, but it's a premium number, so will cost me.

There are apparently some sketchy download links for various versions of SPP, including indian and torrent sites. I can't link those here obviously. I believe they've split the SPP's into production and post production after 2017.04.0.

In order to reduce the frequency at which servers need to be updated, HPE is changing the packaging and delivery of the SPP.

Going forward, two types of SPPs will be released:

• A single “Production SPP” that contains the firmware and drivers for the all of the production server generations, and

• Multiple server generation specific “Post-Production SPPs” that contain the firmware and drivers for a specific post-production server generation (e.g. Gen8 only Post-Production SPP that contains the firmware and drivers for all of Gen8 server models).

When a server generation transitions to “Post-Production,” the firmware and drivers for that server generation are frozen, packaged together on a “Post-Production SPP,” and released. The “Post-Production SPP” is supported through the EOS (end of support) for that server generation, and all future firmware and driver updates are released as hot fixes.  This eliminates the need for yearly updates.

The 2017.04.0 SPP is the last production SPP to contain components for the G7 and Gen8 server platforms. For additional information, please refer to Reducing Server Updates.

So for gen 8 proliant servers, you really want the 2017.04.0 SPP, then install any remaining driver updates/hot fixes. Later SPP's aren't useful for gen 8 servers. 

I will likely attempt to use one of the sketchy download links since this server didn't cost me that much....worst comes to worse I have to replace the system board, which is current plan B now anyways. 

Anyone have any answers to my above questions?

[Jimmy Vance]

I haven't done it in some time now,.  If you download the HPE Scripting Toolkit for Linux, you can create a booable USB key.  It is Linux based. Download the .scexe file that contains the iLO firmware you need (might need to extract it from an RPM file)  I don't think the toolkit has all the necassary bits to run the rpm command directly, but the .scexe should run, or hpsetup

There is also the chance that the iLO itself is defective and the only resolve there would be a new system board

[Torsten.]

As far as I understand you want to format the ILO NAND now.

You need the hpeoncfg tool to push the script to the ILO in question, if you have a bootable ISO it needs at least the ILO driver loaded in order to perform ILO operations.

Do the format, once finished pull the plug of the server for a couple of minutes.

Then check if ILO is working again.

format operation needs ILO firmware 2.42, in older versions the format is not available.

[clusterman]

Sorry, I'm a noob at all of this. 

It looks like the scripting toolkit for linux and windows is not free. I think I managed to find the scripting toolkit for linux download here. I think the ilo firmware .scexe can be downloaded separate from an rpm, so I shouldn't have to extract it. How do I use the toolkit to make the usb bootable such that the .scexe should run? What is hpsetup?

[clusterman]

I think I want to format the NAND. I'm not sure what else to do. 

You need the hpeoncfg tool to push the script to the ILO in question

Which script am I pushing and how do I use that tool to do so? It looks like hpeoncfg requires an OS to be installed, and it currently isn't. I'm not sure it would work because ilo doesn't seem to be responsive.

if you have a bootable ISO it needs at least the ILO driver loaded in order to perform ILO operations

What is on the bootable iso? So the ilo driver has to be part of the iso in order to reformat ilo?

format operation needs ILO firmware 2.42, in older versions the format is not available.

The ilo already on the server is older than that. Does that mean this won't work? Or does that mean I need the ilo firmware 2.42 in the bootable iso?

[Torsten.]

What OS is running on your servers in question?

[clusterman]

No OS at the moment. I haven't installed it yet, but planning to use CentOS. I figured it would be a good idea to get most of the onboard firmware updated before installing the OS.

[Torsten.]

There are 2 utilities you can use.

hpeoncfg  - can run on the local server OS and can manage ILO

hpqlocfg - can manage ILO remotely

First you need ILO make work again, otherwise you cannot do anything except replacing the system board.

Shut down the server and pull the plugs.

Connect the power cords again and check if ILO is alive. If yes, capture the ILO diagnostics web page and post.

Once ILO is alaive, update ILO firmware to 2.55, if older than 2.42.

Then use hpqlocfg to execute the following XML file

<!-- RIBCL Sample Script for HP Lights-Out Products -->
<!--Copyright (c) 2016 Hewlett-Packard Enterprise Development Company,L.P. -->

<!-- Description: This is a sample XML script to force format ll -->
<!-- the iLO partitions. -->
<!-- iLO resets automatically for this operation to take effect -->

<!-- Warning: This command erases all data on the partition(s) -->
<!-- External providers will need to be re-configured if -->
<!-- partition is formatted -->

<!-- Input: VALUE tag: all - format all available partitions -->

<!-- NOTE:You will need to replace the USER_LOGIN and PASSWORD values -->
<!-- with values that are appropriate for your environment -->

<!-- See "HP Integrated Lights-Out Management Processor Scripting -->
<!-- and Command Line Resource Guide" for more information on -->
<!-- scripting and the syntax of the RIBCL XML -->

<!-- Firmware support information for this script: -->
<!-- iLO 4 - Version 2.42 or later. -->
<!-- iLO 3 - None. -->
<!-- iLO 2 - None. -->

<RIBCL VERSION="2.0">
<LOGIN USER_LOGIN="Administrator" PASSWORD="password">
<RIB_INFO MODE="write">
<FORCE_FORMAT VALUE="all" />
</RIB_INFO>
</LOGIN>
</RIBCL> 

If you can the "success" message, pull the plugs again. Re-insert the power cords and check ILO log if the format was done.

[clusterman]

iLO is not currently available, even after pulling cords. During boot, instead of giving F8 option, it says " Warning - iLO 4 not responding." and "1500: iLO4 configuration is temporarily unavailable". It does give the IP address, and I can access the login page using the ilo web interface, but it won't let me log in due to a communication error (see above). 

I'm confused...if iLO is working, why do I need to do the format?

[Jimmy Vance]

---

@clusterman wrote:

Sorry, I'm a noob at all of this. 

It looks like the scripting toolkit for linux and windows is not free. I think I managed to find the scripting toolkit for linux download here. I think the ilo firmware .scexe can be downloaded separate from an rpm, so I shouldn't have to extract it. How do I use the toolkit to make the usb bootable such that the .scexe should run? What is hpsetup?

---

The scripting toolkit is free, The main page is here https://www.hpe.com/us/en/product-catalog/detail/pip.5219389.html

Click the download link, you should then see a link for the Latest Linux and Windows versions

Linux
 https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=1008917055&swItemId=MTX_914b2313ebe849708ccc005eb8&swEnvOid=4184

Windows
https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=1008917055&swItemId=MTX_229ba1a8c74d4c6286c41e4a8d&swEnvOid=4184

You shouldn't need to extract the scexe, it is basically a tarball wrapped in a shell script. 

hpsetup is the utility used to install the firmware whan using the rpm file

There is a customer advisory that explains how to use the toolkit to reformat the NAND, but not sure that is the issue here. 

[clusterman]

Thanks, I think you gave me enough info to track down the required documents for learning how to do all that.

[clusterman]

Update attempting this method (direct flash of ilo). I was able to obtain SPP 2017.04.0 and put the iso on a usb key using the usb key creator utility thing. 

The server will boot into this, I select interactive mode, but it hangs while trying to find the system hardware. I'm able to open the script like in that link, but I'm unable to run the .scexe because hpilo is in use. The prompt gives directions on stopping/unloading it:

Hp Lights-Out Driver “hpilo” must be unloaded (it can be unloaded with commands “/etc/init.d/hp-snmp-agents stop”, “/etc/init.d/hp-health stop”, “/etc/init.d/hp-ams stop”, “rmmod hpilo”)

But those 3 scripts are nowhere to be found, so I have no way of stopping those processes, and no way of unloading hpilo. So this is currently a dead end. Does anyone know why those scripts wouldn't be there?

Moving forward, I'm going to try an older SPP, old firmware dvd iso, and then the other method(s) suggested here. 

[clusterman]

I couldn't find any download links for 2016.04 or 2016.10. I tried the dvd iso, and that worked (SUM started and it analyzed the system), though it is quite old (ilo 4 v1.05), so I didn't install. I may go back to that eventually because any ilo 4 is better than no ilo 4. 

I then tried SPP 2017.04.0 again, but in automatic mode. It installed a couple things, and attempted to update ilo 4 from 2.2 to 2.55, but failed: "ERROR: Unable to start flash. [ilo4_255.bin]". I rebooted, and it's currently hung on "please wait, analyzing the system...". I might try putting the ilo4 rpm on there instead of the scexe and see what happens. 

[Torsten.]

Review this advisory again:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04996097

The ILO may have the evil NAND problem or it is really broken.

In order to format the NAND, we need to update the firmware to 2.55 first.

If the driver cannot be loaded, we cannot update the firmware from the server side.

It's complicated.

Try this:

use putty and point to the IP address of the ILO.

Can you logon?

Try, if a command works:

</>hpiLO-> show /map1/firmware1

If it works, we can proceed.

[clusterman]

Thanks for the link. It seems that in order to run the RIBCL XML script Force_Format I need to use those tools, which require an OS to be installed. I don't currently have an OS installed.

Putty didn't seem to connect. What settings do I use? SSH, port 22? 

I tried the web interface again. It initially gave me a "could not connect to ilo" (or something like that) error, which was what I was getting before. However, I was actually able to login this time (maybe trying to install ilo 4 with the SPP reset something?), though it is extremely slow. I was able to select the new ilo bin file (extracted from the windows ilo 4 firmware download), and hit upload. It's going to take about half an hour to upload at this rate, so we'll see what happens. 

[Torsten.]

Don't care about the tools now, we can run the format command from a booted SPP or just from another PC.

[clusterman]

It gets to about 40% upload, then the session expires. I'm going to try to find a way to increase timeout of the web interface.

[Torsten.]

try a putty ssh session.
but if the ILO resets during the update for error, there is probably no way to update it.

[clusterman]

I was able to increase web interface time out to infinite, which should help, because changing windows takes 10-30s each time.

Ok, tried putty again. I'm using the given ipv6 address in brackets and port 22, then connecting. Putty gives me a black box, but nothing else, so I don't think it's connecting. Maybe I'm not using putty right?

[Torsten.]

i never usedipv6 with ILO so far, try ipv4.