topic Re: BL20p G4 ILO issue in BladeSystem - General

BL20p G4 ILO issue

raoul_schaffner — Thu, 09 May 2013 11:56:33 GMT

I updated the ILO of a BL20p G4 blade to version 2.15 (current as of now for ILO2). Only afterwards did i learn, that while the 2.x series of ILO is installable via the Support Pack for Proliant, it is no longer supported on the p-series of blades.

So, right now the ILO is no longer pingable, nor does a manual press of the power button start the blade. The private ILO address 192.168.1.1 (via dongle / cable exploder) also does not ping.

Is there any way to boot a blade when the ILO is hosed? If so, I could downgrade ILO from within the booted operating system.

Thank you very much for any hints,

Raoul Schaffner.

Re: BL20p G4 ILO issue

TVVJ — Fri, 10 May 2013 08:04:38 GMT

iLO’s functionality includes enabling the server to power on. In case of p-Class Blades, the server’s iLO communicates with the power management module of the p-Class enclosure of the server’s power requirements and based on available power available in the enclosure allows the server to power on.

In the present situation this communication is not occurring as the firmware of the server’s iLO is not able to communicate with the enclosure’s power management module.

I am not able to see a solution other replacing the system board. Having said that, try connecting the Blade to a p-Class Diagnostic Station and check if it powers on. If not then perform the following steps, with the server connected to Diagnostic Station:

a. Clear the server’s NVRAM. To do so, the following steps may be performed:

i. Power down the server and disconnect it from the Diagnostic Station.

ii. Please refer to item 14 of the System board components diagram, on page 56 of 66 of the server’s maintenance and service guide to identify the system maintenance switch. The guide is locatable at the following link:

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c00715963/c00715963.pdf

iii. Locate individual switch S6, and set it from the default position of ‘Off’ to ‘On’. Replace the server in the bay and power it on. Wait till the message indicates that the server has been reset. Then power off the server, remove it from the bay, revert the switch S6 to position back to ‘Off’ and then power it on.

After clearing NVRAM, and the server power’s on, please re-enter the serial number in the server's RBSU. You may perform the following steps:

i. During the server startup sequence, press the F9 key to access RBSU.

ii. Select the System Options menu.

iii. Select Serial Number. The following warning is displayed:

WARNING! WARNING! WARNING! The serial number is loaded into the system during the manufacturing process and should NOT be modified. This option should only be used by qualified service personnel. This value should always match the serial number sticker located on the chassis.

iv. Press the Enter key to clear the warning.

v. Enter the serial number and press the Enter key.

vi. Select Product ID.

vii. Enter the product ID and press the Enter key.

viii. Press the Esc key to close the menu.

ix. Press the Esc key to exit RBSU.

x. Press the F10 key to confirm exiting RBSU. The server will automatically reboot.

b. If the server is able to power on after clearing NVRAM, then attempt troubleshooting steps detailed in the solution section “Maintenance and Configuration Options > Recovering from a failed iLO 2 firmware update” contained in HP Integrated Lights-Out 2 (iLO 2) HP Guided Troubleshooting. This troubleshooting solution can be located at the following url:

http://h20584.www2.hp.com/hpgt/guides/select?lang=en&cc=us&prodTypeId=329290&prodSeriesId=1146658

c. The iLO firmware latest supported on the server is1.70 (B), downloadable from the following link:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=1121598&swItem=MTX-613c4a32c1ec4192b0e9b5fd41&prodNameId=3288176&swEnvOID=4024&swLang=8&taskId=135&mode=4&idx=1

Please review steps to perform the update under Tab Installation Instructions.

d. If the server does not power on when connected to the Diagnostic Station, then it would mean the system board may require replacement. Contact HP support with the server’s serial number:

http://h20565.www2.hp.com/portal/site/hpsc/public/help/contactHP/home/

Regards,

Vijayasarathy

Re: BL20p G4 ILO issue

Oscar A. Perez — Thu, 16 May 2013 14:25:33 GMT

Since p-Class blades are no longer supported in iLO2 versions 2.xx, we did not test iLO2 on the BL20p G4 and did not see that with firmware version 2.12 and 2.15, iLO2 is crashing on boot on these systems.

There is a way to restore iLO2 using the following steps:

EDIT: Use HP Smart Update Firmware DVD version 10.10 instead of Firmware CD 8.70 or Firmware DVD 9.30

Download the Use HP Smart Update Firmware DVD version 10.10. The ISO image is available for download from the following URL:

http://bizsupport1.austin.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?swItem=MTX-9ed665a89aba447d925937f38b

Download the iLO 2 firmware v1.82 Component for Linux (CP012567.scexe) from the following URL:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?swItem=MTX-b1c7bb6f42054394b8012695e9

Download the HP USB Key Utility for Windows and create a bootable USB Flash drive using HP Smart Update Firmware DVD version 10.10.
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?swItem=MTX-40e268a99cc243a299352b7b20

Copy the CP012567.scexe to the USB Key under hp/swpackages folder.

Set the iLO 2 Security Override Switch on the system board to the ON position. The location of the switch is printed on a label located on the inside of the hood cover.

Connect the p-Class dongle to the front of the p-Class blade server, attach the USB key, keyboard and monitor and Power ON the p-Class server. Note that because iLO2 isn't responding, the server could take up to 30 minutes to show POST.

Once the server boots from the USB key, select Interactive mode and wait until the Select Language screen shows up.

Use the following key strokes to exit out of HP Smart Update Manager. A command prompt will be displayed over the HP Smart Update Manager window. The command prompt takes approximately 30 seconds to appear:
CTRL + ALT + d + b + x (Keep the CTRL and ALT keys pressed when typing d b x )

At the command prompt, navigate to the HP Smart Update Firmware DVD directory containing the supplemental iLO 2 firmware update by using the following command:

cd mnt/cdrom/hp/swpackages

Use the following command to unload the HPILO module:

rmmod hpilo

Use the following command to execute the iLO 2 firmware update in direct mode:

sh CP012567.scexe --direct (This parameter requires two dash (-) characters).

Once the iLO 2 firmware upgrade is completed, power OFF the server and set the iLO 2 Security Override Switch on the system board to the OFF position.

Re: BL20p G4 ILO issue

raoul_schaffner — Wed, 15 May 2013 09:34:49 GMT

Hi all,

thank you very much Vijay and Oscar for your valuable hints.

However, I did not yet manage to get into the command promt mode of the Firmware CD 8.70

1) I can boot the blade with the security override switch #1 -- OK

2) I can boot from the 8.70 USB Boot Stick or the 8.70 FW CD-R (CD drive also via USB) -- OK

3) there is no interactive mode to select, just "Boot into firmware CD 8.70" -- it seems to be interactive by default

4) I tried the key combination ctrl + alt + d + b + x (pressed all at once, or holding ctrl + alt and typing d b x, tried both) at several stages of the following process:

- booting

- loading drivers

- starting Application (Title: Proliant Firmware Maintenance CD)

- select Language (here I had to unplug the Keyboard to use the Mouse)

- License Agreement

- Home

- Tab "Firmware Update"

- "Install Firmware" (here "HP Smart Update Manager" gets loaded)

5) after loading HP SUM, I switched back Mouse and Keyboard and again tried the key combo, to no avail.

--> Is it confirmed that this works with version 8.70 of the Firmware Maintenance CD? I have read elsewhere of this key combo, but in combination with FW CD 9.30...

Greetings and many thanks again,

Raoul Schaffner.

Re: BL20p G4 ILO issue

Oscar A. Perez — Wed, 15 May 2013 14:35:37 GMT

Let me test this on my Bl20p G4 real quick and I will post the results.

Re: BL20p G4 ILO issue

Oscar A. Perez — Wed, 15 May 2013 15:36:42 GMT

Ok, the key combination Ctrl + Alt + d + b + x does NOT work in Firmware CD 8.70 for some reason but, it does work in Firmware DVD 9.30 so, I edited the recovery procedure above to use DVD 9.30 instead.

Please notice that with Firmware DVD, the location of swpackages folder changed and I'm now using CP012567.scexe which is iLO2 version 1.82

Let me know the results.
Oscar

Re: BL20p G4 ILO issue

raoul_schaffner — Wed, 15 May 2013 16:20:52 GMT

OK, here's the newest status:

The blade without ILO

1) can boot FW CD 8.70

2) cannot boot FW CD 9.10C (stalling at blue progress bar)

3) cannot boot FW CD 9.30 (stalling at blue progress bar)

Somehow, the missing ILO is affecting the boot process of 9.10C and 9.30

Just out of curiosity, I tried FW CD 9.30 on a different blade with working ILO. This boots fine, and I can confirm the key combo is working at the "Select Language" screen.

So my next step is to test FW CD 9.00 and 9.10B... my only hope, it seems. :-/

Cheers and thanks for any other hints,

Raoul.

Re: BL20p G4 ILO issue

Oscar A. Perez — Wed, 15 May 2013 16:36:37 GMT

It seems Firmware DVD won't start if iLO isn't responding.

Can you just boot the server to Linux?

All we need is to execute the below command while the iLO Security Switch is ON

sh CP012567.scexe --direct

Re: BL20p G4 ILO issue

Jimmy Vance — Wed, 15 May 2013 16:39:43 GMT

or boot into the HP Scripting toolkit shell. Same kernel the other utility images are based from. Download and run the script to make a bootable image and then copy the required .scexe file to the image

older 8.70 toolkit

http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_bd9b6997fbc7fc515f4cf4626f5c8d01=wsrp-navigationalState%3Didx%3D%7CswItem%3DMTX_2387cbb29fd245509e5e171001%7CitemLocale%3D%7Cmode%3D%7Caction%3DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.tpst=bd9b6997fbc7fc515f4cf4626f5c8d01&ac.admitted=1368635718559.876444892.492883150

latest version 9.40

http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_bd9b6997fbc7fc515f4cf4626f5c8d01=wsrp-navigationalState%3Didx%3D%7CswItem%3DMTX_482e23f4092048faadf41a71bc%7CitemLocale%3D%7Cmode%3D%7Caction%3DdriverDocument&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.tpst=bd9b6997fbc7fc515f4cf4626f5c8d01&ac.admitted=1368635812955.876444892.492883150

Re: BL20p G4 ILO issue

Oscar A. Perez — Wed, 15 May 2013 17:41:01 GMT

I just tested HP Smart Update Firmware DVD version 10.10 and it works fine. I updated the recovery procedure once again.

Re: BL20p G4 ILO issue

raoul_schaffner — Wed, 15 May 2013 18:56:58 GMT

This looks very promising!

As I didn't have time to create a Boot Stick, i went with the FW CD 10.10 directly and managed to install ILO 2.09.

Sort of a Proof of Concept :-)

Meanwhile I created the Boot Stick, copied CP012567.scexe onto it, and could therefore install ILO 1.82!

Much joy around here!

You guys are officially invited to a beer when you come to Zurich, Switzerland. :-)

I have now a last question concerning this whole situation:

Is there an authoritative list of Firmware and ILO versions that are recommended to be installed on a particular blade?

If I look at the "Download drivers and software" list for BL20p G4, I see:

Firmware 2008.09.11

ILO 1.70B

FW CD 9.10C

If I run the FW CD 9.10C, I get:

Firmware 2007.06.04

ILO 1.82

There's an obvious mismatch concerning Firmware and ILO...

Cheers and thanks again,

Raoul.

Re: BL20p G4 ILO issue

Oscar A. Perez — Wed, 15 May 2013 20:30:05 GMT

There isn't an official authoritative list of iLO2 versions for each particular system but, I would say that the Readme.txt file that we include with the Windows Online Flash component would be the best reference moving forward because, it comes directly for the iLO team and it is reviewed by a lot of people including myself.

So, If your platform isn't mentioned in the Readme.txt then, it means that the QA team did not qualify that firmware version on your platform. As of today, we are only listing G5 and G6 servers as supported platforms.

Beer sounds good. I will let you know next time the company sends me over to Switzerland. :-)