HPE DL380p GEN8 iLO4 Self Test Error

[Eggman_2112]

Ok We have a few servers showing iLO 4 has detected a self-test error. For details, consult the iLO 4 Server and iLO 4 Diagnostics page.

The diagnostics page indicates:

Embedded Flash/SD-CARD: Embedded media initialization failed due to media write-verify failure.

Embedded Flash/SD-CARD: Media controller exception 01.

We can still get an IP and log in with no problems but we see the message on every reboot.

Actually after we updated the firmware to 2.55, the message went away on the next reboot but then returned on subsequent reboots.

We have tried:

-NVRAM initialization

-Factory Resetting the server through BIOS

-Updating all firmwares to the latest

We actually have maybe 1 in 10 servers giving this issue so thats pretty signifigant.

These are being processed for resale so we would like to find out:

1. Why is the error happening

2. Is there a way to fix that problem

3. Is this a system board issue as the iLO 4 is embedded on the system board.

Thank you for your time and comments. I appreciate it!

[growley]

How did you acccess the iLO4 diag page? I am seeing the same error.

[Eggman_2112]

You just log in to iLO and go to the iLO Event Log under Information. Here is a better shot of the menu... If you can't get an IP through DHCP or use F8 to configure the iLO when the server is booting then I would guess it is a defective iLO chip.

Right now this is looking like a system board issue as the iLO is embedded so that would not be very good quality control if every 1 in 10 iLO chips are failing after 3 - 5 years....especially when that feature is such plays a major role in server administration.

[Eggman_2112]

There is an advisory for fixing this but I haven't tried it yet as it seems like a lot of steps but it seems right in line with the issue.

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04996097

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: c04996097

Version: 6

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors

Surprised no one let me know about that after hundreds of views but check it out... it's actually to big to post the solution here.

Hardware Platforms Affected: HPE ProLiant ML30 Gen9 Server, HPE ProLiant DL20 Gen9 Server, HPE ProLiant SL230s Gen8 Server, HP ProLiant SL250s Gen8 Server, HP ProLiant SL270s Gen8 Server, HPE ProLiant BL460c Gen8 Server Blade, HPE ProLiant DL360p Gen8 Server, HPE ProLiant DL380p Gen8 Server, HP ProLiant DL380p Gen8 Server, HPE ProLiant ML350p Gen8 Server, HPE ProLiant BL465c Gen8 Server Blade, HP ProLiant DL160 Gen8 Server, HPE ProLiant BL420c Gen8 Server Blade, HPE ProLiant DL360e Gen8 Server, HPE ProLiant DL385p Gen8 Server, HPE ProLiant ML350e Gen8 Server, HPE ProLiant DL380e Gen8 Server, HPE ProLiant BL660c Gen8 Server Blade, HPE WS460c Gen8 Graphics Expansion Blade, HPE ProLiant DL320e Gen8 v2 Server, HPE ProLiant ML310e Gen8 v2 Server, HPE ProLiant SL210t Gen8 Server, HPE ProLiant ML350e Gen8 v2 Server, HPE ProLiant DL580 Gen8 Server, HPE ConvergedSystem 700x (CS700x), HP ProLiant XL220a Gen8 v2 Server, HPE ProLiant XL730f Gen9 Server, HPE ProLiant DL160 Gen9 Server, HPE ProLiant DL180 Gen9 Server, HPE ProLiant DL360 Gen9 Server, HPE ProLiant BL460c Gen9 Server Blade, HPE ProLiant DL380 Gen9 Server, HPE ProLiant XL230a Gen9 Server, HP ConvergedSystem 700x v1.1 VMware Kit, HP ConvergedSystem 700x v1.1 Microsoft Kit, HPE ProLiant XL740f Gen9 Server, HPE ProLiant XL750f Gen9 Server, HPE ProLiant ML150 Gen9 Server, HPE ProLiant DL60 Gen9 Server, HPE ProLiant DL80 Gen9 Server, HPE ProLiant SL4540 Gen8 1 Node Server, HPE ConvergedSystem 700 (CS700), HPE ConvergedSystem 700, HP ProLiant ML110 Gen9 Server, HPE ProLiant XL170r Gen9 Server, HPE ProLiant XL190r Gen9 Server, HPE ProLiant WS460c Gen9 Graphics Server Blade, HP ProLiant DL580 Gen9 Server, HPE ProLiant BL660c Gen9 Server Blade, HPE ProLiant DL560 Gen9 Server, HPE ProLiant XL450 Gen9 Server

[Torsten.]

the last screenshot is showing ILO firmware 1.40, so you never did any firmware update ... and this is a firmware issue.
it is worth to try the NAND format procedure, most of the times it works.

[Eggman_2112]

I was just showing growley where the ilo log was located in the ilo management screen. Its not even a screenshot from the actual server, I just found it on google.

We actually have been noticing this issue on large numbers of servers returned from off-lease equipment. I picked one DL380 Gen8 and did the firmware update only and the issue returned upon reboot but that was before I found the advisory. As you can see in my first screenshot the firmware update alone may not fix the issue.

I did not have time to complete all the steps in the guide including NAND format as we have hundreds of servers with that issue so I would just like to have a process for any customers that want to try fixing any servers with that problem. I imagine it would take a while to do this if you had a few hundred servers. Maybe some day I will get to try it out and estimate a average time to perform the process.

As well, I noticed some servers that have that issue can add 5-10 minutes to the post time as thet just seem to sit at that iLO error for a while but they will eventually get past it.

Thanks for your interest and have a great day! Hope this can help for someone searching for that issue.

[Torsten.]

according to the advisories older ILO firmware may corrupt the file system/ partition table of the NAND, hence the several ILO failure symtoms. the fw update cannot fix the issue that already happened, but a format of the NAND can. format th NAND, and check the ILO log and diag page if it works now. you probably need to pull the plugs of the server to get an ILO cold boot.

[ITREG]

Incase you need to format NAND on multiple servers I wrote a really quick batch script. Copy and past into notepad, and save as xxxxxxx.bat

SET /p ILO=What is the iLo IP address of the server? :
SET /p USERNAME=What is the iLo username? :
SET /p PASSWORD=What is the iLo password? :

"C:\Program Files (x86)\Hewlett-Packard\HP Lights-Out Configuration Utility\hpqlocfg.exe" -s %ILO% -l c:\hpqcfg.log -f c:\Force_Format.xml -v -t user=%USERNAME%,password=%PASSWORD%

pause

[Eggman_2112]

Sweet! For now we are just letting the customers that are buying the servers know about the advisory as we've had about 5000 servers resold so it would be a little much to install operating systems or do any work besides booting the servers to erase the drive and see if the error exists. I will certainly keep that handy as others will certainly need it and we can use it on our production servers should that issue ever come in the future.

Just wondering why I've never seen this issue with iLO3 so you would think they could have a firmware update that allows the chip to do that NAND format all by itself but more wishful thinking. I did have one server showing the error then a few days later it was gone so pretty weird!

[Bob358]

Hi Torsten,

I will be formatting the NAND on at least two DL380p Gen8 server and was wondering about the order of operatio.. Should the plugging of the plugs for an iLO cold boot be done prior to formating the NAD or after?  If it even maters.

Thankks and have anice day.

[KevinMoon]

Hi Bob,

How did that format work out for you? Formatting of the NAND has never worked for me. It has always been replacement of the system board. 

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04996097

[Rafel503]

I am planning to replace the system board so will it effect any ILO settings do i need to take any ILO backup configurations,moreover what happens if i dont replace the system board or fix the issue will it cause any issue?becuase i really dont want to bring my server down as it's critical one

[Rafel503]

Hi Kevin ,

Any issues after repalcing the system board do i need to take any backup of any configuration settings

[Eggman_2112]

I would certainly do a full backup of the server with Acronis or other backup software as you may want to to revert back if any problems occur. I would also ensure all the latest matching firmwares/bios are installed on both system boards.

I also found an article on backing up iLO4 with scripts but I have never tried that as we just do hardware asset management. I don't usually work with operating systems of HPE server management software:

https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/ilo4-configuration-dump/td-p/6765114

My rule - always do full backups and plan double the time you think it will take as unexpected problems will arise when you least expect it especially when dealing with system board which may have been refurbished or recertified.

[Eggman_2112]

I think I would definitely try the steps in the advisory before replacing the system board. Having said that, I had servers where they have the iLO error and then on the next reboot it was gone but we still add a lab comment to our inventory that we saw the error just in case it is a system board issue.

We just do asset recovery disposal so I believe most of our servers are sold to companies that break down the servers and sell the individual components.  As a result we don't really have any customers complaining or returning servers due to this issue as they know about the error or other problems when they are buying the server.

I am a big fan of ProLiant Servers and we still have Compaq DL380 G1 servers in production. I was really hoping that the new Gen10 Servers with iLO5 would have had a separate addon card for iLO so that it can be changed for troubleshooting like they did with the network adapter on the Gen8 Gen9 but they kept the iLO5 on board so maybe an engineer will read this and take my advice? |-)

I see some companies that have thousands of servers that just replace the server with another one so at the end of the lease we usually get a pallet with all the bad servers that give the memory initialization / iLO errors and lots of bad drives / bad memory / bad system boards. I guess there are so many memory and cpu pins and connections that it's a wonder they even work at all!

[Rafel503]

After replacing the system board my issue got fixed,so far didnt observe any issues but need to update firmware again.

[mlasidab]

Hold those horses guys. replacing a motherboard should be the last fix.

Hi all,

We had same problem here in my company with a couple of G8 servers. ILO Version were 2.55 (Aug 16 2017).

"Controller firmware revision 2.10.00 Embedded media manager failed initialization".

The only way we could fix it was downgrading the ILO firmware to 2.54 (Jun 15 2017) for that, addi the .rpm file to a bootable  USB with SPP2018030.2018_0226.84 inside. you need to locate the folder were the packages are located and just drop the file with ILO 2.54 in.

Boot from the USB and select interactive updates. Then advanced options and select upgrade/downgrade in order to have the 2.54 listed. 

After downgrading, boot a couple of times and the issue will dissapear. Then you will able to update again to ILO 2.55 if you wouold like to.

Cheers

[Torsten.]

I really doubt that a downgrade will fix this issue or if it just don't show some symptoms ... a NAND format followed by pulling the power plugs will solve it in most cases.

[John Albrektson]

I finally got the victory on this nagging problem!  The instructions at https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04996097 list 6 steps to try, roughtly . . .

1.   Do the NAND flash
2.  Do a power-off/unplug reset
3.  Boot and check ILO to see if if error state is gone (it wasn't for me), if not THEN
4.  Do *another* NAND flash

 . . . and voila!  When ILO finished coming up (no server reboot required) the error was gone, and replaced by a glorious blue (or green, can't tell) check mark!

[Eggman_2112]

Lots of good advice and instructions so hopefully looks like this would be resolved now. Thanks to everyone who provided information and instructions.

[Torsten.]

Next ILO-4 version 2.60 will have a button in the web interface to force the NAND format ...

[Eggman_2112]

Now thats good progress! I can't wait! Thanks for giving the update...

[DblDThe3rd]

how where?

I got this pesky problem and I cant see where to force NAND format 

running iLO4 2.60

[Torsten.]

see advisory

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00048622en_us

[SWWIT]

Hi. I try to use the Powershell Script.

How to determine the Bay Number of the Embedded Flash/SD-CARD?

Help would be appreciated.