HPE DL380p GEN8 iLO4 Self Test Error

[Eggman_2112]

Ok We have a few servers showing iLO 4 has detected a self-test error. For details, consult the iLO 4 Server and iLO 4 Diagnostics page.

The diagnostics page indicates:

Embedded Flash/SD-CARD: Embedded media initialization failed due to media write-verify failure.

Embedded Flash/SD-CARD: Media controller exception 01.

We can still get an IP and log in with no problems but we see the message on every reboot.

Actually after we updated the firmware to 2.55, the message went away on the next reboot but then returned on subsequent reboots.

We have tried:

-NVRAM initialization

-Factory Resetting the server through BIOS

-Updating all firmwares to the latest

We actually have maybe 1 in 10 servers giving this issue so thats pretty signifigant.

These are being processed for resale so we would like to find out:

1. Why is the error happening

2. Is there a way to fix that problem

3. Is this a system board issue as the iLO 4 is embedded on the system board.

Thank you for your time and comments. I appreciate it!

[growley]

How did you acccess the iLO4 diag page? I am seeing the same error.

[Eggman_2112]

You just log in to iLO and go to the iLO Event Log under Information. Here is a better shot of the menu... If you can't get an IP through DHCP or use F8 to configure the iLO when the server is booting then I would guess it is a defective iLO chip.

Right now this is looking like a system board issue as the iLO is embedded so that would not be very good quality control if every 1 in 10 iLO chips are failing after 3 - 5 years....especially when that feature is such plays a major role in server administration.

[Eggman_2112]

There is an advisory for fixing this but I haven't tried it yet as it seems like a lot of steps but it seems right in line with the issue.

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04996097

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: c04996097

Version: 6

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors

Surprised no one let me know about that after hundreds of views but check it out... it's actually to big to post the solution here.

Hardware Platforms Affected: HPE ProLiant ML30 Gen9 Server, HPE ProLiant DL20 Gen9 Server, HPE ProLiant SL230s Gen8 Server, HP ProLiant SL250s Gen8 Server, HP ProLiant SL270s Gen8 Server, HPE ProLiant BL460c Gen8 Server Blade, HPE ProLiant DL360p Gen8 Server, HPE ProLiant DL380p Gen8 Server, HP ProLiant DL380p Gen8 Server, HPE ProLiant ML350p Gen8 Server, HPE ProLiant BL465c Gen8 Server Blade, HP ProLiant DL160 Gen8 Server, HPE ProLiant BL420c Gen8 Server Blade, HPE ProLiant DL360e Gen8 Server, HPE ProLiant DL385p Gen8 Server, HPE ProLiant ML350e Gen8 Server, HPE ProLiant DL380e Gen8 Server, HPE ProLiant BL660c Gen8 Server Blade, HPE WS460c Gen8 Graphics Expansion Blade, HPE ProLiant DL320e Gen8 v2 Server, HPE ProLiant ML310e Gen8 v2 Server, HPE ProLiant SL210t Gen8 Server, HPE ProLiant ML350e Gen8 v2 Server, HPE ProLiant DL580 Gen8 Server, HPE ConvergedSystem 700x (CS700x), HP ProLiant XL220a Gen8 v2 Server, HPE ProLiant XL730f Gen9 Server, HPE ProLiant DL160 Gen9 Server, HPE ProLiant DL180 Gen9 Server, HPE ProLiant DL360 Gen9 Server, HPE ProLiant BL460c Gen9 Server Blade, HPE ProLiant DL380 Gen9 Server, HPE ProLiant XL230a Gen9 Server, HP ConvergedSystem 700x v1.1 VMware Kit, HP ConvergedSystem 700x v1.1 Microsoft Kit, HPE ProLiant XL740f Gen9 Server, HPE ProLiant XL750f Gen9 Server, HPE ProLiant ML150 Gen9 Server, HPE ProLiant DL60 Gen9 Server, HPE ProLiant DL80 Gen9 Server, HPE ProLiant SL4540 Gen8 1 Node Server, HPE ConvergedSystem 700 (CS700), HPE ConvergedSystem 700, HP ProLiant ML110 Gen9 Server, HPE ProLiant XL170r Gen9 Server, HPE ProLiant XL190r Gen9 Server, HPE ProLiant WS460c Gen9 Graphics Server Blade, HP ProLiant DL580 Gen9 Server, HPE ProLiant BL660c Gen9 Server Blade, HPE ProLiant DL560 Gen9 Server, HPE ProLiant XL450 Gen9 Server

[Torsten.]

the last screenshot is showing ILO firmware 1.40, so you never did any firmware update ... and this is a firmware issue.
it is worth to try the NAND format procedure, most of the times it works.

[Eggman_2112]

I was just showing growley where the ilo log was located in the ilo management screen. Its not even a screenshot from the actual server, I just found it on google.

We actually have been noticing this issue on large numbers of servers returned from off-lease equipment. I picked one DL380 Gen8 and did the firmware update only and the issue returned upon reboot but that was before I found the advisory. As you can see in my first screenshot the firmware update alone may not fix the issue.

I did not have time to complete all the steps in the guide including NAND format as we have hundreds of servers with that issue so I would just like to have a process for any customers that want to try fixing any servers with that problem. I imagine it would take a while to do this if you had a few hundred servers. Maybe some day I will get to try it out and estimate a average time to perform the process.

As well, I noticed some servers that have that issue can add 5-10 minutes to the post time as thet just seem to sit at that iLO error for a while but they will eventually get past it.

Thanks for your interest and have a great day! Hope this can help for someone searching for that issue.

[Torsten.]

according to the advisories older ILO firmware may corrupt the file system/ partition table of the NAND, hence the several ILO failure symtoms. the fw update cannot fix the issue that already happened, but a format of the NAND can. format th NAND, and check the ILO log and diag page if it works now. you probably need to pull the plugs of the server to get an ILO cold boot.

[ITREG]

Incase you need to format NAND on multiple servers I wrote a really quick batch script. Copy and past into notepad, and save as xxxxxxx.bat

SET /p ILO=What is the iLo IP address of the server? :
SET /p USERNAME=What is the iLo username? :
SET /p PASSWORD=What is the iLo password? :

"C:\Program Files (x86)\Hewlett-Packard\HP Lights-Out Configuration Utility\hpqlocfg.exe" -s %ILO% -l c:\hpqcfg.log -f c:\Force_Format.xml -v -t user=%USERNAME%,password=%PASSWORD%

pause

[Eggman_2112]

Sweet! For now we are just letting the customers that are buying the servers know about the advisory as we've had about 5000 servers resold so it would be a little much to install operating systems or do any work besides booting the servers to erase the drive and see if the error exists. I will certainly keep that handy as others will certainly need it and we can use it on our production servers should that issue ever come in the future.

Just wondering why I've never seen this issue with iLO3 so you would think they could have a firmware update that allows the chip to do that NAND format all by itself but more wishful thinking. I did have one server showing the error then a few days later it was gone so pretty weird!

[Bob358]

Hi Torsten,

I will be formatting the NAND on at least two DL380p Gen8 server and was wondering about the order of operatio.. Should the plugging of the plugs for an iLO cold boot be done prior to formating the NAD or after?  If it even maters.

Thankks and have anice day.