ProLiant Servers (ML,DL,SL)
cancel
Showing results for 
Search instead for 
Did you mean: 

HPE DL380p GEN8 iLO4 Self Test Error

Eggman_2112
Frequent Advisor

HPE DL380p GEN8 iLO4 Self Test Error

Ok We have a few servers showing iLO 4 has detected a self-test error. For details, consult the iLO 4 Server and iLO 4 Diagnostics page.

The diagnostics page indicates:

Embedded Flash/SD-CARD: Embedded media initialization failed due to media write-verify failure.

Embedded Flash/SD-CARD: Media controller exception 01.

We can still get an IP and log in with no problems but we see the message on every reboot.

Actually after we updated the firmware to 2.55, the message went away on the next reboot but then returned on subsequent reboots.

We have tried:

-NVRAM initialization

-Factory Resetting the server through BIOS

-Updating all firmwares to the latest

We actually have maybe 1 in 10 servers giving this issue so thats pretty signifigant.

These are being processed for resale so we would like to find out:

1. Why is the error happening

2. Is there a way to fix that problem

3. Is this a system board issue as the iLO 4 is embedded on the system board.

Thank you for your time and comments. I appreciate it!20171003_164536_resized.jpg20171003_164035_resized.jpg

IT Manager
25 REPLIES
growley
Occasional Visitor

Re: HPE DL380p GEN8 iLO4 Self Test Error

How did you acccess the iLO4 diag page? I am seeing the same error.

Eggman_2112
Frequent Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

You just log in to iLO and go to the iLO Event Log under Information. Here is a better shot of the menu... If you can't get an IP through DHCP or use F8 to configure the iLO when the server is booting then I would guess it is a defective iLO chip.

Right now this is looking like a system board issue as the iLO is embedded so that would not be very good quality control if every 1 in 10 iLO chips are failing after 3 - 5 years....especially when that feature is such plays a major role in server administration.

wpid-photo-20140811171714.jpg

 

IT Manager
Eggman_2112
Frequent Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

There is an advisory for fixing this but I haven't tried it yet as it seems like a lot of steps but it seems right in line with the issue.

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04996097

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: c04996097

Version: 6

Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors
 
Surprised no one let me know about that after hundreds of views but check it out... it's actually to big to post the solution here.
 

Hardware Platforms Affected: HPE ProLiant ML30 Gen9 Server, HPE ProLiant DL20 Gen9 Server, HPE ProLiant SL230s Gen8 Server, HP ProLiant SL250s Gen8 Server, HP ProLiant SL270s Gen8 Server, HPE ProLiant BL460c Gen8 Server Blade, HPE ProLiant DL360p Gen8 Server, HPE ProLiant DL380p Gen8 Server, HP ProLiant DL380p Gen8 Server, HPE ProLiant ML350p Gen8 Server, HPE ProLiant BL465c Gen8 Server Blade, HP ProLiant DL160 Gen8 Server, HPE ProLiant BL420c Gen8 Server Blade, HPE ProLiant DL360e Gen8 Server, HPE ProLiant DL385p Gen8 Server, HPE ProLiant ML350e Gen8 Server, HPE ProLiant DL380e Gen8 Server, HPE ProLiant BL660c Gen8 Server Blade, HPE WS460c Gen8 Graphics Expansion Blade, HPE ProLiant DL320e Gen8 v2 Server, HPE ProLiant ML310e Gen8 v2 Server, HPE ProLiant SL210t Gen8 Server, HPE ProLiant ML350e Gen8 v2 Server, HPE ProLiant DL580 Gen8 Server, HPE ConvergedSystem 700x (CS700x), HP ProLiant XL220a Gen8 v2 Server, HPE ProLiant XL730f Gen9 Server, HPE ProLiant DL160 Gen9 Server, HPE ProLiant DL180 Gen9 Server, HPE ProLiant DL360 Gen9 Server, HPE ProLiant BL460c Gen9 Server Blade, HPE ProLiant DL380 Gen9 Server, HPE ProLiant XL230a Gen9 Server, HP ConvergedSystem 700x v1.1 VMware Kit, HP ConvergedSystem 700x v1.1 Microsoft Kit, HPE ProLiant XL740f Gen9 Server, HPE ProLiant XL750f Gen9 Server, HPE ProLiant ML150 Gen9 Server, HPE ProLiant DL60 Gen9 Server, HPE ProLiant DL80 Gen9 Server, HPE ProLiant SL4540 Gen8 1 Node Server, HPE ConvergedSystem 700 (CS700), HPE ConvergedSystem 700, HP ProLiant ML110 Gen9 Server, HPE ProLiant XL170r Gen9 Server, HPE ProLiant XL190r Gen9 Server, HPE ProLiant WS460c Gen9 Graphics Server Blade, HP ProLiant DL580 Gen9 Server, HPE ProLiant BL660c Gen9 Server Blade, HPE ProLiant DL560 Gen9 Server, HPE ProLiant XL450 Gen9 Server

 

IT Manager
Torsten.
Acclaimed Contributor

Re: HPE DL380p GEN8 iLO4 Self Test Error

the last screenshot is showing ILO firmware 1.40, so you never did any firmware update ... and this is a firmware issue.
it is worth to try the NAND format procedure, most of the times it works.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Eggman_2112
Frequent Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

I was just showing growley where the ilo log was located in the ilo management screen. Its not even a screenshot from the actual server, I just found it on google.

We actually have been noticing this issue on large numbers of servers returned from off-lease equipment. I picked one DL380 Gen8 and did the firmware update only and the issue returned upon reboot but that was before I found the advisory. As you can see in my first screenshot the firmware update alone may not fix the issue.

I did not have time to complete all the steps in the guide including NAND format as we have hundreds of servers with that issue so I would just like to have a process for any customers that want to try fixing any servers with that problem. I imagine it would take a while to do this if you had a few hundred servers. Maybe some day I will get to try it out and estimate a average time to perform the process.

As well, I noticed some servers that have that issue can add 5-10 minutes to the post time as thet just seem to sit at that iLO error for a while but they will eventually get past it.

Thanks for your interest and have a great day! Hope this can help for someone searching for that issue.

IT Manager
Torsten.
Acclaimed Contributor

Re: HPE DL380p GEN8 iLO4 Self Test Error

according to the advisories older ILO firmware may corrupt the file system/ partition table of the NAND, hence the several ILO failure symtoms. the fw update cannot fix the issue that already happened, but a format of the NAND can. format th NAND, and check the ILO log and diag page if it works now. you probably need to pull the plugs of the server to get an ILO cold boot.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
ITREG
Occasional Visitor

Re: HPE DL380p GEN8 iLO4 Self Test Error

Incase you need to format NAND on multiple servers I wrote a really quick batch script. Copy and past into notepad, and save as xxxxxxx.bat

 

SET /p ILO=What is the iLo IP address of the server? :
SET /p USERNAME=What is the iLo username? :
SET /p PASSWORD=What is the iLo password? :

"C:\Program Files (x86)\Hewlett-Packard\HP Lights-Out Configuration Utility\hpqlocfg.exe" -s %ILO% -l c:\hpqcfg.log -f c:\Force_Format.xml -v -t user=%USERNAME%,password=%PASSWORD%

pause

 

Eggman_2112
Frequent Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

Sweet! For now we are just letting the customers that are buying the servers know about the advisory as we've had about 5000 servers resold so it would be a little much to install operating systems or do any work besides booting the servers to erase the drive and see if the error exists. I will certainly keep that handy as others will certainly need it and we can use it on our production servers should that issue ever come in the future.

Just wondering why I've never seen this issue with iLO3 so you would think they could have a firmware update that allows the chip to do that NAND format all by itself but more wishful thinking. I did have one server showing the error then a few days later it was gone so pretty weird!

IT Manager
Bob358
Occasional Contributor

Re: HPE DL380p GEN8 iLO4 Self Test Error

Hi Torsten,

I will be formatting the NAND on at least two DL380p Gen8 server and was wondering about the order of operatio.. Should the plugging of the plugs for an iLO cold boot be done prior to formating the NAD or after?  If it even maters.

Thankks and have anice day.

KevinMoon
Occasional Visitor

Re: HPE DL380p GEN8 iLO4 Self Test Error

Hi Bob,

 

How did that format work out for you? Formatting of the NAND has never worked for me. It has always been replacement of the system board. 

 

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04996097

Rafel503
Occasional Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

I am planning to replace the system board so will it effect any ILO settings do i need to take any ILO backup configurations,moreover what happens if i dont replace the system board or fix the issue will it cause any issue?becuase i really dont want to bring my server down as it's critical one

Rafel503
Occasional Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

Hi Kevin ,

Any issues after repalcing the system board do i need to take any backup of any configuration settings

Eggman_2112
Frequent Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

I would certainly do a full backup of the server with Acronis or other backup software as you may want to to revert back if any problems occur. I would also ensure all the latest matching firmwares/bios are installed on both system boards.

I also found an article on backing up iLO4 with scripts but I have never tried that as we just do hardware asset management. I don't usually work with operating systems of HPE server management software:

https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/ilo4-configuration-dump/td-p/6765114

My rule - always do full backups and plan double the time you think it will take as unexpected problems will arise when you least expect it especially when dealing with system board which may have been refurbished or recertified.

IT Manager
Eggman_2112
Frequent Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

I think I would definitely try the steps in the advisory before replacing the system board. Having said that, I had servers where they have the iLO error and then on the next reboot it was gone but we still add a lab comment to our inventory that we saw the error just in case it is a system board issue.

We just do asset recovery disposal so I believe most of our servers are sold to companies that break down the servers and sell the individual components.  As a result we don't really have any customers complaining or returning servers due to this issue as they know about the error or other problems when they are buying the server.

I am a big fan of ProLiant Servers and we still have Compaq DL380 G1 servers in production. I was really hoping that the new Gen10 Servers with iLO5 would have had a separate addon card for iLO so that it can be changed for troubleshooting like they did with the network adapter on the Gen8 Gen9 but they kept the iLO5 on board so maybe an engineer will read this and take my advice? |-)

I see some companies that have thousands of servers that just replace the server with another one so at the end of the lease we usually get a pallet with all the bad servers that give the memory initialization / iLO errors and lots of bad drives / bad memory / bad system boards. I guess there are so many memory and cpu pins and connections that it's a wonder they even work at all!

 

IT Manager
Rafel503
Occasional Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

After replacing the system board my issue got fixed,so far didnt observe any issues but need to update firmware again.

mlasidab
Frequent Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

Hold those horses guys. replacing a motherboard should be the last fix.

 

Hi all,

We had same problem here in my company with a couple of G8 servers. ILO Version were 2.55 (Aug 16 2017).

"Controller firmware revision 2.10.00 Embedded media manager failed initialization".

 

 

The only way we could fix it was downgrading the ILO firmware to 2.54 (Jun 15 2017) for that, addi the .rpm file to a bootable  USB with SPP2018030.2018_0226.84 inside. you need to locate the folder were the packages are located and just drop the file with ILO 2.54 in.

Boot from the USB and select interactive updates. Then advanced options and select upgrade/downgrade in order to have the 2.54 listed. 

After downgrading, boot a couple of times and the issue will dissapear. Then you will able to update again to ILO 2.55 if you wouold like to.

Cheers

Torsten.
Acclaimed Contributor

Re: HPE DL380p GEN8 iLO4 Self Test Error

I really doubt that a downgrade will fix this issue or if it just don't show some symptoms ... a NAND format followed by pulling the power plugs will solve it in most cases.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
John Albrektson
Occasional Contributor

Re: HPE DL380p GEN8 iLO4 Self Test Error

I finally got the victory on this nagging problem!  The instructions at https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04996097 list 6 steps to try, roughtly . . .

1.   Do the NAND flash
2.  Do a power-off/unplug reset
3.  Boot and check ILO to see if if error state is gone (it wasn't for me), if not THEN
4.  Do *another* NAND flash

 . . . and voila!  When ILO finished coming up (no server reboot required) the error was gone, and replaced by a glorious blue (or green, can't tell) check mark!

Eggman_2112
Frequent Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

Lots of good advice and instructions so hopefully looks like this would be resolved now. Thanks to everyone who provided information and instructions.

IT Manager
Torsten.
Acclaimed Contributor

Re: HPE DL380p GEN8 iLO4 Self Test Error

Next ILO-4 version 2.60 will have a button in the web interface to force the NAND format ...


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Eggman_2112
Frequent Advisor

Re: HPE DL380p GEN8 iLO4 Self Test Error

Now thats good progress! I can't wait! Thanks for giving the update...

IT Manager
DblDThe3rd
Occasional Visitor

Re: HPE DL380p GEN8 iLO4 Self Test Error

how where?

I got this pesky problem and I cant see where to force NAND format 

running iLO4 2.60

 

Torsten.
Acclaimed Contributor

Re: HPE DL380p GEN8 iLO4 Self Test Error

see advisory

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00048622en_us


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
SWWIT
Occasional Visitor

Re: HPE DL380p GEN8 iLO4 Self Test Error

Hi. I try to use the Powershell Script.

How to determine the Bay Number of the Embedded Flash/SD-CARD?

Help would be appreciated.