DL380p Gen8 | ILO Health Degraded & Display is turning black soon after OS starts to boot

[BladeSteels]

Team,

I'm facing the problem mentioned in the below url's. To start with, one day i noticed my server has stopped responding to mstsc but reachable via ping and C$ share is accessible. When I checked in Data Centre, I found that Display is not coming on KVM screen. I hard booted the server many times but no luck. Every time I used to see the Initial Boot Screen for HP Server, then actual HP logo with options to choose from F9, F10, F11 etc and finally windows used to start loading with the activity wheel running under Microsoft Logo for a while and then disappear to black screen. I connected an external monitor to front VGA port and later tried with back port too but the same results.

Since ILO was not configured, I configured ILO hoping to work from my desk and also see the errors in IML log. I noticed that my firmware version was very old (1.xx) so I immediately downloaded the latest SPP for DL380 Gen8 and tried to update firmware. Everytime ILO kept disconnecting with session time-out error. I somehow managed to create a USB and luckily after many attempts it installed the updated firmware. ALso, 1 hard drive from Raid 1 (2 drive mirror) was failed and I replaced the drive.

I manually downloaded ILO firmware and updated separately to make sure I'm running the latest for the server. I then noticed in my ILO page about ILO health is degraded and further navigating lead to the articles below.

I followed the articles and formatted the embedded flash/sd card and unplugged the server from power source and later plugged it in back. I noticed that the error used to disappear and again appear after 30 seconds. I tried two more time but same result.

REF URL:

https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/HPE-DL380p-GEN8-iLO4-Self-Test-Error/td-p/6979847#.XoHNOvFRXmw

https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/SD-CARD-Embedded-media-initialization-failed/td-p/7017878#.XoHN8_FRXmx

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c04996097

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00048622en_us

Having described the problem, my queries are:
1. Since it is ILO embedded flas/sd card, can we disable the ILO completely for now and proceed with OS boot.

2. What could be the issue with OS booting? Is it the display or vga or OS itself?

3. The articles are telling about motherboard replacement, is that correct?

Thanks in advance

Khan

[AmRa]

Hi Khan,

As per customer advisory it may be the issue with NAND chip which is embedded on system board and if you already checked by formatting the NAND then only option left is system board replacement. 

However before that  please login to ILO and confirm are you getting any below error message on iLO Diagnostic tab

"Embedded media manager failed initialization"

or

"The AHS file system mount failed with (No such device)

or

"The AHS file system mount failed with (I/O error)"

or

"Controller firmware revision 2.09.00 Could not partition embedded media device."

or 

Unable to download Active health System (AHS) report.

If any one of the above issue displays in iLO web console then need to check with system board replacement.

[BladeSteels]

Thanks for your reply. I will certainly take a look at the logs.

Just to brief you, this Server is out of warranty and now we don't want to renew warranty and get the board replaced. I just want to get this fixed.

Is there a proper way of disabling ILO? I read somewhere that by going through F8 that option is available. If so, what harm would it do if i disable ILO? 

Why is system not booting and entering a kind of loop? The display goes black once it starts to display MIcrosoft OS Logo. I don't mind disabling F10 also. I just want the system to boot with or without these error getting noticed. what is the best way.

[BladeSteels]

I logged into ILO & under Diagnostics page I see the below info,

Self Test.....Embedded Flash or SD Card
Status.....Yellow triangle, Warning Symbol, like degraded
Notes....Controller Firmware Revision 2.10.00 Embedded Media Initialization failed due to media write-verify test failure.
WIll this ILO problem cause system to not boot into OS or bring the display to black screen.

[Torsten.]

This is the well known "ILO NAND issue".

It only affects OS boot if you boot from internal SD card.

[BladeSteels]

Hi Torsten,

I have 2 drives for Operating System in Raid 1 and 3 drives in Raid 5.
Boot controller order is raid card followed by sata controller, as far as I remember from home.
I didn’t set boot from Internal Drive. Still, how I can check this and confirm.
In any case, why is the display going black or into repair mode when I boot from iso (OS) file.
I did ran chkdsk and scannow which fixed few issues.
If I don’t want Ilo can I disable it? How?