ILO 5 shows health and storage degraded but there is no error

[pirx]

I updated some ILOs last week to version 3.02 - not sure if this is related, also tried 3.03. Now I noticed that some DL360Gen10 show a warning regarding health and storage. But when I check the storage page I don't see any error, there is also nothing in log. 

Any idea how get rid of this warning or how to find the root cause?'

[support_s]

System recommended content:

1. HPE iLO 5 v3.02 Release Notes | Issues and workarounds

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[shiva_jr]

Hi @pirx , 
      The server health and storage shows the degraded status, even if the system components such as fan, power supply, DIMM, drives not seated properly or symptom of failure. Please try -
1) Check the IML logs, you may get the warnings about system components. Take necessary action.
2) Diagnose the DIMM and Drives, check any issues.
3) Check each of the component one by one in iLO, if any warnings or alerts.

Regards,
Shiva_JR

[pirx]

The storage health is degraded but storage page shows no error, not sure why I should DIMMs then. As I already wrote, no error in ILM either. And this is not just one server, its multiple at different locations. Only change, new ILO firmware 3.02.

[Mislav Puškar]

Hi

We have same problem on 2 of 3 DL360 Gen10 we upgraded, first to 3.02, then to 3,03.

[shiva_jr]

Hi  @pirx ,
     As it is showing as one of the disk failed. I suggest to diagnose the disk once.   If it is found true, replace the disk. 
     If similar issue for multiple servers with iLO 3.03 update, then try downgrading the firmware to previous version (may be to 3.00 (15 Dec 2023)
     Even the same issue persists, please contact HPE support.
Regards,
Shiva_jR

[pirx]

As it is showing as one of the disk failed. I suggest to diagnose the disk once.

@shiva_jr Is it showing this? Which disk then? All disks have ok state, that's my point. I already opend a case but support has no clue either. 

Edit: update from 3.02 to 3.03 did not solve the issue. But reverting to 3.00 does. So please HPE fix this ongoing ILO 5 mess!

[pirx]

I updated HPE support case with the information that the issue is related to newer ILO versions. The reaction: update controller firmware. There is no advisory that newer ILO version need latest controller firmware. There is also nothing in ILO changelogs. And support does not test if the update of controller fw really solved this issue.,

My expectation is: support should report this to R&D and its then fixed in next version. But instead I should trial and error with downtime of services on our end. The HPE support just does not exist for other things then sending out replacement parts.

[bubbafrombama]

I am having the same issue as all here, even with V3.04 iLO firmware. I did upgrade to the latest controller firmware and they did correct the issue for me. They should note this in the requirements of the new iLO firmware.

We've also seen the new firmware past V2.99 cause the storage to "disappear" from the iLO, although it still shows up in tools like iLO Amplifier and OneView. I assume this new storage controller firmware will rectify that, as well. Sigh!

[Andreabiro]

Same to ML350

Thanks HPE

I'll wait the next HP SPP

[pirx]

I did what HPE support asked me, updated the P408 controller firmware. No luck.

[Reza_F]

I have this problem too. My ilo vèsion is 3.06 and this happened after sudden power outage. What is the solution?

[Andreabiro]

I update the ilo firmware

[4ig]

ILO 5 FW downgrade to 3.01 solved the problem. The health status OK, but the RAID controller in the device inventory is still in warning state. Server reboot needed to clear degraded staatus in the device inventory.

[Scott_341]

I have the same issue on 2ea. DL380 G10 Servers after a long power outage that the UPS's couldn't hold overnight so the servers came down hard.
Same described issues with the degraded state and flashing orange on front panel, but everything is green. Even Oneview doesn't see any issues.

P408i-a controller is already version 6.52
iLO version is 3.04

Tried Clearing iLO and IML logs - Didn't work.
Tried Resetting iLO - Didn't work.
Tried Updating iLO Firmware to version 3.07 - Didn't work.

I'm perplexed and out of ideas.

[Scott_341]

I downgraded to 3.01 and that resolved the problem for me.

[pirx]

For me the only version that fixed the issue for a longer time is version 3.00. Everything version after that starts to have the problem again after a few days/weeks or a hard reboot.

The support I got from HPE is pure comedy. They have no clue and requested logs now for the 4th+ time. I asked them to close my case as it is just eating up my time with updating controller firmware the x time.

ILO 5 verion 3.08 also does not fix it. There is now version 7.11 for HPE Smart Array controler with a lot of fixes regarding redfish mentioned. Maybe this fixes it.

[sjiangclst]

My server was on the combination of ilo 3.08 and controller firmware 7.11 and it still crashed. After readying this post, I downgraded ilo to 3.01 and it has been stable so far.  3.09 just came out few days ago. Not sure if it has the fix.

[pirx]

Only 7 months later after many trial and error firmware updates for ILO, Array etc, support now acknowleges the issue. They even used my screenshots, I should sue them!

Advisory: HPE Integrated Lights-Out 5 -In iLO 5 Version 3.01 or Later, the iLO GUI Health Summary Displays an Error as "Storage Degraded" When No Storage Issue Has Actually Occurred

[DGT33]

Just as an update. P408i-a at version 7.11, iLO5 at version 3.10 (dec 2024) and the issue persist.

In our case, a nice DL380 Gen10