[ProLiant DL360e Gen8] NAND Error

[Trle94]

 Hello everyone,
I am currently having issues with NAND error, currently error i have is: 
Controller firmware revision 2.10.00 NAND read failure: Media is in a WRITE-PROTECTED state

But it shuffles always.. Let me first tell you fully what have i tried doing, so i upgraded iLo to latest version 2.61 and there was error about NAND I did click 'Format Embedded Flash and reset iLO' after reboot it came up working BUT Intelligent Provision was not working still, so I tried updating it on latest version, after update i started seeing again NAND read failure, now I tried several times clicking 'Format Embedded Flash and reset iLO' it didnt work, then after a while i tried also over HPQLOCFG to run 'FormatFlash' script, which did not do anything, I even tried shuting down server and taking out power cords for few minutes and putting them back.

Here is log from script:

Spoiler

<?xml version="1.0"?>
<RIBCL VERSION="2.23">
<RESPONSE
STATUS="0x0000"
MESSAGE='No error'
/>
</RIBCL>
<?xml version="1.0"?>
<RIBCL VERSION="2.23">
<RESPONSE
STATUS="0x0000"
MESSAGE='No error'
/>
</RIBCL>
<?xml version="1.0"?>
<RIBCL VERSION="2.23">
<RESPONSE
STATUS="0x0000"
MESSAGE='No error'
/>
</RIBCL>
<?xml version="1.0"?>
<RIBCL VERSION="2.23">
<RESPONSE
STATUS="0x0000"
MESSAGE='Forcing a format of the partition after the iLO reset.'
/>
</RIBCL>
<?xml version="1.0"?>
<RIBCL VERSION="2.23">
<RESPONSE
STATUS="0x0000"
MESSAGE='No error'
/>
</RIBCL>
<?xml version="1.0"?>
<RIBCL VERSION="2.23">
<RESPONSE
STATUS="0x0000"
MESSAGE='No error'
/>
</RIBCL>
<?xml version="1.0"?>
<RIBCL VERSION="2.23">
<RESPONSE
STATUS="0x0000"
MESSAGE='No error'
/>
</RIBCL>

Script succeeded

Any ideas what should I do to get it working again, and solve Intelligent Provision?
Looking forward to your answers!

[Torsten.]

Click format NAND, then pull the power plugs.

Now check in ILO log if the format was really done.

[Trle94]

Hello,
I am getting following error:
1789
04/02/2019 18:09 04/02/2019 18:09 1 Embedded Flash/SD-CARD: Embedded media initialization failed due to media write-verify test failure.
1788
04/02/2019 18:08 04/02/2019 18:08 1 Embedded Flash/SD-CARD: Failed restart..

And in diagnostics it changed to following:

Controller firmware revision 2.10.00 NAND read failure

Then after next only iLo reset it was changed back to:

Controller firmware revision 2.10.00 NAND read failure: Media is in a WRITE-PROTECTED state

[Torsten.]

Unfortunately the NAND is probably permanently damaged.

[Trle94]

Is there anything else we could try?
Would it work if we put sd card?

[Erdogan Temur]

Hi,

You try the following action plan step by step.

• Select the iLO Security Override Switch to ON
• Boot the server using USB SPP with CPxxxxxx.scexe copied to hp\swpackages
• Select Interactive
• CTRL + ALT + D + B + X
• Navigate to mnt/cdrom/hp/swpackages
• Unload HP iLO modüle rmmod hpilo
• sh scxxxxxx.scexe –direct

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_00392fb1f9214da7a5de4cf9a7

If problem persist the replace mainbaord.

[Trle94]

Hello,
Can I do it over booted OS? I have Centos right now installed, cant i just execute it?

[Erdogan Temur]

Hi @Trle94 

It needs to be done by the method suggested above.

[George Vardikos]

Hi Erdogan,

In SPP 2018.03 after I select "Interactive" and come up to the language selection, Ctrl+Alt (and D + B + X with the Ctrl+Alt pressed) does not bring anything up. I tried another combination (D + B + G) for another purpose and it works, but I can't seem to be able to get D + B + X working.

[Torsten.]

This is disabled for security reasons.

To make it work, switch the "ILO security" DIP switch on the systemboard.

[John_from_ea]

good process

[Bojnit]

Hi, did the presented solution by Erdogan Temur work? Thanks for the reply.

[jouniota]

Hi,

I'm also curious to know if the procedure worked.

I have two dl360 gen8 with the same iLO health problem, both upgraded to the last firmware (2.77 from dec 2020).

[0x3333]

Just to register here, I had this problem of WRITE-PROTECTED and doing what Torsten said(Set iLO DIP switch to on) fixed the issue!

I set the DIP Switch number 1 to ON, did the flash format, and reverted it to OFF. Fixed the issue.

[Shalak]

@0x3333  I'm facing the same issue. What ilo firmware version did you use?

[0x3333]

Hi Shalak,

Unfortunately, the issue wan't fixed... I was not able to fix it. The server is running though.

[Shalak]

@0x3333  So basically you say that it helped you for a moment? And then it went back to "Media is in a WRITE-PROTECTED state"?

I'm a little worried about disabling the security DIP. Currently my iLO seems to work properly, I can reach the console, powerup the machine and redfish API works, I don't use HPE OneView and the AHS logs are useful only if I have HPE support license, right?

I do not want to make things worse. I tried several time to use the "Format Embedded Flash and reset iLO" option in iLO GUI, with and without AC powercycling in between - but the log "Embedded Flash/SD-CARD: One or more storage devices have been formatted" does not appear in the iLO Event Log.

[0x3333]

Yep, for a little moment, and then, WRITE-PROTECTED again. Still is...

Mine is with security DIP disabled, for quite a while, it runs in a private network so not an issue, but I'll enable it again as it was unable to fix the issue.

I believe I'll need to live with it. Just leave it as is... as it is working.

[nems8]

We are seeing the same problem on a DL360p Gen8. 

ILO has been updated, and currently running 2.78.

The below seems to suggest updating ILO? (or replace mb, not an option on this old hardware)

Hi,

You try the following action plan step by step.

• Select the iLO Security Override Switch to ON
• Boot the server using USB SPP with CPxxxxxx.scexe copied to hp\swpackages
• Select Interactive
• CTRL + ALT + D + B + X
• Navigate to mnt/cdrom/hp/swpackages
• Unload HP iLO modüle rmmod hpilo
• sh scxxxxxx.scexe –direct

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_00392fb1f9214da7a5de4cf9a7

If problem persist the replace mainbaord.

Kind Regards,
Erdogan.

Just wondering if you guys noticed any more side effects from nand error? I can see that IP is not available, but server seems to be working as it should. No problem logging into ILO and no other errors reported.

Trying to figure out if i should try format of nand and powercycle (ref: https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00048622en_us ) or just leave it.

[0x3333]

Same here, server and iLO works OK, I'm used to the error after so many yers...