Boot problems, iLO 4 self-test error, Smart Array Boot drive missing

[empeiro]

Hello, my ProLiant MicroServer Gen8 won't boot anymore after I switched it of for a couple of days.
Now it shows during boot:

>>iLO 4 has detected a self-test error. For details, consult the iLO 4 Server and iLO4 Diagnostics page. 1502: iLO 4 is disabled. Use the Security Override Switch and iLO 4 F9 ROM-Based Setup Utility to enable iLO functionality.<<
(iLO was deactivated already before)
>>HP Dynamic Smart Array B120i RAID Controller (v3.54.0, 0MB) 0 Logical Volume 1785-Drive Array Not Configured Run Array Configuration Utility

Press <F5> to configure .... 

Boot Logical Drive is configured but is Missing or Offline<<
(Array was configured, I tried to reset it but not working)

Entering Intelligent Provisioning (F10) is not possible, it shows a prompt for a view seconds and then try to boot from NIC.

Is there maybe a plug detached?

Tank you for your help!

[Ihaqueit]

Power on the server within a minimal configuration

[empeiro]

Hello Ihaqueit,
thank you for your Answer! I disconnected 1 Dimm and Drives (Also SAS plug - so just 1 Dimm, Keyboard, VGA and Network connected) but it still shows ilo error and Intelligent Provisioning just shows a promt for a short time.
Anything elese I could check?

[Ihaqueit]

What is the firmware version?

Try to reformat the NAND.

If still issue replaced the systemboard.

[empeiro]

Hello, how can I reformat the NAND without OS?

here are the fw-versions:

Smart Array v3.54.0 (1.07)

Bios J06 (04/04/19)

HP AHCI SATA Controller v0.84

Broadcom UNDI PXE-2.1 v17.2.1

[ksram]

Hi empeiro,

Please refer the link : https://internal.support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00048622en_us to get information related to NAND format.

You may see multiple options here.

Regarding the Intelligent Provisioning, try flashing IP to the next version or the latest version and try accessing.

Let us know the results.

Thank you

RamKS

[Ihaqueit]

Can you attach error screenshot.?

[empeiro]

Hello ksram,

thank you for your answer, somehow I'm not able to perform the reset. It seems the script is not working.

I installed Open Suse live (15.3) on a flash drive, and downloaded hponcfg from here and installed hponcfg-5.6.0-0.x86_64.rpm.

Worked weill but shows missing signature / key:

Spoiler

linux@localhost:~/Downloads> sudo zypper install hponcfg-5.6.0-0.x86_641.rpm
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following NEW package is going to be installed:
hponcfg

1 new package to install.
Overall download size: 59.0 KiB. Already cached: 0 B. After the operation,
additional 174.0 KiB will be used.
Continue? [y/n/v/...? shows all options] (y):
Retrieving package hponcfg-5.6.0-0.x86_64 (1/1), 59.0 KiB (174.0 KiB unpacked)
hponcfg-5.6.0-0.x86_641.rpm:
Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY
V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY

Looking for gpg key ID 26C2B797 in cache /var/cache/zypp/pubkeys.
Repository Plain RPM files cache does not define additional 'gpgkey=' URLs.
hponcfg-5.6.0-0.x86_64 (Plain RPM files cache): Signature verification failed [4-Signatures public key is not available]
Abort, retry, ignore? [a/r/i] (a): i

Checking for file conflicts: .............................................[done]
(1/1) Installing: hponcfg-5.6.0-0.x86_64 .................................[done]
Additional rpm output:
warning: /var/cache/zypper/RPMS/hponcfg-5.6.0-0.x86_641.rpm: Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY

I created the xml file as shown in your link.

Then tried to execute hponcfg-5.6.0-0.x86_64.compsig but just errors apperd:

Spoiler

sudo ./hponcfg-5.6.0-0.x86_64.compsig -l Force_Format.xml
./hponcfg-5.6.0-0.x86_64.compsig: line 1: /Begin: No such file or directory
./hponcfg-5.6.0-0.x86_64.compsig: line 2: Fingerprint: command not found
./hponcfg-5.6.0-0.x86_64.compsig: line 3: Key:: command not found
./hponcfg-5.6.0-0.x86_64.compsig: line 4: Hash:: command not found
./hponcfg-5.6.0-0.x86_64.compsig: line 5: Signature:: command not found
./hponcfg-5.6.0-0.x86_64.compsig: line 6: Fingerprint: command not found
./hponcfg-5.6.0-0.x86_64.compsig: line 7: /End: No such file or directory
./hponcfg-5.6.0-0.x86_64.compsig: line 8: $&apos;HPIMAGE\001\001ݷ\250\347\300\246V@\261J\227\274B=\334{@\377\377\377\3777wpD\020\344\a&apos;: command not found
./hponcfg-5.6.0-0.x86_64.compsig: line 9: syntax error near unexpected token `(&apos;
./hponcfg-5.6.0-0.x86_64.compsig: line 9: `5.6.0-0HPE Lights-Out Online Configuration Utility for Linux (AMD64...�� 3�qL��������&apos;
linux@localhost:/opt/hp/tools&gt; sudo zypper install hash
Loading repository data...
Reading installed packages...
&apos;hash&apos; not found in package names. Trying capabilities.
<font color="#AA0000">No provider of &apos;hash&apos; found.</font>
Resolving package dependencies...
Nothing to do.

I didn't get where to find the key so I tried a reset by using the Online ROM Flash Component.

I installed it and executed the script, worked well:

Spoiler

LASH_iLO4 v1.17 for Linux (Sep 30 2015)
(C) Copyright 2002, 2015 Hewlett-Packard Enterprise Development Company, L.P.
Firmware image: ilo4_278.bin
Current iLO 4 firmware version 2.50; Serial number ILOCZ144200G3

Component XML file: CP046465.xml
CP046465.xml reports firmware version 2.78
This operation will update the firmware on the
iLO 4 in this server with version 2.78.
Continue (y/N)?y
Current firmware is 2.50 (Sep 23 2016 00:00:00)
Firmware image is 0x1001b1c(16784156) bytes
Committing to flash part...
******** DO NOT INTERRUPT! ********
Flashing is underway... 100 percent programmed. \
Succeeded.
***** iLO 4 reboot in progress (may take more than 60 seconds.)
***** Please ignore console messages, if any.
iLO 4 reboot completed.

I rebooted and tried to enter Intelligent Provisioning by pressing F10 but still nothing happens, just showing a promt. Also entering raid configuration (F5) is not possible. During boot it displays the same iLO self test error as before.

How can I fix the hponcfg script?

[empeiro]

Hello Ihaqueit,

it just shows a promt when i try to enter Intelligent Provisioning (black screen and promt in the upper left corner). Entering raid config is not possible, it just continues booting. Ilo error message like in my first post

[Ihaqueit]

Please update the BIOS. Once done then please check if you are still getting the issue.
If the issue is still seen then please contact HPE Technical Support for any additional information or troubleshooting this issue.

[empeiro]

Hello,

seems nothing works.

I installed the last BIOS J06 (was already installed):

Spoiler

linux@localhost:~/Downloads> sudo zypper -v install firmware-system-j06-2019.04.04-1.1.i386.rpm
Verbosity: 2
Non-option program arguments: 'firmware-system-j06-2019.04.04-1.1.i386.rpm'
'firmware-system-j06-2019.04.04-1.1.i386.rpm' looks like an RPM file. Will try to download it.
Initializing Target
Checking whether to refresh metadata for Update repository of openSUSE Backports
Retrieving: repomd.xml ...................................................[done]
Checking whether to refresh metadata for Non-OSS Repository
Retrieving: repomd.xml ...................................................[done]
Retrieving: media ........................................................[done]
Checking whether to refresh metadata for Main Repository
Retrieving: repomd.xml ...................................................[done]
Retrieving: media ........................................................[done]
Checking whether to refresh metadata for Update repository with updates from SUSE Linux Enterprise 15
Retrieving: repomd.xml ...................................................[done]
Checking whether to refresh metadata for Main Update Repository
Retrieving: repomd.xml ...................................................[done]
Checking whether to refresh metadata for Update Repository (Non-Oss)
Retrieving: repomd.xml ...................................................[done]
Checking whether to refresh metadata for Plain RPM files cache
Loading repository data...
Reading installed packages...
Force resolution: No
Selecting 'firmware-system-j06-2019.04.04-1.1.i386' from repository 'Plain RPM files cache' for installation.
Resolving package dependencies...
Force resolution: No

The following NEW package is going to be installed:
firmware-system-j06 2019.04.04-1.1

1 new package to install.
Overall download size: 1.9 MiB. Already cached: 0 B. After the operation,
additional 4.8 MiB will be used.
Continue? [y/n/v/...? shows all options] (y): y
committing
Retrieving package firmware-system-j06-2019.04.04-1.1.i386
(1/1), 1.9 MiB ( 4.8 MiB unpacked)
firmware-system-j06-2019.04.04-1.1.i386.rpm:
Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY
Header SHA1 digest: OK
V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY
MD5 digest: OK

Looking for gpg key ID 26C2B797 in cache /var/cache/zypp/pubkeys.
Repository Plain RPM files cache does not define additional 'gpgkey=' URLs.
firmware-system-j06-2019.04.04-1.1.i386 (Plain RPM files cache): Signature verification failed [4-Signatures public key is not available]
Abort, retry, ignore? [a/r/i] (a): i

Checking for file conflicts: .............................................[done]
(1/1) Installing: firmware-system-j06-2019.04.04-1.1.i386 ................[done]
Additional rpm output:
warning: /var/cache/zypper/RPMS/firmware-system-j06-2019.04.04-1.1.i386.rpm: Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY

Executing %posttrans scripts .............................................[done]
CommitResult (total 1, done 1, error 0, skipped 0, updateMessages 0)
linux@localhost:~/Downloads> rpm -qlp firmware-system-j06-2019.04.04-1.1.i386.rpm
warning: firmware-system-j06-2019.04.04-1.1.i386.rpm: Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY
/usr/lib/i386-linux-gnu/firmware-system-j06-2019.04.04-1.1
/usr/lib/i386-linux-gnu/firmware-system-j06-2019.04.04-1.1/.cpq_package.inc
/usr/lib/i386-linux-gnu/firmware-system-j06-2019.04.04-1.1/.setup
/usr/lib/i386-linux-gnu/firmware-system-j06-2019.04.04-1.1/CP039541.xml
/usr/lib/i386-linux-gnu/firmware-system-j06-2019.04.04-1.1/CPQJ0613.684
/usr/lib/i386-linux-gnu/firmware-system-j06-2019.04.04-1.1/hpsetup
/usr/lib/i386-linux-gnu/firmware-system-j06-2019.04.04-1.1/setup
/usr/lib/i386-linux-gnu/hp-scexe-compat
/usr/lib/i386-linux-gnu/hp-scexe-compat/CP039541.scexe
/usr/lib/i386-linux-gnu/scexe-compat
/usr/lib/i386-linux-gnu/scexe-compat/CP039541.scexe
linux@localhost:~/Downloads> cd /usr/lib/i386-linux-gnu/firmware-system-j06-2019.04.04-1.1/
linux@localhost:/usr/lib/i386-linux-gnu/firmware-system-j06-2019.04.04-1.1> sudo ./hpsetup
Flash Engine Version: Linux-1.5.9.5-2

Name: Online ROM Flash Component for Linux - HP ProLiant MicroServer Gen8 (J06) Servers
New Version: 04/04/2019

Current Version: 04/04/2019

The software is installed and up to date.
The installation does not need to continue,
however, you may still install the software.

Do you want to install this version (y/n) ?y

Flash in progress do not interrupt or your system may become unusable.
Working.............................................
The installation procedure completed successfully.

A reboot is required to finish the installation completely.

Then tried to enter Intelligent Provisioning, again nothing happens, entering Smart Array is also not possible.

Spoiler

I tried to find a replacement mainboard but it seems it isn't for sale? Could you give me the spare part number or a link to the product?

[empeiro · Accepted solution]

Hello there,

I finally found the solution! 

I did a coulple of things like replacing the CPU (Intel Celeron - had fear the system went too hot), but I think most important was the following points:

- Reset Bios configuration, reboot, try to activate Raid B120i, if its not working,

- Diskonnect all drives but leave the boot drive attatched

- Set in BIOS "embedded SATA Configuration" to "Enable SATA AHCI Support" and reboot

- now the system drive should boot

- reboot and set in BIOS "Embedded SATA Configuration" to Enable Dynamic HP Smart Array .." 

- reboot, now pressing F5 or F10 for Smart Array Configuration should work -> Configure Boot drive as Raid 0 and set it as Boot drive

-> Done! System fixed!

- if the last step isn't working set the BIOS settings back to "Enable SATA AHCI .." and unplug the server for a while (like a couple of days so it looses the configuration) then

- repeat the steps from the beginning

What I noticed: no error logs, but features like iLO still not working - when I use intelligent provisining (F10) and try to reset iLO configuration it says "undefined".