Diagnose iLo 5 access on a ProLiant DL 360 Gen 10 server

[stvk]

BLUF

I'm trying to login to iLo but I can't because it is not working properly based on the following evidence I can only describe since I don't have enough permissions to attach pictures.  

The server has been out of support for a while.

Do the findings below indicative of a defective iLo card ?

Do I need to buy support and engage with HP ? 

Background

The server was dormant unplugged from power for a more than a year. Upon starting the server I tried to config iLo following the steps shown at:

https://techlibrary.hpe.com/docs/iss/EL8000t/setup_install/GUID-D7147C7F-2016-0901-06D0-00000000066F.html

The iLo port was already connected to a Cat5E going to a network switch. Both lights on the iLo port and the netwrok switch are flashing green. 

In my config I disabled DHCP and assigned a static IP and after completing the rest of the setup I am able to SSH to iLo with the user credentials defined during its configuration but I am unable to login via a web browser.

Problem 

The iLo 5 logon https web page pops up with a delay but when I try to login (using the same logon credentials I'm using to SSH) I get the error

Login Error:
error=0, message=timeout, details=undefined

I can ping the iLo's IP and I can see it during POST as an IPv4 iLo IP address but the "Click iLO 5 Configuration Utility." does not show up anymore in the iLO setup after pressing F9 and then in UEFI System Utilities. 

Running MSoft's Port Qry on iLo's IP from another machine on the network shows that ports 22, 80 and 443 are listening.

I tried to update to the latest Online ROM Flash Firmware Package for HPE Integrated Lights-Out 5  downloading the software from  

https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_085e2069704d494788d9b37948

and the RESTful Interface Tool from here

https://buy.hpe.com/us/en/software/infrastructure-management-software/system-server-management-software/system-server-software-management-software/restful-interface-tool/p/7630408

but that did not change the situation, meaning I still cannot logon to iLo using a web browser (tried Chrome, Foirefox and Edge but no luck), only via SSH.

After my attempt to upgrade the Online ROM Flash Firmware, I SSH to iLo, and ran the "show map1/firmware" only to find out that the version showing is not the latest one, and the iLo access via the web is still returing the above error

Login Error:
error=0, message=timeout, details=undefined 

I also used Smart Update Manager (SUM) iso and tried to run an overall update on all components. SUM gave a list of updates that ran up to the point that needs access to iLo but because the credentials I'm using (that work when I SSH to iLo) do not work, I can't login so I cannot finish the updates, among which is also the Online ROM Firmware update.

I also get 3 errors when the server boots up

270 - iLO FW Communication Issue - Unable to communicate with iLO FW. Certain
management functionality is not available.
Action: Remove power. If issue persists, update iLO FW. If issue still
persists, call HPE service.

Starting required devices. Please wait, this may take a few moments....

338 - HPE RESTful API Error - Unable to communicate with iLO FW. BIOS
configuration resources may not be up-to-date.
Action: Restore Manufacturing Defaults in RBSU.

312 - HPE Smart Storage Energy Pack 1 Communication Failure - Communication
with the energy pack failed. Its output may not be enabled.
Action: Verify energy pack is properly installed. Update System ROM and iLO
firmware. Confirm energy pack status in the iLO GUI. Reboot server. Contact HPE
support if condition persists.

Here's a summary of the server's HW info. Win 2019 is completely updated. 

OS Name Microsoft Windows Server 2019 Standard
Version 10.0.17763 Build 17763
Other OS Description Not Available
OS Manufacturer Microsoft Corporation
System Manufacturer HPE
System Model ProLiant DL360 Gen10
System Type x64-based PC
System SKU 874459-S01
Processor Intel(R) Xeon(R) Silver 4112 CPU @ 2.60GHz, 2594 Mhz, 4 Core(s), 8 Logical Processor(s)
Processor Intel(R) Xeon(R) Silver 4112 CPU @ 2.60GHz, 2594 Mhz, 4 Core(s), 8 Logical Processor(s)
BIOS Version/Date HPE U32, 2/9/2023
SMBIOS Version 3.2
Embedded Controller Version 2.81
BIOS Mode UEFI
BaseBoard Manufacturer HPE
BaseBoard Product ProLiant DL360 Gen10
BaseBoard Version Not Available
Platform Role Enterprise Server
Secure Boot State Off
PCR7 Configuration Not Available
Windows Directory C:\WINDOWS
System Directory C:\WINDOWS\system32
Boot Device \Device\HarddiskVolume1
Locale United States
Hardware Abstraction Layer Version = "10.0.17763.4644"
User Name Not Available
Time Zone Eastern Daylight Time
Installed Physical Memory (RAM) 64.0 GB
Total Physical Memory 63.7 GB
Available Physical Memory 60.0 GB
Total Virtual Memory 73.2 GB
Available Virtual Memory 69.5 GB

[support_s]

System recommended content:

1. HPE Integrated Lights Out (iLO 5) - " The iLO health monitoring status of the device / adapter located in Embedded is not responsive " is shown in iLO Event log

2. HPE Integrated Lights Out (iLO 5) for Gen10 Servers - iLO Key Features Moderator [above link is no longer valid, please visit https://support.hpe.com/connect/s/  to find the latest info ]

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[stvk]

1. ILO 5 is on ver. 2.81  (>2.3). Please see below results of show command when I login to iLo via SSH. 

</>hpiLO-> show /map1/firmware1

status=0
status_tag=COMMAND COMPLETED
Sun Aug 13 01:08:17 2023

/map1/firmware1
Targets
Properties
version=2.81
date=Mar 07 2023
name=iLO 5
Verbs
cd version exit show load

2. I cannot login to the web interface via https using my credentials for an unknown reason (I'm trying to find out), so I reset the iLO from the command line.

I still cannot login from the web interface but now there's also a warning that says:

iLO Self-Test reports a problem with: Embedded Flash. View details on Diagnostics page. 

So, I researched the above warning and I found the following hpe.com URL where user ksram recommended a 3-step process. 

https://community.hpe.com/t5/proliant-servers-ml-dl-sl/ilo-self-test-reports-a-problem-with-embedded-flash-sd-card-view/td-p/7109055 

I followed the steps of that process and ran:

step 1 (used the RESTful Interface Tool and iLO 5 is now on its latest & greatest version 2.95)

step 2 (reset iLO) but I'm still unable to login to iLO from a web interface getting the error:

Login Error:error=0, message=timeout, details=undefine

Should I perform step 3 (format NAND) or  I should buy HP support to eval the server's system board ?

[stvk]

Server updates:

1. iLO5 is now on ver. 2.95 (the process finished successfully using the RESTful Interface Tool)

2. reset iLO from SSH CLI

Persisting issue(s):  Cannot login to iLO from Web interface. The error I'm getting is:

Login Error:error=0, message=timeout, details=undefine

I also get a Warning that says:   iLO Self-Test reports a problem with: Embedded Flash. View details on Diagnostics page.

Question: Should I perform a NAND format or I should be better buying HP support to eval the server's system board that may be defective ?   

Thanks,  S-

[Tam92]

Hello,

You may try below options

1) Reset BIOS to manufacturing defaults

2)Perform a complete power cycle of the server
3) Connect your Laptop directly to the iLO port of the server (Make sure the laptop's IP address and the iLO IP address are in the same range)
4) Open a browser and try to access iLO

5) You can try using the default iLO credentials available on the serial pullout tag on the server
6) If the issue persists, try NAND flash
7)If nothing works then this could be a hardware related issue and would require the system board to be replaced.

You can contact HPE support for further information and advise on the replacement process for Out of warranty products.

Thanks,
TAM

[thutchings · Accepted solution]

Hello,

Did the other POST errors seen get corrected once the iLO was updated or do the 270, 338, and 312 errors continue? Those errors would typically indicate that the iLO / BIOS communication is not working. This might also result in the error you noticed about the embedded flash since the iLO is not longer the device to manage the flash (NAND). The iLO 4 used to manage the flash and would sometimes result in corrupted NAND. 

You can try to reformat the NAND on the system, but I am not sure that is causing the problem. As previously mentioned, I would recommend setting the BIOS to manufacturing defaults. If this does not correct the problem, then you can try removing the CMOS battery for approx 5 minutes, reinsert and power the server up. These steps along with potentially a factory reset of the iLO should correct this.

Regards

[stvk]

Thanks TAM92,

I went through steps 1-5 and my situation persists. At this point I'm reluctant to run Step 6 since I'll be talking to HP support anyways. 

I'll use the HPE support case number user support_s created when I engage with an HP engineer and go from there. 

I'll also update this thread when I've answer to the issue at hand but that may take some time. Thus, if keeping the thread open for an unknown amount of time is not an option, then it's my request for a moderator to let me know. In that case I'll accept a previous response and close the thread. 

Appreciate everyone's help,

S-        

[Tam92]

Hello,

It will be difficult to keep this thread open for a long time. You can open a new thread anytime with reference to this thread if you have any queries and we shall advise you further.

Thanks,

TAM

[stvk]

Thanks thutchings,

Q: Did the other POST errors seen get corrected once the iLO was updated or do the 270, 338, and 312 errors continue?

A: Unfortunately, the errors 270, 338, and 312 persisted after setting the BIOS to default system settings and even after setting the BIOS to system and manufacturing default settings. 

Admittedly, the following is a very useful comment.

Those errors would typically indicate that the iLO / BIOS communication is not working. This might also result in the error you noticed about the embedded flash since the iLO is not longer the device to manage the flash (NAND). The iLO 4 used to manage the flash and would sometimes result in corrupted NAND. 

Per your recommendation I also tried the following sequence.

1. Set the BIOS to manufacturing defaults (once again).

2. Since item 1 did not correct the problem I removed the CMOS battery for approx 7 minutes, reinserted and powered the server back up.

3. Performed an iLO factory reset 

And the result when I tried to login to iLO from the Web Interface was the message below in a red box right below Log in

Login Error:error=0, message=timeout, details=undefine

and after a couple of minutes the following warning also popped up.

iLO Self-Test reports a problem with: Embedded Flash. View details on Diagnostics page. age the flash (NAND). The iLO 4 used to manage the flash and would sometimes result in corrupted NAND. 

While my hopes are significantly dwindling, my mind points me to either a server retrofit or a trade-in. A quick cost-benefit analysis will tell which financial route is the most cost effective.

Thanks again.

[stvk]

Understood Tam92.

I'll follow your advice if and when I come to a resolution.

I thank the community for their valuable insight. 

Over & out,

S-