SPP update: Node Inventory failed on Gen10 ProLiant DL360 after running SUM

[ay_jay]

We have a I have a ProLiant DL360 Gen10 server we are trying to update the firmware on using a custom SPP package built from 2022-03.

I created a custom bootable ISO which we uploaded to a web server.  In the iLO console for the server, I attached the "CD/DVD" ISO via Virtual media URL to the ISO, and it seemd to connect to it with no issues (at least once we tld the server to boot from iLO virtual media in the settings).

I selected the option to do the interactive update once it came up, then it showed something about the server being set up securely and prompted for Administrator credentials, which I enetered since we have an Administrator password set up for iLO.  That seemed to work, up until I got the error message as follows:

Localhost, Node inventory failed.
"Cannot perform the operation now. Reason: Connection failed to node . Verify no firewalls are blocking traffic between the system running SUM and the remote node."

So I tried to disable the RSBU admin password in System Settings only to find out that one was not set.  I found that out because when I entered one (then reentered it to confirm), the next time I booted it prompted me for the admin creds where it did not before to allow system config changes.  I tried to disable it using the only HPE instructions I found, which said to add a "/" to the end of the password to disable it, but it said that was an invalid character when I tried to change or add it that way.  Now I don't know how to disable the password I have definitely set, which gave me the same result (it only prompted for Administrator or iLO credentials, but fails anyway) when attempting to update the SPP.

Any help here would be appreciated.  Thanks.

[support_s]

System recommended content:

1. Release Notes for Gen10 Service Pack for ProLiant, v2022.03.0

2. Service Pack for ProLiant - Downloading and Installing an SPP

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[ay_jay]

I'm not sure how that helps.  I've already created and downloaded the correct custom SPP, I'm just having some trouble starting it properly.  Could you be more specific with the release notes or within the link?  Thanks.

[ksram]

Hi,

Thank you for the Post.

Could you please try using a full SPP iso file and check if it progress further.

Also try it on different Server just to check if there are any Network related or any other issues.

Thank you
RamKS

[ay_jay]

I've tried uploading the ISO media to the iLO5 HTML5 console of the server driectly from my computer.  I had the same results at the same spot.  It loaded and began the Inventory, but failed at the same place.  I've used this method before without issue. It took longer to load, but it did work.  I'm not sure what has changed with this version of SUM or the SPP package contents.

The reason we initially worked with a custom SPP file is because the full SPP ISO files have been probelmatic to load and store in the past and difficult to work with since they're so much larger on the disk.

[ksram]

Hi,

Please try the below : 

1)  Launch the "launch_sum.bat" from your DVD / Virtual Media / USB on your Windows Server.  Make sure you're running with an Administrator account (it doesn't have to be the Administrator account - a simple "Run as Administrator" will do.)

2)  Open a browser and navigate to the link the "launch_sum.bat" provides.  Type in your Windows administrative account username and password.

3)  Now - before you do anything - click the little down arrow to the left of "Smart Update Manager 8.4.0" in the upper right-hand corner of the screen.  Select the "Nodes" option in the menu that pops out.

4)  It will probably start looking for the node and show a "Invalid password ..." That's OK - ignore it for now.

5)  Instead, click on "localhost" in the left-hand pane to make sure it is highlighted and selected.

6)  Now, look at the upper-right-hand side of the screen for the green "Actions" button with a down arrow.  Click the down arrow next to "Actions".

7)  Choose the "Edit" option from the menu.

  Scroll down just a bit and you'll see TWO credentials sections:  "Credentials" and "iLO Credentials".

9)  Enter a Windows account username and password (that has administrative permissions) in the username and password boxes in the "Credentials" section.  It doesn't have to be THE built-in Administrator account.

10)  Next, enter your iLO username and password in the username and password boxes under "iLO Credentials."

11)  Now click on the green "OK" button at the bottom.


Basically you just have to try starting Hpsum (hpsum.bat) as Administrator and let us know if that helped.

Thank you
RamKS

[ay_jay]

We are NOT using any Windows machines and this is SUM version 8.9.5.  All we have are Red Hat servers on our network and this particular one we are starting with is an ESXi 6.7U3 host in a vSphere cluster which we are trying to update firmware on.  That is why I have loaded the ISO directly to the server, so there should be no firewall issues because it is the "DVD" ISO on the localhost connecting to itself.  The ISO file seems to load SUM from the remote https URL connection where the ISO, so I do not suspect that is the error we are getting for "connection failed to node" since it is the Localhost.  Please assist.

[ksram]

Hi,

Could you please try booting from the ISO directly using One Time Boot Menu and check.

And also share Screenshots if possible.

Thank you
RamKS

[ay_jay]

If you read the first post, I've already done that.  So far, I've attempted to mount the full ISO (~10Gb) and then the Customized (~6Gb) as virtual media in the iLO console from both a web server on our network using a URL, and also from my local machine (laptop connected to iLO5 via VPN).  I'm working remotely, so I do not have access to use a thumb drive.  However, the connection between the webserver and my laptop mounting the ISO to iLO of the server seems to be fine, since SUM loads after booting... I can get past the menu screen where you can choose "interactive mode" and to update firmware.  In the Localhost Guided Update, after completing the "Inventory of baseline" that starts automatically, SUM tries to run the "Inventory of Localhost" in the second part of Step 1 but it fails, as described in the original post.  I'm not sure I could be moe clear.  See the screeshot.

[ksram]

Hi,

Thank you for explaining again.

Understood the issue, but here we are unable to confirm what is blocking.

You may try below :

>> Try using a different SPP file (older versio) just to check if that proceeds.

>> Any other additional Security Settings (from BIOS / iLO) - if yes, remove them and try

>> Update only BIOS and iLO, then run SPP

>> Different Browsers

>> Reset BIOS and try

>> Clear NVRAM (this would require Physical intervention)

>> or I understand you are in Remote Place, but try to boot using a USB Thumd and check.

If nothing helps you may have to log a Case with HPE for suppot.

Thank you
RamKS

[ay_jay]

I already have a case open and the best response I have gotten thus far is to revert to using the previous release (2021.10) - the support engineer I spoke with said others have reported issues with 2022.03 as well.  The only path forward that seems to be working for me so far is by running SUM 8.9.5 (unpacked from the 2022.03 SPP ISO) from a Windows 10 VM that I have on my laptop.  When other system admins on my team need to support this task, they cannot since I am the only one who has access to this Windows machine.  We do, however, have a RHEL7 management server on the same network as the ESXi servers which everyone has access to and which we would like to set up to deploy updates from.

Questions:

1) Is it possible to launch a web UI for SUM ftom a Red Hat 7 server that has no Desktop UI?  If so, are there step by step instructions for how to launch a web UI for SUM remotely (through a tunnel or proxy or something similar, perhaps)? All documentation seems to imply that it's only scriptable via CLI using an input file.  (We are nearly exclusively using Red Hat 7 on our systems and manage them remotely via SSH using the CLI on all our hosts - none have the GNOME Desktop).

2) It seems that the latest update guide is for 8.7.0 and the ISO came with SUM 8.9.5.  Is there a guide available for 8.9.5?

Please advise

[ay_jay]

Also,  I've installed iSUT on the ESXi host.  I set the mode of operation to AutoDeploy and rebooted.  When it came back up, only about 3 of the "Pending" items in the Installation Queue began to install and completed.  After setting the iLO credentials for iSUT, when I ran "sum -status" it shows the following error:

"Communication to iLO failed. If iLO is configured in any of the higher security modes, then use sut -set ilousername=<username> ilopassword=<password> to set the iLO credentials. If iLO is in CAC mode, then use sut -addcertificate <path_to_certificate_file> to set the certificate details
The configuration changes for the command will be saved once the details are provided"

Then it shows me the output for the status which includes the operating mode of AutoDeploy.

SUM running remotely on the Windows machine shows that the install is "done", though about 11 more items are still on the list in the Installation queue that show as "Pending" and are not updated.  Please advise.

[ay_jay · Accepted solution]

So I figured out what the issue was - perhaps HPE can make a note of this in documentation somewhere:

Short answer: the iLO password (we are using the local 'Administrator' account for this SPP deployment task) cannot have a leading hyphen in it. ESXi interfaces with iLO at some point during the update process (and vice versa) to deploy VMware-specific updates and requires iLO credentials, but doesn't like passwords starting with a hyphen. It stores those credentials in iSUT or SUT which is an OS program used to update the OS-specific firmware.

Long answer:

HPE includes iSUT with the SPP package (full version or a customized build package). When you boot the server from the ISO attached to the iLO console (either as a virtual media URL or direct ISO file), a lightweight version of Linux is installed to the system's memory (I believe it is SUSE Linux), which also loads and runs a lightweight version of Firefox to run it's GUI of SUM (Software Update Manager 8.9.5) in the server console. After you select 'Automatic' or 'Interactive' updates and select Firmware to begin, it will prompt you for the server's iLO credentials before it begins running the inventory checks of the server's current firmware against it's baseline in the SPP package (base/full or the custom one you build on HPE's site). If that password is incorrect, it will tell you immediately and re-prompt you to enter the credentials. We were using the correct password, so it would continue on to "Discover" the host and then perform the first inventory checks, but it would fail at the second step of the inventory process.

The error message received was shown in a screenshot previously posted in this thread, but it implies that there is a network or a firewall blocking a connection to the server. This sent us on a "wild goose chase" trying to figure out why iLO can't communicate with the server it is running on, though it CAN communicate with a remote server serving the ISO file.

I installed SUM on a remote host (Windows 10 VM) and added this "Node" (iLO)... and it "Discovered" the server iLO interface successfully and even managed to update the HPE server firmware remotely after running the inventory. HOWEVER, after running the updater several times, it came back sometimes as meeting the baseline, but oddly, running it again later showed that there were upwards of 11 or 12 updates that were also still "Pending" in the iLO's Installation Queue, and there were warnings in SUM for that host which showed that iSUT (Integrated Smart Updates Tools) was not running on the OS. My understanding of iSUT is that it deploys from the SPP attached to SUM, then installs to the server and runs as a service to allow the firmware updates... but for some reason, it was not running in the OS (ESXi), though SUM completed several other firmware updates.

I tried to install SUM directly on the ESXi server via ESXCLI, and had success since the service was registered and showed as running. But changing the config to "AutoDeploy" and checking the SUT status kept showing the following error before still showing the results anyway, and that it was running:

"Communication to iLO failed. If iLO is configured in any of the higher security modes, then use sut -set ilousername=<username> ilopassword=<password> to set the iLO credentials. If iLO is in CAC mode, then use sut -addcertificate <path_to_certificate_file> to set the certificate details
The configuration changes for the command will be saved once the details are provided"

So iSUT wasn't running per SUM's warning, and now SUT running directly on the host wasn't "communicating to iLO", even though I had set the iLO Administrator password correctly in both environments (you set the Node/iLO credentials in the remote SUM UI, and set iLO creds in the ESXi SUT installation via CLI as shown above).. AND I had also noticed in the iLO UI that all the "Pending" firmware left in the Installation Queue to be installed was VMware-specific firmware/drivers. So, I figured SUT (running on ESXi) or iSUT (running from SUM) both couldn't communicate with iLO for the VMware-specfic dependencies only, which means something was funky with the communication between ESXi and iLO. The only thing I could think of to address was the password since there's no networking between them:

Our last few iLO account password rotations all began with a hyphen, and I had even previously checked VMware's and HPE password requirements, which ours met, since it worked when logging in and completing some of the updates. It was allowing the SPP process to continue, and would say it was incorrect when it was entered incorrectly... so I had no idea, initially, that a leading hyphen in the password would prevent the communication between ESXi and iLO (apparently, ESXi interfaces with iLO and vice versa, but the credentials are stored in ESXi's SUT/iSUT configs and applied only when updating VMware specific firmware).

VMware even recommends using special characters - specifically noting that a hyphen was allowed, but there's no mention of having a hyphen as the first character of the password, and this is iLO's Administrator password anyway, not ESXi's root password.

Once I changed it and ran everything again, the Pending items were finally detected by SUM and installed or cleared. I was even able to run the whole process directly from the ISO loaded to the iLO console, as we normally would have, but SUM worked without the iSUT warnings, as well. After rebooting, SUM showed the server to meet the SPP baseline.