Trouble with iLO upgrade using powershell

[uzimmermann]

So this has worked many times before but now I am getting problems.

Linux Powershell 7.1.1 with HPEiLOCmdlets 3.0.0.1:

> $conn2 = $ilos2 | Connect-HPEiLO -Credential $ilocred -DisableCertificateAuthentication

> Update-HPEiLOFirmware -Location "/home/XXX/iLO/firmware/iLO5/ilo5_233.bin" -Connection $conn2

Confirm
Are you sure you want to perform this action?
Performing the operation "Update-HPEiLOFirmware" on target "10.237.75.22".
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): a
WARNING: Update is in progress, this might take several minutes.
Update-HPEiLOFirmware: [10.237.75.22]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.23]: The terminal URI could not be found for odataType UpdateService with traversal path ['Oem'].['Hpe'].['Actions'].['#HpeiLOUpdateServiceExt.AddFromUri'].['target']
Update-HPEiLOFirmware: [10.237.75.72]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.73]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.74]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.75]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.76]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.77]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.172]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.173]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.174]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.175]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.231]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.232]: Object reference not set to an instance of an object.
Update-HPEiLOFirmware: [10.237.75.233]: Object reference not set to an instance of an object.

Same with Powershell on Windows:

Powershell version 5.1 build 14383 Revsion 3866 with HPEiLOCmdlets 3.0.0.1

PS C:\Users\XXX > $conn2 = $ilos2 | Connect-HPEiLO -Credential $ilocred -DisableCertificateAuthentication

PS C:\Users\XXX> Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin -Connection $conn2

Confirm
Are you sure you want to perform this action?
Performing the operation "Update-HPEiLOFirmware" on target "10.237.75.22".
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): a
WARNING: Update is in progress, this might take several minutes.
Update-HPEiLOFirmware : [10.237.75.22]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.23]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.72]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.73]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.74]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.75]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.76]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.77]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.172]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.173]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.174]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.175]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.231]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.232]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

Update-HPEiLOFirmware : [10.237.75.233]: Object reference not set to an instance of an object.
At line:1 char:1
+ Update-HPEiLOFirmware -Location C:\Users\XXX\Downloads\ilo5_233.bin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Update-HPEiLOFirmware], Exception
+ FullyQualifiedErrorId : HPE.iLO.Cmdlet.UpdateHPEiLOFirmware

This used to work just fine, I have done at least 10 rounds of upgrades this way. I am stumped.

Any ideas?

[PowerShellLover]

We have the same problem with the new ilo5_233.bin firmware image. It happens only with this file. The Update-HPEiLOFirmware works with the new ilo4_277.bin without problems. There seems to be a problem in the filehandling (buffer underun?). Nothing we can do about it without a fix by hpe.

Tested with HPEiLOCmdlets 3.0.0.1 and HPEiLOCmdlets 2.x

For reference and google purposes the german error messages:
FullyQualifiedErrorId >> HPE.iLO.Cmdlet.UpdateHPEiLOFirmware
Exception.Message >> Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.

PS: This is NOT a "FullyQualifiedErrorId" dear HPE programmers...

[uzimmermann]

At least under Linux I have the same problem with the iLO 2.77 file, I don't think I tried that one under Windows.

[AmRa]

We have seen this issue with iLO 5 2.33 and Windows PowerShell iLO Cmdlets Version:3.0.0.1,  Same issue has been highlighted to HPE Engineering team. 

HPE backend team is working on this issue and this may fix in new HPE Windows PowerShell iLO Cmdlets release.

[Questions]

I hope the "may fix" for this is an understatement.

When is an ETA for an actual fix for it?

/Henke

[Steam_E]

Dears

We have also this bug this some months. At this moment we are not more able to maintain all of our servers up-to-date.

Do you have in meantime a corrective or a workaround? Thanks

[uzimmermann]

HPEiLOCmdlets 3.1.0.0 which I just installed a couple days ago made this work again.

[Luke Hsieh]

While the upgrade feature was fixed with HPEilocmdlet 3.1.0.0 and ILO5 2.3x, I found the cmdlet to configure Directory Group is still broken.  I complained about it in March and got response sayind it's being worked on and they're aiming to release update in April.  Not too many days in April left though...

[PowerShellLover]

I can confirm this. With HPEiLOCmdlets 3.1.0.0 the Update-HPEiLOFirmware commandlet works again for binary iLO-Update files.

But it seems there was a change in the behaviour of Update-HPEiLOFirmware when sending an iLO-Update file. It returns as soon as the file is completely uploaded. In the versions before it returned after the update was flashed. Now I seek a way to check if the flashing of the update is still running (like the the box "Flashing Firmware x%" shown when logged into the ilo via browser). But I cannot find a cmdlet showing this info. The only thing I found was checking the eventlog for firmware upgrade events (via (Get-HPEiLOEventLog).EventLog | where {$_.Message -like '*Firmware*Upgrade*'} ). But this is not perfect in any way...

Any ideas?

[PowerShellLover]

I have looked at this problem a little bit deeper.

iLO Generation 4 uploads and flashes until Update-HPEiLOFirmware returns an empty object.

iLO Generation 5 only uploads until Update-HPEiLOFirmware returns an StatusInfo object with message ResetInProgress.

More details on ResetInProgress message:

$output = Get-HPEiLORedfishMessageInfo -Connection $hpe_ilo_con

$output.ResetInProgress

Description : A management processor reset is in progress.
Message : A management processor reset is in progress.
NumberOfArgs : 0
ParamTypes : {}
Resolution : Wait for management processor reset to complete, and then retry the operation.
Severity : Warning
Oem :

 Which is strange because when the warning is returned by Update-HPEiLOFirmware the ilo starts to verify the uploaded data and then flashes the bin file. So there is no reason for the function to return an error...

tested with: HPEiLOCmdlets 3.1.0.0, iLO4 2.77/2.78, iLO5 2.41/2.44

Has anybody else seen this?