HPEBIOS cmdlets 2.1.0.1: Object reference not set to an instance of an object

[Sasler]

Hi,

We have been using HPEBIOS CmdLets quite extensively on our Gen10 servers. In most cases it works just fine, but on some we get the "Object reference not set to an instance of an object" error on any HPEBIOS Set cmdlet, although HPEBIOS Get cmdlets work just fine. Some times a simple reboot fixes the issue, but in others multiple reboots, cold boots and lot of waiting is needed. 

Here is an example script:

Enable-HPEBIOSLog -Verbose
$connection = Connect-HPEBIOS -IP 10.10.10.10 -Credential (Get-Credential) -DisableCertificateAuthentication
Get-HPEBIOSWorkloadProfile -Connection $connection
Set-HPEBIOSWorkloadProfile -WorkloadProfile VirtualizationMaximumPerformance -Connection $connection
Disconnect-HPEBIOS -Connection $connection
Disable-HPEBIOSLog

And here is the log output:

2018-11-15 08:29:56,848 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Parameter processing started. 
2018-11-15 08:29:56,850 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Processing parameter - IP.
2018-11-15 08:29:56,851 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Processing parameter - Username.
2018-11-15 08:29:56,852 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Processing parameter - Password.
2018-11-15 08:29:56,853 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Processing parameter - AdminPassword.
2018-11-15 08:29:56,854 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Processing parameter - ClientLoggedInUserPassword.
2018-11-15 08:29:56,855 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Processing parameter - Credential.
2018-11-15 08:29:56,856 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Processing parameter - DisableCertificateAuthentication.
2018-11-15 08:29:56,857 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Processing parameter - Port.
2018-11-15 08:29:56,858 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Processing parameter - Timeout.
2018-11-15 08:29:56,864 HPE.Framework.Logging.Logger.LogMessage INFO - [Connect-HPEBIOS] Executing the cmdlets with 1 task serially.
2018-11-15 08:29:56,866 HPE.Framework.Logging.Logger.LogMessage INFO - Connect-HPEBIOS ProcessCommand() -- Starts
2018-11-15 08:29:56,867 HPE.Framework.Logging.Logger.LogMessage INFO - GetCommunicationModel() -- Start
2018-11-15 08:29:56,868 HPE.Framework.Logging.Logger.LogMessage INFO - GetCommunicationProtocolType() -- Start
2018-11-15 08:29:56,879 HPE.Framework.Logging.Logger.LogMessage INFO - GetCommunicationProtocolType() -- End
2018-11-15 08:29:56,881 HPE.Framework.Logging.Logger.LogMessage INFO - GetCommunicationModel() -- End
2018-11-15 08:29:56,882 HPE.Framework.Logging.Logger.LogMessage INFO - BuildCommand() -- Start
2018-11-15 08:29:56,883 HPE.Framework.Logging.Logger.LogMessage INFO - Connect() -- Start
2018-11-15 08:29:56,885 HPE.Framework.Logging.Logger.LogMessage INFO - CreateConnection() -- Start
2018-11-15 08:29:57,118 HPE.Framework.Logging.Logger.LogMessage INFO - InitializeConnectionProperties() -- Start
2018-11-15 08:29:57,276 HPE.Framework.Logging.Logger.LogMessage INFO - InitializeConnectionProperties() -- End
2018-11-15 08:29:57,278 HPE.Framework.Logging.Logger.LogMessage INFO - CreateConnection() -- End
2018-11-15 08:29:57,278 HPE.Framework.Logging.Logger.LogMessage INFO - Connect() -- End
2018-11-15 08:29:57,280 HPE.Framework.Logging.Logger.LogMessage INFO - BuildCommand() -- End
2018-11-15 08:29:57,281 HPE.Framework.Logging.Logger.LogMessage INFO - Connect-HPEBIOS ProcessCommand() -- Ends
2018-11-15 08:29:57,283 HPE.Framework.Logging.Logger.LogMessage INFO - [Get-HPEBIOSWorkloadProfile] Parameter processing started. 
2018-11-15 08:29:57,288 HPE.Framework.Logging.Logger.LogMessage INFO - [Get-HPEBIOSWorkloadProfile] Processing parameter - Connection.
2018-11-15 08:29:57,290 HPE.Framework.Logging.Logger.LogMessage INFO - [Get-HPEBIOSWorkloadProfile] Processing parameter - OutputType.
2018-11-15 08:29:57,291 HPE.Framework.Logging.Logger.LogMessage INFO - [Get-HPEBIOSWorkloadProfile] Executing the cmdlets with 1 task serially.
2018-11-15 08:29:57,292 HPE.Framework.Logging.Logger.LogMessage INFO - Get-HPEBIOSWorkloadProfile ProcessCommand() -- Starts
2018-11-15 08:29:57,293 HPE.Framework.Logging.Logger.LogMessage INFO - GetCommunicationModel() -- Start
2018-11-15 08:29:57,294 HPE.Framework.Logging.Logger.LogMessage INFO - TestConnection() -- Start
2018-11-15 08:29:57,307 HPE.Framework.Logging.Logger.LogMessage INFO - TestConnection() -- End
2018-11-15 08:29:57,309 HPE.Framework.Logging.Logger.LogMessage INFO - GetCommunicationModel() -- End
2018-11-15 08:29:57,310 HPE.Framework.Logging.Logger.LogMessage INFO - BuildCommand() -- Start
2018-11-15 08:29:57,311 HPE.Framework.Logging.Logger.LogMessage INFO - InitializeConnectionInfo() -- Start
2018-11-15 08:29:57,312 HPE.Framework.Logging.Logger.LogMessage INFO - ReadBIOSRegistryData() -- Start
2018-11-15 08:29:57,323 HPE.Framework.Logging.Logger.LogMessage INFO - ReadBIOSRegistryData() -- End
2018-11-15 08:29:57,325 HPE.Framework.Logging.Logger.LogMessage INFO - InitializeConnectionInfo() -- End
2018-11-15 08:29:57,370 HPE.Framework.Logging.Logger.LogMessage INFO - BuildCommand() -- End
2018-11-15 08:29:57,372 HPE.Framework.Logging.Logger.LogMessage INFO - Get-HPEBIOSWorkloadProfile ProcessCommand() -- Ends
2018-11-15 08:29:57,377 HPE.Framework.Logging.Logger.LogMessage INFO - [Set-HPEBIOSWorkloadProfile] Parameter processing started. 
2018-11-15 08:29:57,378 HPE.Framework.Logging.Logger.LogMessage INFO - [Set-HPEBIOSWorkloadProfile] Processing parameter - Connection.
2018-11-15 08:29:57,379 HPE.Framework.Logging.Logger.LogMessage INFO - [Set-HPEBIOSWorkloadProfile] Processing parameter - WorkloadProfile.
2018-11-15 08:29:57,380 HPE.Framework.Logging.Logger.LogMessage INFO - [Set-HPEBIOSWorkloadProfile] Executing the cmdlets with 1 task serially.
2018-11-15 08:29:57,381 HPE.Framework.Logging.Logger.LogMessage INFO - Set-HPEBIOSWorkloadProfile ProcessCommand() -- Starts
2018-11-15 08:29:57,382 HPE.Framework.Logging.Logger.LogMessage INFO - GetCommunicationModel() -- Start
2018-11-15 08:29:57,383 HPE.Framework.Logging.Logger.LogMessage INFO - TestConnection() -- Start
2018-11-15 08:29:57,396 HPE.Framework.Logging.Logger.LogMessage INFO - TestConnection() -- End
2018-11-15 08:29:57,397 HPE.Framework.Logging.Logger.LogMessage INFO - GetCommunicationModel() -- End
2018-11-15 08:29:57,398 HPE.Framework.Logging.Logger.LogMessage INFO - ValidateParameterRecord() -- Start
2018-11-15 08:29:57,400 HPE.Framework.Logging.Logger.LogMessage INFO - ValidateParameterRecord() -- End
2018-11-15 08:29:57,401 HPE.Framework.Logging.Logger.LogMessage INFO - BuildCommand() -- Start
2018-11-15 08:29:57,402 HPE.Framework.Logging.Logger.LogMessage INFO - InitializeConnectionInfo() -- Start
2018-11-15 08:29:57,403 HPE.Framework.Logging.Logger.LogMessage INFO - ReadBIOSRegistryData() -- Start
2018-11-15 08:29:57,412 HPE.Framework.Logging.Logger.LogMessage INFO - ReadBIOSRegistryData() -- End
2018-11-15 08:29:57,414 HPE.Framework.Logging.Logger.LogMessage INFO - InitializeConnectionInfo() -- End
2018-11-15 08:29:57,415 HPE.Framework.Logging.Logger.LogMessage INFO - ValidateParameterAndValue() -- Start
2018-11-15 08:29:57,436 HPE.Framework.Logging.Logger.LogMessage ERROR - Failed for 10.10.10.10: Object reference not set to an instance of an object.
2018-11-15 08:29:57,457 HPE.Framework.Logging.Logger.LogMessage INFO - [Disconnect-HPEBIOS] Parameter processing started. 
2018-11-15 08:29:57,458 HPE.Framework.Logging.Logger.LogMessage INFO - [Disconnect-HPEBIOS] Processing parameter - Connection.
2018-11-15 08:29:57,459 HPE.Framework.Logging.Logger.LogMessage INFO - [Disconnect-HPEBIOS] Executing the cmdlets with 1 task serially.
2018-11-15 08:29:57,460 HPE.Framework.Logging.Logger.LogMessage INFO - Disconnect-HPEBIOS ProcessCommand() -- Starts
2018-11-15 08:29:57,461 HPE.Framework.Logging.Logger.LogMessage INFO - GetCommunicationModel() -- Start
2018-11-15 08:29:57,463 HPE.Framework.Logging.Logger.LogMessage INFO - TestConnection() -- Start
2018-11-15 08:29:57,477 HPE.Framework.Logging.Logger.LogMessage INFO - TestConnection() -- End
2018-11-15 08:29:57,478 HPE.Framework.Logging.Logger.LogMessage INFO - GetCommunicationModel() -- End
2018-11-15 08:29:57,479 HPE.Framework.Logging.Logger.LogMessage INFO - BuildCommand() -- Start
2018-11-15 08:29:57,481 HPE.Framework.Logging.Logger.LogMessage INFO - Disonnect() -- Start
2018-11-15 08:29:57,500 HPE.Framework.Logging.Logger.LogMessage INFO - ClearRedfishSession() -- Start
2018-11-15 08:29:57,501 HPE.Framework.Logging.Logger.LogMessage INFO - ClearRedfishSession() -- End
2018-11-15 08:29:57,502 HPE.Framework.Logging.Logger.LogMessage INFO - Disonnect() -- End
2018-11-15 08:29:57,503 HPE.Framework.Logging.Logger.LogMessage INFO - BuildCommand() -- End
2018-11-15 08:29:57,504 HPE.Framework.Logging.Logger.LogMessage INFO - Disconnect-HPEBIOS ProcessCommand() -- Ends

As you can see, the Get-HPEBIOSWorkloadProfile worked just fine but Set-HPEBIOSWorkloadProfile returned the "Object reference not set to an instance of an object" error.

This doesn't seem to make any sesne. We are seeing this perhaps every 6th or 7th  server (mostly DL380 Gen10) and since we provision several servers evey week, we do see this issue a lot. Is anyone else encountering this issue?

Thanks

[rookie1082]

Saser

We provision many Gen10 servers as well and this issue popped up for us earlier this year. Then it went away after it was discovered a reboot fixes it.

Now, it has come back in the ML110 Gen10 model, none of the set hpebios commands work only some of the time. I am not sure what the issue is. Reboots do not fix it.

[Sasler]

---

@rookie1082 wrote:
Saser

We provision many Gen10 servers as well and this issue popped up for us earlier this year. Then it went away after it was discovered a reboot fixes it.

Now, it has come back in the ML110 Gen10 model, none of the set hpebios commands work only some of the time. I am not sure what the issue is. Reboots do not fix it.

---

We also do quite few ML110 Gen10 models, but I see this mostly on DL380 Gen10 and only on some ML110. Also, in most cases a reboot fixes it, but not always.

[rookie1082]

This also occurs on the ML350 Gen10 models I believe, because my techs have reported that model to me as well, although the quantities are minimal.

The majority of the Gen10 servers we are currently configuring are just firmware and ILO settings, no BIOS settings - so this issue hasn't been that critical for us recently.

I have messaged Rajesh on this issue but he has not replied back yet, they are usually pretty busy.

The gen9 models worked perfectly and still do.

Have you tried the HPE Redfish commands from the HPE Powershell Gallery to try and set?  When I used that to trouble shoot, it gave me a more clear "reboot needed" error.  Maybe this time it would give a more verbose error message telling you what the exact issue.  I don't have any new servers at the moment to test this on.  

[Sasler]

---

@rookie1082 wrote:

Have you tried the HPE Redfish commands from the HPE Powershell Gallery to try and set?  When I used that to trouble shoot, it gave me a more clear "reboot needed" error.  Maybe this time it would give a more verbose error message telling you what the exact issue.  I don't have any new servers at the moment to test this on.  

---

No, I haven't really used HPE Redfish. Would you have an exmaple how to get that "reboot needed" data?

Thanks

[Vijay_B]

HPERedfishCmdlets can be installed from the PowerShell gallery by executing the cmdlet

 Install-Module HPERedfishCmdlets

Modifying the WorkloadProfile requires a system reboot. You could try the following script to modify the WorkloadProfile using HPERedfishCmdlets.  This script also resets the server and prints the value of the WorkloadProfile setting after reboot.

$s = Connect-HPERedfish $address $username $password -DisableCertificateAuthentication 

$settingl1 = @{}
$settingl1.Add("WorkloadProfile","Virtualization-MaxPerformance") # check redfish/v1/registries for BIOS attribute names and allowable values to be used
$setting = @{}
$setting.Add("Attributes",$settingl1)
 
$msg1 = Set-HPERedfishData -Odataid redfish/v1/systems/1/bios/settings/ -Setting $setting -Session $s -DisableCertificateAuthentication
$msg1.error


# check setting update before reset
$get = Get-HPERedfishDataRaw /redfish/v1/systems/1/bios/settings/ -Session $s -DisableCertificateAuthentication
$get.Attributes.WorkloadProfile
 

# reset server for BIOS settings to take effect
$sys = Get-HPERedfishDataRaw /redfish/v1/systems/1/ $s -DisableCertificateAuthentication

$payload = @{}
$payload.Add("ResetType","ForceRestart")

$msg2 = Invoke-HPERedfishAction -Odataid $sys.Actions.'#ComputerSystem.Reset'.target -Data $payload -Session $s -DisableCertificateAuthentication
$msg2.error


# sleep is to wait till reset
Start-Sleep -Seconds 300
 

# check if the setting applied correctly
$get2 = Get-HPERedfishDataRaw redfish/v1/systems/1/bios/ -Session $s -DisableCertificateAuthentication
$get2.Attributes.WorkloadProfile


Disconnect-HPERedfish -Session $s -DisableCertificateAuthentication

The variable $msg1 in the above script would have the details if the system reboot is needed on the setting change.

Thank you,

Vijay Bhojwani

[Sasler]

---

@Vijay_B wrote:

HPERedfishCmdlets can be installed from the PowerShell gallery by executing the cmdlet

Modifying the WorkloadProfile requires a system reboot. You could try the following script to modify the WorkloadProfile using HPERedfishCmdlets.  This script also resets the server and prints the value of the WorkloadProfile setting after reboot.

The variable $msg1 in the above script would have the details if the system reboot is needed on the setting change.

Thank you,

Vijay Bhojwani

---

Thanks, I shall test that.

Another thing that I noticed is that when reboot or cold boot doesn't help, shutting down and removing power cables does help. 

Thanks

[rookie1082]

Saser,

Have you had a chance to test the steps provided to see if there are any more details provided by the redfish cmdlets?

[Sasler]

---

@rookie1082 wrote:

Saser,

Have you had a chance to test the steps provided to see if there are any more details provided by the redfish cmdlets?

---

No, sorry. I've been too busy to test that. Our server installers have been using that full power of cycle as a workaround in the meanwhile.

Thanks

[Sasler]

---

@rookie1082 wrote:

Saser,

Have you had a chance to test the steps provided to see if there are any more details provided by the redfish cmdlets?

---

Hi,

So I tested your redfish script against a server which was getting that "Object reference not set to an instance of an object" error and it did indeed work. It did change the workload profile correctly. However, when trying to run the HPEBIOS cmdlets after that, including the reset, I was still getting the same error.

So it seem like there is something wrong within the HPEBIOS module, since the redfish module is able to do the same thing without any issues.

Thanks