ILO Amplifier 2.0 : "Required services are not running"

[Sven_Probst]

Hello,

after deploying ILO Amplifier 2.0 we are getting "Required services are not running. For further assistance refer to the user guide." after a while. No login is possible, only restart via "ctrl-alt-del" on console-screen..

We habe about 250 ILOs registered into the Amplifier. In the syslog we find:

Dec 8 13:18:29 xxx Wolfram: MqSyncSendRequest: Sem wait on MqSyncSemaphore[5] failed from 3 to 5, Connection timed out
Dec 8 13:18:29 xxx Wolfram: MqSyncSendRequest: Sem wait on MqSyncSemaphore[5] failed from 5 to 5, Connection timed out
Dec 8 13:18:29 xxx Wolfram: MqSyncSendRequest: Sem wait on MqSyncSemaphore[3] failed from 255 to 3, Connection timed out
Dec 8 13:18:29 xxx Wolfram: MqSyncSendRequest: Sem wait on MqSyncSemaphore[5] failed from 255 to 5, Connection timed out
Dec 8 13:18:30 xxx Wolfram: MqSyncSendRequest: Sem wait on MqSyncSemaphore[5] failed from 255 to 5, Connection timed out
Dec 8 13:18:31 xxx Wolfram: MqSyncSendRequest: Sem wait on MqSyncSemaphore[3] failed from 255 to 3, Connection timed out
Dec 8 13:18:31 xxx Wolfram: MqSyncSendRequest: Sem wait on MqSyncSemaphore[3] failed from 255 to 3, Connection timed out
Dec 8 13:18:34 xxx Wolfram: MqSyncSendRequest: Sem wait on MqSyncSemaphore[3] failed from 255 to 3, Connection timed out

The 1.95 version does not have this problem. (Configuration was restored via backup/restore after installing 2.0).

Any Idea?!

Best regards

Sven Probst

[support_s]

System recommended content:

1. iLO Amplifier Pack 2.00 Release Notes

2. iLO Amplifier Pack 1.95 User Guide

If the above information is helpful, then please click on "Thumbs Up/Kudo" icon.

Thank you for being a HPE community member.

[Sven_Probst]

Login-Screen looks like this when the error occurs:

The problem is not permanent(!)... after reboot everything is normal from 30 min up to half of a day...

[Sven_Probst]

Now I know how to trigger the error.

I just need to create a event on a managed ILO (eg. reboot). After that it I have two strange things:

- iLO Amplifier Pack: HPECentralManagementDeviceAlert: Event Name : UnauthenticatedEventReceived ::: Event Occurred : 2021-12-09T15:41:59Z
::: Severity : Critical ::: Summary : An event of type iLORestServicesStarted Event was received from xxx.xxx.xxx.xxx but the sender couldnot be authenticated. This could be a security issue and the event has been ignored. :::

(the server was added after erasing all servers from configuration... it is a "managed server"!)

- Wolfram: MqSyncSendRequest: Sem wait on MqSyncSemaphore[5] failed from 5 to 5, Connection timed out

is logged at the same second in syslog.

That triggers a "breakdown" of the server......... than it takes a long time to recover, because of the > 250 server entries. (I think, every following event start that again.)

[Birchi]

We have the same issue with the "required services are not running" and with the notifications "An event of type ...". 

The HPE engineering team (ERT) is currently working on my case 5361188162. 

[aireynol]

Same issue on one my 2.0 instances.

[aireynol]

Update, found another instance with this issue. Re-deployed both of them utilising backup/restore config and the issue came back. Re-deployed without backup/restore and so far instances are stable.

Edit: and it's gone again, something seriously unstable about this 2.0 update

[sean0brien]

I'm having similar behaviour after upgrading from 1.95 to 2.10(new appliance and restore config).

I can correlate to events in the system called ‘UnauthenticatedEventReceived’. Typically this event is triggered and then I get and 'Unknown Event' flag in the UI. This is followed by unresponsive system and if I try to reconnect in a different session, the UI eventually loads with 'Required services are not running'
After reviewing the collection of events. They all appear to be originating from the same cluster.

UnauthenticatedEventReceived 2326 / 2777
172.29.18.102 - 290 / 2777 UnauthenticatedEventReceived
172.29.18.104 - 248 / 2777 UnauthenticatedEventReceived
172.29.18.113 - 294 / 2777 UnauthenticatedEventReceived
172.29.18.118 - 202 / 2777 UnauthenticatedEventReceived
172.29.18.120 - 294 / 2777 UnauthenticatedEventReceived
172.29.18.129 - 246 / 2777 UnauthenticatedEventReceived
172.29.18.134 - 401 / 2777 UnauthenticatedEventReceived
172.29.18.136 - 364 / 2777 UnauthenticatedEventReceived

After reviewing these servers for some common configuration, they're all ProLiant BL460c Gen10 with System ROM I41 v2.32 and Ilo 5 2.18. There are other servers that do not meet the prerequisites but work fine.

I have removed and re-added some of the affected servers to test, but the issue remains.  After deploying a new instance of Ilo Amplifier 2.10 I'm able to discover these servers(not restore config) and the errors are not present.  

So perhaps there is an issue with the restore process? 

For now I've went back to the 1.95 instance and will look to get the Ilo versions upgraded.  But it seems like it would work fine if I didn't restore config and just rediscovered all. 

[aireynol · Accepted solution]

HPE support said to expect a fix in 2.12. In the meantime a workaround is to remove the event service subscription for iLO Amplifier Pack from all managed servers iLO, reboot Amplifier Pack and it will be added back once the servers have run inventory.

For Gen 10:
Step 1: Perform GET operation and read all existing subscription.
curl -k -X GET -u username:password https://<iLO IP>/redfish/v1/EventService/Subscriptions?$expand=.
Iterate over all existing subscriptions and identify the subscription map to iLO Amplifier Pack
For Example:
curl -k -X GET -u username:password https://<iLO IP>/redfish/v1/EventService/Subscriptions/<Subscription Id>
Steps 2: You can delete the subscription using the below command
curl -k -X DELETE -u username:password https://<iLo IP>/redfish/v1/EventService/Subscriptions/<Subscription Id>

For Gen 9/ Gen 8 Server:
Step 1: Perform GET operation and read all existing subscription.
curl -k -X GET -u username:password https://<iLO IP>/rest/v1/EventService/EventSubscriptions
Iterate over all existing subscriptions and identify the subscription map to iLO Amplifier Pack
For Example:
curl -k -X GET -u username:password https://<iLO IP>/rest/v1/EventService/EventSubscriptions/<Subscription Id>
Steps 2: You can delete the subscription using the below command
curl -k -X DELETE -u username:password https://<iLo IP>/rest/v1/EventService/EventSubscriptions/<Subscription Id>

"Information on why subscription to be removed

Basically the subscription info that gets restored from the backup is the problem.

The new AMP is unable to process these old subscription info.

 Hence we need to ensure the old subscription info from the backup-restore activity is not present.

 So we need remove the existing subscriptions, restart the amp and then run discovery to ensure that new subscriptions are created. Once that is done, we need to check for issue.(Note: We need to give it enough time to settle down.)"

[admtemplomar]

HI, We too face the same issue.

HPE has released version 2.12, but we are facing issue there also.

Is there any solution provided from HPE for your case ?