"HPE Oneview did not receive an expected event notification from the server..." after 5.50 upgrade

[vx_it]

This has been bugging me for a few days now. After upgrading to OneView 5.50, 16 out of 22 servers have this error in the logs, and I can't figure out if it's active or just an old, stuck error - it's locked so I can't clear it. All of the ILOs are on the same subnet. 

I tried following the steps suggested, but it still remains.

[AmRa]

Have you checked by implementing the action plan suggested under resolution section.

Also please let us know if its HPE Synergy OneView or OneView for ProLiant?

[MLW]

We have the same issue after update to 5.50. Refresh and iLO reset have not fixed it.

[Dennis314159]

Today I have upgrade my HPE OneView appliance from 5.20 to 5.40 to 5.50. In 5.40 everything looked OK, but in 5.50 I have this same issue for all 17 servers. Servers vary from Gen9 and Gen 10. Resetting the iLO and then refreshing makes no difference.

[vx_it]

I have implemented the action plan suggested, with no result. 443 is not blocked. This is regarding OneView for ProLiant.

[Geoff Schunicht]

The issue is that the Redfish event sending service on the iLO appears to have stopped sending events, so OneView logs this message. It was found that an iLO reset will correct (does not need or imply a server OS reboot). I would suggest logging a case to make sure this unexpected issue is tracked. I see above resetting iLO does not seem to help so for sure we need support to gather more details.

OneView uses these Redfish events to help remain in-sync with changes on the server, reducing the need to poll servers.

[wildpanda_nz]

Hi Geoff,

I'm also seeing this issue after upgrading to Oneview 5.5 on around 70 of the 160 servers monitored by Oneview. I've followed the recommendations for a number of servers, refreshing the server in Oneview, resetting the iLO, in one case I've even removed the server from Oneview completely and re-added it and I'm still getting this "expected event notification" error on the servers.

[Geoff Schunicht]

Thanks for the additional information, please work with you local support to get a case open then so they can capture needed data, this is not expected behavior.

[cesarpegado]

Also seeing the same issue since the update didi anyone get a fix if not i will also log a call.

Thanks

[DrDoc]

After checking firewall log I've seen indeed a drop from the ilo back to oneview. Added the ilo-ip to an appropriate rule for https ... and voila: after clicking refresh the problem disappeared.
hint: check if certificates are all valid and "trusted"

[vx_it]

Hi DrDoc,

Thanks for coming back to share your solution - could you give me a little more info here, please?

Are you referring to ILO certificates or OneView certificates?

You added a rule in your firewall to allow HTTPS for each ILO IP? 

[wildpanda_nz]

I've just checked our firewall logs and we're also seeing dropped 443 requests from the iLO to the Oneview appliance. We'll update our firewall to allow those requests and confirm that it fixes the issue - that might take a few days due to change control.

I assume that the 5.5 release put in a new check to raise an event when the Oneview appliance picks up iLO alerts via outbound polling but never received the alert via an inbound push from the iLO itself?

[DrDoc]

Hi vx_it,

for the certificates I'm referring to ssl certificates, for oneview and for ILOs (because https is used). Of course you can use the self-generated ones, but they have to be added as "trusted".

For the firewall: yes I've configured a rule which allows https (and some more) from the ILO back to OneView.
Best to check the Firewall ports required for HPE OneView on the help-page within your appliance.

https://<%yourOneview5.5%>/help/cic/en/index.html#s_security-open-ports-cic.html

cheers

[Geoff Schunicht]

Hi Wildpanda_nz,

The change in OV 5.5 was to provide a new alert generated by OneView when it does not receive events POSTed by the iLO to OneView. OneView does generate a "test" event for this purpose and if that HTTPS REST event is not recieved in a timely manner you will see this.  As noted above the most common reason for this has been firewall configurations for TCP port 443 coming from the iLO to the OneView appliance. These events help inform OneView of changes on the server such as progress of a firmware update vs hardware alerts.

OneView uses SNMP traps for most HW alerts, these come in on UDP port 162. in most cases OneView is not polling for alerts, though there is a health validation poll every 20 minutes that gets a "high-level" status from the iLO.

[Marcel_D]

Same Problem here, but I had the FW-Ports open.

Still shows me the Error. What did the Job for me, is doing a iLo-Reset and a Refresh of the iLo in OneView. But what now sometimes shows up, is a Warning: "Unable to fetch the firmware installation status from the server hardware."

It is not really satisfying to do this for ~110 iLo's even when you are able to Script it.

[HeikoB]

Same issue here. I can confirm, that "doing a iLo-Reset and a Refresh of the iLo in OneView" fixed it for me.

Fortunately only one of twenty servers was affected (all same Gen & firmware).

[Admin_CPCPIPE]

After upgrading to 5.5, the same error occurred. Restarting and refresh ILO helped me.

[amirgilan]

Hi, I have a same issue after upgrading to 5.5 and Restarting ILO and refresh doesn't helped me.

Would it possible to explain more how this error apear? I've checked my firewall and all ports (Inbound and Outbound) are opens.

[DanCernese]

This new message is reporting an existing latent issue in your environment.  It literally means the iLO RESTful API events are not making it back to OneView.  That can only mean either they're being lost/blocked or there's an issue in the iLO itself that is no longer sending events.   Without those events, the reported power state may be wrong, inventory updates, and other status is not getting to OneView.  In some configurations if an iLO was reset or modified directly, the OneView 'subscription' to it's events may have been disabled.  The reset and refresh actions recreate that subscription just in case.

In some customer environments, we discovered one direction was enabled by a firewall but the other direction was not (orthat the near 'end' of a network was configured correctly but the firewall on the other end was not).

inbound/outbound; TCP; https; 443

[Dennis314159]

I fixed my problem by implementing the firewall rules mention in article https://support.hpe.com/hpesc/public/docDisplay?docId=mmr_kc-0113804.

[cesarpegado]

I previously thought upgrdaing the firmware fixes it, but i had a batch of servers in 1 subnet having this issue, and that ended up being a firewall issue. So i'm now trying to get networks to check the connection between iLOs and OneView server.

[Rajini_Saini]

Hi,

If there are no issues on the server, then to clear the alert;
Refresh the server hardware and wait for 5 minutes for the alert to be cleared.
If refreshing the server hardware did not clear the alert, Reset the iLO, and then refresh the server hardware.
Ensure that port 443 is not blocked by a firewall.
Refresh the server hardware after enabling port 443.
Link: https://support.hpe.com/hpesc/public/docDisplay?docId=mmr_kc-0113804

Thank you,

Regards,
Rajini

[amirgilan]

Thanks Dear DanCernese,

With your help, I was able to solve the problem on some servers, but some of the servers (or firewalls on their network) belong to a company or group outside my reach, which unfortunately can not be fixed. For example, the BareMetal servers we use for CDN may have networks of several different companies in different parts of the world whose network ports cannot be changed by me..
My advice is to change STATE of these errors from "Locked" to "Warning" so that we can at least understand the errors more accurately and cleared manually.
 
Thanks a lot.
Amir Gilan