HPE OneView Alerting

[Mike64]

Hello,

We get a number of alerts per day, from HPE OneView, as below.

"Critical Connectivity lost for adapter in slot 0, port 2. ILO-<server name>
Resolution If this server is powered off, booting, or shutting down, occasional connectivity alerts may occur as the network adapter negotiates with the switch or interconnect. These alerts can be ignored and should clear automatically. If this server is booted up and running an operating system, this alert indicates a loss of connectivity between the network adapter and the switch or interconnect. Check the physical connection from the server to its destination device such as interconnect ,blade, switch etc, including any cables. Also check the configuration of the operating system and destination device."

These critical connectivity alerts invariably clear and OneView goes back green.

Unfortunately, a legitimate critical alert, relating to a hardware failure, was overlooked due to OneView alert fatigue.

Our network engineer advises the network is fine and will not investigate.

OneView ping utility is limited in it's flexibility so I cannot put a continuous ping, to a server, and record the results to a text file. Also, as the OneView appliance does not reply to a ping, I cannot setup a continuous from the server to OneView.

I need to reduce the number of connectivity alerts. If we do have a NIC failure, or other hardware failure, knowing about it after a five minute delay is acceptable to me.

Is there a way to modify the polling frequency of OneView?

Has anyone else had this issue and, if so, how was it resolved satisfactorily?

Many thanks,

Mike

[TKop · Accepted solution]

Hello,

This alert is coming from the iLO.  A polling interval is not in play as OneView is taking in what is sent to it.  The iLO is reporting (often via the NIC info) the connection was there but now gone. It would clear itself when recognizing the connection is back. Causes of this behavior may be:

1. The network connction via the port is indeed going down/up. (realizing you noted the network engineer believes things are fine)
2. Issue in either the NIC or iLO FW - is newer FW being used?  There have been some adjustments made over time in the FW to adjust how this event is determined.
3. HW issue - (if this is Synergy, the interconnect HW/FW is another factor)

OneView could itself have an issue however with this alert it is reporting back information being sent to OneView. A support case would be recommended if all the above checks out, to further dig into the root cause.

[support_s]

System recommended content:

1. HP OneView 1.05 UI Online Help (.zip - save to local system for off-appliance browsing)

2. HP OneView 1.10 UI Online Help (.zip - save to local system for off-appliance browsing)

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[Mike64]

Hi TKop,

Excellent feedback, thank you.

I've reviewed all the alerts and found they are coming from older DL380 Gen9 servers and not our newer DL380 Gen10 plus servers which have a later iLO firmware version.

My next steps will be to update the iLO firmware, on our Gen9 servers, and see if that resolves the issue.

Thanks again,

Mike

[bradawk1]

We've had OneView send us alerts about interfaces that aren't even connected or used.  For a while, we went into RBSU and disabled the NIC if possible, but that was a big pain.  Eventually we started sending the alerts to splunk and let splunk sort through it and only send the real alerts.  We are also using nagios to check the status of things.  We really don't use OneView anymore for status alerts.

[TKop]

I believe iLO has made improvements such that events aren't sent for the not connected/unused scenario (would need to double check).  IOW, the events are not to be sent if the port has not been used.  Only when a connection has been seen on the port, and then disconnected, the event would be sent.

Note, you can configure OneView's email notification filtering to only send the alerts that are of interest.  That allows specifying what to include, but not what to explicitly "ignore."  Just mentioning if it helps someone...