OneView Alert Mails - Filtering out Multiple Conditions

[Michael Leu]

I am trying to create a Email Filter in OneView 3.00.08 which should trigger on status:critical and status:warning but ignore the following events:

healthCategory:Appliance eventTypeID:Alerts.DBMaxCapacity
healthCategory:Network eventTypeID:Trap.cpqNic3ConnectivityLost
healthCategory:Network eventTypeID:Trap.HealthStatusArrayCategoryStatus

Any idea how this could be accomplished? Because it seems only one "NOT" is allowed in a query...

[ChrisLynch · Accepted solution]

You will need to change your search criteria to something like the following:

NOT alertTypeID:trap.cpqHeThermalTempDegraded NOT alertTypeID:trap.cpqFca2PhyDrvStatusChange

[Geoff Schunicht]

I had luck leveraging a format like below, I used the Activity view and search option to help isolate a search string that seems to work.

status:critical status:warning NOT alertTypeID:trap.cpqHeThermalTempDegraded NOT alertTypeID:trap.cpqFca2PhyDrvStatusChange

Since I don't have the same data, I can't validate, however for the example provided I would try:

status:critical status:warning NOT alertTypeID: Alerts.DBMaxCapacity NOT alertTypeID::Trap.cpqNic3ConnectivityLost NOT alertTypeID::Trap.HealthStatusArrayCategoryStatus

[David Claussen]

In your question, I see a lot of alert/email formatting. I have been searching for this type of information for months.

I have a new install of OneView Standard 3.10.07-0310774. I have a handfull of ProLiant DL380s and 360s added to it.

As stated in other posts, when setting an allert for simply warning and critical, I receive 6 alert emails when a server reboots.

I would like to be able to create an alert something like this:   status:warning status:critical Physical Drive Status Change  

Is there a document anywhere that can help me with creating alerts that are very specific? I am looking to alert on failed hard drives, failed power supplies, etc, but only things like that. When a server reboots, I receive and alert that the network connectivity to iLO has been lost. I don't want to see that hence the request for being very specific with alert definicians.

Thanks all!

- Dave Claussen

[Mainecoon]

Trying to setup some filter in Oneview and it seems to work fine as long as i can find the cpq string in event details, but how to exclude one of these? as there is no "cpq" string in the error details.

The overall status of the system network is in the error state.

1. Health category InterconnectBay

1. Event details

1. portId

   571956-B21-7C940800V5:d9

2. portStatus

   SubportStateChanged

3. resourceUri

   /rest/interconnects/61b8cbf6-495a-4e79-8ad0-43e4a633cfc0

[nbhms]

I've been looking for the same thing, no luck so far.  Very sparce documentation on some of the finer points of this product.

[Mainecoon]

Any luck with this, i would like to exclude network, interconnect and security alerts like these every hour about leave certificates. so far not much luck.

status:critical status:warning NO associatedresourcecategory:interconnects

or

status:critical status:warning NOT alertTypeID:trap.cpqNicMibCondition NOT associatedresourcecategory:appliance NOT associatedresourcecategory:interconnects

[David Claussen]

With no sufficient documentation and zero support here in the forums, I gave up.

I see that there is a new version, so I might try again, but it is likely the same thing will happen.

Forcing us away from System Insight Manager, in to OneView, and not supporting OneView - not cool.

[Mainecoon]

Yes it is frustrating not be able to filter this in a easy way, i mean why would you send an security alert about an expired leave certificate every hour!? who cares!, why all this fuzz about certificates anyway. I'm running the latest version 4.10.03-0364293.

testing now with this one:

status:critical status:warning NOT associatedresourcecategory:interconnects

not sure if it works, als no clue how to filter the security alerts about crtificates, maybe this:

NOT associatedresourcecategory:appliance

Update: unfortunataly i just got about 30+ alerts saying "An error has occurred on connection 2. Interconnect..........." so who can explain me how to filter this...this topic is closed but there is no real answer.

Looking at this comment in one of the manuals:

TIP: Filters have the same syntax as the Smart Search box in the Activity screen, so you can copy them and paste them in this field.

I started to play with the smartfilter and made the query below that i'm now testing:

status:critical status:warning NOT crm.connectionstatechange NOT alerts.certificatestatus.expired NOT remote-support.notconnectedtohp

Not working....now updated to this

status:critical status:warning NOT alertTypeID:crm.connectionstatechange NOT alertTypeID:alerts.certificatestatus.expired NOT alertTypeID:remote-support.notconnectedtohp

[ChrisLynch]

---

@David Claussen wrote:

With no sufficient documentation and zero support here in the forums, I gave up.

I see that there is a new version, so I might try again, but it is likely the same thing will happen.

Forcing us away from System Insight Manager, in to OneView, and not supporting OneView - not cool.

---

Please remember that these forums are for the community to interact with.  It is not for official product support.

We are updating our documentation to address this very issue.  So, with the next OneView release, we will have better documentation for how to use the Smart Search and Email Filtering capabilities of OneView.

[ChrisLynch]

Yes it is frustrating not be able to filter this in a easy way, 

I'm sorry that this is frustrating.  As I just mentioned, we are working on updating our user documentation on how to use the Smart Search and Email Filtering capabilities.  

i mean why would you send an security alert about an expired leave certificate every hour!? who cares!, why all this fuzz about certificates anyway.

It is very important for you to know that you have expired certificates.  HPE OneView will not be able to negotiate TLS with the iLO or endpoint whos certificate has expired.  One way to address this is to use an internal enterprise PKI (i.e. Microsoft Certificate Services) to manage endpoint SSL certificates.  That way, you add the issuing certificate authorities public certificate to the appliance once, and that is it.  If the endpoints SSL certificate expires, you simply renew it from the issuer, and we will the implicitly trust that certificate, as long as it is valid and not revoked.

status:critical status:warning NOT alertTypeID:crm.connectionstatechange NOT alertTypeID:alerts.certificatestatus.expired NOT alertTypeID:remote-support.notconnectedtohp

This should work, and is similar to the updated examples we are adding to the user documentation and online help.

[Mainecoon]

Hi Chris, thank you for the responce!

About the certificates is this also the case when you disable th cert. verification? as i had many expired certs and did not see any connection issues to the iLO but got fludded every hour with ton's of email. Also now that i clear most of these there are still old cert warning (that are already updated) that are in a locked state that i cannot clear also not via commandline.

Also about this line:

status:critical status:warning NOT alertTypeID:crm.connectionstatechange NOT alertTypeID:alerts.certificatestatus.expired NOT alertTypeID:remote-support.notconnectedtohp

i can tell you it is not working at least not for the connection state part, for the other two i will have to verify the activity log to see if we had any of these.

I would like to filter the connection alert below that happens during server reboot:

Critical

 An error has occurred on connection XX. Interconnect XXXXXXXXXX, interconnect X port X subport X is unlinked. XXXXXXXX
 Resolution If the server XXXXXXXX, bay X is power cycling or powered off, connectivity alerts may occur as the network adapter
 is either disconnected or negotiates connectivity with the interconnect. These alerts can be ignored and should clear automatically.
 If this server is booted up and running an operating system, this alert indicates a loss of connectivity between the network adapter
 and the interconnect. Verify that the CNA supports the current interconnect downlink speed. Verify the configuration of the
operating system, health and link status of the downlink ports on which the connection depends. If the problem persists,
please contact your authorized support representative and provide them with a support dump.
• Alert details

Updates:

I can confirm that the following is working:

12/17/2018  - "NOT alertTypeID:remote-support.notconnectedtohp"  filters out "Remote support is unable to connect to HPE. Remote Support" .

12/18/2018 - "NOT alertTypeID:profilemgr.connections.connection_scmb_error"  filters out
"An error has occurred on connection X. Interconnect XXXXXXXXX, interconnect X port XX subport a is unlinked" .

12/18/2018 - " NOT alertTypeID:crm.connectionstatechange" filters out "Connection on downlink port XX, subport a has failed. The subport is unlinked".

Now looking into "The overall status of the system network is in the error state."

 I'm NOT yet working for HPE.

[Daniel-L]

Hello Mainecoon,

thanks for providing your insight into the alert filtering posibilites - this really helped me a lot!

Did you have any luck in filtering "The overall status of the system network is in the error state."?

[BPOS]

This is a confusing statement as the help menu in OneView has a link to the community forum.  What's the purpose of a community forum if not to SUPPORT the community who BUYS your PRODUCTS?  Also, I'm on the latest version and can see no improvement with documentation on this setting.  It would be very helpful if more examples were provided.  Maybe glean over the most commonly asked questions on the forum and include in the docs?

[ChrisLynch]

Hello and welcome to the HPE Community @BPOS.  The community here is to allow HPE customers and partners to discuss HPE specific topics.  It is not the official avenue for product support.

We have made some changes to our user documentation, which will be posted soon when our next major release is generally available.  It will provide more guidance on how to use the Search box beyond just stating to provide API query syntax as the documentation currently states.  Once the updated user guides are posted, I'll follow back up with the URL and where to locate that information.

[Daniel-L]

Hello Chris,

thanks for your update on this.

While waiting for this next release, could you maybe give me a hint, how to retrieve the alertTypeID for any Altert listed in the activity view. Could this be archived via API call / PowerShell or do we just have to forward all Alerts via syslog or snmp to a third system to do the analysis there?

Regards,

Daniel

[Mainecoon]

Well yes and no in sense that i decided to remote "status:warning" from the filter as this is a warning message only. So far i don't see anything filtered out that i would have like to see. Currently i only see network error's when we do firmware upgrade on the iLO.

@chris, thank for the sharing this news, looking forward for the update.

[Daniel-L]

Spend some time analysing the alerts for a single server directly with the API using Postman.

GET https://oneview.lab/rest/alerts?start=0&count=-1&filter="severity EQ 'CRITICAL'"&filter="alertState EQ 'Cleared'"&filter="healthCategory EQ 'Network'"&filter="resourceUri EQ '/rest/server-hardware/31333138-3839-5A43-3338-313457374356'"

Pulled the interesting parts:

"description": "The overall status of the system network is critical.",
"alertTypeID": "Trap.HealthStatusArrayCategoryStatus",

"description": "Connectivity lost for adapter in slot 2, port 4.",
"alertTypeID": "Trap.cpqNic3ConnectivityLost",

So my notification filter now looks like this:

status:critical NOT alertTypeID:profilemgr.connections.connection_scmb_error NOT alertTypeID:crm.connectionstatechange NOT alertTypeID:Trap.cpqNic3ConnectivityLost NOT alertTypeID:Trap.HealthStatusArrayCategoryStatus

Hope this helps anybody else.

[pcrooney]

I'm going through the same pain as you all. I also ended up finding the strings using the REST API, using powershell and Invoke-RestMethod.

One thing it states in the online OneView help file, is that only one NOT operator is allowed, any others are treated as a string and not as an operator. However, a space between files works like and an AND, so you could do NOT yourstring anotherstring anotherstring.

I used the following code to authenticate to the client via REST, using Powershell, to see the guts of the alert emails, that included the AlertID strings.

In the alert email you get now, there is a hyperlink to the rest URI for each alert (ex: https://10.45.8.18/rest/alerts/11704), just plug that in to the end of this script and output to a file.

All you need to change in the below script is username/password, if using Local authentication, plus your appliance IP address, and at the bottom the rest URI for your alert email, like the example above. Don't change anything else, especially the PUT, that is you sending your authentication. This isn't my code, I found it online somewhere. Cheers to whoever wrote it originally.

*******************************start script****************

$appliance = "10.45.8.18"

$url = "https://$appliance"

$web = New-Object Net.WebClient

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }

$output = $web.DownloadString($url)

$user = @{

    userName= "Administrator"

    password= "YOURPASSWORD"

    authnHost= "$appliance"

    authLoginDomain= "Local"

}

$json = $user | ConvertTo-Json

$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"

$headers.Add("Content-type", 'application/json')

$headers.Add("Accept", 'application/json')

$uri = $url + '/rest/login-sessions'

$response = Invoke-RestMethod -Uri $uri -Method POST -Headers $headers -Body $json -ContentType 'application/json'

$auth = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"

$auth.Add("Auth", $response.sessionID)

$auth.Add("X-Api-Version", '300')

$uri = $url + '/rest/alerts'

Invoke-RestMethod -Method GET -Headers $auth -Uri $url/rest/alerts/26669 -outfile failedILOEntry.txt

***********************************end script********************

I hope this helps anyone who is struggling with this. please let me know if you have any questions.

Cheers,

Paul

[NJK-Work]

Hi pcrooney

Thanks for the tip about using only one NOT in the filter.  So I am doing something like this:

status:critical status:warning NOT alertTypeID:Trap.cpqNic3ConnectivityLost alertTypeID:Trap.cpqNicAllLinksDown

Is this what you are saying?  Only one NOT and then a space between the type alertTypeID entries?

Thanks

NK

[NJK-Work]

This did not seem to work for me:

status:critical status:warning NOT alertTypeID:Trap.cpqNic3ConnectivityLost alertTypeID:Trap.cpqNicAllLinksDown

When I set it to that, I could not receive any emails alerts including that test alert generated from the iLO.  When I changed it to this:

status:critical status:warning NOT alertTypeID:Trap.cpqNic3ConnectivityLost NOT alertTypeID:Trap.cpqNicAllLinksDown

Then I could at least receive the test alert.  Going to leave it at this for a while and see how it works for actual alerts.

Thanks

NK

[sbr140]

Any updates to documentation I have been missing. We are running 4.2 the DOCS are still bad.

[BradV]

Not sure if this will help, but instead of trying to filter the alerts in the API call, I just extract all of the active alerts to a file, then filter then with standard shell commands.  This is on a Linux system.

# Get active alerts:
ACTIVE=$(curl --insecure \
              --header "X-API-Version: ${currentVersion}" \
              --header "auth: ${sessionID}" \
              --request GET "${OneView}/rest/alerts?start=0&count=-1&filter="\%22alertState+EQ+%27Active%27%22")
URIS=$(echo ${ACTIVE} | jq -r'.' | grep uri\"\: | grep -v count | awk '{ print $2 }' | cut -d, -f1 | cut -d\" -f2)
declare -A URI
i=0
for f in $(echo ${URIS}); do
   URI[${i}]=${f}
   ((i++))
done
j=${i}
i=0
DAT=$(date '+%d-%m-%Y')
while [ ${i} -lt ${j} ]; do
   if [[ ${URI[${i}] != *"AlertChangeLog"* ]]; then
      echo 'i = ${i}"
      curl --insecure \
           --header "X-API-Version: ${currentVersion}" \
           --header "auth: ${sessionID}" \
           --request GET ${OneView}${URI[${i}]} | jq -r '.' | tee -a alerts-${DAT}
   else
      echo "This URI is invalid in this context"
   fi
   ((i++))
done
cat alerts-${DAT} | jq -r '.description,.correctiveAction,.uri,.severity,.alertState,.resourceUri,.associatedResource.resourceName'

I can pipe that to grep or whatever I am looking for.

[SullivanMarkA]

This guidance does NOT seem to work for, "alertTypeID:server-hardware.opStatus.nandIssueDetected."
Need to filter these errors to ID crit issues not of this type in OneView, and if possible would like to filter from SMTP alerts.

OneView verion 5.20.01-0420365
iLO 4 version 2.75
Have already tried all steps in Document ID: c04996097

Any help would be appreciated.

[ChrisLynch]

@SullivanMarkA , I believe you are including an invalid item in the alertID value.  Remove server-hardware, as that is a category, not alertTypeID.  You can validate this by looking at the alert resource object, then expanding Details and looking at the alertTypeID field.  Can you provide a screenshot of the alert you are trying to filter, including the expanded Details section?