MSA 2060 FC Alert: Update Server Connection /// Weekly Health mail missing

[DidiSeven]

Hello everybody,

currently we use (11) eleven MSA2060 and all of them talk through the same proxy to check the Update server for new firmware for controllers and disks. Suddenly (4) four systems started to make troubles with alerts like ...

The system was unable to connect to or parse information from the update server.

These alerts are thrown on an irregular basis but 100-200 alerts per day are not rare and even more at weekends. Often those MSA2060 systems show ...

Last Check: 2024-09-24 14:16:32 /// Last Status: Command completed successfully.

This only happens on 4 of 11 MSA2060 systems, which are all placed in different locations.

Furthermore, we have troubles with weekly Health SMTP mails. Most of them are not sent to the target mail-address, manual SMTP-Tests always work fine. We even have one MSA2060 that throws this regular weekly Health mail 2 times at the same time per week.

What could be the cause for these behaviors? Anyone else with these problems?

Regards,

DidiSeven

[JonPaul]

@DidiSeven 
The best way to get the right level of support is by logging a support ticket.
Part of your stated issue is not unfamiliar, support can help you with the Update-Server / Manifest download.

For the email issues, a couple things to look at:
CLI> show email-parameters
Make sure 'Email Notification' is "Enabled" -- You may have to set the 'notification-level' to something other than "None" to get this to enable

If your 'Server Port' is set to default, i.e. blank, set it to your SMTP server port.  default == 25

[DidiSeven]

Hello JonPaul,

that's the output of 'show email-parameters' ...

# show email-parameters
Email Parameters (SMTP)
-----------------------
Email Notification: Enabled
Email Address 1: example@email.org
Email Address 2:
Email Address 3:
Log Destination:
Security Protocol: None
Server Port:
Email Server: smtp-target.email.org
Email Domain: email.org
Email Sender: msa2060-test
Email Sender Password: Not Configured
Alert Notification: all
Event Notification: none
Persistent Alerts: Enabled
Include Logs: Disabled

Success: Command completed successfully. (2024-09-25 10:01:43)
#

Are you sure, that Notification Level can be changed in MSA2060 and if yes, how? I remember, this can be done on an MSA2050 storage system, but I have troubles to find a setting in the MSA2060.

Regards

DidiSeven

Btw, regarding the Update-Server and Manifest, I wonder why this works on 7 other systems with the same settings and this suddenly throws many alerts on these four MSA2060 storage-systems.

[DidiSeven]

Ok, I found this thread ...

Solved: Re: MSA 2060 notifications customization - Hewlett Packard Enterprise Community (hpe.com)

... but I had no luck.

Also, if you turn off 'notification-level' to none, how do you configure it back to all, like ...

set email parameters alert-notification-level all

but that's not sufficent (the system requires another parameter, but which one?).

Furthermore, I still receive no Health alerts ... I always set manual time to 23:57 (24hours) on last saturday.

[JonPaul]

@DidiSeven 
For more hands on troubleshooting please log a support case.
Please note that there are 2 types of notiifcations that can be used with the MSA 6th Gen.   Event Notifications or Alert notifications.
Event Notifications - VERY chatty, every event that happens on the system.  Multiple levels of severity (none, info, resolved, warn, error, crit)
Alert Notifications - Only actionable events, ALL or NONE
The events notification is the legacy feature.  It is sometime difficult to sift through the details.  For example Events set to WARN level will not notify you that an array host port is not connected, that's a INFO level event.  The host port could put you in a single pathed (degraded) state so you set the notification level to INFO and now you receive 10s of notifications per day of non-actionable events, ex scrub start/end.
But since that is actionable, i.e. you can (re)connect the port, it will be shown in ALERTS without filling your email with just INFO about the events on the array.

Email parameters  includes a command line option for both 'notification-level' and 'alert-notification-level'

You can use tab-completion to see the options for a command
ex. 
  type: 'set email' <tab>  
  This will complete the command and show:  set email-parameters
  Type:  <tab>
  This will show all the options for email-parameters

Most likely the missing parameter is  'security-protocol'.  Even if it is 'none' and has not changed it is required.

[DidiSeven]

Hello JonPaul,

I very much appreciate you are replying again. I read about those two type of notifications in the other thread I mentioned.

Who is responsible for the weekly Health alert? The Event notifications or the Alert notifications? I still have trouble to get Health Alerts from most MSA2060 systems. They once worked and suddenly vanished, which is very annoying.

There is a third party website from Stephen Wagner mentioning troubles with MSA2050 notifications and I remember having similar problems with the 5th genereation as well, but they were solved long ago ...

HPE MSA 2040 (and 2050/2052) - SMTP/E-Mail Notification issue (through SMTP proxy/relay servers) - The Tech Journal (stephenwagner.com)

Furthermore, I was very well aware about the TAB possibilities using commands in CLI on the MSA but it's still not working. I used the following command and got the following result ...

# set email-parameters alert-notification-level all security-protocol none
Error: The command is missing at least one parameter. (2024-09-22 06:27:53)
#

I did not find any good explanation how to use the commands, even in SMU Reference Guide.

So, how is that command used? Maybe some more hints, why the Health Alert is missing, whereas standard SMTP Test mails work?

Regards

DidiSeven

[JonPaul]

@DidiSeven 
Sorry this was not a quick fix.  The best way to get the right level of attention/support is to log a case with HPE Support.
I'm afraid that our back and forth is ultimately slowing down resolution for you.  Plus if there is a hardware component needing replacement the support team will be able to supply that.

[ENSAS]

Hello DidiSeven,

Just to let you know I have the same issue with update server notifications by email that I can't turn off and this is very annoying.

I have set notification level to warn but these emails are alert and error level so they continue to be delivered...

FYI for the command line, you should download the CLI reference guide.
What is not between brackets are mandatory parameters.

It says :
Syntax
set email-parameters
[alert-notification-level all|none]
domain <domain>
email-list <email-addresses>
[include-logs enabled|disabled|on|off]
[notification-level crit|error|warn|resolved|info|none]
[persistent-alerts enabled|disabled|on|off]
[port <port-number>]
security-protocol tls|ssl|none
[sender <sender>]
[sender-password <password>]

I have multiple problems with these MSA2060, I have two bays. Same problems on both. Such as replication schedule not working, IP parameters showing crazy settings in SMU or in CLI, very slow interface... Ticket to support ? Answer is : firmware is buggy, wait for a new firmware or downgrade.
Well.... that is a shame. I have to reboot my controllers (SC and MC) to get it sorted for a while. Not working to sort out this notification problem though...
Of course the IP and DNS settings are verified and correct. Bays were able to notify me when firmware was available in the past but it was with firmware IN210R004 and now I have IN210P001.

[SRPRC]

We get the "MSA 2060 SAS Alert: Update Server Connection" alert a lot. Were you able to fix this and how?

We had a support ticket and the 'resolution" was to wait for the next firmware update.

[ENSAS]

Hello,

No, there seems to be no way of fixing this.

I had a meeting with the HPE regional sales manager in charge of my area, because I have way too much problems with these products... He said he will get in touch with support level 2 & 3 as well as storage specialits ingeneers to investigate the problems and try to find answers as quickly as possible but probably the advised solution will be to downgrade the firmware and wait for a new one to come with corrections...

[SRPRC]

That's what I was afraid of.
Thanks for replying so quickly! If you do get a solution and think about it, please reply here again. I would appreciate that.

________________________________

DISCLAIMER: This e-mail message and any attachments are intended solely for the use of the individual or entity to which it is addressed and may contain information that is confidential or legally privileged. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately and permanently delete this message and any attachments.

[JonPaul]

@SRPRC @ENSAS 
Please direct message me your support case numbers and I will try to direct them the right way.
Trying to get support through a forum is hit or miss.  Support through a sales person... ?

[ENSAS]

Thanks for your answer but I don't expect anything from the support, the answer to my ticket was pretty clear : downgrade.
I wasn't seeking help through a salesperson, he has been made aware of my huge disappointment about the whole process (order, delivery, installation and the product) because I had problems at every single step and he decided to meet me as for him this is not normal and he couldn't do nothing about it.
Anyway... I'm done for now, the only thing I can do is either downgrade or wait and uprgade when a new firmware will be made available.

[DidiSeven]

@JonPaul 

Ok, vacation is over! I remember that I first thought it might be a firmware bug, but as some systems work orderly and some not, I smashed this opinion.

But nevertheless, it makes sense and downgrading would be very bad, if necessary.

Ok HPE, show us that you can do it better ...

[SRPRC]

Update for the thread:

I updated to the latest firmware (IN210P002) and it fixed this issue, however because of this issue I had changed all alerts to go directly to me only and not our alerting DL. So, now I can't change the To: email address. Even though the form is completed correctly,  I get:

Unable to set the email notifications.

The command is missing at least one parameter. (2024-12-11 08:24:58) 

Oh well. Just check your notification email addresses before updating the firmware.

[JonPaul]

@SRPRC 
Please include the non-optional parameter:  notification-level   in the command line.
It can be set to "none" .
A similar issue is present in the MSA Gen7 systems and is noted in the release notes:  https://support.hpe.com/hpesc/public/docDisplay?docId=dp00005411en_us
Search for 'email-parameters'

[DidiSeven]

Long time no see!

A couple of days ago I had the opportunity to upgrade all MSA2060 systems to the new firmware version IN210P002. The bug with the Update Server Connection is indeed solved, but unfortunately the Health Report bug is still present.

In the meantime I also learned how to use the 'set email-parameter' via command line as well and it works perfectly, e.g. in cases where you have no port set for SMTP, like port 25. If you try to do this by GUI, it is not accepted. Doing it via command line, it works (must be a bug as well).

We have 12x the MSA2060 storage systems. All are configured the same way, SMTP mails can be sent without problems, but I do wonder, why only 4 MSA2060 systems sent a weekly health report and 8 of the 12 systems do not send a weekly Health report. Makes no sense to me and it must be still a bug in the firmware!

Anyone else?

Cheers,

DidiSeven

[JonPaul]

@DidiSeven 
FYI - here is the notification about the SMU setting email parameters:  https://support.hpe.com/hpesc/public/docDisplay?docId=a00144737en_us&docLocale=en_US
If you are not already, you should look at getting added to the email list for MSA.
  - Sign into the HPE support center
  - Look for the "Sign up for support Alerts"

For the emails not being received, try   CLI>  restart mc both full
Verify that you can get emails with the test email
Beyond that a call into HPE support would be needed.