Receiving false HP AlertMail intermittently for multiple HP Blade Servers (not all) ?

[Server-Support]

People,

I'm having some strange issue with my HP AlertMail sending out email mentioning about:

HP AlertMail-152: (CRITICAL) Blade in bay 16: Failed and then followed by

HP AlertMail-159: (NORMAL) Blade in bay 16: OK after few seconds.

There is no error or alert reported when I logged in to the Onboard Admin page for all of the Blade servers and also the enclosure.

Why is that I've received that alert email intermittently at random times when nothing is changed or really broken ?

[Server-Support]

This is my OA information:

Onboard Administrator #1 information:

                Product Name  : BladeSystem c7000 DDR2 Onboard Administrator with KVM

Firmware Ver. : 3.71 Dec 07 2012

Onboard Administrator #2 information:

                Product Name  : BladeSystem c7000 DDR2 Onboard Administrator with KVM

Firmware Ver. : 3.71 Dec 07 2012

I cannot see any firmware updates which specifically mentions about this is a known issue with the firmware version above.

So updating the firmware may not fix this issue but could possibly causing more problem with unnecessary work later on.

[Torsten.]

This could also depend on the servers BIOS and/or ILO.

What server model and what versions?

[Server-Support]

Hi Torsten,

it is happening on mixed type of Blades BL 465c G7 and G8.

[Torsten.]

I would check the OA log for related messages and update all related firmware first. Since I don't know what versions you have installed I cannot tell more.

[Server-Support]

There is nothing logged in the OA log TOrsten, see the below screenshot:

and also the Firmware versions:

>SHOW FIRMWARE SUMMARY

Enclosure Firmware Summary

Onboard Administrator Firmware Information
Bay Model                                              Firmware Version
--- -------------------------------------------------- ----------------
1   BladeSystem c7000 DDR2 Onboard Administrator with KVM 3.71
2   BladeSystem c7000 DDR2 Onboard Administrator with KVM 3.71

Enclosure Component Firmware Information
Device  Name                                          Location Version    NewVersion
-----------------------------------------------------------------------------------
TRAY  | BladeSystem c7000 Onboard Administrator Tray  | -     | 1.7      | 1.7    
LCD   | BladeSystem c7000 Insight Display             | -     | 2.4.3    | 2.4.3  
FAN   | Active Cool 200 Fan                           | 1     | 2.9.4    | 2.9.4  
FAN   | Active Cool 200 Fan                           | 2     | 2.9.4    | 2.9.4  
FAN   | Active Cool 200 Fan                           | 3     | 2.9.4    | 2.9.4  
FAN   | Active Cool 200 Fan                           | 4     | 2.9.4    | 2.9.4  
FAN   | Active Cool 200 Fan                           | 5     | 2.9.4    | 2.9.4  
FAN   | Active Cool 200 Fan                           | 6     | 2.9.4    | 2.9.4  
FAN   | Active Cool 200 Fan                           | 7     | 2.9.4    | 2.9.4  
FAN   | Active Cool 200 Fan                           | 8     | 2.9.4    | 2.9.4  
FAN   | Active Cool 200 Fan                           | 9     | 2.9.4    | 2.9.4  
FAN   | Active Cool 200 Fan                           | 10    | 2.9.4    | 2.9.4  

>SHOW FIRMWARE SUMMARY CSV

Bay Number, Device Name, Discovered, Firmware Component, Current Version, Firmware ISO Version
1, ProLiant BL465c G7, No, System ROM, A19 12/08/2012, , iLO3, 1.55 Jan 24 2013, , Power Management Controller, 1.6, ,
2, ProLiant BL465c G7, No, System ROM, A19 01/29/2011, , iLO3, 1.80 Jul 11 2014, , Power Management Controller, 1.6, ,
3, ProLiant BL465c G7, No, System ROM, A19 05/08/2012, , iLO3, 1.70 Jan 09 2014, , Power Management Controller, 1.6, ,
4, ProLiant BL465c G7, No, System ROM, A19 02/02/2014, , iLO3, 1.80 Jul 11 2014, , Power Management Controller, 1.6, ,
5, ProLiant BL465c Gen8, No, System ROM, A26 02/06/2014, , iLO4, 2.00 Jul 30 2014, , Power Management Controller, 3.3, ,
6, ProLiant BL465c Gen8, No, System ROM, A26 02/06/2014, , iLO4, 2.00 Jul 30 2014, , Power Management Controller, 3.3, ,
7, ProLiant BL465c Gen8, No, System ROM, A26 02/06/2014, , iLO4, 2.00 Jul 30 2014, , Power Management Controller, 3.3, ,
8, ProLiant BL460c Gen8, No, System ROM, I31 02/10/2014, , iLO4, 1.40 Jan 14 2014, , Power Management Controller, 3.3, ,
9, ProLiant BL465c G7, No, System ROM, A19 12/08/2012, , iLO3, 1.61 Aug 01 2013, , Power Management Controller, 1.6, ,
10, ProLiant BL465c G7, No, System ROM, A19 02/02/2014, , iLO3, 1.80 Jul 11 2014, , Power Management Controller, 1.6, ,
11, ProLiant BL465c G7, No, System ROM, A19 02/02/2014, , iLO3, 1.80 Jul 11 2014, , Power Management Controller, 1.6, ,
12, ProLiant BL465c Gen8, No, System ROM, A26 02/06/2014, , iLO4, 2.00 Jul 30 2014, , Power Management Controller, 3.3, ,
13, ProLiant BL465c Gen8, No, System ROM, A26 02/06/2014, , iLO4, 2.00 Jul 30 2014, , Power Management Controller, 3.3, ,
14, ProLiant BL465c Gen8, No, System ROM, A26 02/06/2014, , iLO4, 2.00 Jul 30 2014, , Power Management Controller, 3.3, ,
15, ProLiant BL460c Gen8, No, System ROM, I31 02/10/2014, , iLO4, 1.40 Jan 14 2014, , Power Management Controller, 3.3, ,
16, ProLiant BL465c Gen8, No, System ROM, A26 09/03/2014, , iLO4, 2.03 Nov 07 2014, , Power Management Controller, 3.3, ,

[Torsten.]

I would consider to update all the BIOS and ILO and also the OA to the latest versions, some are really old meanwhile. Just to rule out there are known issues solved by the newer versions.

[Server-Support]

Thanks for the suggestion Torsten.

Somehow the change management advisory board does not approve the suggestion performing high risk firmwdare update on the  OnBoard admin which can breaks in other parts or causing outage with the Virtual Connect module.

[Torsten. · Accepted solution]

The question is when the risk is higher:

Stay with old (probably unsupported) firmware or update to a current supported version.

See also

http://h20195.www2.hp.com/V2/GetPDF.aspx/c03681570.pdf and included links.

[Server-Support]

Yes, you are right. updating to the latest OA enclosure firmware v 4.40 resolved this issue.

This was found in OA syslog:

Apr 27 10:46:42  OA: Management Processor on Blade 2 appears unresponsive.

Apr 27 10:46:42  OA: Management Processor on Blade 6 appears unresponsive.

Apr 27 10:46:42  OA: Management Processor on Blade 8 appears unresponsive.

Apr 27 10:46:42  OA: Management Processor on Blade 10 appears unresponsive.

Apr 27 10:46:42  OA: Management Processor on Blade 12 appears unresponsive.

Apr 27 10:46:42  OA: Management Processor on Blade 14 appears unresponsive.

Apr 27 10:46:42  OA: Management Process on Blade 14 appears responsive again.

Apr 27 10:46:42  OA: Management Process on Blade 8 appears responsive again.

Apr 27 10:46:42  OA: Management Process on Blade 6 appears responsive again.

Apr 27 10:46:42  OA: Management Process on Blade 2 appears responsive again.

Apr 27 10:46:42  OA: Management Process on Blade 10 appears responsive again.

Apr 27 10:46:42  OA: Management Process on Blade 12 appears responsive again

This is a known issue, notice the OA in this case is 3.71. 

http://h20565.www2.hp.com/hpsc/doc/public/display?sp4ts.oid=3188477&docId=emr_na-c04619734&docLocale=en_US

hope this helps.