Fan Subsystem Status Changed: From: ok To: degraded

[Lincker]

HP BladeSystem Enclosure reports fan failed. 

Hardware: c3000 + Active Cool 100 Fans

Problem take place on a different fan bays, sometimes two fans failed together. 

Typical failed messages in Syslog OA:

 Jan 7 03:23:17 OA: Fan Subsystem Status Changed: From: ok To: degraded
 Jan 7 03:23:17 OA: Enclosure Status changed from OK to Degraded. (Cooling Subsystem)
 Jan 7 03:23:17 OA: Fan Status Changed: Bay #4 From: ok To: failed

After re-insert fan, Fan Subsystem is ok:

 Jan 9 09:14:21 OA: Fan Removed: Bay #4
 Jan 9 09:14:21 OA: Required fan is missing from fan bay 4
 Jan 9 09:14:28 OA: Fan Inserted: Bay #4
 Jan 9 09:14:28 OA: Fan Subsystem Status Changed: From: degraded To: ok
 Jan 9 09:14:28 OA: Enclosure Status changed from Degraded to OK.

After reset OA, re-flash OA or change OA, Subsystem is ok. But errors will return after some days.

I know about it because, we replaced 6 fans, 2 module OA and chassis midplane.

I found interesting page in knowledge base mmr_kc-0100236, which says, that our problem can fixed by firmware upgrade.

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/mostViewedDisplay?javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_efb5c0793523e51970c8fa22b053ce01=wsrp-navigationalState%3DdocId%253Dmmr_kc-0100236-16%257CdocLocale%253Den_US&javax.portlet.tpst=efb5c0793523e51970c8fa22b053ce01&sp4ts.oid=3540808&ac.admitted=1401...

We upgraded OA firmware two times, but this can not fix the problem, it contnues more than year.

 Jan 17 18:43:17 OA_Flash: Flashing firmware image from 3.71 to 4.01 
 Jan 17 18:44:54 OA_Flash: Firmware image flashed from 3.71 to 4.01

 May 8 15:35:54 OA_Flash: Flashing firmware image from 4.01 to 4.21 
 May 8 15:37:31 OA_Flash: Firmware image flashed from 4.01 to 4.21