Cooling Unit Failure on HP RX6600

[Pierre Marie Kingne]

Hi All,

Could someone help anlysing theses logs for me. I've been having issue with the three rear FAN on RX6600. The logs are attached.

Thanks in advance.

Pierre

[Michal Kapalka (mikap)]

hi,

could you check also an error messages in the syslog ???

grep EMS /var/adm/syslog/syslog.log

if you find some EMS message put the resmon output.

mikap

[Pierre Marie Kingne]

Hello

Kindly find attached the output of /var/adm/syslog/syslog.log


Regards.

[cnb]

Feb 2 13:15:19 ECM-HO-BACKUPGLOBUS EMS [3164]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 207355921 -r /system/events/ia64_corehw/core_hw -n 207355906 -a
Feb 3 13:15:28 ECM-HO-BACKUPGLOBUS EMS [3164]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 207355921 -r /system/events/ia64_corehw/core_hw -n 207355907 -a

You'll need to post the /var/opt/resmon/log/event.log file or perform the above commands in order to see the issue(s).

You can also look in the MP logs.


Rgds,

[cnb]

Power Supply 0 Predictive Failure asserted
Power Supply 1 Normal

Fans State
-----------------------------------------------------------
System Fan 1 Normal
System Fan 2 Normal
System Fan 3 Normal
System Fan 4 Failure
System Fan 5 Failure
System Fan 6 Failure

PS0 shows power issues when the fans are failed. Might want to replace PS0.


Service Guide:
http://www.docs.hp.com/en/AB464-9003C-ed7/AB464-9003C-ed7.pdf

Rgds,

[cnb]

FWIW, your system iLO-2 Firmware appears to be old (2.17). Some critical issues were fixed with newer firmware:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=3257031&swItem=ux-77052-1&prodNameId=3257032&swEnvOID=4001&swLang=13&taskId=135&mode=5

Rgds,

[Pierre Marie Kingne]

Hello cnd,

The PS0 is working, if you check very well the logs you will find that. But the real concern is the cooling unit. I wonder how three cooling FANs can failed at the same time. Could someone discover in the logs what can cause the failure.


Thanks in advance.

[Michal Kapalka (mikap)]

Hi Pierre,

if i would heva some HW trouble with the server, and it would a production server, i worul open a HW call in HP, to solve this issue immediatelly.

mikap

[cnb]

"The PS0 is working, if you check very well the logs you will find that. But the real concern is the cooling unit. I wonder how three cooling FANs can failed at the same time. Could someone discover in the logs what can cause the failure. "

Pierre,

Sorry, but I disagree and this Power Supply is not "working" as you stated:

*Power Supply 0 Predictive Failure asserted*

If is was "working" and checking the "logs very well" the logs would never show this warning, ever.

The multiple fan failure could be several FRUs:

"Fans are controlled by a Hex Fan Controller (HFC) Field Programmable Gate Array (FPGA) chip located on the Common I/O Backplane (CIOB) FRU. The baseboard Management Controller (BMC) chip located on the Unified Core I/O board (UCIO) FRU monitors ambient air temperatures, chip temperatures, server configuration, and fan operation or failure from the HFC. The BMC then sends commands to the HFC to increase/decrease fan speed, etc. Air is drawn through the front of the chassis and pushed out the rear by the cooling fans. ****A failure of multiple fans can be caused by the HFC on the CIOB or the UCIO.****"

Call your Hardware Service Provider and get PS0 replaced. It is INTERMITTENTLY failing with a predictive failure. Replace the CIOB or the UCIO.

Your PDC is old and needs updating. Power recovery and FAN issues were resolved in later versions.

"Hardware Revision S0 Firmware Revision F.02.17 Oct 5 2007,16:53:04"

http://www13.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c02018376

Of course I know nothing and you're asking for help determining the cause of the system failures...

http://www13.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c01331437

Best Regards,

cnb
http://forums13.itrc.hp.com/service/forums/helptips.do?#33

[Pierre Marie Kingne]

Hello All,

I have just replaced the 3 FANs on the server, but the status is still the same. nothing has changed. Please advise.

Regards.

[Torsten.]

Could be a problem with the fan controller on the system board.

[cnb · Accepted solution]

Not surprised and you didn't even take the time to read the Fan Logic Support Document provided by HP.

As stated by HP:

****A failure of multiple fans can be caused by the HFC on the CIOB or the UCIO.****"


http://www13.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c01331437

Replace PS0
Replace CIOB and UCIO

But maybe you will still go on ignoring everyone who is trying to help you.

Rgds,

[Pierre Marie Kingne]

Hello CNB,

I'm sorry but I didn't ignore something. I told you that the PS0 was showing failed because the power cable was removed from the power source, that's why you saw that in the log PS0 failed.Hope you understood me. I'll try to order the new parts.

Thanks once more

Regards.

[S.N.S]

Bonjour Pierre,

>>>But maybe you will still go on ignoring everyone who is trying to help you.

Simply you havent assigned any point to those who spend their time to type to your question , to assign you with the issue -- imagine 3 Fans Failing!

Now for this Fan behaviour - I have seen it to be erratic -
Best to Log a case with HP Support, and let them deal with it...They should be doing something to clear such problems with these rx6600s...

When I had the same but intermitent problem, I had made it run with the lid open - not recommened - but I hate to see the Failure message...

Best let HP deal with it, if under warranty!!!

HTH
SNS

[Zackt]

Hi All,

I have  the problem with RX 6600 server like Mr. Pierre..

Could someone help anlysing theses logs for me. I've been having issue with the three rear FAN on RX6600.

The logs are :

 System Firmware:  2.03 [4648]   

 BMC Version:      5.14           

 MP Version:       F.01.58    

  Installed Memory: 4096 MB    

For System Processor Status see the SS command.

System Power state: On

Temperature       : Normal

Power supplies                State

-----------------------------------------------------------

Power Supply 0                Normal

Power Supply 1                Normal

Fans                          State

-----------------------------------------------------------

System Fan 1                  Normal

System Fan 2                  Normal

System Fan 3                  Normal

System Fan 4                  Failure

System Fan 5                  Failure

System Fan 6                  Performance Lag

MP FW     : F.01.58

 BMC FW    : 05.14

 EFI FW    : ROM A 05.66, ROM B 05.66

 System FW : ROM A 02.03, ROM B 02.03, Boot ROM A

 PDH FW    : 50.04

 DHPC FW   : 01.23

 UCIO FW   : 03.0a

 PRS FW    : 00.08 UpSeqRev: 0c, DownSeqRev: 08

 HFC FW    : 00.03 SetRev: 00

Log Entry 83315: 03 Jun 2013 04:56:13

Alert Level 2: Informational

Keyword: COOLING_UNIT_OK

Cooling unit OK

Logged by: Baseboard Management Controller;

Sensor: Cooling Device - System Fan 6

Data1: transition to OK

0x2051AC21ED025AA5 FFFF0007F50A0300

Log Entry 83314: 03 Jun 2013 04:56:13

Alert Level 2: Informational

Keyword: COOLING_UNIT_OK

Cooling unit OK

Logged by: Baseboard Management Controller;

Sensor: Cooling Device - System Fan 5

Data1: transition to OK

0x2051AC21ED025AA5 FFFF0007F40A0300

Log Entry 15: 03 Jun 2013 05:05:01

Alert Level 2: Informational

Keyword: COOLING_UNIT_OK

Cooling unit OK

Logged by: Baseboard Management Controller;

Sensor: Cooling Device - System Fan 4

Data1: transition to OK

0x2051AC23FD020100 FFFF0007F30A0300

Log Entry 14: 03 Jun 2013 05:04:48

Alert Level 3: Warning

Keyword: COOLING_UNIT_WARNING

Cooling unit warning

Logged by: Baseboard Management Controller;

Sensor: Cooling Device - System Fan 6

Data1: transition to Non-Critical from OK

0x2051AC23F00200F0 FFFF0107F50A0300

Hope any one can give me the idea..

Thanks in advance.

sysnetkt@yahoo.com

Zackt

[Holger G]

Hi Zackt,

if you have issues on fans 4-6 (or 1-3) at the same time in an HP Integrity rx6600 server, this is most likely related to an issue with the fan controller.

You could of course swap e.g. fans 1 and 4 offline to be sure the issue really remains on the fan slot, and you do not have the very rare case of three failing fans.

There is even a minimal chance to fix a fan controller issue by updating / reflashing the fan controller firmware (HFC FW). In your case the HFC FW is 00.03; latest is 00.04. - As even your complete system has pretty old FW, you may update the complete system.

If the FW update does not fix the issue, the Common IO Backplane, which contains the HFC, needs to be replaced.

Kind Regards

Holger

I am an HP employee.
Was this post useful? - You may click the KUDOS! Star.

[Zackt]

Hi ..Holger.

Iam very appreciated with your response and tqvm...

For my first step, i will make FW update. Where can i download and step to update that FW ?

Is it  may effect my current install Operating System and  Database System ? Currently iam using Redhat and its running as mysql database server.

Thanks..

[Holger G]

Latest FW can be downloaded at HP Support Center:
http://www.hp.com/go/hpsc
 -> search for rx6600 and select your product from the list
  -> choose “Drivers, Software & Firmware”

   -> choose your Operating System

    -> go to "Firmware - System"

Use "rx3600 and rx6600 FPGA and PSoC Firmware" to update HFC FW etc.
Use "rx3600 and rx6600 System, BMC, and iLO2 MP Firmware" to update SFW, iLO and BMC.

Read and follow the INSTALLATION INSTRUCTIONS in the RELEASE NOTES on that download page.