Integrity Servers
cancel
Showing results for 
Search instead for 
Did you mean: 

Cooling Unit Failure on HP RX6600

 
SOLVED
Go to solution

Cooling Unit Failure on HP RX6600

Hi All,

Could someone help anlysing theses logs for me. I've been having issue with the three rear FAN on RX6600. The logs are attached.

Thanks in advance.

Pierre
17 REPLIES 17
Michal Kapalka (mikap)
Honored Contributor

Re: Cooling Unit Failure on HP RX6600

hi,

could you check also an error messages in the syslog ???

grep EMS /var/adm/syslog/syslog.log

if you find some EMS message put the resmon output.

mikap

Re: Cooling Unit Failure on HP RX6600

Hello

Kindly find attached the output of /var/adm/syslog/syslog.log


Regards.
cnb
Honored Contributor

Re: Cooling Unit Failure on HP RX6600

Feb 2 13:15:19 ECM-HO-BACKUPGLOBUS EMS [3164]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 207355921 -r /system/events/ia64_corehw/core_hw -n 207355906 -a
Feb 3 13:15:28 ECM-HO-BACKUPGLOBUS EMS [3164]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 207355921 -r /system/events/ia64_corehw/core_hw -n 207355907 -a

You'll need to post the /var/opt/resmon/log/event.log file or perform the above commands in order to see the issue(s).

You can also look in the MP logs.


Rgds,

cnb
Honored Contributor

Re: Cooling Unit Failure on HP RX6600

Power Supply 0 Predictive Failure asserted
Power Supply 1 Normal

Fans State
-----------------------------------------------------------
System Fan 1 Normal
System Fan 2 Normal
System Fan 3 Normal
System Fan 4 Failure
System Fan 5 Failure
System Fan 6 Failure

PS0 shows power issues when the fans are failed. Might want to replace PS0.


Service Guide:
http://www.docs.hp.com/en/AB464-9003C-ed7/AB464-9003C-ed7.pdf

Rgds,

cnb
Honored Contributor

Re: Cooling Unit Failure on HP RX6600

Re: Cooling Unit Failure on HP RX6600

Hello cnd,

The PS0 is working, if you check very well the logs you will find that. But the real concern is the cooling unit. I wonder how three cooling FANs can failed at the same time. Could someone discover in the logs what can cause the failure.


Thanks in advance.
Michal Kapalka (mikap)
Honored Contributor

Re: Cooling Unit Failure on HP RX6600

Hi Pierre,

if i would heva some HW trouble with the server, and it would a production server, i worul open a HW call in HP, to solve this issue immediatelly.

mikap
cnb
Honored Contributor

Re: Cooling Unit Failure on HP RX6600

"The PS0 is working, if you check very well the logs you will find that. But the real concern is the cooling unit. I wonder how three cooling FANs can failed at the same time. Could someone discover in the logs what can cause the failure. "

Pierre,

Sorry, but I disagree and this Power Supply is not "working" as you stated:

*Power Supply 0 Predictive Failure asserted*

If is was "working" and checking the "logs very well" the logs would never show this warning, ever.

The multiple fan failure could be several FRUs:

"Fans are controlled by a Hex Fan Controller (HFC) Field Programmable Gate Array (FPGA) chip located on the Common I/O Backplane (CIOB) FRU. The baseboard Management Controller (BMC) chip located on the Unified Core I/O board (UCIO) FRU monitors ambient air temperatures, chip temperatures, server configuration, and fan operation or failure from the HFC. The BMC then sends commands to the HFC to increase/decrease fan speed, etc. Air is drawn through the front of the chassis and pushed out the rear by the cooling fans. ****A failure of multiple fans can be caused by the HFC on the CIOB or the UCIO.****"

Call your Hardware Service Provider and get PS0 replaced. It is INTERMITTENTLY failing with a predictive failure. Replace the CIOB or the UCIO.

Your PDC is old and needs updating. Power recovery and FAN issues were resolved in later versions.

"Hardware Revision S0 Firmware Revision F.02.17 Oct 5 2007,16:53:04"

http://www13.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c02018376

Of course I know nothing and you're asking for help determining the cause of the system failures...

http://www13.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c01331437

Best Regards,

cnb
http://forums13.itrc.hp.com/service/forums/helptips.do?#33

Re: Cooling Unit Failure on HP RX6600

Hello All,

I have just replaced the 3 FANs on the server, but the status is still the same. nothing has changed. Please advise.

Regards.
Torsten.
Acclaimed Contributor

Re: Cooling Unit Failure on HP RX6600

Could be a problem with the fan controller on the system board.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
cnb
Honored Contributor
Solution

Re: Cooling Unit Failure on HP RX6600

Not surprised and you didn't even take the time to read the Fan Logic Support Document provided by HP.

As stated by HP:

****A failure of multiple fans can be caused by the HFC on the CIOB or the UCIO.****"


http://www13.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c01331437

Replace PS0
Replace CIOB and UCIO

But maybe you will still go on ignoring everyone who is trying to help you.

Rgds,


Re: Cooling Unit Failure on HP RX6600

Hello CNB,

I'm sorry but I didn't ignore something. I told you that the PS0 was showing failed because the power cable was removed from the power source, that's why you saw that in the log PS0 failed.Hope you understood me. I'll try to order the new parts.

Thanks once more

Regards.
S.N.S
Valued Contributor

Re: Cooling Unit Failure on HP RX6600

Bonjour Pierre,

>>>But maybe you will still go on ignoring everyone who is trying to help you.

Simply you havent assigned any point to those who spend their time to type to your question , to assign you with the issue -- imagine 3 Fans Failing!

Now for this Fan behaviour - I have seen it to be erratic -
Best to Log a case with HP Support, and let them deal with it...They should be doing something to clear such problems with these rx6600s...

When I had the same but intermitent problem, I had made it run with the lid open - not recommened - but I hate to see the Failure message...

Best let HP deal with it, if under warranty!!!

HTH
SNS

"Genius is 1% inspiration, 99% Perspiration" - Edison
Zackt
Occasional Visitor

Re: Cooling Unit Failure on HP RX6600

 

Hi All,

 

I have  the problem with RX 6600 server like Mr. Pierre..

Could someone help anlysing theses logs for me. I've been having issue with the three rear FAN on RX6600.

The logs are :

 

 System Firmware:  2.03 [4648]   

 BMC Version:      5.14           

 MP Version:       F.01.58    

  Installed Memory: 4096 MB    

                               

 

 

For System Processor Status see the SS command.

 

System Power state: On

Temperature       : Normal

 

Power supplies                State

-----------------------------------------------------------

Power Supply 0                Normal

Power Supply 1                Normal

 

Fans                          State

-----------------------------------------------------------

System Fan 1                  Normal

System Fan 2                  Normal

System Fan 3                  Normal

System Fan 4                  Failure

System Fan 5                  Failure

System Fan 6                  Performance Lag

 

MP FW     : F.01.58

 BMC FW    : 05.14

 EFI FW    : ROM A 05.66, ROM B 05.66

 System FW : ROM A 02.03, ROM B 02.03, Boot ROM A

 PDH FW    : 50.04

 DHPC FW   : 01.23

 UCIO FW   : 03.0a

 PRS FW    : 00.08 UpSeqRev: 0c, DownSeqRev: 08

 HFC FW    : 00.03 SetRev: 00

 

Log Entry 83315: 03 Jun 2013 04:56:13

Alert Level 2: Informational

Keyword: COOLING_UNIT_OK

Cooling unit OK

Logged by: Baseboard Management Controller;

Sensor: Cooling Device - System Fan 6

Data1: transition to OK

0x2051AC21ED025AA5 FFFF0007F50A0300

 

 

Log Entry 83314: 03 Jun 2013 04:56:13

Alert Level 2: Informational

Keyword: COOLING_UNIT_OK

Cooling unit OK

Logged by: Baseboard Management Controller;

Sensor: Cooling Device - System Fan 5

Data1: transition to OK

0x2051AC21ED025AA5 FFFF0007F40A0300

 

  

Log Entry 15: 03 Jun 2013 05:05:01

Alert Level 2: Informational

Keyword: COOLING_UNIT_OK

Cooling unit OK

Logged by: Baseboard Management Controller;

Sensor: Cooling Device - System Fan 4

Data1: transition to OK

0x2051AC23FD020100 FFFF0007F30A0300

 

 

Log Entry 14: 03 Jun 2013 05:04:48

Alert Level 3: Warning

Keyword: COOLING_UNIT_WARNING

Cooling unit warning

Logged by: Baseboard Management Controller;

Sensor: Cooling Device - System Fan 6

Data1: transition to Non-Critical from OK

0x2051AC23F00200F0 FFFF0107F50A0300

 

 

Hope any one can give me the idea..

 


Thanks in advance.

 

sysnetkt@yahoo.com

Zackt

  

Holger G
HPE Pro

Re: Cooling Unit Failure on HP RX6600

Hi Zackt,

 

if you have issues on fans 4-6 (or 1-3) at the same time in an HP Integrity rx6600 server, this is most likely related to an issue with the fan controller.

You could of course swap e.g. fans 1 and 4 offline to be sure the issue really remains on the fan slot, and you do not have the very rare case of three failing fans.

There is even a minimal chance to fix a fan controller issue by updating / reflashing the fan controller firmware (HFC FW). In your case the HFC FW is 00.03; latest is 00.04. - As even your complete system has pretty old FW, you may update the complete system.

If the FW update does not fix the issue, the Common IO Backplane, which contains the HFC, needs to be replaced.

 

 

Kind Regards

Holger

I am an HP employee.
Was this post useful? - You may click the KUDOS! Star.

- I work for Hewlett Packard Enterprise -
Zackt
Occasional Visitor

Re: Cooling Unit Failure on HP RX6600

Hi ..Holger.

 

Iam very appreciated with your response and tqvm...

 

For my first step, i will make FW update. Where can i download and step to update that FW ?

 

Is it  may effect my current install Operating System and  Database System ? Currently iam using Redhat and its running as mysql database server.

 

Thanks..

Holger G
HPE Pro

Re: Cooling Unit Failure on HP RX6600

Latest FW can be downloaded at HP Support Center:
http://www.hp.com/go/hpsc
 -> search for rx6600 and select your product from the list
  -> choose “Drivers, Software & Firmware”

   -> choose your Operating System

    -> go to "Firmware - System"

 

Use "rx3600 and rx6600 FPGA and PSoC Firmware" to update HFC FW etc.
Use "rx3600 and rx6600 System, BMC, and iLO2 MP Firmware" to update SFW, iLO and BMC.

 

Read and follow the INSTALLATION INSTRUCTIONS in the RELEASE NOTES on that download page.

- I work for Hewlett Packard Enterprise -