topic Re: DL585 G1 System Overheating Issue in Operating System - Linux

DL585 G1 System Overheating Issue

digriz_1 — Thu, 13 May 2010 04:52:04 GMT

We recently upgraded a bunch of DL585 G1 (running RH Linux 4.7) to PSP 8.30 (hp-health-8.3.0.43-30.x86_64)

Now several of them are reporting System overheating problems, i.e.

messages:May 10 17:03:29 xxxxxxxx hpasmd[12590]: WARNING: hpasmd: System Overheating (Zone 5, Location CPU, Temperature 111C)
messages:May 10 17:03:29 xxxxxxxx hpasmd[12590]: CRITICAL: hpasmd: Automatic Operating System Shutdown Initiated Due to Overheat Condition

Temperature seems to high to be real to me, assume its either a software or hardware bug

Anyone got any ideas ?
And will upgrading to PSP 8.40 solve this issue, theres nothing in the 8.40 release notes related to this

Re: DL585 G1 System Overheating Issue

Michal Kapalka (mikap) — Thu, 13 May 2010 06:02:23 GMT

hi,

if this is the same issue on all servers, it could be some bud on PSP layer, but if its only on one machine, i would recomend to make HW healt check.

maybe the upgrade to the neves version of PSP it could be help, sometimes not all bugs will be reported in the release notes.

mikap

Re: DL585 G1 System Overheating Issue

digriz_1 — Fri, 14 May 2010 00:25:15 GMT

We have 64 DL 585 G1, 46 have been upgraded to PSP 8.30 and of those 6 servers have had this overheating problem, in 2 different data centers and we never had an over heating problem before the upgrade

Re: DL585 G1 System Overheating Issue

SERKAN AKÇIN — Tue, 25 May 2010 06:01:19 GMT

Hi,

We have 4 dl585 servers.
2 of them rhel 4.7 and have no problem.
But
2 of them rhel 5.4 and giving temp errors.

We don't have temp problems before upgrade.

HP says, you must change the mainboard but, I think it is occur after the firmware update and psp updates.

Do you find a solution?

Re: DL585 G1 System Overheating Issue

Steven E. Protter — Tue, 25 May 2010 13:51:25 GMT

Shalom,

Use the web based PSP interface ( http://hostname:2301 ) to check actual temp.

These servers are pretty old, and you may have poor airflow or bad fans. All of these issues can cause overheating.

Clearly the software is more sensitive.

If this were a very severe problem, the system would have failed long ago. Still, run through the checklist, give them the eyeball check, make sure the fan openings are not covered with dust accumulation.

SEP

Re: DL585 G1 System Overheating Issue

Gerardo Arceri — Thu, 27 May 2010 00:23:06 GMT

Please upgrade firmware on the servers, we hit a similar bug couple of years ago and Firmware fixed it.
IF YOU ARE SURE THAT THE SYSTEM IS NOT OVERHEATING, use RBSU (Bios Setup) to disable Thermal Shutdown.

Re: DL585 G1 System Overheating Issue

kiheiman — Wed, 29 Dec 2010 16:48:11 GMT

We are seeing the CPU over-heating events on a number of DL585G1 Linux servers. The problem started about 6 months ago and seems to come in spurts. After the server powers down, it continues to generate the over-heating events in the IML - but the CPU heat sinks are not hot. The fix is to pull both AC power cords and boot the server back up. We have tried replacing motherboards, moving CPU modules around, etc. The problem appears to happen with both iLO driver versions 8.40 and 8.50 (not HP supported). It is also happening with Linux version 4.7 and 4.8. A number of the servers are on iLO firmware version 1.8x, but I do not see anything in the fix info that older versions would cause an over-heating event.

Re: DL585 G1 System Overheating Issue

Alzhy — Mon, 03 Jan 2011 18:56:36 GMT

Update your Firmware.
Use the hpsum to check and update the various firmware of you G5.

Re: DL585 G1 System Overheating Issue

kiheiman — Fri, 07 Jan 2011 14:41:40 GMT

Here is some more info on our situation.

Servers with the latest version of the motherboard firmware and iLO firmware are crashing. Servers with old versions of iLO firmware are not crashing. Servers running RHEL 4.7 are not crashing. The server crashes did not start until we upgraded some servers to RHEL 4.8. We are not running a PSP other than the 8.40 iLO driver. The smoking gun, based upon forum comments, seems to point to any server running a RHEL release newer than 4.7.

Re: DL585 G1 System Overheating Issue

SERKAN AKÇIN — Fri, 07 Jan 2011 14:56:26 GMT

We changed the heatsink of the 4 processors and the problem is gone.

HP says that in the production of G1 servers in the years 2004 and 2005, there is a heatsink metal alloy problem. We changed them and the craches are gone.

Re: DL585 G1 System Overheating Issue

digriz_1 — Sun, 09 Jan 2011 23:12:42 GMT

As I started this thread, thought I would give an update.

We have tried upgrading PSP and BIOS/ILO firmware combinations with no success. Servers still overheat, crash and refuse to boot until they have cooled down. Although physical inspection shows no heat issues

What we have done is "Disable Thermal Shutdown" in Bios Setup and this has stopped the crashes without issue

Re: DL585 G1 System Overheating Issue

kiheiman — Tue, 11 Jan 2011 14:50:31 GMT

On a couple of servers, we have disabled the high temperature shutdown in the RBSU and that has stopped the servers from crashing. On those servers, we now see a high temperature alert in the IML and the server continues to work. I guess the temporary fix is to disable the temperature shutdown option in all of our 585G1 servers, but it would be nice to know why this is happening. HP is going to analyze our sosreport to see if they can find anything.

Re: DL585 G1 System Overheating Issue

Alzhy — Tue, 11 Jan 2011 21:46:41 GMT

"What we have done is "Disable Thermal Shutdown" in Bios Setup and this has stopped the crashes without issue"

VERY dangerous amigo...It could fry your server.

Press HP on what the issue is and if it is a bad hp-healthh release or a firmware needing an update to match your new PSP bits.

Re: DL585 G1 System Overheating Issue

Dave. — Thu, 14 Apr 2011 13:56:57 GMT

Did you ever get this resolved? Did you upgrade to PSP 8.40 or later?

I found this (update to hp-health driver and wondered if it applies:

"Fixed a problem where if ambient temperature is very low, hpasmlited can incorrectly determine the system is over a temperature threshold and improperly shut the system down."
Link to RHEL 5 x86:-
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=398220&swItem=MTX-2f2a4208241845c485f7a847e6&prodNameId=3288126&swEnvOID=4006&swLang=8&taskId=135&mode=4&idx=2
Please let us know the outcome, Dave.

Re: DL585 G1 System Overheating Issue

Dave. — Fri, 15 Apr 2011 08:45:29 GMT

Please ignore my question on how you got it resolved. I missed the answer already posted.

But I would be interested if the hp-health driver update fix I posted has been used by anyone successfully or not.

Re: DL585 G1 System Overheating Issue

kiheiman2 — Wed, 08 Jun 2011 17:02:30 GMT

An update to our current high temp alerts. We had one server that was generating alerts about every day - the thermal shutdown had been disabled. We started the process for upgrading to the 8.40 version System Health Agent (current version was 8.50). By the time we did the upgrade, the alerts had been gone for one month. Very strange. The previous referenced HP Advisory about "the alerts being generated because the server is too cold" is most interesting. We had been looking to see if any server cabinet was getting too hot.