High CPU Temp on Proliant G7 Servers

[Oscar A. Perez]

 I sent you a PM

[Stephan G]

Did not reoccur.

Neither on the one with 1.58 nor on the ones with 1.28.

[BuildTheRobots]

Just thought I'd pipe up as we've had similar problems.

DL380 (G7) with an E5606 CPU.

iLO 1.5, BIOS: 02/12/2012.

iLO reported the CPU was around 82oc and then initiated a thermal shutdown. When we reset the iLO temperatures returned to normal.

We have a handful of these servers in production though this is the only one that's been having problems.

It was mentioned by one of our engineers that this server had the BIOS set to HP Power/temp control rather than OS Control -though I don't know if this makes any difference.

-Can anyone advice on how I can disable the power-off on thermal shutdown, preferably without rebooting the boxes acting as our ESXi hosts?

[Oscar A. Perez]

Hi Stephan,

I sent you a PM.  I uploaded a newer 1.58 to the FTP yesterday. Could you please test this newer one?

ftp://ilo4me:G!v3t2me@ftp.usa.hp.com

[Oscar A. Perez]

Hi BuildTheRobots,

Could you please send me a PM with the iLO event log and IML log of the iLO that initiated the thermal shutdown? 

Oscar

[BuildTheRobots]

Hi Oscar,

More than happy to provide logs, however I'm currently booted off the HP Insight Diagnostics CD via the iLO and the hardware is on a different continent to me.  -If you can tell me how to get you the logs (when I ask it to save it says there's no USB device conncted) then you're more than welcome to them :)

[Stephan G]

"I sent you a PM.  I uploaded a newer 1.58 to the FTP yesterday. Could you please test this newer one?"

I installed it 2 hours ago. Sorry forgot to respond.

[Oscar A. Perez]

Can you log in to iLO3 web UI and take a screenshot of the event log and iml log? I only need the events around the time iLO triggered the server shutdown.

[Stephan G]

Here you go

[BuildTheRobots]

The Event Log only has two entries from last year complaining about the power supplies not being redundant.

The Platform event log is attached.

[Oscar A. Perez]

Hi BuildTheRobots,

Your case looks similar to Stephan except that you mentioned that it goes away only after rebooting iLO. How often do you see this issue?  If it happens often, I would suggest trying the 1.58 beta on the worst offender or go back to version 1.28 until we figure out the root cause and provide a permanent solution.

Oscar

[BuildTheRobots]

Hi Oscar,

We have a handful of DL380 (G7)'s though we have only had this issue on one of them -upsettingly a box sat in production.

We reset the iLO when it first happened and the issue does not seem to have occured again.

As a precaution, we've demoted the server from it's active roll and have disabled auto-shutdown on thermal.

Am happy to change the iLO firmware, but as it's only had the problem once, it not having the problem on the new firmware doesn't really prove anything in my mind.

Any advice appriciated,

With thanks :)

[Stephan G]

Just an update. Did not happen again.

But sometimes (once in a month) we are getting these errors:

server: (SNMP) Remote Insight/ Integrated LightsOut Interface Error (9006):

After resetting the iLO the System Management Homepage can reach the iLO again. The iLO itself operates normally.

[Oscar A. Perez]

Hi Stephan,

This is caused by an old CPQSMIF.DLL on your system. Basically, you need to upgrade the iLO3 Channel Interface driver (CHIF). Keep in mind that this driver isn't the same as the iLO3 Management Driver (CORE).

In the same FTP site, I have uploaded the release candidate for both the CHIF and CORE drivers v3.9.0.0. They come with even more fixes to random 9006 issue plus Event ID 78/79. These drivers will be officially released with the next SPP.

[Oscar A. Perez]

Hi Stephan,

Any updates on the 1.58 beta?

[Stephan G]

Well yes or no ;)

Nothing happened. The problem did not reappear on this server.

Thanks for your help.

[Oscar A. Perez]

Ok, the issue was finally reproduced in our lab and we were able to find the root cause of these false CPU overheating reports in the IML log. 

I've uploaded to the FTP a new iLO3 v1.59 Beta that contains the fix. 

ftp://ilo4me:G!v3t2me@ftp.usa.hp.com/Beta/iLO3

[Stephan G]

That's great news. Thanks for the fix. I will test it in the next days.

[Torsten.]

So did this really affect all or at least a few different servers with ILO3?

When will the fix be officially released?

[Oscar A. Perez]

Any G7 with Intel processor could potentially exhibit the issue.

Our Quality Assurance team wants to test this fix thoroughly so, the fix is being officially added to the next iLO3 release planned for the Jan/Feb SPP.

BTW, I did add to the 1.59 beta the critical fix for the SSRT101250 (CVE-2013-4805) issue.

[anthony11]

Might enabling "enhanced cooling" or whatever it's called in BIOS help? Back when I was having C-state issues with my DL580G7's HP advised me to do this on them - and I've seen discussions of NIC-related problems on these systems (I unfortunately have two Qlogic cards in one of them) recommending the same to keep 10GE NICs cooler.

[Suresh_Mani]

Hi ,

FTP Site contains only the .bin file.

Could you please provide us with the complete package for windows and linux distribution?

Could you also upload the .exe file for Windows & the .scexe file for linux?

[Stephan G]

You can download the newest iLO drivers including the fix here:
http://h20000.www2.hp.com/bizsupport/TechSupport/DriverDownload.jsp?prodNameId=4154847&locale=en_us&taskId=135&prodSeriesId=4154735&prodTypeId=18964

[Oscar A. Perez]

Hi Mani,

If you are refering to the iLO3 version 1.59, this is a beta release. There is no Online Components for it.

[Suresh_Mani]

Hi Oscar,

With this Beta Firmware 1.59, ilo logs informational messages every minute in the ilo logs.

Severity             Class   Last Update                Initial Update            Count    Description

Informational  iLo3    10/10/2013 12:40   10/10/2013 12:40   1            Ilo Updated the host Date and Time.

Informational  iLo3    10/10/2013 12:39   10/10/2013 12:39   1            Ilo Updated the host Date and Time.

Informational  iLo3    10/10/2013 12:38   10/10/2013 12:38   1            Ilo Updated the host Date and Time.

It is filling up the ilo logs every minute. How to go about this ?