HP Proliant ML350 Gen10 Sensor 12-HD Max Consistently showing warmer than anything else

[WafflesMcDuff]

Hi all,

This summer we had a very hot week and one day it got warm enough that 1 of my SATA disks failed. Ever since, my fans all want to run at 69% all the time.

According to the temperature page, Sensor 12-HD Max is at 49C (Caution 60C).

I've managed to override the fans using the API so they only run at 39% but it's still annoyingly loud. If I go any lower, with the fans, I've had a couple days where that sensor registers 62 and shuts the system down.

Nothing else apart from the CPUs, BMC and PSUs are even *CLOSE* to that temperature. But their expected operation temperature is much higher to begin with. (see pic below)

I'm assuming at this point that something must have happened during the overheat event that broke this sensor and that it's registering an incorrect temperature. I've just ordered a laser thermometer to double check this theory.

Following the overheat event this summer, I replaced the failed disks, I rebuilt the RAID, and transformed it from RAID5 to RAID6.

I've updated all firmware, BIOS, drivers, iLO, anything I could think of. Nothing has helped. This 1 sensor continues to force the fans to run at 69% (or 39% when modified with the API setting).

Is there something I'm missing?  Short of replacing the whole systemboard (a $1200 prospect) is there any way of dealing with a faulty temperature sensor? Is there some way to disable the sensor? Or adjust its caution threshold?

[support_s]

System recommended content:

1. Gen10 Service Pack for ProLiant 2021.10.1 - Component Release Notes

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[WafflesMcDuff]

I'm assuming this is an automated response, since I said I had updated all the drivers and firmware already, and the posted link leads to an SPP that is more than a year out of date.

[ksram]

Hi,

Thank you for the Post.

Here HD Max usually related to the Array Controller.

Please confirm the replaced SATA Drives are HPEs (if any 3rd Part / un-supported Drives are installed, that may cause this issues as the Controller / iLO fails to collect the Temperature readings right)

You may try the following : 

1) Leave the Server on POST for 10 - 15 minutes and check for the Fan speed / Noise

Then Enter to SSA / Intelligent Provisioning and stay there for 10 minutes.

Restart and let the Server boot to OS 

>> By doing these, confirm where the Fan issue starts (as soon as the Server Powered On / when it loads the OS.

>> If its as soon as the Server is Powered On follow : Plan A

>> If the fan issue starts once the OS is booted, follow PLan B

Plan A : 

1) Perform Power Drain

2) Test using HPE Drives (if its already HPEs, then ignore this)

3) Try updating the iLO and BIOS to very latest (ignore if the SPP has latest iLO Firmware - check the Content report of the SPP)

4) If nothing helps, replace the Controller (if its Internal Controller - then System Board)

Plan B : 

1) Update the iLO and BIOS  to very latest (ignore if the SPP has latest iLO Firmware - check the Content report of the SPP)

2) Update the AMS - Agentless Management Service

3) Power Drain the Server

Thank you
RamKS

[WafflesMcDuff]

Please confirm the replaced SATA Drives are HPEs (if any 3rd Part / un-supported Drives are installed, that may cause this issues as the Controller / iLO fails to collect the Temperature readings right)

I had considered this possibility. And indeed, the disks are not HPE. But I can see in SSA that the temperature is reading just fine.

While trying to troubleshoot this, I did find one older SSD that was not reading a temperature in SSA so I replaced it. No luck.

If memory serves, I have noticed that the fans seem to ramp up when the OS finishes booting. 

As stated in my original post, I'm pretty sure I've already updated iLO, BIOS and all relevant Firmwares to the latest versions. If you spot something in the screenshot that is pertinent that I've missed, please let me know

What steps do you recommend for a power drain? Disconnect both PSUs and then just hold down the power button for 30 seconds to clear the capacitors? Or do I need more radical steps like removing RAID batteries? 

[Moderator note: Erased the confidential info.]

[ksram]

Hi @WafflesMcDuff 

Good day!

We are awaiting for the details further which were requested.

Alternatively you may log a Case following :  https://support.hpe.com/help/en/Content/productSupport/supportCaseManager.html 

Thank you
RamKS

[BFischer]

Hi,

Did you receive an answer or find a solution yet? I've got a dozen servers all with non-HPE drives I've built over the last 13 years. Not one of them has had this issue. I'm building one today that is wiggin out of the 12-HD Max sensor. Is there a way to disable this sensor? In case HPE is looking at this post: Replacing the drives with HPE drives is NOT an answer.

Thank you!

Brian

[WafflesMcDuff · Accepted solution]

I finally solved the root cause.

Originally I had an array of Seagate ST50000LM000 5400rm 5TB SATA disks in my server. This had never been a problem until this past august when 2 of the disks died in a short time span. I replaced them from a local supplier, and the RAID rebuilt. But the fans constantly spun at 69%. This was odd because I had 6 other drives just like in there... or so I thought!

Seagate has 2 revisions of this drive:

• ST5000LM000 with PN 2U8170 and
• ST5000LM000 with PN 2AN170

PN 2AN170 is unoficially compatible with HPE Systems

PN 2U8170 was making my HD Max sensor go crazy, reading 10-20 degress Celsius higher than actual temp.

I've replaced the 2 * 2U8170 drives that I bought this summer with 2 x 2AN170 drives and my fans are now back to 6% where they belong.

[WafflesMcDuff]

@BFischer - Please see my previous reply (yesterday) but also check this link: http://dascomputerconsultants.com/HPCompaqServerDrives.htm

It contains a list of (un)officially supported third-party drives.

Hopefully you can find a drive in this list that fits your needs without making your sensor go nuts.

[BFischer]

That's an awesome link thank you! Very informative. I did change up the drives I'm using a bit (neither were on his list). I've been using Samsung 860 Pro 2TB (MZ-76P2T0E) on probably the last 5 builds going back to 2019 with no issues. This time I bought Samsung 850 Pro 2TB (MZ-7KE2T0BW) because the prices of the 860's went up. So I'm going to try the firmware route on the 850's and see if that will fix it. Otherwise it sounds like I'm screwed and will need to pony up for the 860's. I'm outside the return window for the 850's so hopefully I'll find another use for them. Otherwise I'm out $4k (bought 8 of them at $500/piece). 3rd party drives were working so well for me that I didn't even know this issue was even a thing.