HPE Gen10 Plus high BMC temperature

[paradoxhub]

I have my fan on 100% all the time to keep CPU cool while it's transcoding a lot of media 24/7, the CPU never goes above 55c according to TrueNAS.

But that BCM temp sometimes hits 80, usually hovers around 75. Looking at pics of the board on google images there's the iLO chip there, so I'm gonna put a little copper heatsink on it.

Moderator edit: Confidential info erased. 

[Raithmir]

I just bought a Gen10 Plus too. My temperatures seem the same as yours, and it seems like it's normal.
https://www.servethehome.com/hpe-proliant-microserver-gen10-plus-review-this-is-super/3/

I will be interested to hear if you adding a heatsink makes a difference. I've seen someone has gone to even more extreme measures!
https://imgur.com/a/7npqgdc

[paradoxhub]

Fan on the side is an interesting idea.

Where did you find that image?

I added little adhesive aluminium heatsinks allover the board on any exposed chips or anything that looked like a power regular or felt warm to the touch after shutting down.

I also glued a pure copper block to my IBM ServeRaid card.

[paradoxhub]

I did the fan mod:

https://imgur.com/a/8jQuXUn

[Floyd71]

Maybe it works, but for one "mature" and not so cheap product the solution should be done by manufacturer and properly implemented on all devices with claimed problem.

I am wondered if simple replacement of motherboard will improve the problem.

Nowdays, I observed new problem apart of the BMC temperature, this is LOM sensor readings - also very high values, close to limit.

Comparing to my older ProLiant Microserver Gen8, very unsatisfied with newer server.

Get Outlook for Android<>

[paradoxhub]

I just found out today that the gen10 plus motherboard doesn't support quicksync on CPUs. and my server is a 70tb plex/media server..  I wish I had kept my gen8. what's the next best option in the same price range? thanks

[--RW--]

I am also pretty disappointed in the tempatures of the gen 10 plus v2 microserver. I have two gen8 boxes, one with a dell 12/gb RAID and the other with w P420i; neither of them get anywhere close to 83C at idle at 25C ambient server room temp. This is down from nearly 90C since I placed some small heatsinks on the chip. I realize the Gen10 is a smaller form factor, but someone at HP should have tested the server with a RAID adapter and full bays at some point, right? I tried to place a Noctua 5V USB fan near the BMC, but it was impossible to place properly since there is little to no clearance when the PCI slots are filled.

Are these machines supposed to run so hot?

Product ID is: P54644-001, with 64GB RAM, two 8TB SAS 12/Gb HP drives, two Intel DC S4600 SATA SSDs, iLO enablement, and HPE Smart Array E208i-p SR Gen10. All firmware is at the latest levels.

Gen8 w/ p420i

With P420iWith Dell 12Gb RAIDGen10 Plus v2Gen10 Plus v2 Firmware

[Floyd71]

This is sad to see that tnis HPE Microserver Gen10 plus (also as V2) has the same issue with the temperature.

I am quite positive that it is not due to case size, but due to downestimate temperature development in some components.

They most probably did not made proper testing (as this is bottom line of the Servers).

In my case, if I increase significantly the fan speed, the temperature drop down for few degrees of C, but nothing to serious and server becomes to loud.

Anyhow I can conclude that in long terms it means following:

- lower lifetime of the server due to (I could call it) overheating of the important components

- more heat development in oder components, especially on the HDD which are closer to the motherboard

It should be good if HPE could show or help us to identify which components are to hot (or with the temperatures which are at least above 65-70% of the allowable temperature) in order to seal passive radiators on them and to cool them down. 

[paradoxhub]

I cringe every time I look at the temperatures tab and see those huge mountains at the back where my IBM1015 card is.. even with the 80mm noctua fan on the side my temps in that area (BMC) only went down 5-7c.

The weird thing is even if you run the fans on like 75% it *just barely* seems to cool it sufficiently.. but as I'm sure you know it's very loud even past 30%. 

These are my temps right now with the fan mod and the main fan at 45% all day:

(The CPU *ALWAYS* shows 40c and never changes, I think the sensor broke / burnt out? Does the CPU temp change for you guys? How do I know if it's overheating and needs new thermal paste etc?)

These are the drive temps:

This is pretty BAD with the main fan at 45% all day. It sounds like a jet engine - why are the HDDs nearing 40c?

[--RW--]

These are your temps after the fan mod? You cut your case and put in a fan, correct?

Does the fan push air in or exhust?

[Floyd71]

No. Please read entire conversqtion. I consider temlerature in my server to high and only what i did was firmware / bios update and fan speed increasement.

So - everything is still under warranty conditions. I am looking forward if HPE officially has any resolution of this problem.


Get Outlook for Android<>

[--RW--]

My question was for @paradoxhub 

[marciocosta]

Hi! I have the same issue here with 3 x DL360 Gen10.  Before post, I've upgraded all firmware with the last SPP package (system ROM U32 v2.76 (02/09/2023) / ILO5 2.81 Mar 07 2023).  I really think the BMC have a high temperature.  Not contacted HPE yet (all servers still in warranty).  I setup fan minimal speed at 60%.  Please check item #23:

01-Inlet Ambient Intake 15 0 OK 21C Caution: 42C; Critical: 47C
02-CPU 1 CPU 11 5 OK 40C Caution: 70C; Critical: N/A
03-CPU 2 CPU 4 5 OK 40C Caution: 70C; Critical: N/A
04-P1 DIMM 1-6 Memory 8 4 OK 33C Caution: 90C; Critical: N/A
06-P1 DIMM 7-12 Memory 13 4 OK 33C Caution: 90C; Critical: N/A
08-P2 DIMM 1-6 Memory 1 4 OK 33C Caution: 90C; Critical: N/A
10-P2 DIMM 7-12 Memory 6 4 OK 33C Caution: 90C; Critical: N/A
12-HD Max SystemBoard 11 0 OK 35C Caution: 60C; Critical: N/A
14-Stor Batt 1 SystemBoard 5 0 OK 22C Caution: 60C; Critical: N/A
15-Front Ambient Intake 9 1 OK 23C Caution: 60C; Critical: N/A
16-VR P1 SystemBoard 11 3 OK 35C Caution: 115C; Critical: 120C
17-VR P2 SystemBoard 4 3 OK 36C Caution: 115C; Critical: 120C
18-VR P1 Mem 1 SystemBoard 8 2 OK 28C Caution: 115C; Critical: 120C
19-VR P1 Mem 2 SystemBoard 13 2 OK 28C Caution: 115C; Critical: 120C
20-VR P2 Mem 1 SystemBoard 1 2 OK 28C Caution: 115C; Critical: 120C
21-VR P2 Mem 2 SystemBoard 6 2 OK 30C Caution: 115C; Critical: 120C
22-Chipset SystemBoard 13 9 OK 37C Caution: 100C; Critical: N/A

23-BMC SystemBoard 9 12 OK 59C Caution: 110C; Critical: 115C

24-BMC Zone SystemBoard 9 13 OK 35C Caution: 90C; Critical: 95C
25-HD Controller SystemBoard 8 9 OK 28C Caution: 78C; Critical: 83C
26-HD Cntlr Zone SystemBoard 8 8 OK 28C Caution: 85C; Critical: 90C
27-LOM SystemBoard 7 13 OK 50C Caution: 110C; Critical: 115C
28-LOM Card SystemBoard 14 14 OK 46C Caution: 110C; Critical: 120C
29-I/O Zone SystemBoard 14 11 OK 30C Caution: 90C; Critical: 95C
31-PCI 1 Zone SystemBoard 11 9 OK 32C Caution: 90C; Critical: 95C
33-PCI 2 Zone SystemBoard 11 9 OK 32C Caution: 90C; Critical: 95C
35-PCI 3 Zone SystemBoard 5 13 OK 28C Caution: 90C; Critical: 95C
38-Battery Zone SystemBoard 7 10 OK 30C Caution: 75C; Critical: 80C
39-P/S 1 Inlet PowerSupply 1 10 OK 30C Caution: N/A; Critical: N/A
40-P/S 2 Inlet PowerSupply 4 10 OK 31C Caution: N/A; Critical: N/A
41-P/S 1 PowerSupply 1 13 OK 40C Caution: N/A; Critical: N/A
42-P/S 2 PowerSupply 3 13 OK 40C Caution: N/A; Critical: N/A
43-E-Fuse PowerSupply 4 9 OK 26C Caution: 100C; Critical: N/A
44-P/S 2 Zone PowerSupply 4 7 OK 33C Caution: 75C; Critical: 80C
49-CPU 1 PkgTmp CPU 11 5 OK 43C Caution: N/A; Critical: N/A
50-CPU 2 PkgTmp CPU 4 5 OK 47C Caution: N/A; Critical: N/A

Best Regards.

[gernoox]

DL380 Gen10 24SFF
iLO 5 2.97
Innovation Engine (IE) Firmware 0.2.3.0
System ROM U30 v2.90 
23-BMC 76°C & 27-LOM 68°C for years without problems
Because of GPU and second Raid controller added in Pcie slots, fans noise is too loud.
Solution is to cheat with ILO and force 50% fans adjustment (don't try this at home!) :

curl --request PATCH --url 'https://ILO-IP/redfish/v1/Chassis/1/Thermal/' --user 'USER:PASSWORD' --header 'content-type: application/json' --insecure --data '{"Oem": {"Hpe": {"FanPercentAdjust": 50}}}'

[gabrielkozsar]

Hi , I am new here....

we have the same issue with the FAN noise. The two fans blowing the CPU heatsink sit at 9% with no load/idle but the center FAN is running at 38-50% which is a turbo jet and very annoying even though the server is idling. I have set the performance settings to the lowest and the changed the cooling profiles too. Nothing seems to help to bring the FAN down. I also see >70C on the BMC inside iLo. Everyting on the server is updated to the latest - as of 16.APR.2024 .  But where exactly is the BMC. I tried to figure out the hotspot with a thermal camera. 

The iLo chip is constatnly on ,obviously ..... I even spotted periodical writes into a NAND chip. 
The iLo chip is the hottest on IC on right side (looking from the front) which is cooled by the two CPU fans on the right.
The INTEL chip is the warmest (40C) in center of the board right under the raiser and the second hotspot is a winding inside the choke.
Obviously temps change under conditions ,the photos are with open "hood" and IDLE.  

The server has two Kinston server SSDs (not HPE drives) in RAID1.

The fact that it is not possible to post pictures here is pretty lame.  I will upload them to MEGA and post a link to it.

https://mega.nz/folder/G8cDAKzb#GiDvjhCduA_hw-MDIW6p7w

EDIT: I am going to put a small (15x15mm) heatsink on the ICs (iLO+Intel) to see if it helps to lower the FAN speed.
EDIT2: I also do not understand when and how iLo polls the temperaturess across the sensors. It's seems to take ages. Or not at all ? Are those values an AVG ?

[--RW--]

Do you have non-HP components besides the SSDs? Tell me about those 4 DIMMs.

[gabrielkozsar]

4x 16GB DDR4 HPE 879527-091

I was able to read it through CPU-Z. iLo doesn't care seems like.

The server power consumption at IDLE is 50W.

01-Inlet Ambient  Intake  10  1   OK  26C  Caution: 42C; Critical: 46C    02-CPU 1  CPU  14  5   OK  40C  Caution: N/A; Critical: N/A    03-P1 DIMM 1-4  Memory  11  7   OK  32C  Caution: 89C; Critical: N/A    05-Chipset  SystemBoard  10  9   OK  46C  Caution: 100C; Critical: N/A    06-VR P1  SystemBoard  14  3   OK  31C  Caution: 115C; Critical: 120C    07-Stor Batt 1  SystemBoard  5  11   OK  34C  Caution: 60C; Critical: N/A    08-BMC  SystemBoard  13  14   OK  75C  Caution: 110C; Critical: 115C    09-BMC Zone  SystemBoard  14  14   OK  42C  Caution: 85C; Critical: 90C    10-System Board  SystemBoard  9  9   OK  34C  Caution: 90C; Critical: 95C    13-PCI 1 Zone  SystemBoard  12  14   OK  36C  Caution: 90C; Critical: 95C    14-PCI 2 Zone  SystemBoard  9  14   OK  39C  Caution: 90C; Critical: 95C    15-HD Controller  SystemBoard  7  13   OK  58C  Caution: 100C; Critical: N/A    16-HD Cntlr Zone  SystemBoard  8  14   OK  36C  Caution: 90C; Critical: 95C    18-LOM  SystemBoard  9  13   OK  51C  Caution: N/A; Critical: N/A    19-P/S 1 Inlet  PowerSupply  1  8   OK  29C  Caution: N/A; Critical: N/A    20-P/S 2 Inlet  PowerSupply  4  8   OK  30C  Caution: N/A; Critical: N/A    21-P/S 1  PowerSupply  1  12   OK  40C  Caution: N/A; Critical: N/A    22-P/S 2  PowerSupply  4  12   OK  40C  Caution: N/A; Critical: N/A    23-P/S Zone  PowerSupply  2  7   OK  28C  Caution: 75C; Critical: 80C    25-System Board  SystemBoard  14  15   OK  41C  Caution: 85C; Critical: 90C    27-M2 Zone  SystemBoard  15  13   OK  34C  Caution: 85C; Critical: 90C 

Fan        Location     Redundant    Status     Speed
Fan 1     System        N/A                 OK         6%
Fan 2    System        N/A                 OK         6%
Fan 3    System         N/A                OK         52%

• Minimum Fan Speed
  0%
• Thermal Configuration
  Optimal Cooling

Power Supply Summary

Present Power Reading 51 Watts 

Power Management Controller Firmware Version 1.1.4 

Power Status  Redundant 

Power Discovery Services Status N/A 

High Efficiency Mode Balanced

> Static Low Power Mode

Firmware Name Firmware Version Location

Drive SCEKH5.1 Port=1I:Box=2:Bay=1
Drive SCEKH5.1 Port=1I:Box=2:Bay=2
Embedded Video Controller 2.5 Embedded Device
HPE Ethernet 1Gb 2-port 361i Adapter - NIC 1.3564.0 Embedded LOM
HPE Smart Array P408i-a SR Gen10 3.00 Embedded RAID
HPE Smart Storage Energy Pack 1 Firmware 0.70 Embedded Device
iLO 5 3.03 Mar 22 2024 System Board
Intelligent Platform Abstraction Data 16.1.0 Build 21 System Board
Intelligent Provisioning 3.50.100 System Board
Power Management Controller Firmware 1.1.4 System Board
Power Management Controller FW Bootloader 1.1 System Board
Power Supply Firmware 1.00 Bay 1
Power Supply Firmware 1.00 Bay 2
Redundant System ROM U43 v2.80 (07/20/2023) System Board
Server Platform Services (SPS) Firmware 5.1.4.913 System Board
System Programmable Logic Device 0x0D System Board
System ROM U43 v3.00 (02/01/2024) System Board

[gabrielkozsar]

I have added a heatsink to iLO and Intel chip. The BMC temperature went down approx 10C (60-65C), but unfortunately did not help to lower the FAN speed. I have done some thermal images , will upload them to my link mentioned in my previous post.

[gabrielkozsar]

HI, the FAN 3  (70%) was finally tamed to 6% . Now each three FANS are running at 6% when IDLE.

The issue was a non-HPE SSD. The moment a HPE SSD was plugged into the system iLO immediately was getting temperature information from the drives i.e. 04-HD Max - SystemBoard. FAN 3 went immediately silent. So the onboard chip temperatures had no relationship with the high FAN speed in our case.

The OS is still on non-HPE SSDs in RAID1 , the two HPE SSDs were plugged in to the other two slots an that was enough.

g