HPE Gen10 Plus high BMC temperature

[dszpisjak]

Hello,

I purchased the microserver a few months ago and started noticing that the BMC temperature is way above everything else even under no load, while the fan in the server is running at 10% speed.

This is the table I see under Power & Thermal-Temperature Information in iLO 5.

Sensor Location X Y Status Reading Thresholds
01-Inlet Ambient  Ambient  12  3   OK  27C  Caution: 42C; Critical: 46C 
02-CPU  CPU  5  7   OK  40C  Caution: N/A; Critical: N/A 
03-P1 DIMM 1-2  Memory  1  7   OK  40C  Caution: 89C; Critical: N/A 
05-Chipset  System  6  12   OK  59C  Caution: 100C; Critical: N/A 
06-VR P1  System  9  6   OK  46C  Caution: 115C; Critical: 120C 
07-BMC  System  13  9   OK  80C  Caution: 105C; Critical: N/A 
08-BMC Zone  System  13  11   OK  53C  Caution: 90C; Critical: 95C 
10-PCI 1 Zone  I/O Board  11  13   OK  39C  Caution: 90C; Critical: 95C 
11-LOM  System  9  12   OK  44C  Caution: 100C; Critical: N/A 
12-AHCI HD Max  System  8  10   OK  35C  Caution: 60C; Critical: N/A 

Is this normal behavior?

regards,
DAn

[SanjeevGoyal]

Hello,

I would request to update the bios and all firmware with the latest.

1. Bios

2.ILO firmware

3.Server Platform Services (SPS)
4. Innovation Engine (IE) Firmware 

If you feel this was helpful please click the KUDOS! thumb below!   

Regards,

[dszpisjak]

I may be off-track here but I can't seem to update either the BIOS or the ILO firmware. 

I downloaded these files: 
BIOS: https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_579f55d77b1c4028bfa4f0740c#tab3
ILO: https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_11b0bf7deb9d4b5aa46ee921ef#tab3

In both cases, when I upload them via the ILO web interface I get the following error message:

The file signature is invalid. Make sure you are using a valid, signed flash file and try again.

I checked the posted SHA256 hashes and they match the downloaded file's hash.

Now I am downloading a 9GB SPP ISO, hoping that will help.

System version details:
iLO 5 2.11 Nov 12 2019
System ROM U48 v2.10 (03/09/2020)
Intelligent Platform Abstraction Data 1.4.0 Build 11 
System Programmable Logic Device 0x09 
Redundant System ROM U48 v2.10 (03/09/2020)
Intelligent Provisioning 3.40.192 
Server Platform Services (SPS) Firmware 5.1.3.94

Not that it should matter, but I am running Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-58-generic x86_64)

There seems to be something off here with the invalid signatures. Any ideas on where to look? 

regards,
DAn

[SanjeevGoyal]

Hello,

You have shared the bios & iLO firmware VMware link due to this you are getting the below error.

The file signature is invalid. Make sure you are using a valid, signed flash file and try again.

You have to download the OS-specific file.

or you can update by the latest SPP.

If you feel this was helpful please click the KUDOS! thumb below!   

Regards,

[dszpisjak]

I managed to upgrade the BIOS, but can't find a firmware for the iLO. I searched for Gen10+ and filtered for iLO firmware, I tried every downloadable file from the HPE site. No success yet.

I will try using the latest SPP and report back. Once iLO is updated I will update the SPS and IE and check the temperature again. So far it's still high.

Thank you for your help so far.

[dszpisjak]

I managed to update anything there is to update using the latest SPP. Here is where I stand now regarding software versions:

iLO 5 2.30 Aug 24 2020 System Board
Intel(R) I350 Gigabit Network Connection 1.2684.0 Embedded LOM
Intelligent Platform Abstraction Data 7.1.0 Build 13 System Board
Intelligent Provisioning 3.40.192 System Board
Redundant System ROM U48 v2.10 (03/09/2020) System Board
Server Platform Services (SPS) Firmware 5.1.4.113 System Board
System Programmable Logic Device 0x09 System Board
System ROM U48 v2.20 (10/27/2020) System Board

And the BMC temperature is still 75 when everything else is <=55 celsius. That's 20C difference, I find this very hard to believe. Any idea on where to go next?

regards,
DAn

[Torsten.]

Consider to check SPSgen and Innovation Engine firmware versions too.

[dszpisjak]

Alright, I managed to update every firmware on the system to the latest version.  Innovation Engine seems to be unavailable for my config. It isn't listed on the HPE support site for Gen10+, and even if I try to apply the Gen10 version the updater says the target is incorrent and SUM doesn't pick it up.

Here is where I stand now:
Embedded Video Controller 2.5 Embedded Device
iLO 5 2.33 Dec 09 2020 System Board
Intel(R) I350 Gigabit Network Connection 1.2766.0 Embedded LOM
Intelligent Platform Abstraction Data 7.1.0 Build 13 System Board
Intelligent Provisioning 3.40.192 System Board
NVM Express Controller PCI-E Slot 1
NVMe Drive 2B2QEXM7 Slot 1
Redundant System ROM U48 v2.10 (03/09/2020) System Board
Server Platform Services (SPS) Firmware 5.1.4.208 System Board
System Programmable Logic Device 0x09 System Board
System ROM U48 v2.20 (10/27/2020) System Board

The temperature is still 75C+ on the BMC right after a cold boot. What's next?

I also notice another strange behavior. Sometimes my system boots up twice. The first boot shuts down shortly after memory initialization is completed. Then it boots without problems. It doesn't always occur and I can't figure out why it does when it does. Is this normal? I have seen this behavior on an overclocked PC before...

[dszpisjak]

So, where exactly can I turn with this issue? It is causing me a headache as I occasionally receive a thermal shutdown warning when everything except the BMC is within the accepted thermal range.

I highly doubt this is normal behavior, and it's been present since I unboxed the machine.

[dszpisjak]

I tried reaching HPE support by opening a case. To my complete surprise, after providing all the information and clicking "Submit case" the following message greeted me: 

The service or information you requested is not available at this time. Please copy and paste the information provided into a text editing application and try your submission at another time.

What is going on here? How do I submit a case?

The copy-paste instructions are beyond my imagination...

regards,
Daniel

[Sione]

I just bought the same server and seeing the same thing, did HP Support ever get back to you on this?

[dszpisjak]

I managed to get in touch with HPE support. They sent me a new mainboard. I replaced it and saw rhe same high BMC temperature. The L1 support engineer could *not* confirm this is normal behavior. He said they don't have the capcity to test it out on stock hardware and I am running this at home not in a datacenter environment so they cannot provide a reliable comparison.

I left it as is and told the engineer that I refer to my support ticket should this become a problem in the long term.

I asked an electrical engineer friend about the issue and he said these BMC chips are likely built using layered circuits which makes it harder to cool them. I also checked the area of the mainboard where this chip is located and found no passive cooling.

Given the above here is *my* conclusion. This seems to be normal behavior. The chip is indeed hotter than anything else in the chasis (70+ compared to ~50) yet when under heavy load it doesn't go over 85C and its critical threshold is above 100C. I had a thermal shutdown which was caused by a different chip reaching a warning temperature. The BMC was doing fine in this scenario.

[Sione]

Good to know the BMC was not the cause of your thermal shutdown, I was worried that my BMC was running to hot as well so I did some googling and found your post here on it and was trying to figure if it was a issue that I need to worrie about but seems like its the nomral temp range for this computer.

[Floyd71]

I have the same problem with the 07-BMC temperature.

I tried other method with increased minimum fan speed.

For example, when 01-Intel Ambient says that is 19C the 07-BMC is arround 70, but when increase fan speed to minimal 40%, then I got BMC-07 of 65C with 01-IntelAmbient 17C.

From this experiment (for just 10minutes), I feel, that passive cooler would be very helpfull to this BMC to decrease temperature if increased air flow helps to reduce the temperature. 

Please, could you provide photo and identify exactly which is this BMC chip which is hot, in order to put passive radiator on it to reduce temperature?

[dszpisjak]

I am sorry, I cannot provide a photo identifying the BMC as I haven't done that myself.

Your best shot is asking HP to identify this part or take a cheap thermal camera and measure it yourself.

[paradoxhub]

Did you ever resolve this? I got my Gen10+ a few weeks ago and the BCM-07 temp has been around 75c even when idle or just after a bootup. I have tried running the fan at 100% for 6 hours and it has 0 effect on the BCM temperature.

Am I fully up to date?

[paradoxhub]

[Floyd71]

No, the issue was never resolved. I believe that something with motherboard is not correct.


Get Outlook for Android

[Floyd71]

No I could not resolve it. I am pretry sure that motherboard is problematic and due to this I am only ocassioally run the server.



Get Outlook for Android

[paradoxhub]

Is it the ILO chip that is running hot? I could put some copper heatsinks on it.

[Floyd71]

Could you tell me how to identify that iLO chip? Is it on the iLO enablement kit card or it is on the motherboard with letters iLO on them? If that one, when operating it is not observed excessive temperature of them. Shortly after turn off and touching with well-grounded finger, there is no big heat on the surface of the chip. I would say maybe 50C. 

I am concerned about this high temperature which persists always on the Motherboard. 

During the summer time, when room temperature is close to 35 - 36 degrees Celsius the BMW temperature is getting to high (arround 85 C , where the maximum temperature shall not be greater than 105 degrees. 

Here is print-screen (server turned off for more than 24hours) from iLO main-page:

I am pretty sure that coper cooler will not solve the problem. At least when server not operate, the temperature will remain the same.
When operating, with (default) cooler settings temperature will be not more than 2 - 3 C lower which is not significant and it will cause much more loud, cooler usage and more power consumption for cooling - which will open other issues.

I believe that new motherboard design shall be done which will solve the temperature issue.

I will appreciate if could be considered new motherboard to be send to me to serve as spare part, as usage of server (on month usage) is less than 10 hours total (explicitly due high risk of loose production machine due to temperature issue). Due to this, I am still using old HP Microserver G8 (6 years old machine) as my primary server. 

Thank you for considering my message.

[Torsten.]

Did you ever tell the server model?

Consider to use the latest SPP to upgrade all the firmware, Innovation Engine may also cause false temperatures.

[Floyd71]

My firmware and BIOS are upgraded to latest versions, but no changes with sensor 07-BMC temperature. 

P/N of the server is P16006-421 (Intel Xeon + 16Gb RAM). 

What is "Innovative engine"? could you please explain, so I could try also this one.

Thank you. 

[Torsten.]

It is obviously a HPE ProLiant MicroServer Gen10+, need to have a look, since it is different to ther proliants.

[Torsten.]

Not sure if this applies to your server too:

https://support.hpe.com/hpesc/public/docDisplay?docId=a00099826en_us&docLocale=en_US