HP ML150 G6 System Health Blinking Red Only in Dual CPU Configuration; Single CPU Config is OK

[JuliusPIV]

Problem:

I have an HP ML150 G6 with a single E5504 and 24GB (6x 4GB) modules that works just fine.
After installing an E5540 matched-pair CPU kit and balancing the DIMMs across both CPU DIMM banks (in slots 2A, 4B, 6C), the system powers on but doesn't POST.
After 4-5 seconds of powering on, the System Health light blinks red indicating a critical system failure was detected, which according to the documentation could be: Processor, Memory, Regulator, Thermal Event, Fan or NMI.
I need help narrowing the scope of the issue for further troubleshooting.

Troubleshooting:

• Reset the BIOS to failsafe settings
• Ran the SmartUpdate Firmware (9.30) ISO which confirmed no updates were required
• I removed all other components (CD/DVD ROM, Hard Drives etc.) to eliminate those possibilities
• Reverting to the original configuration = OK
• CPU A in Processor 1 socket with 3x4GB (modules 1, 2 & 3) in CPU1_DIMM slots 2A, 4B, 6C = OK
• CPU B in Processor 1 socket with the same 3x4GB modules in CPU1_DIMM slots 2A, 4B, 6C = OK
• CPU A in Processor 1 socket with the other 3x4GB modules (4, 5 & 6) in CPU1_DIMM slots 2A, 4B, 6C = OK
• CPU B in Processor 1 socket with the same 3x4GB modules CPin U1_DIMM slots 2A, 4B, 6C = OK
• CPU A in Processor 1 socket with 3x4GB in CPU1_DIMM slots 2A, 4B, 6C and 3x4GB in CPU2_DIMM slots 2A, 4B, 6C = OK (although the system only registers 12GB and complains about the memory configuration)
• CPU B in Processor 1 socket with 3x4GB in CPU1_DIMM slots 2A, 4B, 6C and 3x4GB in CPU2_DIMM slots 2A, 4B, 6C = OK (although the system only registers 12GB and complains about the memory configuration)
• CPU A in Processor 1 socket with 6x4GB in CPU1_DIMM slots = OK
• CPU B in Processor 1 socket with 6x4GB in CPU1_DIMM slots = OK
• Swapped heat sinks between both CPU's
• I also tested several memory configuration variation, swapping memory modules around into different slots (example: 2A moved to 6C, 6C to 4B, 4B to 2A etc) = All the combinations were OK with 1 CPU; Adding the second CPU resulted in a failure.
•  Basically, when a CPU is in the second processor socket, the system exhibits the issue.  No matter what CPU is in the second processor socket, the system fails to boot.  (i.e.: CPU A in Processor 1 socket & CPU B in Processor 2 socket; CPU B in Processor 1 socket & CPU A in Processor 2 socket)

Based on the above, the CPU's and memory modules are sound, suggesting it's something else.

• I didn't see any dual CPU configuration prerequisites listed in the documentation: http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c01706108-8.pdf
• Unsure if the system doesn't like the memory configuration (In a dual CPU configuraiton, does it require 12x memory modules of the same type & speed?
• Unsure if a BIOS update is required and if it is, it's not available without a warranty or support plan
• Unsure of how to validate the issue isn't caused by the actual second processor socket
• I cleared the SEL, added the second CPU, booted & powered it off and saw an event with CPU2 ThernTemp. 
• Related to the above: Unsure if the system requires one or two additional fans for a dual CPU configuration.
• Related to the above: I have a fans plugged into SYS_FAN3 and SYS_FAN5 but on SYS_FAN2 there's the quivalent of a jumper installed; all others are empty.
• Unsure if it's the power supply although it's important to note that the system works with 6 SATA Hard Drives, CD/DVD ROM & Video card, 6x4GB modules in a single CPU configuration.
• Unsure if there's a jumper on the motherboard that needs to be toggled.
• Can't find the offline edition of HP Insight Diagnostics for this server.
• Can't find the Intelligent Provisioning Recovery Media for Gen6 servers.
• Didn't see anything relevant here (although I'm still sorting through it): http://h20566.www2.hpe.com/portal/site/hpsc/public/psi/troubleshootResults/?sp4ts.oid=3884323&sp4ts.pn=518174-005
• Didn't see anything relevant here (although I'm still sorting through it): http://h20566.www2.hpe.com/portal/site/hpsc/public/psi/mostViewedResults/?sp4ts.oid=3884323&sp4ts.pn=518174-005
• Didn't see anything relevant here (although I'm still sorting through it): http://h20566.www2.hpe.com/portal/site/hpsc/public/psi/advisoriesResults/?sp4ts.oid=3884323&sp4ts.pn=518174-005
• The link for the ML150 G6 here leads me to a dead end: http://h20566.www2.hpe.com/hpsc/doc/public/display?docLocale=en_US&docId=emr_na-c00308226&sp4ts.oid=3884323

At this point, I'm essentially out of ideas. so any advice/guidance is greatly appreciated.

Thanks!

[Moderator edit: Removed the broken link. Please refer to https://support.hpe.com/]

[Jimmy Vance]

Did you try Both CPU's and minimal memory?

[JuliusPIV]

Hi Jimmy Vance and thanks for the reply.

With both CPU's and 1x4GB module in both 2A DIMM slots, the same error persists.  I do have some 2GB modules around so I suppose I can give those a try later today.

I also tried testing with that 'jumper' that's currently plugged into SYS_FAN2:

• Original Configuration: SYS_FAN2 = Jumper; SYS_FAN3 = Fan, SYS_FAN5 = Fan; All others are empty.
• SYS_FAN2 = Empty; SYS_FAN3 = Fan, SYS_FAN5 = Fan
• SYS_FAN1 = Jumper; SYS_FAN2 = Empty; SYS_FAN3 = Fan, SYS_FAN5 = Fan

Aside from reviewing the SEL while in the BIOS, is there another option for better understanding why System Health is blinking red?

[JuliusPIV]

The SEL in LO100 shows:

Source        Description
FAN_1         Lower Critical-going low
CPU2 IErr   State Asserted
CPU1 IErr   State Asserted

I pulled the power cable, CMOS & RAM & reinstalled everything, and as soon as it was plugged in, it powered on with the same blinking red light. 

I'm at a loss: Board? CPU A? CPU B? Other?

[TTr]

How many fans total are installed? Same about the power supplies and rating Watts.

[JuliusPIV]

Good morning TTr, and thanks for the reply!

For what its worth, I think I'm exactly where your head is at: Earlier this morning I re-read the Maintenance and Service Guide as well as the Quick Start, and the Quick Start makes references to 2P systems shipping with 3 fans.  But I'm not seeing where 3x Fans and/or a 750W (or anything higher than a 460W) PSU is a requirement.   That said, the system has:

2x Fans:

SYS_FAN1 = Empty
SYS_FAN2 = Some type of jumper ?
SYS_FAN3 = Installed
SYS_FAN4 = Empty
SYS_FAN5 = Installed

1x 460W PSU - Also, at the moment the system doesn't have much installed:

• Both CPU's
• 1x4GB module in DIMM 2A of each bank
• No HDD's, No CD/DVD Rom, No peripherals, No cards etc.

Thanks again.

[sgunelius]

I wonder if moving a fan to Sys Fan 1 might address the issue.  We ran into a problem with a newer DL360p Gen8 not booting with a missing fan even though it was barely loaded.  Are you seeing an errors at Power On Self Test (POST) that might indicate the problem?  We saw 1611 "Fan Solution Not Sufficient" at POST.

You should be able to purchase these fairly cheap (Spare# 496066-001, Hot-plug fan, 60 mm).

[JuliusPIV]

Evening sgunelius, thanks for the contribution.

I moved the fan from SYS_FAN3 to SYS_FAN1 but the system continues to report the same errors, with and without the 'jumper' in SYS_FAN2.

To make matters worse:

• The slots for Fan 1, Fan 1.5 and Fan 2 are every so slightly closer together than the slots for Fan 3, meaning I can't slide the current fan into the proper location.
• Not realizing this trickery, I've already got HP Part 487109-001 / Spare Part 519740-001 on order.
  Fan 2Fan 1 & Fan 1.5Fan 3

Also, the spare part number you mentioned doesn't seem to correspond to the ML150 G6.  Everything I've seen suggests 513927-B21; maybe 513737-001.  All of these part numbers appear to correspond to the same fan I have: 487108-001, 487109-001, 487099-001, 519740-001.

At this point, I don't think I have much choice but to revert back to a single CPU configuration.
And from the looks of all the prices I've seen from the fans above to the drive cage (519733-001, 519734-001, 519736-001, 598657-001, 599161-001, 487737-b21, 466509-001, 466510-001), I might be able to part out the box and recoup nearly all my costs!

<rant>I've got to say: I'm really disappointed in the direction this went in.  I've had several Dell PowerEdge's over the years and it's never been this hard to get parts for them.  After this experience, I'm quite fearful of picking up another machine because who knows what lies (<g>) ahead!</rant>

Thanks all for lending your time and effort into this.  Notifications are on so I'll be keeping an eye on this if someone replies.

Cheers!
J

[Myze]

Hello,
I have exactly the same problem. Have you found a solution?
Best regards,

[Sunitha_Mod]

Hello @Myze,

Thank you for posting! 

We would recommend you to create a new topic using the "New Discussion" button, so the experts can check and assist you further. 

[Fritex]

@Myzeand @JuliusPIV Lately, have had to re-make 18 of those single CPUs and made 8 usable dual CPU Xeon E5504 with 3 fans.

Since having bunch of those HP ProLiant ML150 G6, I've had spare parts.

For the fan solution, I cut one plastic "leg holder", or smaller part of the fan, therefrom I have two fans on front and one back, dual CPU setup.

Here is what I've cut: https://drive.google.com/file/d/1dhxdSX_UA5u2sUjFUabUcWMi0nTbVvnf/view?usp=sharing

I've also put a workaround plastic wire-jumper (2-3 pin) on SYS_FAN_2. This is how it looks like: https://drive.google.com/file/d/1FVYHsysRCAZMdw1-XLahgt0Ba3qFUHzO/view?usp=sharing

Meaning, from the FAN Configuration image here: https://serverfault.com/a/588597, the "cutted" fan sits in-place with one plastic leg holder (left one), while the other (right one) is cut out. This is how it looks with 2 front fans: https://drive.google.com/file/d/1llct4T18Q0Dk2onmMlJCPM6-lxz6pn2x/view?usp=sharing

Here is the part which I've cut (because one of them is okay, other not fit): https://drive.google.com/file/d/182eGHysF2lvGc8SX6eeiRcBoRGn_kU_H/view?usp=sharing

Moving it one position further, see in-picture here: https://drive.google.com/file/d/1FKNLLXG3fSfO0WlM1xB0_yPg9qDjkyJ5/view

To conclude, dual CPU, 3 fans (2 front SYS_FAN1 and SYS_FAN3, 1 back SYS_FAN4) and SYS_FAN2 with "jumper" works pretty fine and a blue plastic tunnel cover fits well and PC case closes normally. 

View from front through the tunnel with that modified "fan": https://drive.google.com/file/d/13E-eC2bi1rJaSWbfdFDxrmObFtKstxxV/view?usp=sharing

Here it how it looks like: https://drive.google.com/file/d/1lCuqrApJaCjb3-xn7Bt0wMDh_HJORk0Z/view?usp=sharing

The fan stands still in this position with one leg (the "clip" didn't clicked as well since the metal whole isn't meant for this type of the fan), but the fan doesn't bounces/jumps around, and in both holes with it's standing legs on the bottom. Since the PC stays in this position further, I just add the blue tunnel cover while removing the small part which holds something (I guess the smaller fan?) and it fits well to close with metal cover plate.