ML350 G6 will not boot with 2 X5670 CPUs

[thebaronn]

Hi, I'm really hoping someone can help with this as it is driving me crazy.

I have an ML350 G6 with 1 CPU (X5670, SLVB7) and 9 x 8GB Registered DIMMs (72GB). The server powers up and works perfectly. The CPU was recently upgraded from an E5506 SLBF8.

I would like to install a second CPU and another 9 x 8GB Registerd DIMMs (taking the RAM to 144GB).

The trouble is, when I install the second (matched) CPU the server will not boot.

When the second CPU is installed, if I go into iLO then it does not show the latest hardware, it shows the hardware from the previous boot, which suggest it is not POST'ing correctly. None of the hardware sensors show any data.

I have been swapping parts to try to identify the issue, and am now really scratching my head. So far I have tried:-

• Adding a second 750W power supply.
• Replaced the system board. Note the system board has product code 606019 which supports the higher end Xeons.
• Added a fourth system fan.
• Switched around the CPUs; both boot in socket one with only one CPU installed.
• Switched around the RAM. All 19 DIMMs work in the sockets for CPU one.
• Reset BIOS/cleared NVRAM using the dip switch on the system board.

I did suspect a faulty system board, but unless I am incredibly unlucky and have two that are DoA, then I'm not sure what else this could be.

Am I missing an obvious BIOS setting?

I should add that the two LEDs on the front panel are green, the LEDs on the two power supplies are green and there are no error LEDs showing on the system board.

ANY help would be greatly appreciated!

[thebaronn]

Sorry to bump this, but this is still an issue and I'd really like to know if anyone has experienced something similar.

[freddy_bat]

PPM2 (Processor Power Module) for processor №2 is installed? PPM2 is needed on ML350 G5 (not sure about G6).

[Edwin Eppel]

No PPM mentioned in the manual for the G6. Poster seems to have met the requirements:

• identical part numbers for the both CPUs
• additional fan required for 5600 series CPUs (System supports 4 fans max. Manual lists for dual CPU redundant fan config. 2 rear fans, 2 front fans, 2 CPU fans, 2 DIMM baffles and large redundant air baffle.)
• reset NVRAM with main board dipswitch

Main board 511775-001 supports 5500 series CPU; 606019-001 is required for 5600 series CPU. Manual states that CPU2 memory is optional. The one thing not mentioned is the BIOS version. Older BIOS may not support dual X5670 CPUs. Unfortunately a service contract with HP is required for the latest BIOS as the G6 series is retired, and there is no guarantee that it will work. If you can find a second E5506 SLBF8 I would try installing 2 of those and see if it will post with the older model CPUs..

[thebaronn]

Thanks for the tip, I've located another E5506 CPU and will install it tomorrow.

The current BIOS version is 07/02/2013. This appears to be the penultimate version for this server, with the latest version being 2015/08/16.

The server is out of warranty so I cannot try the later BIOS.

[thebaronn]

Update, if you look at the BIOS revision history (link below), the 07/02/2013 release mentions unrelated fixes for X5675 and X5690, so the BIOS should in theory support the X5670 too.

http://h20564.www2.hpe.com/hpsc/swd/public/detail?sp4ts.oid=3884318&swItemId=MTX_a343bf8e9164467e95d689a9b6&swEnvOid=1005#tab-history

[thebaronn]

Update: I have now tried with two E5506 CPUs, same result though, system will not POST. Before I installed the second E5506 CPU I booted up with just one (the new one to make sure it is not a dud) and it booted fine. Should I reset NVRAM/BIOS having changed CPUs? 

I have double checked the pins in CPU socket 2, they look as good as new.

Andy.

[Edwin Eppel]

Any ideas Jimmy Vance? Poster seems to have covered all of the bases I can think of. I haven't dabbled much with the BIOS, but install both PSUs, both E5506 CPUs and one stick of memory for each CPU. Set DP switch 6 to clear NVRAM and then see if you can get into the BIOS to look for any settings that need to be changed; or does the posting not go far enough for you to access the BIOS?. Seems unlikely that you would have 2 defective motherboards with the same issue, unless it is the BIOS that is the issue. I would concentrate on getting it to work with the older CPUs first, but you've eliminated the PSU, CPU and memory as the problem and that just leaves the motherboard and BIOS. Wish I had a second Xenon CPU handy as we have the same ML350 G6 with spare PSU and memory, so I'm just missing the CPU to try it out myself.

[thebaronn]

Thanks Edwin, if you are super curious you can buy an E5506 for around £5 :)

I'll try some more combinations with the two E5506 CPUs tomorrow (e.g. removing all but two DIMMs).

Sadly when the second CPU is installed I cannot get as far as the BIOS. I get nothing on the screen and cannot access BIOS remotely. When I have one CPU installed I've trawled the BIOS for anything that looks like it might be causing this, but to no avail. One would have also thought something as fundamental as this would be covered in the manuals/other forum posts.

Thanks,
Andy.

[Edwin Eppel]

I went through the RBSU User Guide and didn't see anything that would affect your situation, except maybe Power Redundancy settings, but the default is Balanced any way. If the BIOS is reset to defaults you should be fine. With the 2nd CPU installed the system checks for adequate power and number of fans at post, and you seem to have that covered so I'm stumped as well. :(

[Jimmy Vance]

My only thought is there is probably something wrong with the 2nd CPU socket. I know the original poster says it looks OK, but if you can put both CPU's in socket 1 and everything is OK, it pretty much only leaves the 2nd socket as the issue, especially since he now has two of the original processors. 

[thebaronn]

Thanks Jimmy, I think it is looking that way. The pins all look fine though, and this is the second board so I may just be incredibly unlucky.

[Rob_at_NSIT]

I seem to remember a rash of these with bad boards.  The Ver1 (and Ver2?) of the hex board had a couple of traces too close to the CPU 2 socket bracket.  If you remove the socket hold down you see a small screw pad and shiny traces.  If the traces are uncut (or cut finely enough you know how to reflow them) one of those little red plastic washers may save the board.   The ver 3 of the board enlarged the pad and moved the memory trace.  On that note; try it with no memory in socket two that would defiantly point to the above. .

Note: Adding the washer will make the CPU lose adding a shim may / will be necessary.

 Bios setting: CPU Node interleave should be disabled.

Also make sure the CPU power cable is completely and fully seated it uses independent pins to power the two CPUS.  

Final note remove the AC and Battery short the two battery connectors for about a minute (or leave overnight), I have seen strange "firmware ghosts" that the erase function just doesn’t clear.  For example I recently had a Gen8 with an extra memory spare locked on.  If you enabled ADG with hot spare you lost 2 DIMMs per CPU and memory mirroring reduced a 256GiB server to 64GiB. 

[Rob_at_NSIT]

Small clarification, I mean the CMOS battery and CMOS Battery contact that are available after it is removed.  Also for safety if there is a BBWC disconnect those batteries as well.  

[thebaronn]

Thanks for the excellent suggestions Rob, I'll try these out and post back.

[thebaronn]

Both boards are revision "0A".

When you say "If you remove the socket hold down you see a small screw pad and shiny traces", do you mean use a Torx screwdriver to remove the cover that holds down the CPU (highlighted below)?

Thanks.

[Rob_at_NSIT]

I’m going to be honest, I do not remember.  It was 3 to 4 years ago.  But I seem to remember two out of the three boards I inspected the damage was under the upper right black - silver screw and the third was lower right silver – sliver screw (circled).  

Looking at your picture if the board was in front of me I would:

Closer inspect the 3rd from the left CAP.  From the picture it looks to be bulging.  If it is from one of the known bad cap makers I would replace it and any others on the board of the same make / model.

Check to see if the white spot under the lever between the two top silver-silver screws is a reflection or damage from a screwdriver being used to lift the CPU retainer.  If it is not a reflection I would remove the bracket and inspect very close - possibly file and reflow it.

Inspect the spot on the memory trace between C379 and 327, is it dust or cut?

This could be a photo artifact, but in the upper right area by the silver bracket between C3017 and C3009 there is a dark area that like like it could be a burned trace in layer 2,  directly under the C3017 silkscreen it appears to have a crack or a bulge, again leaning to a failure in lay two.  Repairing an inner layer issue is mostly impossible to fix.  (It can be by passed if you know the layout and/or have a circuit diagram.) 

Again this could be photo issues but it appears the entire upper right row of CPU pins are burned in to the socket.  As if they got hot enough to char the nylon.  And the logic board appears fried in the same area.   I’ve attached an edit of the photo.

EDIT: How did you get the photo in the post?  The edited image is showing up in the side bar--> as an attachment.

[Dennis Handly]

>How did you get the photo in the post?

Yes, how?

When I have to include a picture, I have to upload an image.  (Perhaps if you point to a URL, that's all you need?)

Anyway looking at the page source I see:

http://s21.postimg.org/edwm65nwn/IMG_1262.jpg

Ah! And using Insert Photos, you get:

[thebaronn]

That's right Dennis, I used Insert Photo from the WYSIWYG editor and pointed to a hosted image.

[thebaronn]

Thanks for this Rob, I'll take the CPU cover off and examine the board more closely. A few of the items you mention are photo artefacts/shadows; I'll summarise once I've completed the inspection.

[thebaronn]

Apologies for the delay in my response, I've been busy with other priorities. I recently found time to turn my attention back to this old G6.

I visually examined both system boards (the original and the replacement with the CPU and heatsink holders removed) and the capacitors seem to be fine (no bulging caps or burnt out traces). The capacitors on the power supply board also look fine.

Rather than start to electrically test and swap out caps, I found a bare bones working ML350 G6 on eBay for £150 so bought it and swapped in my RAM and X5670 CPUs; it works perfectly. Note that the new system board is revision "0B", as opposed to "0A" of the other two boards.

So, it looks like the issue is either a) the two previous "0A" system boards are dud (pretty unlucky) or b) the power supply board is a dud.

As frustrating as it is, given that I now have a working system I'm going to have to give up on the root cause. If I had the time it would be nice to take one of the old "0A" boards and put it in the new server (with the new power supply board) to see if that's the cause.

Thanks again everyone for your input.

[Edwin Eppel]

Thanks for taking the time to update us. We have a ML350 G6 so it's good to have a record of what to watch out for.

[Baranyg]

Hi

I have exactly the same problem, and the solution was, to check the memory population order. The first channel is the nearest three dimms to the cpu. 

[Core_2_Duo]

I have the same issue with a 0A board. I tried a lot of different memory configurations with the 2nd cpu installed but none worked. This really sucks because I already wasted money and effort getting the extra components for the 2nd cpu installation. I suspect the board is flawed and it might even be the reason 0B was released. I tried it with 2 different sets of CPUs and neither worked. It wont POST. I also tried removing the lga1366 metal socket clip by unscrewing the 4 torx screws and taking out the metal bracket. The heatsinks still pushed the cpu down. This was to test out the theory of Rob_at_NSIT that the this part interferes with some of the memory traces. I'm 100% sure the socket pins are all still good. I inspected it very thorougly and I also tried all other suggested things like resetting the board with the DIP switch 6 and removing the coin cell cmos battery and shorting the leads for a while.

[miloman]

For the record: I also have the 0A mobo, and as of today 2 X5670's up dan running with 24GB of RAM (12 x 2GB reg ecc).