Unexpected shutdowns Proliant ML370 G5

[Janet Erbach]

We bought out ML370 G5 in December and are running it under SBS 2008. The server has re-booted itself at least a dozen times since then, and the event logs show only "unexpected shutdown".

Yesterday it re-booted in the middle of the day while we were processing orders, and it re-booted again last night at some point. We're using an APC UPS - is it possible that the UPS is responsible? What else can I do to troubleshoot and resolve this?

[Gary Benavides Meza]

When there are no error on the logs and the diagnostics test passed no issues detected. Most of the time that is realated to power, so it could the power backplane or the UPS. Try working with the server directly connected to outlet, without the UPS, if the server reboots again that might be the power backplane.

[Johan Berqgvist]

Did you solve this problem? I've got the exact same problem with our ML370 G5

[Janet Erbach]

Our Network consultant is convinced that the shutdowns are related to the UPS we're using - an APC "Smart-UPS" Ac 1500 model. He researched it extensively, including repeated conversations with APC, and says that the UPS model we're using does NOT meet the power needs of the ML370. (Something related to the sign wave that the UPS produces not meeting the specs that the ML370 requires.)

We have not yet replaced the UPS. So even though our problem isn't 'solved' until we do, I'm 99% sure that our shutdown issues will disappear with the proper power supply. And since an earlier reply to my post suggested power-related issues, I would check into the same thing on your end!

[Johan Berqgvist]

Interesting... The server in question wasn't attached to any sort of UPS. We just bought a HP T1500 UPS to eliminate any power-related problems in our small server room and the problem was not solved by this.

Our old HP/compaq server does not have any problems. And it's standing right next to the new server.

Note: The server has been working fine for over a year before this started in late april.

Smells like hardware/PSU-problem to me. I'll try with a new PSU. I've got one i can borrow for a couple of weeks.

[Anne Nutt]

Would you let us know what the solution was as I am now having the same problems with a ML350.

[BenS]

Which event logs are you looking at? the Windows ones? or the HP Integrated event log? If you've configured the HP agents correctly on the server, the HP log should catch things like an ASR initiated shutdown, or other potential hardware related issues. It doesn't catch everything, but it's still worth checking (if you haven't done so)

[Anne Nutt]

I have been working with HP and have logged 3 jobs with them trying to solve the servers issue. It shuts down randomly mostly at night and just a while ago it went 'ping' and died and wouldn't restart until I unplugged it for 20 minutes. There never appears to be anything in the logs hp or system as it is too quick to write to anything. HP have replaced the Motherboard and we have updated all the software to the max and for a little while it settled down a bit. But I think now it is getting worse. It is a 21/2 year Proliant ML350 running W2003 sever.

[Johan Guldmyr]

How about power supplies? Have it/they been replaced too?

[Anne Nutt]

No that hasn't been suggested.

[Johan Guldmyr]

Okay, if you have more than one PSU then it might be worth while to try to run it on only one PSU/ cross swap / try a PSU from another server.

[Anne Nutt]

Thank you for the advice I will try to get one to put in it. The server is on a APC UPS - do you think it could still shut down as quickly with this on it.

[Johan Guldmyr]

Yes definately. If it is a power issue then the issue might certainly reside in the UPS itself. Can be identified by connecting directly to the wall.

[Anne Nutt]

Thank you I really appreciate the help, I will let you know how I get on.

[Janet Erbach]

According to two independent network consultants I've talked to, the APC UPS units can be a problem with the HP ML350 and ML370...depending on the output of the UPS. We've been using one that outputs at 800 watts or so, and the ML370 draws 1400 or so. And when the UPS runs a self-check it will cause a server re-start every time. We ordered an APC 1500VA USB Ser RM2u (just arrived yesterday - not hooked up yet.) I'm told the 2200 is an even better choice for longer up-time until the server shutdown initiates.

[Anne Nutt]

That is a wonderful clue for me to investigate. I hope this is the issue as I will be able to resolve it quickly. It has been like finding a needle in a haystack and the server is starting to show signs of the wear and tear of the constant unexpected shut downs - one hard drive has already crashed. Would you let me know how you get on and thanks for the information.

[MikeR_4]

This sounds like almost exactly what I have been dealing with. I have some extra info for you to try. Please let me know what you find as I am trying to get my finding escaladed at HP.

My client has a new ML370G5 and an APC Smart-UPS 1000VA. I was told the same thing by APC about it being underpowered, but the normal output for our ML370G5 is under 300 watts. I did find that it must be a smart-UPS due to the modified sine wave problem. After replacing the power supply, power supply backplane and the UPS I was able to resolve this issue by disabling the ASR (Automatic Server Recover) in BIOS. We are also now running the latest firmware and all MS drivers from HP as of Friday July 3rd, but problem persists if ASR in enabled. I conducted my testing by pulling the UPS from power while sitting at the CMOS screen (I did this to prevent Windows corruption and to rule out driver issues). I found with ASR enabled the server would almost always reboot within three or four pulls from AC power. Once I disabled ASR I was pulled power over ten times without ever experiencing a reboot. I even tested in Windows. After all my testing I really believe the issue lies with the ILO2/ASR feature. Again, I would really like to hear what you all experience.

[Janet Erbach]

Thank you for posting your testing methodology - I am going to try testing the new UPS (which I hooked up last Friday) in the same way you did yours and see what happens. I'll post my results when I've tested!

[Janet Erbach]

I did a quick test of our new APC UPS last night - it's an APC Smart UPS 1500VA USB/SER RM2U. I pulled the power plug and waited 5 minutes - no server hiccups at all, so the sine wave et al that the new unit emits does, indeed, meet the needs of the ML370.

[MikeR_4]

Janet, was your previous unit a smart-UPS 1000? I found my problem occurred when the power was pulled and placed back online several times somewhat quickly. This was the best simulation to emulate several brownouts. The problem I had was not holding the server up on battery, but rather the cut over form AC to battery. Extremely strange, but still would like to know what happens for you with the 1500 as my testing has been done with the Smart-UPS 1000.

[Janet Erbach]

The old unit was a Smart UPS SC1500. When it went into self-test mode, it would generate a re-start on the server. When power was pulled form that unit - the server shut off immediately. The battery support never kicked in at all...which is why I was satisfied for now that the new unit kept things up and running for at least 5 minutes. I plan on testing it more, but probably won't be able to get to that for another week or two since it means coming in after hours. :)

[Johan Berqgvist]

FYI.

My problem solved itself when i switched Powersupply. No reboot/shutdown in 2 weeks now.

regards

[ChrisCJ]

I am sitting on the phone at this very moment waiting for my HP Insight Diagnostics to run per the HP tech. I am working with this same exact problem on my ProLiant ML 310. We had this happen back in early May out of nowhere. This server was delivered in December of 2008. Well, back in May they pretty much gutted the server and it was fixed after 2-3 visits of part swaps. Here I am again today. Ironically I found the server down this AM and while tracing its power wire I found that the tech from May had not plugged it back into the APS RS 1500 we had for this server but rather the wall. I moved the plug back to the APS and powered it up and everything seemed fine. 6 hours later here I am on the phone with HP thinking this is hardware again. I am going to see what he says but think I'll follow the advice here of everyone. Disable the ASR and perhaps see if I can kill the self testing going on this APS.
Other thing of interest is a new light rail system is going live right outside our door here and they are testing in and out of the station right now all morning. Electric train and all. I wonder if there is some relation to this.

[ChrisCJ]

Sorry that is an APC Back-UPS RS 1500.

[MikeR_4]

Chris, I do know that only a true Smart-UPS will work do to a modified sine wave problem.