HPE Gen9 losing all power after shutdown (PSU connected and green light)

[MarkPV]

Hi,

Have new problem... which is quite rare since I have seen lots of issues over the years... Hoping to find anyone who has a clue.

Have couple of servers, since recent we decided to instead of idling, we shut down nodes we don't use. I regularly check with WOL, server goes online, all goes fine, after the check they go back to standby. 3 servers however, lost full power after say 30-60 min. I suddenly noticed all lights were gone, even the orange standby light. So I checked all cables, no issues there. PSU light even was green. I had to remove cable for say 10 sec, replug, and then they worked as usual again.

I thought, if it's a one time deal, then weird, but no worries. However today 2 nodes did not boot and again, all power gone and psu's still green and fine... again remove and plugin works. But what is going on here? 

Thanks in advance.

Edit: ilo only says: Server power removed, but psu is green. 

[Vinky_99]

@MarkPV 

Good day!

While it's difficult to pinpoint the exact cause without further investigation, there are a few potential factors that could contribute to this problem. Here are a few troubleshooting steps you can try:

>> Make sure that all power cables are securely connected to the servers and the power source. Verify that there are no loose connections or damaged cables that could cause intermittent power loss.

>> Even though the PSU lights are green, it's possible that one or both of the PSUs are faulty. Consider swapping the PSUs between servers to see if the issue persists. If the problem follows the PSU, it may need to be replaced.

>> Connect the affected servers to a different power outlet or power strip to rule out any issues with the current power source. Sometimes, unstable or faulty power sources can cause unexpected power loss.

>> Ensure that your servers have the latest firmware and driver updates installed. Outdated firmware or drivers can sometimes cause power-related issues. 

>> Check the server's event logs or integrated management logs (iLO, iDRAC, etc.) for any error messages or warnings that might shed light on the power loss issue. Look for any patterns or commonalities between the affected servers.

>> Evaluate the physical environment in which the servers are located. Excessive heat, humidity, or dust can impact the functioning of the hardware components, including the power supply. Ensure proper ventilation and consider cleaning any accumulated dust.

Hope this helps! Let me know... 

[MarkPV]

HI,

First of all, thanks for your time. 

- Cables are triple checked on psu and pdu end.
- Did swap psu's yesterday as first test (so far so good).
- All servers have latest SPP installed. 
- Even logs already checked, only power removed, other then no faults/clues whatsover. 
- Environment is good. 

Thanks for the pointers.. I'm hoping that the psu swap will do the trick, but still think it's weird they ran over a year without a problem and suddenly 3 machines have the same error at once. 

Kind regards,

Mark

[Vinky_99]

Hey Mark, 

It's good to hear that you have already checked and addressed some of the common potential causes for the power loss issue. Swapping the PSUs and ensuring the latest firmware updates are installed are positive steps towards resolving the problem.

It is indeed unusual for multiple servers to develop the same issue simultaneously after running without problems for a long period. While it could be a coincidence, there could be an underlying factor that triggered this behavior.

Here are a couple of additional suggestions you could consider:

>> Think back to any recent changes or updates that might have been made to the servers or the infrastructure. Did you install any new hardware, update firmware, or make any configuration changes? Sometimes, seemingly unrelated modifications can inadvertently lead to unforeseen issues. If you made any changes before the problem occurred, try reverting them or investigating their potential impact.

>>If the issue persists despite the PSU swap and other troubleshooting steps, it may be worthwhile to reach out to HPE support for further assistance.

[ManBha]

Hello,

We need to review the logs from the servers to further isolate the issue if the issue persists after swapping PSU.

Thanks