DL380G7 HP ilo3 "server status: RESET"

[NSC8]

Hi,
we have an HP DL380G7 (Bios P67 01/30/2011), with ilo3 (1.16 12/17/2010).

We tried to reboot the server from linux using fence_ipmilan (RH Cluster suite 5.5).

The server didn't reboot, and now i have no display, i can access the HP Ilo3 server page, but i can't power on it.

On the Power Management\Server Power item of ilo3 webpage i have only one Virtual Power Button: "Momentary Press" (which didn't work).

The server Status is "Server is currently RESET" ... since 7 hours.

Is there a method to boot up the server ?
(I don't have any PDU ad this time, and server is on a remote Datacenter).

Greetings.

[NSC8]

I have updated ilo3 firmware to 1.20:

I tried powering on/off using SSH cli on ilo interface:

hpiLO-> power
power: server is currently: Off

hpiLO-> power on
power: Server powering on .......

hpiLO-> power off hard
Forcing server power off .......

hpiLO-> power on
power: Server powering on .......

hpiLO-> power
power: server is currently: Off

Any hints ?

[Oscar A. Perez]

Remove power cord for about 10 seconds. If this happens again, contact HP support. You might need a system board replacement.

[RyanTMcBride]

Just to provide another datapoint, I also have a DL380G7 (BIOS: P67 01/30/2011, ILO3: 1.20) with the same problem, which has now occurred twice. Restarting the ILO has no effect.

I've only seen it on box of many that we operate, but I'd really like to know if this is a general problem with all G7-series servers (in which case we would need to to be very cautious about remote server reboots), or just a hardware flaw in this one.

[Oscar A. Perez]

I've been working on a very similar issue on the DL380 G6 and this is the first time I learn that the DL380 G7 could be showing the same symptom.

This is what I've seen in the lab. If I instruct Linux to restart the system, the system goes for a reboot as normal but, if early during system initialization and that the right second, I issue a server "power off" command from iLO then, the server might get into this state.

I know that the fence_ipmilan uses iLO to shutdown the server. I'm wondering if you guys are seeing this issue when the fence_ipmilan command to power off the server is issued when the server has just been rebooted.

[Oscar A. Perez]

Could you guys please confirm that the issue is seen only on few of the DL380 G7 systems you have?

If so, please call HP support, log a case and then post the case number here.

Thanks

[NSC8]

Hi, finally it seems that it was the power supply that defect.

After changing the PSU, fence_ipmilan "seems to work" it's a production cluster so i can't really do more tests.

Actually i use fence_ipmilan with those parameters:
method=cycle, powerwait=4, timeout=20.

Greetings.

[NSC8]

Hi, finally i do some more test today with RedHat ClusterSuite and the use of fence_ipmilan.

I have tried method=onoff instead of method=cycle.

I can't achieve a correct reboot with DL380g7.
The server was in state power on (ssh'ing to ilo to check it), but it does'nt boot, i didnt see the HP BIOS screen, after some "power off hard", and "power on", it successfuly boot.

Is it a place where HP provides the right configuration i must use with fence_ipmilan and HP ILO3 ?

Greetings.

[Oscar A. Perez]

What iLO3 firmware version are you using?

We recently released version 1.25 which fixes an issue where using fence_ipmilan to power off the server could result in a server reboot.
The root cause of this issue is that the fence_ipmilan violates the IPMI specs and doesn't wait 6 seconds before retrying the last power on/off IPMI command.
If the server doesn't power off within a couple of seconds, fence_ipmilan could repeat the last power off command and iLO3 could think the second command is to power the server back on. We added a workaround in 1.25 to filter out these out of specs retries.

[NSC8]

Hi, we are running HP ilo firmware version 1.20 (14 Mar 2011).
We encouter the same problem with previous firmware 1.16.

I use a timeout of 20seconds for fence_ipmilan.

The problem is that the server does'nt reboot, it is stalled (no bios logo or boot process) whereas ilo report it as powered on.

[Oscar A. Perez]

Does this issue happen on few of your systems or it happens on all of them?

Please try rebooting iLO3 from the webGUI. If rebooting iLO3 doesn't change anything then, this is the issue you mentioned in your first post where the server is stuck in Reset and I'm going to need you to log a case number with HP support.

With the case number, I can get that system board captured.

[NSC8]

Hi this append on our 2 New HP DL380G7 servers.

The server status RESET message have been fixed by replacing our PSU.

But now we still have the problem of the server that does'nt reboot after beeing fenced.

If i ssh to the ilo interface of the server the power status is marked as "On".

The only way to reboot it is to do a "power off hard" and a "power on"

[Jimmy Vance]

Update your iLO3 firmware to v1.25. I was seeing some issues with fence_ipmilan and v1.25 appears to resolve the problems.