topic Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM" in ProLiant Servers (ML,DL,SL)

30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

David_746 — Wed, 01 Feb 2006 07:18:07 GMT

We had a problem this morning where nearly 30 servers rebooted at the same time. The event log showed an unexpected shutdown and the HP integrated log states "ASR Detected by System ROM".
All the servers are at least version 7.30 of the management agents and are dl380s or 360s G3 and above.
We did not have any power etc outages and the servers are split between two sites. There were other servers based on the sites that did not have any problems.

I did see something relating to the survey utility causing problems as this runs at 6am by default but the advisory stated it was for servers with 40+ disks attached. (EA04028-CW01) and we have never had this before.

The survey utility seems to be the only common feature?

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Mario_66 — Wed, 01 Feb 2006 10:05:35 GMT

Hello,

have you checked ASR timeout?

M.

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

David_746 — Wed, 01 Feb 2006 11:50:55 GMT

The time outs are all at 10 minutes.

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

SPa — Thu, 02 Feb 2006 03:52:54 GMT

Automatic Server Recovery is initiated when the Server senses an unresponsiveness.There could be Software reasons for the ASR and as 30 Servers simultaneously rebooted it is a remote possibility of any Hardware issue on the server.You may want to check the Event Logs to see if any activity which may be common to all the servers was happening like backup or an application performing some task.

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

David_746 — Thu, 02 Feb 2006 05:10:49 GMT

There is nothing preceeding the unscheduled shutdown in the event logs and in some cases there the previous event log is some hours beforehand. There were no updates or backups etc running at that time and the only common thing was the survey utiity running at that time.

Cheers

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Prashant (I am Back) — Thu, 02 Feb 2006 12:16:52 GMT

Hi,

What is the polling time for the survey utility. Check the time.
open surey.txt.
It will tell you.

Regards,
Prashant S.

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Prashant (I am Back) — Thu, 02 Feb 2006 12:19:28 GMT

Hi,

Check this out.
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=565419

May be relevent.

Regards,
Prashant S.

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

David_746 — Thu, 02 Feb 2006 12:47:04 GMT

Its the standard 7days for the survey utility. I saw that document and I think we are just going to remove the survey utiltiy as it serves no real purpose anymore.
It just would be nice to find the root cause.

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Prashant (I am Back) — Thu, 02 Feb 2006 12:56:12 GMT

Hi,

Disable ASR atleast in one two server.
To check the root cause, if it reboot again.

Prashant S.

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Prashant (I am Back) — Mon, 06 Feb 2006 10:26:29 GMT

Hi David,

Any updates ?

Regards,
Prashant S.

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

David_746 — Mon, 06 Feb 2006 11:26:12 GMT

Hi there
This has not happened again but we will be unistalling the survey utility, just seems very odd that something caused so many to reboot.