ProLiant Servers (ML,DL,SL)
1834650 Members
2149 Online
110069 Solutions
New Discussion

30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

 
David_746
Advisor

30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

We had a problem this morning where nearly 30 servers rebooted at the same time. The event log showed an unexpected shutdown and the HP integrated log states "ASR Detected by System ROM".
All the servers are at least version 7.30 of the management agents and are dl380s or 360s G3 and above.
We did not have any power etc outages and the servers are split between two sites. There were other servers based on the sites that did not have any problems.

I did see something relating to the survey utility causing problems as this runs at 6am by default but the advisory stated it was for servers with 40+ disks attached. (EA04028-CW01) and we have never had this before.

The survey utility seems to be the only common feature?
10 REPLIES 10
Mario_66
Valued Contributor

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Hello,

have you checked ASR timeout?

M.
David_746
Advisor

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

The time outs are all at 10 minutes.
SPa
Trusted Contributor

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Automatic Server Recovery is initiated when the Server senses an unresponsiveness.There could be Software reasons for the ASR and as 30 Servers simultaneously rebooted it is a remote possibility of any Hardware issue on the server.You may want to check the Event Logs to see if any activity which may be common to all the servers was happening like backup or an application performing some task.
David_746
Advisor

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

There is nothing preceeding the unscheduled shutdown in the event logs and in some cases there the previous event log is some hours beforehand. There were no updates or backups etc running at that time and the only common thing was the survey utiity running at that time.

Cheers


Prashant (I am Back)
Honored Contributor

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Hi,

What is the polling time for the survey utility. Check the time.
open surey.txt.
It will tell you.

Regards,
Prashant S.
Nothing is impossible
Prashant (I am Back)
Honored Contributor

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Hi,

Check this out.
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=565419

May be relevent.

Regards,
Prashant S.
Nothing is impossible
David_746
Advisor

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Its the standard 7days for the survey utility. I saw that document and I think we are just going to remove the survey utiltiy as it serves no real purpose anymore.
It just would be nice to find the root cause.
Prashant (I am Back)
Honored Contributor

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Hi,

Disable ASR atleast in one two server.
To check the root cause, if it reboot again.

Prashant S.

Nothing is impossible
Prashant (I am Back)
Honored Contributor

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Hi David,

Any updates ?

Regards,
Prashant S.
Nothing is impossible
David_746
Advisor

Re: 30 servers simultaneously rebooting approx 6am with "ASR Detected by System ROM"

Hi there
This has not happened again but we will be unistalling the survey utility, just seems very odd that something caused so many to reboot.