topic Re: Server crashed ( Reboot after panic ) in Operating System - HP-UX

Server crashed ( Reboot after panic )

manamirhastam — Wed, 14 Nov 2007 01:38:27 GMT

We are using Oracle RAC10g on two rx4640.
One of the servers crashed today.
There is no log in OLDsyslog.log

The content of /etc/shutdownlog :
Reboot after panic: INIT, IIP:0xe0000000014a83d0 IFA:0x64645f656e747279

the output of q4:
q4> trace event 0
stack trace for event 0
crash event was a INIT
Stack unwind reached bottom at depth, 7
send_INIT_monarch+0x70
Send_Monarch_TOC+0x40
kepd_ioctl+0x720
spec_ioctl+0x1d0
vno_ioctl+0x190
ioctl+0x200
syscall+0x4e0
_u_default+0x1246b670

Any idea?

Re: Server crashed ( Reboot after panic )

Sandy Chen — Wed, 14 Nov 2007 02:21:46 GMT

Hi,

Check your cluster quorum, we once have just like that because of heartbeat network intermittent. If I wasn't wrong, if RAC quorum does not meet, it will reboot the server.

Regards,
Sandy

Re: Server crashed ( Reboot after panic )

whiteknight — Wed, 14 Nov 2007 04:20:46 GMT

hi,

it is a ServiceGuard TOC, please check your OLD_syslog.log to know the sequence of events so that you will know the root cause.

WK

Re: Server crashed ( Reboot after panic )

manamirhastam — Wed, 14 Nov 2007 05:40:22 GMT

Hi,
We have not configured HP ServiceGuard because we are using Oracle Clusterware.
The last line of OLD_syslog.log belongs to yesterday.
Do you think any relation with Oracle Clusterware?
Thanks and Regards.

Re: Server crashed ( Reboot after panic )

manamirhastam — Tue, 01 Jan 2008 11:11:20 GMT

We had again crash on server.
The q4 output is:

stack trace for event 0
crash event was a INIT
Stack unwind reached bottom at depth, 5

asio0_intr+0x2a0
sapic_interrupt+0x60
external_interrupt+0x3b0
bubbledown+0x0
idle+0x590
os_rendez_ihandler+0x0

Any idea?

Re: Server crashed ( Reboot after panic )

Steven E. Protter — Tue, 01 Jan 2008 12:04:39 GMT

Shalom,

Perform full q4 analysis.
Send output to HP Response Center for analysis.

This requires a support contract.

ITRC is designed to compliment and not replace a support contract. I don't have the faintest idea of how to process the q4 output.

The HP Response Center does. When I send them the output, I generally get a patch back I can install to correct the problem.

SEP

Re: Server crashed ( Reboot after panic )

NPD USER — Wed, 16 Jan 2008 15:05:44 GMT

Looks like CRS TOC'd the server; Check your CSS log: $CRS_HOME/log/servername/cssd/ocssd.log.

Look for entries that indicated 'missed checkin'. When RAC misses or times out on voting disk or hearbeat, it will TOC the server. Also check the crsd.log on the other nodes for node eviction.