Server crashed ( Reboot after panic )

[manamirhastam]

We are using Oracle RAC10g on two rx4640.
One of the servers crashed today.
There is no log in OLDsyslog.log

The content of /etc/shutdownlog :
Reboot after panic: INIT, IIP:0xe0000000014a83d0 IFA:0x64645f656e747279

the output of q4:
q4> trace event 0
stack trace for event 0
crash event was a INIT
Stack unwind reached bottom at depth, 7
send_INIT_monarch+0x70
Send_Monarch_TOC+0x40
kepd_ioctl+0x720
spec_ioctl+0x1d0
vno_ioctl+0x190
ioctl+0x200
syscall+0x4e0
_u_default+0x1246b670

Any idea?

[Sandy Chen]

Hi,

Check your cluster quorum, we once have just like that because of heartbeat network intermittent. If I wasn't wrong, if RAC quorum does not meet, it will reboot the server.

Regards,
Sandy

[whiteknight]

hi,


it is a ServiceGuard TOC, please check your OLD_syslog.log to know the sequence of events so that you will know the root cause.


WK

[manamirhastam]

Hi,
We have not configured HP ServiceGuard because we are using Oracle Clusterware.
The last line of OLD_syslog.log belongs to yesterday.
Do you think any relation with Oracle Clusterware?
Thanks and Regards.

[manamirhastam]

We had again crash on server.
The q4 output is:

stack trace for event 0
crash event was a INIT
Stack unwind reached bottom at depth, 5

asio0_intr+0x2a0
sapic_interrupt+0x60
external_interrupt+0x3b0
bubbledown+0x0
idle+0x590
os_rendez_ihandler+0x0

Any idea?

[Steven E. Protter]

Shalom,

Perform full q4 analysis.
Send output to HP Response Center for analysis.

This requires a support contract.

ITRC is designed to compliment and not replace a support contract. I don't have the faintest idea of how to process the q4 output.

The HP Response Center does. When I send them the output, I generally get a patch back I can install to correct the problem.

SEP

[NPD USER]

Looks like CRS TOC'd the server; Check your CSS log: $CRS_HOME/log/servername/cssd/ocssd.log.

Look for entries that indicated 'missed checkin'. When RAC misses or times out on voting disk or hearbeat, it will TOC the server. Also check the crsd.log on the other nodes for node eviction.