Node gets rebooted once the RAC database

[UNIX_HP]

Hi,
I have Oracle 11g RAC installed on HP-UX 1131
Any help to solve this issue???

##############################################

Stored message buffer up to system crash:

hostname :
model : ia64 hp superdome server SD32B
mem size : 0
panic mesg : Fault when executing in kernel mode
dumptime : 1289888350 Tue Nov 16 07:19:10 MET 2010
savetime : 1289888764 Tue Nov 16 07:26:04 MET 2010
release : @(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf
modules : 17
vhpt : 0xe0000001b0000000
vhpt size : 8388608
page size : 1000 (4096)
[rng] [btlan] [procsm] [c8xx] [cdfs] [igssn] [fcd] [td] [gelan] [iether] [igelan] [lvmp] [nadv] [ciss] [sasd]
[mpt]
msgbuf + 0x8:
Loaded ACPI revision 2.0 tables.
MMIO on this platform supports Write Coalescing.


Using /stand/ext_ioconfig

Memory Class Setup
-------------------------------------------------------------------------
Class Physmem Lockmem Swapmem
-------------------------------------------------------------------------
System : 31040 MB 31040 MB 31040 MB
Kernel : 31040 MB 31040 MB 31040 MB
User : 29697 MB 26327 MB 26430 MB
-------------------------------------------------------------------------

ktracer is off until requested.
Installing Socket Protocol families AF_INET and AF_INET6
64000/0xfa00 esvroot
Kernel EVM initialized
sec_init(): kernel RPC authentication/security initialization.
secgss_init(): kernel RPCSEC_GSS security initialization.
rpc_init(): kernel RPC initialization.
rpcmod_install(): kernel RPC STREAMS module "rpcmod" installation. ...(driver_install)
NOTICE: nfs_client_pv3_install(): nfs3 File system was registered at index 10.
NOTICE: nfs_client_pv4_install(): nfs4 File system was registered at index 11.
NOTICE: cachefsc_install: cachefs File system was registered at index 13.
btlan_load() Loaded Successfully
0 cell
0/0 sba
0/0/0 lba
0/0/0/1 pci_slot
0/0/1 lba
0/0/1/1 pci_slot
0/0/1/1/0 PCItoPCI
Initializing the Ultra320 SCSI Controller at 0/0/1/1/0/4/0. Controller firmware version is 01.03.35.70
0/0/1/1/0/4/0 mpt
0/0/1/1/0/4/0.0x0 estp
Initializing the Ultra320 SCSI Controller at 0/0/1/1/0/4/1. Controller firmware version is 01.03.35.70
0/0/1/1/0/4/1 mpt
0/0/1/1/0/6/0 iether
0/0/1/1/0/6/1 iether
0/0/2 lba
0/0/2/1 pci_slot
0/0/2/1/0 PCItoPCI
0/0/2/1/0/4/0 iether
0/0/2/1/0/4/1 iether
0/0/2/1/0/6/0 iether
0/0/2/1/0/6/1 iether
0/0/4 lba
0/0/4/1 pci_slot
0/0/5 lba
0/0/5/1 pci_slot
0/0/6 lba
0/0/6/1 pci_slot
0/0/6/1/0 PCItoPCI
fcd: Claimed HP AD194-60001 4Gb Fibre Channel port at hardware path 0/0/6/1/0/4/0 (FC Port 1 on HBA)
0/0/6/1/0/4/0 fcd
fcd: Claimed HP AD194-60001 4Gb Fibre Channel port at hardware path 0/0/6/1/0/4/1 (FC Port 2 on HBA)
0/0/6/1/0/4/1 fcd
0/0/6/1/0/6/0 iether
0/0/6/1/0/6/1 iether
0/0/8 lba
0/0/8/1 pci_slot
0/0/9 lba
0/0/9/1 pci_slot
0/0/10 lba
0/0/10/1 pci_slot
0/0/10/1/0 PCItoPCI
0/0/10/1/0/4/0 iether
0/0/10/1/0/4/1 iether
0/0/10/1/0/6/0 iether
0/0/10/1/0/6/1 iether
0/0/12 lba
0/0/12/1 pci_slot
0/0/12/1/0 PCItoPCI
Initializing the Ultra320 SCSI Controller at 0/0/12/1/0/4/0. Controller firmware version is 01.03.35.70
0/0/12/1/0/4/0 mpt
0/0/12/1/0/4/0.0x0 estp
Initializing the Ultra320 SCSI Controller at 0/0/12/1/0/4/1. Controller firmware version is 01.03.35.70
0/0/1/1/0/4/0.0x0.0x0 eslpt
0/0/1/1/0/4/0.0 tgt
0/0/1/1/0/4/0.0.0 sdisk
0/0/12/1/0/4/1 mpt
0/0/12/1/0/6/0 iether
0/0/12/1/0/6/1 iether
0/0/13 lba
0/0/13/1 pci_slot
0/0/14 lba
0/0/14/1 pci_slot
0/0/14/1/0 PCItoPCI
fcd: Claimed HP AD194-60001 4Gb Fibre Channel port at hardware path 0/0/14/1/0/4/0 (FC Port 1 on HBA)
0/0/14/1/0/4/0 fcd
fcd: Claimed HP AD194-60001 4Gb Fibre Channel port at hardware path 0/0/14/1/0/4/1 (FC Port 2 on HBA)
0/0/14/1/0/4/1 fcd
0/0/14/1/0/6/0 iether
0/0/14/1/0/6/1 iether
0/120 processor
0/121 processor
0/122 processor
0/123 processor
0/124 processor
0/125 processor
0/250 pdh
0/250/0 acpi_node
0/250/1 ipmi
0/250/2 asio0
2 cell
2/120 processor
2/121 processor
2/122 processor
2/123 processor
2/124 processor
2/125 processor
2/250 pdh
2/250/0 acpi_node
0/0/6/1/0/4/0.0x5005076308000129 estp
0/0/6/1/0/4/0.0x5005076308080129 estp
0/0/6/1/0/4/0.0x5005076308080129.0x0 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x0 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4000000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4000000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4002000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4001000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4004000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4001000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4100000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4500000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4400000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4002000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4004000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4100000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4501000000000000 eslpt
0/0/6/1/0/4/0.1 fcd_fcp
0/0/6/1/0/4/0.1.80.255.0 fcd_vbus
0/0/6/1/0/4/0.1.80.255.0.0 tgt
0/0/6/1/0/4/0.1.80.255.0.0.0 sctl
0/0/6/1/0/4/0.1.80.0.0 fcd_vbus
0/0/6/1/0/4/0.1.80.0.0.0 tgt
0/0/6/1/0/4/0.1.80.0.0.0.0 sdisk
0/0/6/1/0/4/0.1.80.0.0.0.1 sdisk
0/0/6/1/0/4/0.1.80.0.0.0.2 sdisk
0/0/6/1/0/4/0.1.80.0.0.0.4 sdisk
0/0/6/1/0/4/0.1.80.0.2 fcd_vbus
0/0/6/1/0/4/0.1.80.0.2.0 tgt
0/0/6/1/0/4/0.1.80.0.2.0.0 sdisk
0/0/6/1/0/4/0.1.80.0.8 fcd_vbus
0/0/6/1/0/4/0.1.80.0.8.0 tgt
0/0/6/1/0/4/0.1.80.0.8.0.0 sdisk
0/0/6/1/0/4/0.1.80.0.10 fcd_vbus
0/0/6/1/0/4/0.1.80.0.10.0 tgt
0/0/6/1/0/4/0.1.80.0.10.0.0 sdisk
0/0/6/1/0/4/0.1.80.0.10.0.1 sdisk
0/0/6/1/0/4/0.0x5005076308080129.0x4400000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4500000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4501000000000000 eslpt
0/0/6/1/0/4/0.1.47.255.0 fcd_vbus
0/0/6/1/0/4/0.1.47.255.0.0 tgt
0/0/6/1/0/4/0.1.47.255.0.0.0 sctl
0/0/6/1/0/4/0.1.47.0.0 fcd_vbus
0/0/6/1/0/4/0.1.47.0.0.0 tgt
0/0/6/1/0/4/0.1.47.0.0.0.0 sdisk
0/0/6/1/0/4/0.1.47.0.0.0.1 sdisk
0/0/6/1/0/4/0.1.47.0.0.0.2 sdisk
0/0/6/1/0/4/0.1.47.0.0.0.4 sdisk
0/0/6/1/0/4/0.1.47.0.2 fcd_vbus
0/0/6/1/0/4/0.1.47.0.2.0 tgt
0/0/6/1/0/4/0.1.47.0.2.0.0 sdisk
0/0/6/1/0/4/0.1.47.0.8 fcd_vbus
0/0/6/1/0/4/0.1.47.0.8.0 tgt
0/0/6/1/0/4/0.1.47.0.8.0.0 sdisk
0/0/6/1/0/4/0.1.47.0.10 fcd_vbus
0/0/6/1/0/4/0.1.47.0.10.0 tgt
0/0/6/1/0/4/0.1.47.0.10.0.0 sdisk
0/0/6/1/0/4/0.1.47.0.10.0.1 sdisk
0/0/12/1/0/4/0.0x0.0x0 eslpt
0/0/12/1/0/4/0.0 tgt
0/0/12/1/0/4/0.0.0 sdisk
0/0/14/1/0/4/0.0x5005076308100129 estp
0/0/14/1/0/4/0.0x5005076308180129 estp
0/0/12/1/0/4/1.0x0 estp
0/0/12/1/0/4/1.0x2 estp
0/0/12/1/0/4/1.0x0.0x0 eslpt
0/0/12/1/0/4/1.0 tgt
0/0/12/1/0/4/1.0.0 stape
0/0/12/1/0/4/1.0x2.0x0 eslpt
0/0/12/1/0/4/1.2 tgt
0/0/12/1/0/4/1.2.0 sdisk
0/0/14/1/0/4/0.0x5005076308180129.0x0 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x0 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4000000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4000000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4001000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4100000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4002000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4500000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4004000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4400000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4501000000000000 eslpt
0/0/14/1/0/4/0.1 fcd_fcp
0/0/14/1/0/4/0.1.160.255.0 fcd_vbus
0/0/14/1/0/4/0.1.160.255.0.0 tgt
0/0/14/1/0/4/0.1.160.255.0.0.0 sctl
0/0/14/1/0/4/0.1.160.0.0 fcd_vbus
0/0/14/1/0/4/0.1.160.0.0.0 tgt
0/0/14/1/0/4/0.1.160.0.0.0.0 sdisk
0/0/14/1/0/4/0.1.160.0.0.0.1 sdisk
0/0/14/1/0/4/0.1.160.0.0.0.2 sdisk
0/0/14/1/0/4/0.1.160.0.0.0.4 sdisk
0/0/14/1/0/4/0.1.160.0.2 fcd_vbus
0/0/14/1/0/4/0.1.160.0.2.0 tgt
0/0/14/1/0/4/0.1.160.0.2.0.0 sdisk
0/0/14/1/0/4/0.1.160.0.8 fcd_vbus
0/0/14/1/0/4/0.1.160.0.8.0 tgt
0/0/14/1/0/4/0.1.160.0.8.0.0 sdisk
0/0/14/1/0/4/0.1.160.0.10 fcd_vbus
0/0/14/1/0/4/0.1.160.0.10.0 tgt
0/0/14/1/0/4/0.1.160.0.10.0.0 sdisk
0/0/14/1/0/4/0.1.160.0.10.0.1 sdisk
0/0/14/1/0/4/0.0x5005076308100129.0x4001000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4002000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4004000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4400000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4100000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4500000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4501000000000000 eslpt
0/0/14/1/0/4/0.1.80.255.0 fcd_vbus
0/0/14/1/0/4/0.1.80.255.0.0 tgt
0/0/14/1/0/4/0.1.80.255.0.0.0 sctl
0/0/14/1/0/4/0.1.80.0.0 fcd_vbus
0/0/14/1/0/4/0.1.80.0.0.0 tgt
0/0/14/1/0/4/0.1.80.0.0.0.0 sdisk
0/0/14/1/0/4/0.1.80.0.0.0.1 sdisk
0/0/14/1/0/4/0.1.80.0.0.0.2 sdisk
0/0/14/1/0/4/0.1.80.0.0.0.4 sdisk
0/0/14/1/0/4/0.1.80.0.2 fcd_vbus
0/0/14/1/0/4/0.1.80.0.2.0 tgt
0/0/14/1/0/4/0.1.80.0.2.0.0 sdisk
0/0/14/1/0/4/0.1.80.0.8 fcd_vbus
0/0/14/1/0/4/0.1.80.0.8.0 tgt
0/0/14/1/0/4/0.1.80.0.8.0.0 sdisk
0/0/14/1/0/4/0.1.80.0.10 fcd_vbus
0/0/14/1/0/4/0.1.80.0.10.0 tgt
0/0/14/1/0/4/0.1.80.0.10.0.0 sdisk
0/0/14/1/0/4/0.1.80.0.10.0.1 sdisk
64000/0xfa00/0x0 esdisk
64000/0xfa00/0x1 esdisk
64000/0xfa00/0x2 estape
64000/0xfa00/0x3 esdisk
64000/0xfa00/0xd esctl
64000/0xfa00/0xe esdisk
64000/0xfa00/0xf esdisk
64000/0xfa00/0x10 esdisk
64000/0xfa00/0x11 esdisk
64000/0xfa00/0x12 esdisk
64000/0xfa00/0x13 esdisk
64000/0xfa00/0x14 esdisk
64000/0xfa00/0x15 esdisk
Boot device's HP-UX HW path is: 0.0.1.1.0.4.0.0.0
iether0: INITIALIZING HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter at hardware path 0/0/1/1/0/6/0
iether1: INITIALIZING HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter at hardware path 0/0/1/1/0/6/1
iether8: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/2/1/0/4/0
iether9: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/2/1/0/4/1
iether10: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/2/1/0/6/0
iether11: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/2/1/0/6/1
iether2: INITIALIZING HP AD194-60001 PCI/PCI-X 1000Base-T 2-port 4Gb FC/2-port 1000B-T Combo Adapter at hardware path 0/0/6/1/0/6/0
iether3: INITIALIZING HP AD194-60001 PCI/PCI-X 1000Base-T 2-port 4Gb FC/2-port 1000B-T Combo Adapter at hardware path 0/0/6/1/0/6/1
iether12: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/10/1/0/4/0
iether13: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/10/1/0/4/1
iether14: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/10/1/0/6/0
iether15: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/10/1/0/6/1
iether4: INITIALIZING HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter at hardware path 0/0/12/1/0/6/0
iether5: INITIALIZING HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter at hardware path 0/0/12/1/0/6/1
iether6: INITIALIZING HP AD194-60001 PCI/PCI-X 1000Base-T 2-port 4Gb FC/2-port 1000B-T Combo Adapter at hardware path 0/0/14/1/0/6/0
iether7: INITIALIZING HP AD194-60001 PCI/PCI-X 1000Base-T 2-port 4Gb FC/2-port 1000B-T Combo Adapter at hardware path 0/0/14/1/0/6/1

System Console is on the Built-In Serial Interface
AF_INET socket/streams output daemon running, pid 293
afinet_prelink: module installed
Starting the STREAMS daemons-phase 1
LVM: Root VG activated
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 67108864
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0000000000000000 - major is 3, minor is 0x0; start = 1497952, size = 33554432
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: B.11.31_LR FLAVOR=perf
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 33417628 Kbytes, lockable: 24349744 Kbytes, available: 27819924 Kbytes

bad_kern_reference: 0x8008031.0x8, fault = 0x53

panic: Fault when executing in kernel mode
Stack Trace:
IP Function Name
0xe000000001e56a80 bad_kern_reference+0xa0
0xe00000000073d860 $cold_pfault+0x270
0xe000000000c00060 vm_hndlr+0xaa0
0xe000000001ab9780 bubbledown+0x0
0xe000000000b835c1 tcp_wput+0x23e1
0xe000000000b79e40 streams_putmsg_fc+0x280
0xe000000000b795e0 inet_push+0x140
0xe000000000b674e0 inet_stream_send+0x400
0xe000000000b714b0 sosend_default+0x5c0
0xe000000000bf2db0 soo_rw+0x110
0xe000000000c1a740 write+0x420
0xe000000000ba75f0 syscall+0x590
End of Stack Trace

linkstamp: Wed Oct 14 15:22:22 METDST 2009
_release_version: @(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf
Calling function e00000000154f7e0 for Shutdown State 1 type 0x2

sync'ing disks (0 buffers to flush):
0 fcache pages still dirty
0 buffers not flushed
0 buffers still dirty

[Shibin_2]

Do you have any backup of old kernel ( good condition )? If available try to boot from there. Then compare the kernel configuration with the one giving problem.

[UNIX_HP]

We have no backup of the kernel. From this reference bad_kern_reference: 0x8008031.0x8, fault = 0x53, you can know what is the problem?

[Shibin_2]

If you have support agreement with HP, contact HP Response Center.

Looks like you are in deadlock situation.

[Hein van den Heuvel]

For the benefit of others, and to help create a better case for support, should you need to go there, please describe the timeline of events a little more clear?
What worked,
What command or event triggered failure?
What we the end state (the crash above).

fwiw,
Hein