System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Node gets rebooted once the RAC database

 
UNIX_HP
Advisor

Node gets rebooted once the RAC database

Hi,
I have Oracle 11g RAC installed on HP-UX 1131
Any help to solve this issue???

##############################################

Stored message buffer up to system crash:

hostname :
model : ia64 hp superdome server SD32B
mem size : 0
panic mesg : Fault when executing in kernel mode
dumptime : 1289888350 Tue Nov 16 07:19:10 MET 2010
savetime : 1289888764 Tue Nov 16 07:26:04 MET 2010
release : @(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf
modules : 17
vhpt : 0xe0000001b0000000
vhpt size : 8388608
page size : 1000 (4096)
[rng] [btlan] [procsm] [c8xx] [cdfs] [igssn] [fcd] [td] [gelan] [iether] [igelan] [lvmp] [nadv] [ciss] [sasd]
[mpt]
msgbuf + 0x8:
Loaded ACPI revision 2.0 tables.
MMIO on this platform supports Write Coalescing.


Using /stand/ext_ioconfig

Memory Class Setup
-------------------------------------------------------------------------
Class Physmem Lockmem Swapmem
-------------------------------------------------------------------------
System : 31040 MB 31040 MB 31040 MB
Kernel : 31040 MB 31040 MB 31040 MB
User : 29697 MB 26327 MB 26430 MB
-------------------------------------------------------------------------

ktracer is off until requested.
Installing Socket Protocol families AF_INET and AF_INET6
64000/0xfa00 esvroot
Kernel EVM initialized
sec_init(): kernel RPC authentication/security initialization.
secgss_init(): kernel RPCSEC_GSS security initialization.
rpc_init(): kernel RPC initialization.
rpcmod_install(): kernel RPC STREAMS module "rpcmod" installation. ...(driver_install)
NOTICE: nfs_client_pv3_install(): nfs3 File system was registered at index 10.
NOTICE: nfs_client_pv4_install(): nfs4 File system was registered at index 11.
NOTICE: cachefsc_install: cachefs File system was registered at index 13.
btlan_load() Loaded Successfully
0 cell
0/0 sba
0/0/0 lba
0/0/0/1 pci_slot
0/0/1 lba
0/0/1/1 pci_slot
0/0/1/1/0 PCItoPCI
Initializing the Ultra320 SCSI Controller at 0/0/1/1/0/4/0. Controller firmware version is 01.03.35.70
0/0/1/1/0/4/0 mpt
0/0/1/1/0/4/0.0x0 estp
Initializing the Ultra320 SCSI Controller at 0/0/1/1/0/4/1. Controller firmware version is 01.03.35.70
0/0/1/1/0/4/1 mpt
0/0/1/1/0/6/0 iether
0/0/1/1/0/6/1 iether
0/0/2 lba
0/0/2/1 pci_slot
0/0/2/1/0 PCItoPCI
0/0/2/1/0/4/0 iether
0/0/2/1/0/4/1 iether
0/0/2/1/0/6/0 iether
0/0/2/1/0/6/1 iether
0/0/4 lba
0/0/4/1 pci_slot
0/0/5 lba
0/0/5/1 pci_slot
0/0/6 lba
0/0/6/1 pci_slot
0/0/6/1/0 PCItoPCI
fcd: Claimed HP AD194-60001 4Gb Fibre Channel port at hardware path 0/0/6/1/0/4/0 (FC Port 1 on HBA)
0/0/6/1/0/4/0 fcd
fcd: Claimed HP AD194-60001 4Gb Fibre Channel port at hardware path 0/0/6/1/0/4/1 (FC Port 2 on HBA)
0/0/6/1/0/4/1 fcd
0/0/6/1/0/6/0 iether
0/0/6/1/0/6/1 iether
0/0/8 lba
0/0/8/1 pci_slot
0/0/9 lba
0/0/9/1 pci_slot
0/0/10 lba
0/0/10/1 pci_slot
0/0/10/1/0 PCItoPCI
0/0/10/1/0/4/0 iether
0/0/10/1/0/4/1 iether
0/0/10/1/0/6/0 iether
0/0/10/1/0/6/1 iether
0/0/12 lba
0/0/12/1 pci_slot
0/0/12/1/0 PCItoPCI
Initializing the Ultra320 SCSI Controller at 0/0/12/1/0/4/0. Controller firmware version is 01.03.35.70
0/0/12/1/0/4/0 mpt
0/0/12/1/0/4/0.0x0 estp
Initializing the Ultra320 SCSI Controller at 0/0/12/1/0/4/1. Controller firmware version is 01.03.35.70
0/0/1/1/0/4/0.0x0.0x0 eslpt
0/0/1/1/0/4/0.0 tgt
0/0/1/1/0/4/0.0.0 sdisk
0/0/12/1/0/4/1 mpt
0/0/12/1/0/6/0 iether
0/0/12/1/0/6/1 iether
0/0/13 lba
0/0/13/1 pci_slot
0/0/14 lba
0/0/14/1 pci_slot
0/0/14/1/0 PCItoPCI
fcd: Claimed HP AD194-60001 4Gb Fibre Channel port at hardware path 0/0/14/1/0/4/0 (FC Port 1 on HBA)
0/0/14/1/0/4/0 fcd
fcd: Claimed HP AD194-60001 4Gb Fibre Channel port at hardware path 0/0/14/1/0/4/1 (FC Port 2 on HBA)
0/0/14/1/0/4/1 fcd
0/0/14/1/0/6/0 iether
0/0/14/1/0/6/1 iether
0/120 processor
0/121 processor
0/122 processor
0/123 processor
0/124 processor
0/125 processor
0/250 pdh
0/250/0 acpi_node
0/250/1 ipmi
0/250/2 asio0
2 cell
2/120 processor
2/121 processor
2/122 processor
2/123 processor
2/124 processor
2/125 processor
2/250 pdh
2/250/0 acpi_node
0/0/6/1/0/4/0.0x5005076308000129 estp
0/0/6/1/0/4/0.0x5005076308080129 estp
0/0/6/1/0/4/0.0x5005076308080129.0x0 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x0 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4000000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4000000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4002000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4001000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4004000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4001000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4100000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4500000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4400000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4002000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4004000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4100000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308000129.0x4501000000000000 eslpt
0/0/6/1/0/4/0.1 fcd_fcp
0/0/6/1/0/4/0.1.80.255.0 fcd_vbus
0/0/6/1/0/4/0.1.80.255.0.0 tgt
0/0/6/1/0/4/0.1.80.255.0.0.0 sctl
0/0/6/1/0/4/0.1.80.0.0 fcd_vbus
0/0/6/1/0/4/0.1.80.0.0.0 tgt
0/0/6/1/0/4/0.1.80.0.0.0.0 sdisk
0/0/6/1/0/4/0.1.80.0.0.0.1 sdisk
0/0/6/1/0/4/0.1.80.0.0.0.2 sdisk
0/0/6/1/0/4/0.1.80.0.0.0.4 sdisk
0/0/6/1/0/4/0.1.80.0.2 fcd_vbus
0/0/6/1/0/4/0.1.80.0.2.0 tgt
0/0/6/1/0/4/0.1.80.0.2.0.0 sdisk
0/0/6/1/0/4/0.1.80.0.8 fcd_vbus
0/0/6/1/0/4/0.1.80.0.8.0 tgt
0/0/6/1/0/4/0.1.80.0.8.0.0 sdisk
0/0/6/1/0/4/0.1.80.0.10 fcd_vbus
0/0/6/1/0/4/0.1.80.0.10.0 tgt
0/0/6/1/0/4/0.1.80.0.10.0.0 sdisk
0/0/6/1/0/4/0.1.80.0.10.0.1 sdisk
0/0/6/1/0/4/0.0x5005076308080129.0x4400000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4500000000000000 eslpt
0/0/6/1/0/4/0.0x5005076308080129.0x4501000000000000 eslpt
0/0/6/1/0/4/0.1.47.255.0 fcd_vbus
0/0/6/1/0/4/0.1.47.255.0.0 tgt
0/0/6/1/0/4/0.1.47.255.0.0.0 sctl
0/0/6/1/0/4/0.1.47.0.0 fcd_vbus
0/0/6/1/0/4/0.1.47.0.0.0 tgt
0/0/6/1/0/4/0.1.47.0.0.0.0 sdisk
0/0/6/1/0/4/0.1.47.0.0.0.1 sdisk
0/0/6/1/0/4/0.1.47.0.0.0.2 sdisk
0/0/6/1/0/4/0.1.47.0.0.0.4 sdisk
0/0/6/1/0/4/0.1.47.0.2 fcd_vbus
0/0/6/1/0/4/0.1.47.0.2.0 tgt
0/0/6/1/0/4/0.1.47.0.2.0.0 sdisk
0/0/6/1/0/4/0.1.47.0.8 fcd_vbus
0/0/6/1/0/4/0.1.47.0.8.0 tgt
0/0/6/1/0/4/0.1.47.0.8.0.0 sdisk
0/0/6/1/0/4/0.1.47.0.10 fcd_vbus
0/0/6/1/0/4/0.1.47.0.10.0 tgt
0/0/6/1/0/4/0.1.47.0.10.0.0 sdisk
0/0/6/1/0/4/0.1.47.0.10.0.1 sdisk
0/0/12/1/0/4/0.0x0.0x0 eslpt
0/0/12/1/0/4/0.0 tgt
0/0/12/1/0/4/0.0.0 sdisk
0/0/14/1/0/4/0.0x5005076308100129 estp
0/0/14/1/0/4/0.0x5005076308180129 estp
0/0/12/1/0/4/1.0x0 estp
0/0/12/1/0/4/1.0x2 estp
0/0/12/1/0/4/1.0x0.0x0 eslpt
0/0/12/1/0/4/1.0 tgt
0/0/12/1/0/4/1.0.0 stape
0/0/12/1/0/4/1.0x2.0x0 eslpt
0/0/12/1/0/4/1.2 tgt
0/0/12/1/0/4/1.2.0 sdisk
0/0/14/1/0/4/0.0x5005076308180129.0x0 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x0 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4000000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4000000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4001000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4100000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4002000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4500000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4004000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4400000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308180129.0x4501000000000000 eslpt
0/0/14/1/0/4/0.1 fcd_fcp
0/0/14/1/0/4/0.1.160.255.0 fcd_vbus
0/0/14/1/0/4/0.1.160.255.0.0 tgt
0/0/14/1/0/4/0.1.160.255.0.0.0 sctl
0/0/14/1/0/4/0.1.160.0.0 fcd_vbus
0/0/14/1/0/4/0.1.160.0.0.0 tgt
0/0/14/1/0/4/0.1.160.0.0.0.0 sdisk
0/0/14/1/0/4/0.1.160.0.0.0.1 sdisk
0/0/14/1/0/4/0.1.160.0.0.0.2 sdisk
0/0/14/1/0/4/0.1.160.0.0.0.4 sdisk
0/0/14/1/0/4/0.1.160.0.2 fcd_vbus
0/0/14/1/0/4/0.1.160.0.2.0 tgt
0/0/14/1/0/4/0.1.160.0.2.0.0 sdisk
0/0/14/1/0/4/0.1.160.0.8 fcd_vbus
0/0/14/1/0/4/0.1.160.0.8.0 tgt
0/0/14/1/0/4/0.1.160.0.8.0.0 sdisk
0/0/14/1/0/4/0.1.160.0.10 fcd_vbus
0/0/14/1/0/4/0.1.160.0.10.0 tgt
0/0/14/1/0/4/0.1.160.0.10.0.0 sdisk
0/0/14/1/0/4/0.1.160.0.10.0.1 sdisk
0/0/14/1/0/4/0.0x5005076308100129.0x4001000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4002000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4004000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4400000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4100000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4500000000000000 eslpt
0/0/14/1/0/4/0.0x5005076308100129.0x4501000000000000 eslpt
0/0/14/1/0/4/0.1.80.255.0 fcd_vbus
0/0/14/1/0/4/0.1.80.255.0.0 tgt
0/0/14/1/0/4/0.1.80.255.0.0.0 sctl
0/0/14/1/0/4/0.1.80.0.0 fcd_vbus
0/0/14/1/0/4/0.1.80.0.0.0 tgt
0/0/14/1/0/4/0.1.80.0.0.0.0 sdisk
0/0/14/1/0/4/0.1.80.0.0.0.1 sdisk
0/0/14/1/0/4/0.1.80.0.0.0.2 sdisk
0/0/14/1/0/4/0.1.80.0.0.0.4 sdisk
0/0/14/1/0/4/0.1.80.0.2 fcd_vbus
0/0/14/1/0/4/0.1.80.0.2.0 tgt
0/0/14/1/0/4/0.1.80.0.2.0.0 sdisk
0/0/14/1/0/4/0.1.80.0.8 fcd_vbus
0/0/14/1/0/4/0.1.80.0.8.0 tgt
0/0/14/1/0/4/0.1.80.0.8.0.0 sdisk
0/0/14/1/0/4/0.1.80.0.10 fcd_vbus
0/0/14/1/0/4/0.1.80.0.10.0 tgt
0/0/14/1/0/4/0.1.80.0.10.0.0 sdisk
0/0/14/1/0/4/0.1.80.0.10.0.1 sdisk
64000/0xfa00/0x0 esdisk
64000/0xfa00/0x1 esdisk
64000/0xfa00/0x2 estape
64000/0xfa00/0x3 esdisk
64000/0xfa00/0xd esctl
64000/0xfa00/0xe esdisk
64000/0xfa00/0xf esdisk
64000/0xfa00/0x10 esdisk
64000/0xfa00/0x11 esdisk
64000/0xfa00/0x12 esdisk
64000/0xfa00/0x13 esdisk
64000/0xfa00/0x14 esdisk
64000/0xfa00/0x15 esdisk
Boot device's HP-UX HW path is: 0.0.1.1.0.4.0.0.0
iether0: INITIALIZING HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter at hardware path 0/0/1/1/0/6/0
iether1: INITIALIZING HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter at hardware path 0/0/1/1/0/6/1
iether8: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/2/1/0/4/0
iether9: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/2/1/0/4/1
iether10: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/2/1/0/6/0
iether11: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/2/1/0/6/1
iether2: INITIALIZING HP AD194-60001 PCI/PCI-X 1000Base-T 2-port 4Gb FC/2-port 1000B-T Combo Adapter at hardware path 0/0/6/1/0/6/0
iether3: INITIALIZING HP AD194-60001 PCI/PCI-X 1000Base-T 2-port 4Gb FC/2-port 1000B-T Combo Adapter at hardware path 0/0/6/1/0/6/1
iether12: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/10/1/0/4/0
iether13: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/10/1/0/4/1
iether14: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/10/1/0/6/0
iether15: INITIALIZING HP AB545-60001 PCI/PCI-X 1000Base-T 4-port 1000B-T Adapter at hardware path 0/0/10/1/0/6/1
iether4: INITIALIZING HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter at hardware path 0/0/12/1/0/6/0
iether5: INITIALIZING HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter at hardware path 0/0/12/1/0/6/1
iether6: INITIALIZING HP AD194-60001 PCI/PCI-X 1000Base-T 2-port 4Gb FC/2-port 1000B-T Combo Adapter at hardware path 0/0/14/1/0/6/0
iether7: INITIALIZING HP AD194-60001 PCI/PCI-X 1000Base-T 2-port 4Gb FC/2-port 1000B-T Combo Adapter at hardware path 0/0/14/1/0/6/1

System Console is on the Built-In Serial Interface
AF_INET socket/streams output daemon running, pid 293
afinet_prelink: module installed
Starting the STREAMS daemons-phase 1
LVM: Root VG activated
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 67108864
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0000000000000000 - major is 3, minor is 0x0; start = 1497952, size = 33554432
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: B.11.31_LR FLAVOR=perf
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 33417628 Kbytes, lockable: 24349744 Kbytes, available: 27819924 Kbytes

bad_kern_reference: 0x8008031.0x8, fault = 0x53

panic: Fault when executing in kernel mode
Stack Trace:
IP Function Name
0xe000000001e56a80 bad_kern_reference+0xa0
0xe00000000073d860 $cold_pfault+0x270
0xe000000000c00060 vm_hndlr+0xaa0
0xe000000001ab9780 bubbledown+0x0
0xe000000000b835c1 tcp_wput+0x23e1
0xe000000000b79e40 streams_putmsg_fc+0x280
0xe000000000b795e0 inet_push+0x140
0xe000000000b674e0 inet_stream_send+0x400
0xe000000000b714b0 sosend_default+0x5c0
0xe000000000bf2db0 soo_rw+0x110
0xe000000000c1a740 write+0x420
0xe000000000ba75f0 syscall+0x590
End of Stack Trace

linkstamp: Wed Oct 14 15:22:22 METDST 2009
_release_version: @(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf
Calling function e00000000154f7e0 for Shutdown State 1 type 0x2

sync'ing disks (0 buffers to flush):
0 fcache pages still dirty
0 buffers not flushed
0 buffers still dirty
4 REPLIES
Shibin_2
Honored Contributor

Re: Node gets rebooted once the RAC database

Do you have any backup of old kernel ( good condition )? If available try to boot from there. Then compare the kernel configuration with the one giving problem.
Regards
Shibin
UNIX_HP
Advisor

Re: Node gets rebooted once the RAC database

We have no backup of the kernel. From this reference bad_kern_reference: 0x8008031.0x8, fault = 0x53, you can know what is the problem?
Shibin_2
Honored Contributor

Re: Node gets rebooted once the RAC database

If you have support agreement with HP, contact HP Response Center.

Looks like you are in deadlock situation.
Regards
Shibin
Hein van den Heuvel
Honored Contributor

Re: Node gets rebooted once the RAC database


For the benefit of others, and to help create a better case for support, should you need to go there, please describe the timeline of events a little more clear?
What worked,
What command or event triggered failure?
What we the end state (the crash above).

fwiw,
Hein