System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Cluster node gets rebooted once the RAC database is started.

raja_stm
Occasional Contributor

Cluster node gets rebooted once the RAC database is started.

Hi,
I have Oracle 11g RAC installed on HP-UX 1131 IA SG 11.18 cluster.When i try to start the database on one of the node,am getting the following error on console and the system gets rebooted.This issue is not there with second node of the cluster and the database is running fine there.

Any help to solve this issue???

##############################################

Stored message buffer up to system crash:

MFS is defined: base= 0xe000000101933000 size= 27008 KB
Found adjacent data tr. Growing size. 0x26cd000 -> 0x66cd000.
Loaded ACPI revision 2.0 tables.
MMIO on this platform supports Write Coalescing.


MCA recovery subsystem disabled, not supported on this platform.
Using /stand/ext_ioconfig

Memory Class Setup
-------------------------------------------------------------------------
Class Physmem Lockmem Swapmem
-------------------------------------------------------------------------
System : 3885 MB 3885 MB 3885 MB
Kernel : 3885 MB 3885 MB 3885 MB
User : 3343 MB 2963 MB 2975 MB
-------------------------------------------------------------------------

Starting ktracer 0 1
Installing Socket Protocol families AF_INET and AF_INET6
64000/0xfa00 esvroot
Kernel EVM initialized
sec_init(): kernel RPC authentication/security initialization.
secgss_init(): kernel RPCSEC_GSS security initialization.
rpc_init(): kernel RPC initialization.
rpcmod_install(): kernel RPC STREAMS module "rpcmod" installation. ...(driver_install)
NOTICE: nfs_client_pv3_install(): nfs3 File system was registered at index 11.
NOTICE: nfs_client_pv4_install(): nfs4 File system was registered at index 12.
NOTICE: cachefsc_install: cachefs File system was registered at index 14.
btlan_load() Loaded Successfully
0 sba
0/0 lba
0/0/1/0 UsbOhci
0/0/1/1 UsbOhci
0/0/1/2 UsbEhci
0/0/2/0 side_multi
0/0/2/0.0 side
0/0/2/0.1 side
0/0/2/0.0.0x0 estp
0/1 lba
Initializing the Ultra320 SCSI Controller at 0/1/1/0. Controller firmware version is 01.03.35.65
0/1/1/0 mpt
Initializing the Ultra320 SCSI Controller at 0/1/1/1. Controller firmware version is 01.03.35.65
0/1/1/1 mpt
0/1/2/0 iether
0/1/2/1 iether
0/2 lba
0/2/1 pci_slot
fcd: Claimed HP A6826-60001 2Gb Fibre Channel port at hardware path 0/2/1/0 (FC Port 1 on HBA)
0/2/1/0 fcd
fcd: Claimed HP A6826-60001 2Gb Fibre Channel port at hardware path 0/2/1/1 (FC Port 2 on HBA)
0/2/1/1 fcd
0/3 lba
0/3/1 pci_slot
0/4 lba
0/4/1 pci_slot
0/5 lba
0/5/1 pci_slot
0/6 lba
0/1/1/0.0x0 estp
0/1/1/0.0x1 estp
0/1/1/0.0x1.0x0 eslpt
0/1/1/0.0x0.0x0 eslpt
0/1/1/0.1 tgt
0/1/1/0.1.0 sdisk
0/1/1/0.0 tgt
0/1/1/0.0.0 sdisk
0/6/1/0 asio0
0/0/2/0.0.0x0.0x0 eslpt
0/0/2/0.0.0 tgt
0/0/2/0.0.0.0 sdisk
0/6/1/1 asio0
0/6/2/0 gvid_core
120 processor
121 processor
250 pdh
250/0 ipmi
250/1 asio0
250/2 asio0
250/3 acpi_node
0/2/1/0.0x50001fe150035298 estp
0/2/1/0.0x50001fe15003529c estp
0/2/1/0.0x50001fe150034a3c estp
0/2/1/0.0x50001fe150034a38 estp
0/2/1/0.0x50001fe1500f2c3e estp
0/2/1/0.0x50001fe1500f2c3a estp
0/2/1/0.0x50001fe150035298.0x0 eslpt
0/2/1/0.0x50001fe150034a38.0x0 eslpt
0/2/1/0.0x50001fe1500f2c3e.0x0 eslpt
0/2/1/0.0x50001fe1500f2c3a.0x0 eslpt
0/2/1/0.156 fcd_fcp
0/2/1/0.156.13.255.0 fcd_vbus
0/2/1/0.156.13.255.0.5 tgt
0/2/1/0.156.13.255.0.5.0 sctl
0/2/1/0.156.13.5.0 fcd_vbus
0/2/1/0.156.13.5.0.0 tgt
0/2/1/0.156.13.5.0.0.0 sctl
0/2/1/0.208 fcd_fcp
0/2/1/0.208.2.255.2 fcd_vbus
0/2/1/0.208.2.255.2.8 tgt
0/2/1/0.208.2.255.2.8.0 sctl
0/2/1/0.208.2.40.0 fcd_vbus
0/2/1/0.208.2.40.0.0 tgt
0/2/1/0.208.2.40.0.0.0 sctl
0/2/1/0.0x50001fe15003529c.0x0 eslpt
0/2/1/0.208.2.255.2.9 tgt
0/2/1/0.208.2.255.2.9.0 sctl
0/2/1/0.208.2.41.0 fcd_vbus
0/2/1/0.208.2.41.0.0 tgt
0/2/1/0.208.2.41.0.0.0 sctl
0/2/1/0.0x50001fe150034a3c.0x0 eslpt
0/2/1/0.156.13.255.0.4 tgt
0/2/1/0.156.13.255.0.4.0 sctl
0/2/1/0.156.13.4.0 fcd_vbus
0/2/1/0.156.13.4.0.0 tgt
0/2/1/0.156.13.4.0.0.0 sctl
0/2/1/1.0x50001fe150035298 estp
0/2/1/1.0x50001fe15003529c estp
0/2/1/1.0x50001fe150034a3c estp
0/2/1/1.0x50001fe150034a38 estp
0/2/1/1.0x50001fe1500f2c3e estp
0/2/1/1.0x50001fe1500f2c3a estp
0/2/1/1.0x50001fe150034a3c.0x0 eslpt
0/2/1/1.0x50001fe150034a38.0x0 eslpt
0/2/1/1.0x50001fe1500f2c3e.0x0 eslpt
0/2/1/1.0x50001fe1500f2c3a.0x0 eslpt
0/2/1/1.156 fcd_fcp
0/2/1/1.156.13.255.0 fcd_vbus
0/2/1/1.156.13.255.0.4 tgt
0/2/1/1.156.13.255.0.4.0 sctl
0/2/1/1.156.13.4.0 fcd_vbus
0/2/1/1.156.13.4.0.0 tgt
0/2/1/1.156.13.4.0.0.0 sctl
0/2/1/1.156.13.255.0.5 tgt
0/2/1/1.156.13.255.0.5.0 sctl
0/2/1/1.156.13.5.0 fcd_vbus
0/2/1/1.156.13.5.0.0 tgt
0/2/1/1.156.13.5.0.0.0 sctl
0/2/1/1.208 fcd_fcp
0/2/1/1.208.2.255.2 fcd_vbus
0/2/1/1.208.2.255.2.8 tgt
0/2/1/1.208.2.255.2.8.0 sctl
0/2/1/1.208.2.40.0 fcd_vbus
0/2/1/1.208.2.40.0.0 tgt
0/2/1/1.208.2.40.0.0.0 sctl
0/2/1/1.0x50001fe150035298.0x0 eslpt
0/2/1/1.208.2.255.2.9 tgt
0/2/1/1.208.2.255.2.9.0 sctl
0/2/1/1.208.2.41.0 fcd_vbus
0/2/1/1.208.2.41.0.0 tgt
0/2/1/1.208.2.41.0.0.0 sctl
0/2/1/1.0x50001fe15003529c.0x0 eslpt
0/2/1/0.0x50001fe15003529c.0x4001000000000000 eslpt
0/2/1/0.0x50001fe150035298.0x4002000000000000 eslpt
0/2/1/0.0x50001fe15003529c.0x4002000000000000 eslpt
0/2/1/0.156.13.255.0.1 tgt
0/2/1/0.156.13.255.0.1.0 sctl
0/2/1/0.156.13.1.0 fcd_vbus
0/2/1/0.156.13.1.0.0 tgt
0/2/1/0.156.13.1.0.0.1 sdisk
0/2/1/0.156.13.1.0.0.2 sdisk
0/2/1/0.0x50001fe150035298.0x4001000000000000 eslpt
0/2/1/0.156.13.255.0.0 tgt
0/2/1/0.156.13.255.0.0.0 sctl
0/2/1/0.156.13.0.0 fcd_vbus
0/2/1/0.156.13.0.0.0 tgt
0/2/1/0.156.13.0.0.0.1 sdisk
0/2/1/0.156.13.0.0.0.2 sdisk
0/2/1/1.0x50001fe150035298.0x4001000000000000 eslpt
0/2/1/1.0x50001fe15003529c.0x4001000000000000 eslpt
0/2/1/1.0x50001fe150035298.0x4002000000000000 eslpt
0/2/1/1.156.13.255.0.0 tgt
0/2/1/1.156.13.255.0.0.0 sctl
0/2/1/1.156.13.0.0 fcd_vbus
0/2/1/1.156.13.0.0.0 tgt
0/2/1/1.156.13.0.0.0.1 sdisk
0/2/1/1.156.13.0.0.0.2 sdisk
0/2/1/1.0x50001fe15003529c.0x4002000000000000 eslpt
0/2/1/1.156.13.255.0.1 tgt
0/2/1/1.156.13.255.0.1.0 sctl
0/2/1/1.156.13.1.0 fcd_vbus
0/2/1/1.156.13.1.0.0 tgt
0/2/1/1.156.13.1.0.0.1 sdisk
0/2/1/1.156.13.1.0.0.2 sdisk
64000/0xfa00/0x0 esdisk
64000/0xfa00/0x1 esdisk
64000/0xfa00/0x2 esdisk
64000/0xfa00/0x3 esctl
64000/0xfa00/0x4 esctl
64000/0xfa00/0x5 esctl
64000/0xfa00/0x9 esdisk
64000/0xfa00/0xa esdisk
Boot device's HP-UX HW path is: 0.1.1.0.1.0
iether0: INITIALIZING HP PCI-X 1000Base-T Dual-port Built-in at hardware path 0/1/2/0
iether1: INITIALIZING HP PCI-X 1000Base-T Dual-port Built-in at hardware path 0/1/2/1

System Console is on the Built-In Serial Interface
AF_INET socket/streams output daemon running, pid 39
afinet_prelink: module installed
Starting the STREAMS daemons-phase 1
LVM: Root VG activated
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 8339456
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0000000000000000 - major is 3, minor is 0x0; start = 2349920, size = 4169728
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: B.11.31_LR FLAVOR=perf
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 4182844 Kbytes, lockable: 2831396 Kbytes, available: 3231368 Kbytes

bad_kern_reference: 0xfff0031.0xc00000014a6b4820, fault = 0x8

Message buffer contents after system crash:

panic: Fault when executing in kernel mode
Stack Trace:
IP Function Name
0xe000000001fa1fc0 bad_kern_reference+0xa0
0xe000000000893680 vfault+0x1230
0xe00000000088d990 vm_hndlr+0x5d0
0xe000000001c27780 bubbledown+0x0
0xe00000000080b4e1 dupb+0x1c1
0xe000000001613090 dupmsg+0x90
0xe000000001218b20 $cold_inet_dgram_usrrecv+0x200
0xe00000000087fcb0 recvit+0x200
0xe000000000639580 recvmsg+0x180
0xe000000000835bb0 syscall+0x530
End of Stack Trace

linkstamp: Mon Dec 01 16:35:55 IST 2008
_release_version: @(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf
Calling function e000000001713980 for Shutdown State 1 type 0x2

sync'ing disks (0 buffers to flush):
0 fcache pages still dirty
0 buffers not flushed
0 buffers still dirty
i 0 pfn 0x1 pages 0x9f
i 1 pfn 0x100 pages 0x3f3ec
i 2 pfn 0x3fc00 pages 0x15c
i 3 pfn 0x4040000 pages 0xbfcda
i 4 pfn 0x40ffd44 pages 0xc4
i 5 pfn 0x40ffe7e pages 0x14a
*** Not enough CPUS for a compressed dump ***

*** A system crash has occurred. (See the above messages for details.)
*** The system is now preparing to dump physical memory to disk, for use
*** in debugging the crash.

*** The dump will be a SELECTIVE dump with
compression OFF and concurrency ON: 938 of 4085 megabytes.
*** To change this dump type, press any key within 10 seconds.
*** Proceeding with selective dump, with compression off and concurrency on.



Primary Dump Header Location :
Device details:
Major number: 31 Minor number:0x21000
Offset: 2349920.
*** The dump may be aborted at any time by pressing ESC.
*** Dumping: 100% complete (938 of 938 MB)
time: 38 seconds, Number of Dump units: 1
***********************************************************
* ROM Version : 03.17
* ROM Date : 03/31/2005
* BMC Version : 03.48
***********************************************************
0 0 0x0015B2 0x0000000018978410 boot time event
1 0 0x0000A4 0x0000000000000000 start memory configuration
0 0 0x0015B2 0x0000000023380173 boot time event
1 0 0x000014 0x0000000000000000 CPU0 starting cell relocation
1 0 0x000009 0x0000000000000000 CPU0 launch EFI
0 0 0x0015B2 0x0000000024940601 boot time event
1 0 0x000207 0x000000000011000C CPU0 starting EFI
POSSE Library version 0.11 is loading...
EFI version 1.10 [14.62]
EFI64 Running on Intel(R) Itanium Processor Family
EFI 1.10 IPF server rx2620 3.14 [Tue Sep 30 14:14:27 2003] - HP
##############################################
4 REPLIES
likid0
Honored Contributor

Re: Cluster node gets rebooted once the RAC database is started.

But looks like your system is not even booting to the SO ??

You get a nice panic when booting


$Revision: vmunix: B.11.31_LR FLAVOR=perf
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 4182844 Kbytes, lockable: 2831396 Kbytes, available: 3231368 Kbytes

bad_kern_reference: 0xfff0031.0xc00000014a6b4820, fault = 0x8

Message buffer contents after system crash:

panic: Fault when executing in kernel mode
Stack Trace:
Windows?, no thanks
raja_stm
Occasional Contributor

Re: Cluster node gets rebooted once the RAC database is started.

The system is booting the OS.but the system is getting rebooted again when i try to start the RAC database.
raja_stm
Occasional Contributor

Re: Cluster node gets rebooted once the RAC database is started.

Hi,
I did one strange thing with the problematic node.Unknowingly i started installing the package(OS-Core B.11.31.9999.%2008_1116 Core Operating System, plus Software Terms & Conditions) with this node and then I realised and immediately stopped the installation.but still some of the components of package got installed.

=> NFS- B.11.31.9999.%2008_1116 ONC/NFS;Network-File System,Information Services,Utilities
=> Networking-B.11.31.9999.%2008_1116 HP-UX_Lanlink_Product
=> OS-Core-B.11.31.9999.%2008_1116 Core Operating System, plus Software Terms & Conditions).

I suspect that this could be the cause of this problem.Please confirm????
Als let me know,if i remove this package,will it affect the kernel???
Duncan Edmonstone
Honored Contributor

Re: Cluster node gets rebooted once the RAC database is started.

>> OS-Core B.11.31.9999.%2008_1116

Where did you "get" this from?

I have to say at first glance that doesn't look like a "released" version of the OS to me...

HTH

Duncan

HTH

Duncan