Unexpected instance shutdown due to ORA-07445

[Joaquin_13]

Hi guys,

Our database is experiencing some OS internal error as implies in the Oracle error ORA-07445. Our database would unexpectedly terminates by itself and returning this error in alert log.
Mon Sep 5 16:20:52 2005
Errors in file /mbs3/app/oracle/product/8.0.5/rdbms/log/ora_19441.trc:
ORA-07445: exception encountered: core dump [10] [2063843280] [2745540544] [1812] [] []
Mon Sep 5 16:20:56 2005
Errors in file /mbs3/app/oracle/product/8.0.5/rdbms/log/pmon_26216.trc:
ORA-07445: exception encountered: core dump [10] [2063845392] [1074118632] [1812] [] []
Mon Sep 5 16:21:09 2005
DBW0: terminating instance due to error 472
Instance terminated by DBW0, pid = 26228

I've tried checking the syslog.log but i couldn't see any error that would make sense or relevant to this problem we encounter.
Does anybody know how to deal with this problem....


Thanks in advance & best regards,

Joaquin

[Jean-Luc Oudart]

It looks like you will have to log an iTAR with Oracle support.

Check Metalink Note:211909.1 "Customer Introduction to ORA-7445 Errors" It will explain what info to collect for suppport.

Regards
Jean-Luc

[Steven E. Protter · Accepted solution]

You are not going to see anything in syslog because this is an oracle area.

Oracle has a utility at metalink.oracle.com that lets you collect data and submit it to support.

Even if you have no support contract the data can be helpful in locating the problem.

SEP

[Yogeeraj_1]

hi joaquin,

you are running a rather old version of Oracle. Consider upgrading.

Had a look at metalink forums, it seems like you are facing a bug.


Explanation
-----------
The errors ORA-604 ORA-02051, introduce a dynamic memory corruption into the SGA which causes the shadow process to abend and generate the ORA-7445 error and dump core.

Pmon wakes up and sees that there is a process that had crashed, and tries to clean up after the abended process. In rolling back the failed transactons, Pmon also encounters the memory corruption and is also unable to handle the corruption causing Pmon to abend. Pmon failure then results in instance failure.


btw, 8.0.5 is not even listed in the "Certify & Availability" section:
Oracle Database - Enterprise Edition Version(s)
10gR2 64-bit
10g 64-bit
9.2 64-bit
9.0.1 64-bit
8.1.7 (8i) 64-bit
8.1.7 (8i)
8.1.6 (8i) 64-bit
8.1.6 (8i)
8.1.5 (8i) 64-bit
8.1.5 (8i)
8.0.6 64-bit
8.0.6

Don't know if Oracle support would be able to help you in that either.

Can you post a copy of your alert.log?

regards
yogeeraj

[Joaquin_13]

Hi,

Thanks for the time guys,
As suggested. Our alert.log is already posted together with this response.


regards,

Joaq's

[Yogeeraj_1]

hi joaq's

Hard to say at this point. The ora-7445 indicates that oracle has detected a core dump when attempting a particular operation. This could happen almost anywhere in the code by definition as it is an unhandled exception.

There is an associated trace file that will help to narrow the problem down a bit.

please post either of:
/mbs3/app/oracle/product/8.0.5/rdbms/log/ora_19441.trc
/mbs3/app/oracle/product/8.0.5/rdbms/log/pmon_26216.trc

regards
yogeeraj

[Joaquin_13]

Hi,

Attached is the PMON trace file...

[Joaquin_13]

& this one is the SQL trace file....



thank's & best regards,

Joaq's

[Indira Aramandla]

Hi Joaquin,

An ORA-7445 error is raised by an Oracle server process when it has received a fatal signal from the operating system. The error may be raised in either a foreground or background process. The process will normally write an error to the alert log write a trace file in either user_dump_dest or background_dump_dest create a core dump in core_dump_dest.

There are many 'illegal' operations that the operating system can trap; a common example is a process writing to an invalid memory location. To protect the system, the offending process will be sent a fatal signal. Typically, the signals seen are SIGBUS (signal 10, bus error) and SIGSEGV (signal 11, segmentation violation).

An ORA-7445 is a generic error, and can occur from anywhere in the Oracle code. The precise location of the error is identified by the trace file it produces.

As Yogeeraj mentioned the error introduces a dynamic memory corruption into the SGA which causes the shadow process to abend and generate the ORA-7445 error and dump core.

ORA-7445: exception encountered: core dump [10] [2122262800] [261978112] [] [] [] tells us:
The error occurred in function run_some_SQL()
The process received signal SIGBUS
The process received signal 10
Some additional information pertinent to the signal


Attached is a Note from metalink that explains the error ORA-07445.


Indira A

[Yogeeraj_1]

hi,

i have tried to looking into the problem using the ORA-7445 Lookup Tool found on Note:208922.1 ("Troubleshoot an ORA-7445 Error Using the ORA-7445 Lookup Tool") but in vain.

You are receiving the signals SIGBUS (signal 10, bus error)....

This is a generic error, and can occur from anywhere in the Oracle code. The precise location of the error is identified by the trace file it produces.

I would suggest that you contact support so that they can analyze it more precisely and give you a workaround.

You will have to upload them the following:
1. the alert log, stretching back to at least the previous instance startup prior to the error. Ensure the startup init.ora parameters are included

2. all ORA-7445 and ORA-600 trace files since the last instance startup This is to stablish if the ORA-7445 being investigated is related to or a symptom of another error. (If a lot of trace files exist only collect the first 3 to 5 - Oracle Support will be able to tell from the alert.log if more trace files are required.)

[Note:211909.1 - "Customer Introduction to ORA-7445 Errors"]

sorry for not being able to help any further.

good luck

kind regards
yogeeraj