topic Re: Oracle reported Ora-07445 error, ugrent! Please help. in Operating System - HP-UX

Oracle reported Ora-07445 error, ugrent! Please help.

刘绍江 — Wed, 16 Nov 2005 22:50:35 GMT

TWO RPT8400
HP-UX 11.11 PA-RISC (64-bit)+ ORACLE8.1.7.4
HP MC
we have two database(db1,db2) on two rpt8400(server1,server2), but recently the db1 and db1 would report ORA-07445 error when the databases were running on server1, but everything is ok when they are running on server2. so I cloned the oracle running environmeng from server2 to server1, but the problem can not be fixed. it looks like a system(os) relative problem because the databases can be run on server2.
sometimes,the error would cause instance crashed.
and I browsed the trace file, it show many following message:

dump of memory from xxxxxxx
repeat 5 times
and so on.

Who can guide me how to fix the problem? thank you in advance.

Re: Oracle reported Ora-07445 error, ugrent! Please help.

RAC_1 — Wed, 16 Nov 2005 23:01:58 GMT

Post the errors here. Difficult to get the attachment. Also do you see any errors in syslog.log/dmesg??

Re: Oracle reported Ora-07445 error, ugrent! Please help.

刘绍江 — Wed, 16 Nov 2005 23:09:16 GMT

Thank for your reply.
no error message in syslog/dmesg.
/uh01/app/oracle/admin/FWHDB/udump/ora_1131_fwhdb.trc
Oracle8i Enterprise Edition Release 8.1.7.4.0 - 64bit Production
With the Partitioning option
JServer Release 8.1.7.4.0 - 64bit Production
ORACLE_HOME = /apps/oracle
System name: HP-UX
Node name: szx021
Release: B.11.11
Version: U
Machine: 9000/800
Instance name: FWHDB
Redo thread mounted by this instance: 1
Oracle process number: 20
Unix process pid: 1131, image: oracle@szx021 (TNS V1-V3)

*** 2005-11-17 06:19:56.522
*** SESSION ID:(47.25448) 2005-11-17 06:19:56.520
Exception signal: 11, code = 800003ffe17c7938
sc_onstack: 000000001656cd50, sc_mask 0000000000000000
ORA-07445: exception encountered: core dump [11] [3783031096] [374787408] [0] [] []
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ssexhd()+344 ? ksedst() 000000000 ?
800003FFE17B0810 ?
000000001 ? 000000040 ?
_sigreturn()+0 ? ssexhd() 000000003 ? 000000000 ?
04BDF3E39 ?
800003FFE17C73F0 ?
kgklookup()+252 ? _sigreturn() C0000000883B8380 ?
800003FFE17C5A78 ?
C0000000A1698710 ?
000000001 ?
kgffgiov()+192 ? kgklookup() 000000000 ? 000000000 ?
800003FF01C2AA50 ?
800003FFE17C5F20 ?
kgffsubmit()+548 ? kgffgiov() 800003FFE17C5E58 ?
80000001000FC2DD ?
800003FFE17C6D70 ?
000000C50 ?
kgfdsubmit()+124 ? kgffsubmit() 000000000 ?
4000000000AFA1D3 ?
4000000000B56F1F ?
00001F727 ?
kgfdimm()+136 ? kgfdsubmit() 00000005E ?
800000010007C140 ?
000000000 ?
C0000000880AC608 ?
kcfrbd()+608 ? kgfdimm() 000000000 ? 000000000 ?
800003FFE17C4380 ?
C0000000879A5D30 ?
kcbzib()+4344 ? kcfrbd() 200002100000001 ?
800000010007C140 ?
40000000002A5D38 ?
C00000008799EE20 ?
kcbget()+3580 ? kcbzib() 000000074 ?
800000010007C140 ?
4000000000B47A37 ?
0E17C2EF0 ?
ktbxchg()+72 ? kcbget() 800003FFE17C36C0 ?
C0000000883B8918 ?
800003FFE17C3748 ?
800003FFE17C3A10 ?
kdifind()+3432 ? ktbxchg() 800000010007C140 ?
4000000000B751F7 ?
800003FFE17C36C8 ?
8000000100308568 ?
kdimod0()+688 ? kdifind() 800003FFE17C35F0 ?
800003FFE17C2FD0 ?
000000000 ?
80000001002F0068 ?
kaudel()+3120 ? kdimod0() 800000010007C140 ?
000000000 ? 10006A140 ?
8000000100308568 ?
delrow()+1244 ? kaudel() 000000000 ? 000C36E93 ?
80000001002F50E8 ?
800000010007C140 ?
qerdlFetch()+332 ? delrow() CDEC702200070010 ?
4000000000B0EE07 ?
C0000000841ECFD0 ?
8000000100308530 ?
delexe()+740 ? qerdlFetch() 8000000100071140 ?
800003FFE17C1210 ?
800000010007C140 ?
C00000008DA8E848 ?
opiexe()+9936 ? delexe() 4000000000252E10 ?
800000010006A140 ?
000000001 ?
8000000100005160 ?
opipls()+13096 ? opiexe() 64437BB068 ?
800000010007A1C0 ?
000000048 ? 1000000AF ?
opiodr()+2092 ? opipls() C0000000869EFB90 ?
800000010007A1C0 ?
000000000 ?
40000000008B61BB ?
rpidrus()+176 ? opiodr() 000000000 ?
400000000021EFB0 ?
800000010006A940 ?
800003FFE17B8470 ?
skgmstack()+148 ? rpidrus() 4000000000B368F3 ?
4000000000BE6DEB ?
C00000008428C320 ?
000000000 ?
rpidru()+144 ? skgmstack() 8000000100000001 ?
000006040 ?
4000000000B0417B ?
000000000 ?
rpiswu2()+912 ? rpidru() 800000010006A940 ?
C00000009820A83F ?
800000010006A940 ?
800003FFE17B8470 ?
rpidrv()+2180 ? rpiswu2() 800003FFE17BC6B8 ?
C0000000885F0230 ?
000000000 ?
800000010006A940 ?
psddr0()+172 ? rpidrv() 80000001000B6858 ?
100000000 ?
800000010006A940 ?
800003FFE17B8470 ?
psdnal()+268 ? psddr0() 000000000 ? 000005648 ?
8000000100005720 ?
80000001000B6A98 ?
pfrrun()+80024 ? psdnal() 800003FFE17B8478 ?
800000010006A940 ?
80000001000BF148 ?
80000001000B6C60 ?
peicnt()+304 ? pfrrun() 800000010007A180 ?
C00000008852DAD8 ?
800000010006A940 ?
800000010007A180 ?
kkxexe()+500 ? peicnt() 000000000 ? 000000000 ?
8000000100005648 ?
C00000008A5A4A78 ?
opiexe()+12064 ? kkxexe() 4000000000BB98A7 ?
4000000000000002 ?
000254910 ? 000000004 ?
opiall0()+3972 ? opiexe() 4000000000B5DAAB ?
000000000 ? 000000000 ?
000000001 ?
kpoal8()+1040 ? opiall0() 100000000 ? 000000000 ?
000000000 ? 000000000 ?
opiodr()+2092 ? kpoal8() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
ttcpip()+4672 ? opiodr() 2000000000000 ?
80000001000A4968 ?
000000001 ?
4000000000C36E93 ?
opitsk()+2184 ? ttcpip() 800003FFC8002E68 ?
C00000000013EE5B ?
800003FFB4014368 ?
800003FFC8002E68 ?
opiino()+1328 ? opitsk() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
opiodr()+2092 ? opiino() 4000000000573948 ?
7F80000000000000 ?
000000032 ?
800003FFB400EDC8 ?
opidrv()+1696 ? opiodr() 00000048C ? 48C00000081 ?
000000947 ?
800003FFE17BBDA0 ?
sou2o()+32 ? opidrv() 000000000 ? 201000000000000 ?
000000000 ? 000000000 ?
main()+200 ? sou2o() 000000000 ? 000000000 ?
800003FFE17B0800 ?
000000000 ?
$START$()+152 ? main() 21408130800 ?
800003FFE17B056F ?
800003FFE17B04DB ?
800003FFE17B000C ?
----- Argument/Register Address Dump -----
Argument/Register addr=000000000.
Dump of memory from 0x000000000 to 0x000000100
00000000 00000000 00000000 00000000 00000000 [................]
Repeat 15 times
Argument/Register addr=800003ffe17b0810.
Dump of memory from 0x800003FFE17B0810 to 0x800003FFE17B0910
E17B0810 800003FF E17BDB60 C0000000 00284A10 [.....{.`.....(J.]
E17B0820 800003FF DFFFF480 00000000 00000000 [................]
E17B0830 00000000 00000000 00000000 00000000 [................]
Repeat 4 times
E17B0880 C0000000 000218B3 00000000 00000000 [................]
E17B0890 800003FF E17B0800 80000001 0007C140 [.....{.........@]
E17B08A0 00000002 E178FC60 800003FF E17B0608 [.....x.`.....{..]
E17B08B0 80000001 000438B0 00000000 00000000 [......8.........]
E17B08C0 800003FF DFFFF480 00000000 00000040 [...............@]
E17B08D0 00000000 00000000 00000000 00000000 [................]
E17B08E0 00000002 00000000 00000000 00000200 [................]
E17B08F0 00000000 00000000 00000000 00000000 [................]
E17B0900 00000000 00000430 40000000 00DB9B78 [.......0@......x]
Argument/Register addr=000000040.
Dump of memory from 0x000000040 to 0x000000140
00000040 00000000 00000000 00000000 00000000 [................]
Repeat 15 times
Argument/Register addr=800003ffe17c73f0.
Dump of memory from 0x800003FFE17C73F0 to 0x800003FFE17C74F0
E17C73F0 00000000 00000000 00000000 00000000 [................]
Repeat 5 times
E17C7450 2F000041 1656CD50 00000000 00B5D000 [/..A.V.P........]
E17C7460 00000000 50BF0040 00000000 00B55800 [....P..@......X.]
E17C7470 00000000 00B53000 00000000 00000000 [......0.........]
E17C7480 00000000 1656CE08 00000000 000000B0 [.....V..........]
E17C7490 00000000 00000000 00000000 0000000C [................]
E17C74A0 00000000 08000008 00000000 00030720 [............... ]
E17C74B0 00000000 00000001 00000000 000331D0 [..............1.]
E17C74C0 00000000 00000801 00000000 FFFFFFFF [................]
E17C74D0 00000000 001F5868 00000000 7FC1C31C [......Xh........]
E17C74E0 00000000 00000003 00000000 00000000 [................]
Argument/Register addr=c0000000883b8380.
Dump of memory from 0xC0000000883B8380 to 0xC0000000883B8480
883B8380 00020031 0005D5A6 04C1B31E D16C1700 [...1.........l..]
883B8390 04C1B31E D16C1700 00000000 00000000 [.....l..........]
883B83A0 079C0000 00000000 00000001 04C1B31E [................]
883B83B0 FFFFFFFF 00000003 00000010 00000000 [................]
883B83C0 00000000 00000000 00000000 00000000 [................]
883B83D0 C0000000 85FEFD70 C0000000 44DEA014 [.......p....D...]
883B83E0 0000000C 00000000 00000000 00000000 [................]
883B83F0 00000000 00000000 00000000 00000000 [................]
Repeat 4 times
883B8440 FFFFFFFF FFFF0000 0000009E 00000013 [................]
883B8450 004BAEB5 00000000 00000001 00803202 [.K............2.]
883B8460 00000002 00000000 0000ED55 00000000 [...........U....]
883B8470 00001E03 00000000 C0000000 8428BCF0 [.............(..]
Argument/Register addr=800003ffe17c5a78.
Dump of memory from 0x800003FFE17C5A78 to 0x800003FFE17C5B78
E17C5A70 C0000000 33D1A05C [....3..\]
E17C5A80 00000000 00000000 00000006 0341D7E6 [.............A..]
E17C5A90 00026E38 00000003 00000001 00000000 [..n8............]
E17C5AA0 C0000000 883B8378 00000000 00000000 [.....;.x........]
E17C5AB0 C0000000 859028F0 C0000000 33D1A014 [......(.....3...]
E17C5AC0 00000000 00000000 00000000 00000000 [................]
E17C5AD0 00000000 00000000 40000000 00761FD8 [........@....v..]
E17C5AE0 00000000 00000000 00000000 00000000 [................]
Repeat 3 times
E17C5B20 CDEBE17A 0007A1C0 0000014D 0000000D [...z.......M....]
E17C5B30 002B246D 00000000 80000001 000901F8 [.+$m............]
E17C5B40 80000001 04C3B31D 04C00C78 00000000 [...........x....]
E17C5B50 C0000000 00000000 01C35C2E 879A5D68 [..........\...]h]
E17C5B60 C0000000 855A91E0 CD889E3B 00070117 [.....Z.....;....]
E17C5B70 00000001 00C80000 [........]
Argument/Register addr=c0000000a1698710.
Dump of memory from 0xC0000000A1698710 to 0xC0000000A1698810
A1698710 C0000000 A1695040 00000000 00000000 [.....iP@........]
A1698720 C0000000 88586248 C0000000 885904C0 [.....XbH.....Y..]
A1698730 00000000 00000000 00000000 00000000 [................]
Repeat 1 times
A1698750 00000000 00000000 C0000000 A1695040 [.............iP@]
A1698760 00000000 00000000 00000000 00000000 [................]

Re: Oracle reported Ora-07445 error, ugrent! Please help.

RAC_1 — Wed, 16 Nov 2005 23:27:56 GMT

It creates a core file. Right?? What does "file core" say??

Re: Oracle reported Ora-07445 error, ugrent! Please help.

刘绍江 — Thu, 17 Nov 2005 00:05:56 GMT

yes,it created core in cdump directory:
the output as following:

core: ELF-64 core file - PA-RISC 2.0 from 'oracle' - received SIGABRT

Re: Oracle reported Ora-07445 error, ugrent! Please help.

刘绍江 — Thu, 17 Nov 2005 01:23:08 GMT

anyone has experience on it?

Re: Oracle reported Ora-07445 error, ugrent! Please help.

RAC_1 — Thu, 17 Nov 2005 01:57:40 GMT

Nothing in my notes. Look at oracle metalinks.

I am searching forums for similar errors.

Re: Oracle reported Ora-07445 error, ugrent! Please help.

刘绍江 — Thu, 17 Nov 2005 02:17:01 GMT

I am very appreciated for your kindly help!

I browsed lots of article from metalink, But I can not find similar error and solution.

Please let me know if you can find the solution. Thank you RAC.

Re: Oracle reported Ora-07445 error, ugrent! Please help.

Yogeeraj_1 — Thu, 17 Nov 2005 02:52:04 GMT

hi,

you may also wish to log an TAR with Metalink so that you get a more prompt and expert investigation on your problem.

They will be asking you various information on your system (OS) and Database (parameters, etc) in the form of an RDA report...

hope this helps
kind regards
yogeeraj

Re: Oracle reported Ora-07445 error, ugrent! Please help.

刘绍江 — Thu, 17 Nov 2005 03:17:19 GMT

thanks for your reminder.

Oracle are searching the workground, they said the issue is not matching with any known issues. and the version is desupport version...

Re: Oracle reported Ora-07445 error, ugrent! Please help.

Frank de Vries — Fri, 18 Nov 2005 04:49:29 GMT

Do you still need help or is it resolved.
Could you otherwise post your pfile here
(init.ora) for both db's

Re: Oracle reported Ora-07445 error, ugrent! Please help.

Ron Gordon — Fri, 18 Nov 2005 09:55:39 GMT

I am having similar issue on my 8.1.7.2.0 databases. (32bit oracle on 64 bit sun server) According to oracle the fix is to upgrade to 9i. My work around is to monitor the alert.log every 10 minutes. If the ORA-7445 error occurs bounce the database automatically.

Re: Oracle reported Ora-07445 error, ugrent! Please help.

Frank de Vries — Mon, 21 Nov 2005 02:00:28 GMT

Uugh pretty nasty - you must have patient users :)
How often I have heard this from Oracle that upgrading to the next release would solve something. As if bugs don't exist there ?
Whatever happened to root-cause analysis.

I would check if you have multiple db_writers
and I would start up a db with a minimum
of parameters (almost Vanilla) to see
if your problem persists

From what I understand the error is
triggered sometime after starting a db.

Check the usual suspects unix kernel params such as NPROC, MAXUSERS and all SEM'S*
and SHARED SEGMENT params.
Then I would check IPCS and see if it is aligned (one memory segment for complete SGA)
and that semaphores behave correctly)

Well see how you get on.
good luck

Re: Oracle reported Ora-07445 error, ugrent! Please help.

刘绍江 — Tue, 13 Dec 2005 00:54:25 GMT

Hi, Dear

Thanks for all your kindly support on this case.
Now The problem was fixed. I would like to
explain where the problem is:

In face, everything will cause the ora-07445 error. such as OS patch, ORACLE BUG, APPLICATION BUG, OR CPU, MEMORY.

I am sure that it is not oracle paramter problem as the oracle instance can be run at other node with same parameter configuration.

So we did a test on memory and CPU, de-active one by one, at least, after we deactived the Main CPU(0), The oracle did not report such error.
I monitored the status for several days, it can work well.
But CPU did not report any error at system level.

In a word, One Cpu caused this trouble.
The problem was fixed drastically after Hp replaced the failed CPU.

Re: Oracle reported Ora-07445 error, ugrent! Please help.

刘绍江 — Tue, 13 Dec 2005 01:43:31 GMT

The cpu caused this problem, Please browse my post for detailed information.