Shared memory probelm with oracle

[uvc_1]

Hi,

We've got a strange problem here. We have a 11.11 sysetm running a very lareg Oracle RAC.

We had a situation when our machine crashed. From the dump analysis, it seems as if the DB tried to start and allocate the shared memory twice - hence causing the machine a sever depletion of memory - and a crash.

When we tried to see what happens on shutdown, it seems that sometimes, upon database shutdown, not all DB processes (oracleXXXX) die with the database. Also, the shared memory segment asociated with the database is not removed. We had to manually kill them sometimes.

My question is: does anyone know of this phenomenon? Is it a normal behaviour of oracle to leave active processes after shutdown was completed? how would you solve this? plant a kill in the script?

[Eric Antunes · Accepted solution]

Hi,

There is a bug in the 9.2.0.4 version, corrected in 9.2.0.5 (Server Patch Set). What is your RDBMS version??

Best Regards,

Eric Antunes

[uvc_1]

indeed, it's 9.2.0.4. It's strange that oracle didn't tell us so - we opened a call! do you have more information such as bug number or something like that?

[Eric Antunes]

Hi,

Yes, it's bug n.º 3305683. See here: http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=3305683.8 and http://metalink.oracle.com/metalink/plsql/showdoc?db=Bug&id=3305683

Best Regards,

Eric Antun

[uvc_1]

Thanks a lot.
After reading the notes, I'm not sure this is our problem, for several reasons:

1. The situation occurs both on shutdown normal, immediaet and abort
2. It does not hang the script. We the script continues after the shutdown, but there are some living server processes.
3. It's not a consistent behaviour - happens only sometimes.

...Am I wrong?

[Eric Antunes]

Hi again,

If it happens only sometimes, then it may be not that bug but this one: http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=2495540.8

Do you get any "ORA-" error message in alert-.log?

Best Regards,

Eric Antunes

[Bill Hassell]

It is extremely unusual to have a server crash in HP-UX due to memory problems. You can request (and get) much more memory than the server has installed as HP-UX is a virtual memory system. The underlying reason for the crash is likely a known problem that can be patched but you'll need your OS crash dump analyzed to find the reason. The HP Response Center can do this analysis even if you don't have a software contract (there may be a small charge). Make sure you have patches up to date.

It is also not uncommon to have modules not release shared memory segments during a shutdown. But the underlying reason is poor programming and not anything to do with the system. It seems many programmers grab shared memory thinking that somehow the system will know what to do with orphaned programs.

It is VERY important to have the shutdown procedure never use kill -9 as the -9 GUARENTEES orphaned shared memory segments as well as possible corrupted indexes and data elements. If a process hangs and cannot be killed with kill -15 or kill -1, flag that program as needing a patch or rewrite.

[Steven E. Protter]

I've seen Oracle do this but it didn't crash my server. It made the server very slow and caused heavy paging to occur.

You may have a problem with ownership of the oracle program binaries, and therefore the shared memory they allocate.

This can be as a result of the bug mentioned above.

You should have /etc/rc.config.d/savecrash configured to save crash dumps and you should send the next crash dump to HP for Q4 analysis.

SEP

[uvc_1]

The server did not crash. It simply hanged due to 100% memory and swap usage. We had to TOC, and when HP personnel analised the dump, they said there was a lot of shared memory occupied by oracle.

There were no ORA-600 from what I can see in the alert log. Simple an active server process that tusc showed to be stuck in a read() function. probably from a socket. normal kill (-15) made him die nicely.

[uvc_1]

sorry, that one came out confusing.
At one time, the server hanged than was TOC'ed.
Afterwards , we put into the script a ps to search for living server processes, and we found on several times a process still existing after the shutdown was completed. a normal kill removed it and freed the shared memory.
The question is why such processes still exist after the shutdown?

[Bill Hassell]

You'll have to manually trace the shutdown script to see errors are reported. Use sh -x oracle_shutdown_script to see what is happening. Each process that fails to stop either did not get signaled properly or it is hung-up and probably needs a patch to prevent this problem.

[Eric Antunes]

Hi,

You can see a full list of bugs for 9.2.0.4 at: http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=245758.1.

There is bug 2905667 ("Instance terminated by CKPT due to ORA-1110 during shutdown in RAC") at the RAC section that may be your issue (see note http://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=2905667.8 for more info...). Can you see, for each instance, in the background_dump_dest (select value from v$parameter where name = 'background_dump_dest') destination if there are any ckpt_*.trc?

Best Regards,

Eric Antunes

[Eric Antunes]

Hi,

You are already at 9.2.0.4 so you don't need to look at the previous link I gave you: look at those:

http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=283897.1 for bugs corrected in 9.2.0.6

and

http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=263791.1 for bugs corrected in 9.2.0.5

Can you see, for each instance, in the background_dump_dest directory if there are any *.trc with the datetime of the last shutdown?

Best Regards,

Eric Antunes

[Eric Antunes]

Hi,

Do you have 3 RAC nodes? I'm asking this because bug 2495540 (and corresponding patch) is about ORA-600 errors on the 3rd node.

Regards,

Eric

[Eric Antunes]

Hi,

Did you solved your issue?

Best Regards,

Eric Antunes