Shared memory probelm with oracle

[uvc_1]

Hi,

We've got a strange problem here. We have a 11.11 sysetm running a very lareg Oracle RAC.

We had a situation when our machine crashed. From the dump analysis, it seems as if the DB tried to start and allocate the shared memory twice - hence causing the machine a sever depletion of memory - and a crash.

When we tried to see what happens on shutdown, it seems that sometimes, upon database shutdown, not all DB processes (oracleXXXX) die with the database. Also, the shared memory segment asociated with the database is not removed. We had to manually kill them sometimes.

My question is: does anyone know of this phenomenon? Is it a normal behaviour of oracle to leave active processes after shutdown was completed? how would you solve this? plant a kill in the script?

[Eric Antunes · Accepted solution]

Hi,

There is a bug in the 9.2.0.4 version, corrected in 9.2.0.5 (Server Patch Set). What is your RDBMS version??

Best Regards,

Eric Antunes

[uvc_1]

indeed, it's 9.2.0.4. It's strange that oracle didn't tell us so - we opened a call! do you have more information such as bug number or something like that?

[Eric Antunes]

Hi,

Yes, it's bug n.º 3305683. See here: http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=3305683.8 and http://metalink.oracle.com/metalink/plsql/showdoc?db=Bug&id=3305683

Best Regards,

Eric Antun

[uvc_1]

Thanks a lot.
After reading the notes, I'm not sure this is our problem, for several reasons:

1. The situation occurs both on shutdown normal, immediaet and abort
2. It does not hang the script. We the script continues after the shutdown, but there are some living server processes.
3. It's not a consistent behaviour - happens only sometimes.

...Am I wrong?

[Eric Antunes]

Hi again,

If it happens only sometimes, then it may be not that bug but this one: http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=2495540.8

Do you get any "ORA-" error message in alert-.log?

Best Regards,

Eric Antunes

[Bill Hassell]

It is extremely unusual to have a server crash in HP-UX due to memory problems. You can request (and get) much more memory than the server has installed as HP-UX is a virtual memory system. The underlying reason for the crash is likely a known problem that can be patched but you'll need your OS crash dump analyzed to find the reason. The HP Response Center can do this analysis even if you don't have a software contract (there may be a small charge). Make sure you have patches up to date.

It is also not uncommon to have modules not release shared memory segments during a shutdown. But the underlying reason is poor programming and not anything to do with the system. It seems many programmers grab shared memory thinking that somehow the system will know what to do with orphaned programs.

It is VERY important to have the shutdown procedure never use kill -9 as the -9 GUARENTEES orphaned shared memory segments as well as possible corrupted indexes and data elements. If a process hangs and cannot be killed with kill -15 or kill -1, flag that program as needing a patch or rewrite.

[Steven E. Protter]

I've seen Oracle do this but it didn't crash my server. It made the server very slow and caused heavy paging to occur.

You may have a problem with ownership of the oracle program binaries, and therefore the shared memory they allocate.

This can be as a result of the bug mentioned above.

You should have /etc/rc.config.d/savecrash configured to save crash dumps and you should send the next crash dump to HP for Q4 analysis.

SEP

[uvc_1]

The server did not crash. It simply hanged due to 100% memory and swap usage. We had to TOC, and when HP personnel analised the dump, they said there was a lot of shared memory occupied by oracle.

There were no ORA-600 from what I can see in the alert log. Simple an active server process that tusc showed to be stuck in a read() function. probably from a socket. normal kill (-15) made him die nicely.

[uvc_1]

sorry, that one came out confusing.
At one time, the server hanged than was TOC'ed.
Afterwards , we put into the script a ps to search for living server processes, and we found on several times a process still existing after the shutdown was completed. a normal kill removed it and freed the shared memory.
The question is why such processes still exist after the shutdown?