Simpler Navigation for Servers and Operating Systems - Please Update Your Bookmarks
Completed: a much simpler Servers and Operating Systems section of the Community. We combined many of the older boards, so you won't have to click through so many levels to get at the information you need. Check the consolidated boards here as many sub-forums are now single boards.
If you have bookmarked forums or discussion boards in Servers and Operating Systems, we suggest you check and update them as needed.
General
cancel
Showing results for 
Search instead for 
Did you mean: 

Shared memory probelm with oracle

SOLVED
Go to solution
uvc_1
Occasional Visitor

Shared memory probelm with oracle

Hi,

We've got a strange problem here. We have a 11.11 sysetm running a very lareg Oracle RAC.

We had a situation when our machine crashed. From the dump analysis, it seems as if the DB tried to start and allocate the shared memory twice - hence causing the machine a sever depletion of memory - and a crash.

When we tried to see what happens on shutdown, it seems that sometimes, upon database shutdown, not all DB processes (oracleXXXX) die with the database. Also, the shared memory segment asociated with the database is not removed. We had to manually kill them sometimes.

My question is: does anyone know of this phenomenon? Is it a normal behaviour of oracle to leave active processes after shutdown was completed? how would you solve this? plant a kill in the script?
14 REPLIES
Eric Antunes
Honored Contributor
Solution

Re: Shared memory probelm with oracle

Hi,

There is a bug in the 9.2.0.4 version, corrected in 9.2.0.5 (Server Patch Set). What is your RDBMS version??

Best Regards,

Eric Antunes
Each and every day is a good day to learn.
uvc_1
Occasional Visitor

Re: Shared memory probelm with oracle

indeed, it's 9.2.0.4. It's strange that oracle didn't tell us so - we opened a call! do you have more information such as bug number or something like that?
Eric Antunes
Honored Contributor

Re: Shared memory probelm with oracle

Hi,

Yes, it's bug n.º 3305683. See here: http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=3305683.8 and http://metalink.oracle.com/metalink/plsql/showdoc?db=Bug&id=3305683

Best Regards,

Eric Antun
Each and every day is a good day to learn.
uvc_1
Occasional Visitor

Re: Shared memory probelm with oracle

Thanks a lot.
After reading the notes, I'm not sure this is our problem, for several reasons:

1. The situation occurs both on shutdown normal, immediaet and abort
2. It does not hang the script. We the script continues after the shutdown, but there are some living server processes.
3. It's not a consistent behaviour - happens only sometimes.

...Am I wrong?
Eric Antunes
Honored Contributor

Re: Shared memory probelm with oracle

Hi again,

If it happens only sometimes, then it may be not that bug but this one: http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=2495540.8

Do you get any "ORA-" error message in alert-.log?

Best Regards,

Eric Antunes
Each and every day is a good day to learn.
Bill Hassell
Honored Contributor

Re: Shared memory probelm with oracle

It is extremely unusual to have a server crash in HP-UX due to memory problems. You can request (and get) much more memory than the server has installed as HP-UX is a virtual memory system. The underlying reason for the crash is likely a known problem that can be patched but you'll need your OS crash dump analyzed to find the reason. The HP Response Center can do this analysis even if you don't have a software contract (there may be a small charge). Make sure you have patches up to date.

It is also not uncommon to have modules not release shared memory segments during a shutdown. But the underlying reason is poor programming and not anything to do with the system. It seems many programmers grab shared memory thinking that somehow the system will know what to do with orphaned programs.

It is VERY important to have the shutdown procedure never use kill -9 as the -9 GUARENTEES orphaned shared memory segments as well as possible corrupted indexes and data elements. If a process hangs and cannot be killed with kill -15 or kill -1, flag that program as needing a patch or rewrite.


Bill Hassell, sysadmin
Steven E. Protter
Exalted Contributor

Re: Shared memory probelm with oracle

I've seen Oracle do this but it didn't crash my server. It made the server very slow and caused heavy paging to occur.

You may have a problem with ownership of the oracle program binaries, and therefore the shared memory they allocate.

This can be as a result of the bug mentioned above.

You should have /etc/rc.config.d/savecrash configured to save crash dumps and you should send the next crash dump to HP for Q4 analysis.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
uvc_1
Occasional Visitor

Re: Shared memory probelm with oracle

The server did not crash. It simply hanged due to 100% memory and swap usage. We had to TOC, and when HP personnel analised the dump, they said there was a lot of shared memory occupied by oracle.

There were no ORA-600 from what I can see in the alert log. Simple an active server process that tusc showed to be stuck in a read() function. probably from a socket. normal kill (-15) made him die nicely.
uvc_1
Occasional Visitor

Re: Shared memory probelm with oracle

sorry, that one came out confusing.
At one time, the server hanged than was TOC'ed.
Afterwards , we put into the script a ps to search for living server processes, and we found on several times a process still existing after the shutdown was completed. a normal kill removed it and freed the shared memory.
The question is why such processes still exist after the shutdown?
Bill Hassell
Honored Contributor

Re: Shared memory probelm with oracle

You'll have to manually trace the shutdown script to see errors are reported. Use sh -x oracle_shutdown_script to see what is happening. Each process that fails to stop either did not get signaled properly or it is hung-up and probably needs a patch to prevent this problem.


Bill Hassell, sysadmin
Eric Antunes
Honored Contributor

Re: Shared memory probelm with oracle

Hi,

You can see a full list of bugs for 9.2.0.4 at: http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=245758.1.

There is bug 2905667 ("Instance terminated by CKPT due to ORA-1110 during shutdown in RAC") at the RAC section that may be your issue (see note http://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=2905667.8 for more info...). Can you see, for each instance, in the background_dump_dest (select value from v$parameter where name = 'background_dump_dest') destination if there are any ckpt_*.trc?

Best Regards,

Eric Antunes
Each and every day is a good day to learn.
Eric Antunes
Honored Contributor

Re: Shared memory probelm with oracle

Hi,

You are already at 9.2.0.4 so you don't need to look at the previous link I gave you: look at those:

http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=283897.1 for bugs corrected in 9.2.0.6

and

http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=263791.1 for bugs corrected in 9.2.0.5

Can you see, for each instance, in the background_dump_dest directory if there are any *.trc with the datetime of the last shutdown?

Best Regards,

Eric Antunes
Each and every day is a good day to learn.
Eric Antunes
Honored Contributor

Re: Shared memory probelm with oracle

Hi,

Do you have 3 RAC nodes? I'm asking this because bug 2495540 (and corresponding patch) is about ORA-600 errors on the 3rd node.

Regards,

Eric
Each and every day is a good day to learn.
Eric Antunes
Honored Contributor

Re: Shared memory probelm with oracle

Hi,

Did you solved your issue?

Best Regards,

Eric Antunes
Each and every day is a good day to learn.