Monitoring Process Memory Usage & Stale Oracle Processes

[EDWARD WYNN_1]

Hi,

Following on from a recent discussion in another thread I have determined that I have 2 memory leak problems on my machine:

1. We have an in-house developed process that has a memory leak. This process is a CORBA server that provides an interface to an Oracle instance.

3. If our in-house process crashes it does not clean up its connections to oracle, this leaves stale processes using memory until the instance is shut down.

I was wondering if anyone had any thoughts on:

a) How to monitor, in detail, the memory usage of a single process - not just the total used but when it is allocated/ freed etc.

b) Given that I cannot shutdown the Oracle instance nightly or even weekly for cold backup how can I get rid of the stale oracle processes - or better still how can I avoid them in the first place ? I have read something in the forums that this situation can happen if the process is in the middle of IO when the controller process crashes...

Many thanks in advance, Eddie

[Alexander M. Ermes]

Hi there.
Tried glance already ?
That gives you a chance, to monitor everything on your system.
Rgds
Alexander M. Ermes

[EDWARD WYNN_1]

Hi,

Haven't installed glance - is this a freebie or do you need a codeword ? If it is a freebie do you happen to know which of the apps CD's it is on ?

Thanks, Eddie

[Carlos Fernandez Riera]

You can find a trial version on App. CDs.

But if you like to know when... maybe tusc is the correct tool. The use of this tool means that you are familiar with system calls.


The other way is look into your oracle, v$process and v$session will help you (?).
Also, if you use listener for your connection, will can specify a timeout:

From ORACLE_HOME/network/admin/samples/sqlnet.ora:

########################
#sqlnet.expire_time = 10
########################
#
#Possible values: 0-any valid positive integer! (in minutes)
#Default: 0 minutes
#Recommended value: 10 minutes
#
#Purpose: Indicates the time interval to send a probe to verify the
# client session is alive (this is used to reclaim watseful
# resources on a dead client)

[EDWARD WYNN_1]

I think I will install the trial version of glance.

The expire_time line in sqlnet.ora sounds interesting, I do use a listener but the processes and connections in question are local ones and use the bequeth connection - do you know if there is a similar mechanism for these connections ?

Thanks, Eddie

[Carlos Fernandez Riera]

See:
http://forums.itrc.hp.com/cm/QuestionAnswer/1,11866,0x82f8c6af36b7d5118ff10090279cd0f9,00.html

[EDWARD WYNN_1]

Thanks for the input so far.

I have tried the Oracle PROFILE method mentioned in the linked thread but unfortunately this does not do what I need.

This mechanism only kills the oracle session if it is re-used after the IDLE timeout period has expired. I need something that will kill both the Oracle session and Unix process without the session having to be re-used.

I would be very grateful if anyone has any further ideas on this.

Thanks, Eddie

[Tracey]

I hope someone has some more suggestions, I am having the same problem.

Thanks

[EDWARD WYNN_1]

Hi Tracey,

I thought I would let you know that I got no-where with this, I even raised a TAR with oracle in the end and they have confirmed that there is no-way of doing what I needed to do - although they do say that this situation should never happen when using BEQ connections.

Here is the text of the TAR
### Detailed Problem Statement: ###
I have a problem with my database whereby occasionally our CORBA Server process
crashes and does not clean up its connections to Oracle. This means I am left
with dead processes that I cannot clean up - the only way to remove them is to
kill -9 them or shutdown the database - neither of which can be done easily in
my operating environment.
What I want to do is implement a form of dead connection detection such that
these dead processes can be killed off automatically.
The problem is that as this CORBA process is running on the same machine as
oracle it is using the BEQ protocol adapter to make its connections, it is also
using this protocol as it is much faster than TCP or IPC. However, my
understanding of DCD is that it only works with TCP and IPC connections.
I have tried using the PROFILE IDLE_TIME mechanism to get around this but this
relies on something re-using an idle connection before it closes the session -
in any case it does not kill the Unix process so it would not be any use to me.
In short I have 3 questions:
1) Is there anyway of killing off BEQ protocol connections like DCD does for
TCP and IPC ?
2) If (1) above cannot be done can you confirm if DCD would do what I require
(close the oracle session and kill the Unix process) after the expiry_time
elapses if I was to use the IPC protocol.
3) Have you any figures on relative/average database connection times using the
3 protocols available to me (BEQ, IPC and TCP) ?
.
### What were you trying to accomplish when the problem occurred? ###
See above explanation
.
### Has this ever worked? ###
No
.
### Can you reproduce the problem at will? ###
Yes
.
### Reproducible on other platforms, systems, environments? ###
No
.
### Known recent changes to your computer environment: ###
None
.
Contact me via : E-mail -> eddiew@oakleafconsultancy.com


11-OCT-01 16:57:32
=== Comments ===
Thank you for logging this TAR via Metalink. We have received your TAR and are
looking into it now. We will contact you shortly with our response.
Regards,
Marc CHARPENTIER


12-OCT-01 11:31:45
=== Analysis ===
1)DCD is only supported on IPC and TCP, there is no solution for BEQ connection
Because DCD has been implemented for network failure and client node crash
as BEQ are for localy connections these two cases could not exist.
When the client dies , it has to send a signal and the server process disapears
2)Normaly , the session is killed and the Unix process server disapears.
But sometime, the session is marked to kill and it can take time to rollback
or release the resources.
3)I don't have any comparative figure with the 3 protocols.
the fastest should be BEQ then IPc (unix domain Sockect) then TCP.


12-OCT-01 11:37:23
For more explanation on DCD you could take a look at note 151972.1
Thanks to update and close these tar .Thanks Alain RICHY
(Without any news , I'll close these tar the 15/10/01 )


12-OCT-01 12:09:02
=== Comments ===
Closed by customer
Many Thanks for your input, I kind of suspected that this would be your answer
-maybe I will raise an enhancement request for something to be done to kill off
stale BEQ connected processes.

I have not tried using the IPC and TCP connections yet - but may do if this is still an issue in the future, although I cannot really afford the performance hit it will mean.

Hope this helps someone !

Eddie