Possible Memory Leak??

[Con O'Kelly]

Hi All

Have a problem with L-3000 11.00 server with 12GB of memory.

After a re-boot the server starts to use increasing amounts of memory. Starts off with about 8GB free and within an hour it's all used and the system starts swapping.

I can't identify the process(es) responsible.
Tried sorting processes by memory usage using Glance and also UNIX95= ps -el command but both don't show any obvious process(es) that are utilising almost 8GB of memory within an hour.

Also process data from Measureware and nothing jumps out. All the additional memory usage is user memory. Buffer cahce is constant around 300-500MB.

The only change was an install of JFS 3.3 last night.

I have sdone multiple re-boots and every time the same thing happens.

Any ideas what could be causing this?
Thanks

Cheers
Con

[Steven E. Protter]

Usually its an application that leaks memory.

You should be able to use glance to identify the process that has growing memory use. Maybe even it will show up on top.

I'd take the following steps:

1) Take a look at any oracle apps and see if there are memory patches
2) Make sure the system itself is up to date on patches. Maybe there is one for Online JFS.

SEP

[Sridhar Bhaskarla]

Hi Con,

If there is any increase in memory usage due to JFS upgrade, I would expect it to be 'system memory' due to vx-inode cache.

Since the increase is in user memory, I would suggest you to look any other changes. Run the following commands periodically and see if you see any increase in the first column for any processes over the time. If so, then those processes may have memory leaks.

UNIX95= ps -e -o 'vsz pid args' |sort -n

-Sri

[bhavin asokan]

hi,

almost same problem was with me.
after i gave the sybase group the privilage of MLOCK it started working fine.

i gave like this.
#setprivgrp sybase12 MLOCK

as per our DBA for oracle also we have to do this.

regds,

[Con O'Kelly]

Hi

Thanks for the replies.

Sri thats exactly the command I was using to try and identify the leak but it didn't show anything. No process was increasing memory usage even though user memory was increasing all the time.

Bhavin - that sounds an interesting idea as we are running Sybase - I'll look into that as that could be the issue.

I'll assign points tomorrow.

Thanks

Cheers
Con

[Ivajlo Yanakiev]

I have same problem.
Patches fix problem.
Apply last patch bundle

[Sridhar Bhaskarla]

Couple more things...

1. Are you sure you have the same number of processes everytime you run that command?.

2. How about 'ipcs -ma'. Do you see massive shared memory segments in it that were not there before?.

3. Can you post the last lines of your glance's 'm' screen?. It shows how the memory is distributed..

MLOCK will

[Sridhar Bhaskarla]

Excuse my impatience to work with the windows. I did something causing it to post the earlier message prematurely.

MLOCK capability will make that particular group to 'lock' memory pages so that they won't get paged out in case of memory pressure. It may improve your application's performance but will not reduce the memory usage.

Patching is always a good idea.

-Sri

PS: 0 points for this post please.

[Eric Antunes]

Con,

I've installed JFS 3.3 on my test system and seem to have no problem. Did you applied patch PHKL_30927 after upgrading to JFS 3.3??

Best Regards,

Eric Antunes

[Con O'Kelly]

Hi

Thanks for the help.

Through trial & error we have identified the application that appears to be causing the issue.
When the app is running, the memory leak occurs. From glance we can see Total Memory utilisation increasing with a corresponding increase in User Memory utilisation. Once the application is shutdown memory utilisation stabilises.

The strange thing is that the app's processes do not show any increase in their memory use. The 3 processes only use about 4MB of memory. I have used glance and the UNIX95= ps -e.... command and memory usage stays the same.
The app is a third party security product not a database.
Shared memory also stays constant.

Any ideas on how we can see what process(es) is using this memory?

Thanks.

Cheers
Con

[Ermin Borovac]

The following KB article may be useful.

Date: 9/9/04
Document description: Memory Usage - What is using all of the memory?"
Document id: MEMORYKBAN00000975

I would get hold of procsize utility which can show you per-process memory usage including UAREA, TEXT, DATA, STACK, Shared Memory (SHMEM), & Memory Mapped files (MMAP).

You can sort processes by total memory usage

# procsize -fcn |sort -rnk 11 | more

[Geoff Wild]

Glance will help you find it - using the "Global Process List" (g key) from within glance...look at the RSS field...select the process with large RSS by using the "s" key...
then a capital "M" to see the "Memory Regions Page"

Check the maxdsiz kernel parameter - maybe your application requires it to be increased?

Rgds...Geoff

[Con O'Kelly]

Hi again

Thanks for the help.

We have managed to isolate the problem to an issue with CA's eTrust product and a new kernel that is built with the JFS 3.3 Filesystem install.

From kmeminfo I can see large amounts of kernel memory for dynamic data being allocated (about 10.5GB - 90% total memory).

The problem only occurs when eTrust is running. As soon as it is shutdown memory utilisation stabilises, though the dynamic memory is not returned. (we need to reboot to do this).

There is also no problem if we boot using the kernel prior to the JFS 3.3 install. The problem only occured after a JFS 3.3 install.
The only changes to the kernel with JFS 3.3 install are the adding of the "vxportal33" and "vxfs" driver/subsystem. The kernel also no longer has the "vxadv" and "vxbase" drivers which I assume were replaced by the above 2 drivers with the JFS 3.3 install.

It appears that eTrust has a problem with these drivers and as such it chews up large amounts of memory.
Has anybody had similar issue with eTrust and JFS 3.3?

Cheers
Con

[Con O'Kelly]

Hi

The issue has been fixed.

I should have read the replies as Eric provided the answer!
I did look at the patch (PHKL_30927) Eric mentioned but it didn't seem to address my issue. I forgot about the patches that it superseeded! Indeed patch PHKL_21077 addressed the memory leak issue with JFS 3.3 and certain system calls.

Still unsure why this problem only occured with eTrust running and not when Oracle or other app's were running.

Thanks Eric, sorry not to give you the rabbit, I'd already assigned points when I got the fix.
BTW where did you find the references to this patch. It is not mentioned in Release Notes nor is it included in the JFS 3.3 foundation patch bundle.

Thanks.

Cheers
Con