More memory issues - UNIX95= ps?

[Greg Stark_1]

I have noticed that in the last few months a lot of our server's available memory has been decreasing. I've been using the following command to look at memory by process:

UNIX95= ps -eo 'vsz pid args' |sort -nr |head -20

First of all, is the result of "vsz" in MB's or 4K pages? Also, why doesn't this column add up to my available memeory?

Attached is the full output of the above command and swapinfo -tm and vmstat 1 10. After looking at this, I know most will recommend we need more memory. It just seems strange that the available memory has gone down on all of these servers. I recently applied patches to the servers and was wondering if there could be another problem before I throw more hardware at it.

What do you guys think?

Thanks again,
Greg

[Ross Zubritski]

When is the last time that you bounced the box? You could some some application "leakage"

Regards,

RZ

[Greg Stark_1]

On two of the servers, I also had been logging swapinfo -tm for the last year to a file. Attached is the output of that in an excel graph. I parsed out only the "memory PCT USED line. I installed the patches on Jan 14.

[Steven E. Protter]

Could not access your spreadsheet.

You're running Oracle.

More users/sessions mean more memory usage.

Leakage is a very real possibility though we have not run into that with our Oracle products. Or maybe we patched our way out of it, don't remember.

This is probably a kernel/performance tuning type issue. Is response time suffering, why are you thinking of throwing moer hardware at it?

SEP

[Bill Hassell · Accepted solution]

vsz is in Kbytes, not pages or megs (man ps). The amount of RAM reported by ps will never add up to actual RAM. The reason is that the following memory resident items are not reported by ps:

kernel memory
memory mapped files
shared libraries
shared memory segments
buffer cache

Of these, the buffer cache is often the largest user, mostly because of a bad default value in the kernel (up to 50% of RAM can be used by this cache).

The total of all processes is not important if you are looking for a memory leak. Instead, look at the largest programs to see if they are growing. If so, then go after the programmers to see why this is happening--it may be normal.

However, the vmstat metric po is the most telling: you are severely short on RAM because you are paging dozens of pages (up into hundreds). The typical penalty is 100:1 for processes that must be swapped out.