cancel
Showing results for 
Search instead for 
Did you mean: 

Where is my memory going?

 
Tony Walker_2
Frequent Advisor

Where is my memory going?

Hi guys,

Got a RH AS 4u3 server that I'd appreciate a second opinion on...

Its got 16GB of phys which is showing used as follows:

# free
total used free shared buffers cached
Mem: 16413860 16391888 21972 0 2048 45396
-/+ buffers/cache: 16344444 69416
Swap: 17800012 6590068 11209944

As you can see even withouth buffers/cache its almost at capacity. This can also be seen in sar -r:

06:08:11 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
06:08:13 PM 23476 16390384 99.86 1412 30692 11209972 6590040 37.02 1789208
06:08:15 PM 23540 16390320 99.86 1420 30684 11209972 6590040 37.02 1789208
06:08:17 PM 23636 16390224 99.86 1436 30668 11209972 6590040 37.02 1789208
06:08:19 PM 23636 16390224 99.86 1436 30668 11209972 6590040 37.02 1789208
06:08:21 PM 23772 16390088 99.86 1456 30716 11209972 6590040 37.02 1789208
Average: 23612 16390248 99.86 1432 30686 11209972 6590040 37.02 1789208

However, I've checked using ps -eo rss,vsz and can only see ~9GB of vsz and ~1.2GB of rss. So, on the face of it I'm wondering why with ~9GB virtual in use I seem to be 99% utilised. I can see pageing activity, but not much swapping:

# sar -B 2 5
Linux 2.6.9-34.ELsmp (loleqseueqd06) 06/26/2007

06:09:57 PM pgpgin/s pgpgout/s fault/s majflt/s
06:09:59 PM 214.29 55.10 554.08 0.00
06:10:01 PM 1051.55 183.51 1784.54 1.03
06:10:03 PM 3675.13 318.78 5072.08 170.56
06:10:05 PM 6297.44 162.05 3318.46 215.90
06:10:07 PM 4826.53 148.98 3083.16 108.67
Average: 3214.72 173.82 2766.26 99.39

06:10:36 PM pswpin/s pswpout/s
06:10:38 PM 6.67 0.00
06:10:40 PM 3.06 0.00
06:10:42 PM 10.71 0.00
06:10:44 PM 0.00 0.00
06:10:46 PM 0.00 0.00
Average: 4.09 0.00

Any help in identifying where the other 8ish GB is would be great. I realise that RH holds onto memory and relinquishes it as required but this seems excessive..

Thanks,

Tony
5 REPLIES
Ivan Ferreira
Honored Contributor

Re: Where is my memory going?

Please run ipcs -a. If you have a database in this server, the remaining memory could be shared memory usage, for example, oracle SGA.
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Steven E. Protter
Exalted Contributor

Re: Where is my memory going?

Shalom Tony,

The OS tends to allocate unused memory into the buffer cache. They seem to hav edone a very good job getting the cache to release it quickly when requested.

I would think that a database or other shared memory user is grabbing the extra memory.

AS to why the buffer cache is not reduced to compensate, that may be a bug.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Tony Walker_2
Frequent Advisor

Re: Where is my memory going?

Thanks guys,

The situation persists. ipcs doesn't throw up anything. There are really only 2 java procs running and a realtime q manager. At present vsz total = ~4g but still 99% memutil. Swapping (inc major page faults is ongoing too):

09:54:55 AM pgpgin/s pgpgout/s fault/s majflt/s
09:54:57 AM 228.57 30.61 49.49 8.67
09:54:59 AM 176.41 8.21 58.97 5.64
09:55:01 AM 175.51 542.86 72.45 5.10
09:55:03 AM 90.26 6553.85 4494.87 3.59
09:55:05 AM 193.88 0.00 203.57 6.12
Average: 173.01 1423.31 973.21 5.83

09:56:15 AM pswpin/s pswpout/s
09:56:17 AM 56.41 0.00
09:56:19 AM 23.59 0.00
09:56:21 AM 37.56 0.00
09:56:23 AM 61.03 0.00
09:56:25 AM 38.58 0.00
Average: 43.41 0.00

Average: 21528 16392332 99.87 6033 83358 16496786 1303226 7.32 308014

Thanks,

Tony

Stuart Browne
Honored Contributor

Re: Where is my memory going?

Tune swappiness down to 0, and see what it does (echo 0 > /proc/sys/vm/swappiness).
One long-haired git at your service...
Stuart Browne
Honored Contributor

Re: Where is my memory going?

Oh, a full process list (ps -elf) will help us give you a more reasonable answer for the swap usage.

But my bet is it's just allocated memory that hasn't been used, so the kernel swapped it out to more efficiently use avaialble resources (physical memory).

swappiness = 0 will stop the kernel from that behaviour though.. and only allocate swap when all physical memory is exhausted.
One long-haired git at your service...