Where is my memory going?

[Tony Walker_2]

Hi guys,

Got a RH AS 4u3 server that I'd appreciate a second opinion on...

Its got 16GB of phys which is showing used as follows:

# free
total used free shared buffers cached
Mem: 16413860 16391888 21972 0 2048 45396
-/+ buffers/cache: 16344444 69416
Swap: 17800012 6590068 11209944

As you can see even withouth buffers/cache its almost at capacity. This can also be seen in sar -r:

06:08:11 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
06:08:13 PM 23476 16390384 99.86 1412 30692 11209972 6590040 37.02 1789208
06:08:15 PM 23540 16390320 99.86 1420 30684 11209972 6590040 37.02 1789208
06:08:17 PM 23636 16390224 99.86 1436 30668 11209972 6590040 37.02 1789208
06:08:19 PM 23636 16390224 99.86 1436 30668 11209972 6590040 37.02 1789208
06:08:21 PM 23772 16390088 99.86 1456 30716 11209972 6590040 37.02 1789208
Average: 23612 16390248 99.86 1432 30686 11209972 6590040 37.02 1789208

However, I've checked using ps -eo rss,vsz and can only see ~9GB of vsz and ~1.2GB of rss. So, on the face of it I'm wondering why with ~9GB virtual in use I seem to be 99% utilised. I can see pageing activity, but not much swapping:

# sar -B 2 5
Linux 2.6.9-34.ELsmp (loleqseueqd06) 06/26/2007

06:09:57 PM pgpgin/s pgpgout/s fault/s majflt/s
06:09:59 PM 214.29 55.10 554.08 0.00
06:10:01 PM 1051.55 183.51 1784.54 1.03
06:10:03 PM 3675.13 318.78 5072.08 170.56
06:10:05 PM 6297.44 162.05 3318.46 215.90
06:10:07 PM 4826.53 148.98 3083.16 108.67
Average: 3214.72 173.82 2766.26 99.39

06:10:36 PM pswpin/s pswpout/s
06:10:38 PM 6.67 0.00
06:10:40 PM 3.06 0.00
06:10:42 PM 10.71 0.00
06:10:44 PM 0.00 0.00
06:10:46 PM 0.00 0.00
Average: 4.09 0.00

Any help in identifying where the other 8ish GB is would be great. I realise that RH holds onto memory and relinquishes it as required but this seems excessive..

Thanks,

Tony

[Ivan Ferreira]

Please run ipcs -a. If you have a database in this server, the remaining memory could be shared memory usage, for example, oracle SGA.

[Steven E. Protter]

Shalom Tony,

The OS tends to allocate unused memory into the buffer cache. They seem to hav edone a very good job getting the cache to release it quickly when requested.

I would think that a database or other shared memory user is grabbing the extra memory.

AS to why the buffer cache is not reduced to compensate, that may be a bug.

SEP

[Tony Walker_2]

Thanks guys,

The situation persists. ipcs doesn't throw up anything. There are really only 2 java procs running and a realtime q manager. At present vsz total = ~4g but still 99% memutil. Swapping (inc major page faults is ongoing too):

09:54:55 AM pgpgin/s pgpgout/s fault/s majflt/s
09:54:57 AM 228.57 30.61 49.49 8.67
09:54:59 AM 176.41 8.21 58.97 5.64
09:55:01 AM 175.51 542.86 72.45 5.10
09:55:03 AM 90.26 6553.85 4494.87 3.59
09:55:05 AM 193.88 0.00 203.57 6.12
Average: 173.01 1423.31 973.21 5.83

09:56:15 AM pswpin/s pswpout/s
09:56:17 AM 56.41 0.00
09:56:19 AM 23.59 0.00
09:56:21 AM 37.56 0.00
09:56:23 AM 61.03 0.00
09:56:25 AM 38.58 0.00
Average: 43.41 0.00

Average: 21528 16392332 99.87 6033 83358 16496786 1303226 7.32 308014

Thanks,

Tony

[Stuart Browne]

Tune swappiness down to 0, and see what it does (echo 0 > /proc/sys/vm/swappiness).

[Stuart Browne]

Oh, a full process list (ps -elf) will help us give you a more reasonable answer for the swap usage.

But my bet is it's just allocated memory that hasn't been used, so the kernel swapped it out to more efficiently use avaialble resources (physical memory).

swappiness = 0 will stop the kernel from that behaviour though.. and only allocate swap when all physical memory is exhausted.