High Memory utilization

[sbahra]

Redhat Linux version : 5.6

[root@sl]# free
             total       used       free     shared    buffers     cached
Mem:      16436192   16313340     122852          0      16752    1674012
-/+ buffers/cache:   14622576    1813616
Swap:     18874360    2769164   16105196

[root@sl]# more /proc/meminfo
MemTotal:     16436192 kB
MemFree:        122604 kB
Buffers:         17112 kB
Cached:        1674340 kB
SwapCached:     353740 kB
Active:        3332188 kB
Inactive:        48784 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     16436192 kB
LowFree:        122604 kB
SwapTotal:    18874360 kB
SwapFree:     16105196 kB
Dirty:             968 kB
Writeback:           0 kB
AnonPages:     1685236 kB
Mapped:        1585500 kB
Slab:            72584 kB
PageTables:     180108 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  20801000 kB
Committed_AS:  8598100 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    316556 kB
VmallocChunk: 34359421495 kB
HugePages_Total:  6144
HugePages_Free:   5318
HugePages_Rsvd:    440
Hugepagesize:     2048 kB

top - 03:05:45 up 95 days, 11:29,  3 users,  load average: 1.92, 1.76, 1.72
Tasks: 320 total,   1 running, 318 sleeping,   0 stopped,   1 zombie
Cpu(s):  5.3%us,  2.4%sy,  0.0%ni, 47.4%id, 44.9%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16436192k total, 16317376k used,   118816k free,    17316k buffers
Swap: 18874360k total,  2769164k used, 16105196k free,  1674636k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30984 oracle    15   0 2791m 1.3g 1.3g S  0.8  8.3  49:51.66 oracle
19241 oracle    15   0 2470m 1.1g  11m S  0.8  6.9 679:08.51 oraagent.bin
 7890 oracle    15   0 2757m 670m 670m S  0.0  4.2   0:01.68 oracle
 5415 oracle    15   0 2765m 393m 388m S  0.0  2.5   0:05.98 oracle
 7894 oracle    15   0 2781m 374m 373m S  0.0  2.3   0:03.79 oracle
 5418 oracle    15   0 2767m 370m 365m S  0.0  2.3   0:05.88 oracle
28110 oracle    15   0 2765m 359m 357m S  0.0  2.2   0:13.80 oracle
 7898 oracle    15   0 2768m 348m 347m S  0.0  2.2   0:03.88 oracle
 7910 oracle    15   0 2765m 317m 313m S  0.0  2.0   0:11.49 oracle
 7166 oracle    15   0 2767m 217m 212m S  0.0  1.4   0:01.93 oracle
12021 oracle    15   0 2769m 213m 201m S  0.0  1.3   0:01.80 oracle
14811 oracle    15   0 2764m 207m 205m S  0.0  1.3   0:02.70 oracle
 7464 oracle    15   0 2762m 179m 179m S  0.0  1.1   0:55.95 oracle
28180 oracle    15   0 2762m 165m 163m S  0.0  1.0   0:28.38 oracle
 7926 oracle    15   0 2765m 160m 157m S  0.0  1.0   0:24.77 oracle
 7147 oracle    15   0 2762m 145m 141m S  0.0  0.9   0:10.35 oracle
28199 oracle    15   0 2762m 139m 136m S  0.0  0.9   0:15.57 oracle
28152 oracle    15   0 2762m 136m 133m S  0.0  0.8   0:13.16 oracle
19899 oracle    15   0 2763m 133m 133m S  0.0  0.8   0:01.98 oracle
 7263 oracle    15   0 2762m 118m 115m S  0.0  0.7   0:06.00 oracle
24326 oracle    15   0 2764m 109m 105m S  0.0  0.7   0:01.41 oracle
 5924 oracle    15   0 2767m 108m 102m S  0.0  0.7   0:00.76 oracle
11042 oracle    15   0 2762m 107m 105m S  0.0  0.7   0:07.57 oracle
 3567 oracle    15   0 2761m 107m 105m S  0.0  0.7   0:00.69 oracle
28144 oracle    15   0 2762m 100m  97m S  0.0  0.6   0:00.54 oracle
 7218 oracle    15   0 2762m  94m  92m S  0.0  0.6   0:00.27 oracle

Question:-  The free buffer/cache size in free commands shows 1813616K. Whereas if i do ps aux command to find out total amount of memory used by all the processes then it comes to around 10GB(see attachment). So as per ps aux command it tells me that i should have 4gb free out of total physical 16gb but the free buffer/cache is only 1.8G free. what else could be eating up the memory.

[craiger_]

Low/reserved ram and additional pages are reserved by the kernel, not displayed in "free" and not typically 4gigs.   I see Oracle is running here, did the oracle installer expand ulimits, increase kernel table sizes perhaps?  I don't know if you have identical hardware handy, but it'd be interesting to see the memory totals with an untweaked kernel (one that possibly doesn't consume 4gigs).

curiously,

-craig