Free Memory always low - What Happened to My Memory?

[Alzhy]

I've a server, recently imaged with RHEL 5.4, patched to the latest and greatest. As I was about to finish everything and start installing Oracle, I notice that out of the 128GB Mem, I always end up with just about half a GB of mem. And there is no apps running yet. It is consistent, even after a fresh reboot -- mem free is about half a GB. MemInfo follows below.

I am clueless what's causing this and am about to just re-image and track my steps and see when this issue start to manifest itself.


# free
total used free shared buffers cached
Mem: 132101528 131514200 587328 0 4552 9224
-/+ buffers/cache: 131500424 601104
Swap: 67076088 66732 67009356

]# cat /proc/meminfo
MemTotal: 132101528 kB
MemFree: 585984 kB
Buffers: 4916 kB
Cached: 9888 kB
SwapCached: 9924 kB
Active: 18452 kB
Inactive: 6772 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 132101528 kB
LowFree: 585984 kB
SwapTotal: 67076088 kB
SwapFree: 67009360 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 8684 kB
Mapped: 5932 kB
Slab: 42140 kB
PageTables: 3948 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 67443396 kB
Committed_AS: 179648 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 302444 kB
VmallocChunk: 34359435055 kB
HugePages_Total: 64144
HugePages_Free: 64144
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

[Steven E. Protter]

Shalom,

-/+ buffers/cache: 131500424 601104

There it is.

When memory is in use Linux puts it in buffer cache to speed up I/O. Unlike older versions of HP-UX it can pull memory from cache very quickly.

SEP

[Alzhy]

Shalom too SEP.

But is this the normal behaviour? I've other servers - freshly installed, similarly patched that I've not:

presented SAN LUNs yet
installed our custom scripts
mod'd sysctl.conf, limits.conf for Oracle

And I do not see any issue.

[Alzhy]

And one more thing, our Proof of Cncept Big LINUX machine was a 256GB 48-way server (diff vendor though) and I never saw all my memory being gobbled up as chance "at once".

I'm dealing with VERY large Linux machines btw.

[Steven E. Protter]

The behavior is perfectly normal. It is the default behavior of RHEL.

Write a program to do some malloc, large chunk of memory, say 256 MB.

Your request will get what it needs instantly. The system will not page and the buffer figures will decline.

SEP

[Alzhy]

That's what RH Support told me but not quite the case. Impossible for a server with 128GB of memory with hardly any filesystems with files on them to gobble up all the memory.

I am now backtracking and I think I found the culprit. I've the latest HP Multipathing package and Redhat's latest multip-pathing package.

When I tried to collpase my LVM VG that sits on a multi-pathed EVA LUN, the OS ran out of Memory and eventually Panic'd.

SO I am rolling back to HPDMmultipath-4.2.0 instead of 4.3 first and then rollback :

device-mapper.i386 1.02.32-1.el5 installed
device-mapper.x86_64 1.02.32-1.el5 installed
device-mapper-event.x86_64 1.02.32-1.el5 installed
device-mapper-multipath.x86_64 0.4.7-30.el5_4.2 installed

[Ivan Ferreira]

Can you please verify the memory usage by running the ps command, ipcs -a command, and lsmod command, and post the results..

[Alzhy]

Ivan, all looks kool. It really looks like it is the device-mapper stuff that's causing my woes.

On other servers where I do not have any persented SAN disks and no VGs and Lvols and Filesystems carved -- my memory is fine.


Now how do I roll-back my device-mapper packages? just "yum downgrade" ?

[Steven E. Protter]

Shalom,

yum remove package_name

Do take a look at the list in case there are unexpected dependencies.

SEP

[Alzhy]

yum downgrade did it.
But I lost lvm tools.

I have a breakthough though on what's causing my issues.

My Kernel Configuration that was passed on to me by a DBA:


vm.nr_hugepages = 70000
vm.zone_reclaim_mode = 1

kernel.shmmni = 4096
kernel.shmmax = 0x4000000000
kernel.sem = 250 32000 250 128
net.ipv4.ip_local_port_range = 1024 65000
fs.aio-max-nr=804800
net.core.rmem_default = 262144
net.core.wmem_max = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 262144
fs.file-max = 6553600


When I remove vm.nr_hugepages = 70000, my mem woes are gone.

Anyone knows what should be the proper value? The DB will most likely have ~32GB max SGA size and I think that's what it is used for --for Oracle to use huge pages.