System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Free Memory always low - What Happened to My Memory?

 
Alzhy
Honored Contributor

Free Memory always low - What Happened to My Memory?

I've a server, recently imaged with RHEL 5.4, patched to the latest and greatest. As I was about to finish everything and start installing Oracle, I notice that out of the 128GB Mem, I always end up with just about half a GB of mem. And there is no apps running yet. It is consistent, even after a fresh reboot -- mem free is about half a GB. MemInfo follows below.

I am clueless what's causing this and am about to just re-image and track my steps and see when this issue start to manifest itself.


# free
total used free shared buffers cached
Mem: 132101528 131514200 587328 0 4552 9224
-/+ buffers/cache: 131500424 601104
Swap: 67076088 66732 67009356

]# cat /proc/meminfo
MemTotal: 132101528 kB
MemFree: 585984 kB
Buffers: 4916 kB
Cached: 9888 kB
SwapCached: 9924 kB
Active: 18452 kB
Inactive: 6772 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 132101528 kB
LowFree: 585984 kB
SwapTotal: 67076088 kB
SwapFree: 67009360 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 8684 kB
Mapped: 5932 kB
Slab: 42140 kB
PageTables: 3948 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 67443396 kB
Committed_AS: 179648 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 302444 kB
VmallocChunk: 34359435055 kB
HugePages_Total: 64144
HugePages_Free: 64144
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
Hakuna Matata.
10 REPLIES
Steven E. Protter
Exalted Contributor

Re: Free Memory always low - What Happened to My Memory?

Shalom,

-/+ buffers/cache: 131500424 601104

There it is.

When memory is in use Linux puts it in buffer cache to speed up I/O. Unlike older versions of HP-UX it can pull memory from cache very quickly.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Alzhy
Honored Contributor

Re: Free Memory always low - What Happened to My Memory?

Shalom too SEP.

But is this the normal behaviour? I've other servers - freshly installed, similarly patched that I've not:

presented SAN LUNs yet
installed our custom scripts
mod'd sysctl.conf, limits.conf for Oracle

And I do not see any issue.
Hakuna Matata.
Alzhy
Honored Contributor

Re: Free Memory always low - What Happened to My Memory?

And one more thing, our Proof of Cncept Big LINUX machine was a 256GB 48-way server (diff vendor though) and I never saw all my memory being gobbled up as chance "at once".

I'm dealing with VERY large Linux machines btw.

Hakuna Matata.
Steven E. Protter
Exalted Contributor

Re: Free Memory always low - What Happened to My Memory?

The behavior is perfectly normal. It is the default behavior of RHEL.

Write a program to do some malloc, large chunk of memory, say 256 MB.

Your request will get what it needs instantly. The system will not page and the buffer figures will decline.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Alzhy
Honored Contributor

Re: Free Memory always low - What Happened to My Memory?

That's what RH Support told me but not quite the case. Impossible for a server with 128GB of memory with hardly any filesystems with files on them to gobble up all the memory.

I am now backtracking and I think I found the culprit. I've the latest HP Multipathing package and Redhat's latest multip-pathing package.

When I tried to collpase my LVM VG that sits on a multi-pathed EVA LUN, the OS ran out of Memory and eventually Panic'd.

SO I am rolling back to HPDMmultipath-4.2.0 instead of 4.3 first and then rollback :

device-mapper.i386 1.02.32-1.el5 installed
device-mapper.x86_64 1.02.32-1.el5 installed
device-mapper-event.x86_64 1.02.32-1.el5 installed
device-mapper-multipath.x86_64 0.4.7-30.el5_4.2 installed


Hakuna Matata.
Ivan Ferreira
Honored Contributor

Re: Free Memory always low - What Happened to My Memory?

Can you please verify the memory usage by running the ps command, ipcs -a command, and lsmod command, and post the results..
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Alzhy
Honored Contributor

Re: Free Memory always low - What Happened to My Memory?

Ivan, all looks kool. It really looks like it is the device-mapper stuff that's causing my woes.

On other servers where I do not have any persented SAN disks and no VGs and Lvols and Filesystems carved -- my memory is fine.


Now how do I roll-back my device-mapper packages? just "yum downgrade" ?
Hakuna Matata.
Steven E. Protter
Exalted Contributor

Re: Free Memory always low - What Happened to My Memory?

Shalom,

yum remove package_name

Do take a look at the list in case there are unexpected dependencies.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Alzhy
Honored Contributor

Re: Free Memory always low - What Happened to My Memory?

yum downgrade did it.
But I lost lvm tools.

I have a breakthough though on what's causing my issues.

My Kernel Configuration that was passed on to me by a DBA:


vm.nr_hugepages = 70000
vm.zone_reclaim_mode = 1

kernel.shmmni = 4096
kernel.shmmax = 0x4000000000
kernel.sem = 250 32000 250 128
net.ipv4.ip_local_port_range = 1024 65000
fs.aio-max-nr=804800
net.core.rmem_default = 262144
net.core.wmem_max = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 262144
fs.file-max = 6553600


When I remove vm.nr_hugepages = 70000, my mem woes are gone.

Anyone knows what should be the proper value? The DB will most likely have ~32GB max SGA size and I think that's what it is used for --for Oracle to use huge pages.
Hakuna Matata.
Jimmy Vance
HPE Pro

Re: Free Memory always low - What Happened to My Memory?

With RHEL 5.4 you don't need the HP multipath kit as it doesn't really have anything but a template for the multipath.conf file. The device-mapper packages that ship with RHEL 5.4 understand HP storage so you don't even need the multipath.conf file unless you want to modify a setting to something other than it's default value.

Older versions of RHEL had modified device-mapper files that included information about HP storage. All that information gets sent upstream to the device mapper maintainer now.






__________________________________________________
No support by private messages. Please ask the forum!      I work for HPE

If you feel this was helpful please click the KUDOS! thumb below!