hp-ux out of memory

[yunardi]

Dear All,

I'm running HP-UX 11.11i on Superdome with 76GB of Memory. Sometimes I can not login to system using telnet/ftp/sftp/ssh or even console. After waiting for some minutes, I can login to system and saw (with glance) the utilization of memory is 100%.

Here is the memory report from glance :
Total VM : 75.5gb Sys Mem : 6.26gb User Mem: 67.9gb Phys Mem: 76.0gb
Active VM: 68.5gb Buf Cache: 1.52gb Free Mem: 323.3mb

This system running Oracle with about 12GB of SGA, but the application (datawarehouse) consume almost 60GB of RAM.

1. Is there any command or tools to limit user memory, so the application memory can be limited to lower level below 60GB.
2. Is it correct, if system out of memory; user or even root can not login to the system ?

Thanks,
EKO

[Patrick Wallek · Accepted solution]

First check and see what your dbc_max_pct is set to. I sincerely hope it is NOT 50. If it is you must reduce it immediately. For 76GB of RAM, I would make dbc_min_pct and dbc_max_pct the same at 2 or 3.

Now if those are not a problem, the first thing to do is check 'swapinfo -tam' and see if you are actually paging out. If you are, then you need more RAM.

If you can't get more RAM you might consider lowering Oracle's SGA.

Users may be having trouble logging in because you are paging out. When things have to be paged out to disk, your performance will suffer terribly.

When you see RAM usage at 100%, what is the top process? If it is vhand, then you have memory problems and things are starting to page out.

[A. Clay Stephenson]

Your best tool is PRM (Process Resource Management) but it's not a free product. However, normally for this case, PRM is not really needed. Instead you need a baseball bat to apply to the applications/database managers to reduce their memory footprint. This is normally a configuration change --- unless there is a memeory leak. You need to determine if the applications grows without bounds over time.

The system is not actually limited by the amount of physical memory but rather the amount of virtual memory. The worst case would be a number of very large processes that are trying to be paged in and out simultaneously. With so little free memory, you are almost certainly paging out and that is what is causing the performance hit.

Your first task should be to talk to the applications/database guys and see what can be done to reduce the memory footprint. If they are using huge caches, the performance gains they hope to get are killed by at least a factor of 100 by the impact of swapping.

[yunardi]

Dear All,

Thanks for quick response.

Actualy I've said to the applications team that utilization of memory controlled by application. If oracle can alocated memory with SGA setting, so others application could do this. But since the deloper of the applications still can't limit memory usage (because this applications "playing" in memory rather than disk), they always asking sysadmin to solve this problem.

I think I really need PRM, I will contact our HP local support for this.

Here I attach the output of kmeminfo, may be you can find something about this.

Thanks,
EKO

[Don Morris_1]

Since it looks like their memory consumption is all in their heap (DATA segment is the big consumer for pid 11334) -- you could lower maxdsiz_64bit to less than 60Gb. That will cause their allocation requests (brk/sbrk/malloc) to fail earlier.

Back to your original question - if all the memory on the system is tied up, you should be able to make new processes / do new allocations -- but you may end up blocked waiting for pages to swap out [via vhand, swapper, etc.]. That may take some time (minutes is not surprising) resulting in what you perceive as a login hang [or even a telent timeout]. Personally, I'd keep a session logged into this machine via console or ssh or whatnot with a real-time priority such that you can always get to it for monitoring the system. It may take a while for some of the monitoring processes to collect data, but at least you'll be able to get on the box if you need to adjust things or shutdown applications.

[Paul Clayton]

Hi,
We have similar issues with our applications, and the only fix for us was to increase swap by 6 times memory. Not the ideal situation, but it did stop the out of memory issues. I know performance will not be optimal, but what can you do when your company won't spend on hardware and it becomes a sysadmin problem.

[yunardi]

Dear Don,

For maxdsize, here is the value on my server :
# kmtune|grep maxdsiz
maxdsiz 1073741824 - 1073741824
maxdsiz_64bit 4396972769279 - 4396972769279

Is this correct setting ?
What is the unit of this parameter ? bytes or kbytes. I dont really remember if I have set this value.

and buffer cache is :
# kmtune|grep dbc
dbc_max_pct 5 - 5
dbc_min_pct 2 - 2

Its should be okay, because maximum of buffer cache is only 5% (3,5GB) from 70GB.

=========
Thanks,

[Don Morris_1]

Ok... so you've allowed 64-bit applications up to 4Tb of data space (virtual, and therefore implicitly physical if you had it). Hence the application has no problem grabbing 60Gb or so [since the system has it, and you've allowed it].

Whether those values are correct strongly depends on what you want here. From the sound of things, you want your system to stay responsive and not allow this single application to consume the entire box.
The big question then becomes how this application handles malloc() failures. If it scales itself back (one would hope for a big commercial application, which is what I assume this is... since who knows how much swap/virtual address space is available to it, much less RAM?), then lowering maxdsiz_64bit is your simplest solution. That prevents this application from consuming more than that amount in data virtual size [what amount you set is up to you... with your system load around 7Gb [sys + buf cache], I would probably try around 55Gb to give yourself enough flex space for other apps, system spikes].

However, if that's unacceptable to the application -- you'll have to go to the more complex solution... don't limit this application's virtual space [i.e. leave maxdsiz_64bit where it is], but limit the physical space it can consume so as to leave memory for the rest of the system. To do that, you'll need the PRM product. That product allows you to partition the physical memory resources (not actual RAM ranges, mind you... just the accounting side) so that you could limit this application to consume no more than X Gb of RAM, leaving Y for other user applications / the system. When the application grows near to X, vhand and other memory reclamation mechanisms would activate and page out other pages in that process... while the rest of your applications only have to compete with each other for memory.

So in short -- I'd try out lowering maxdsiz_64bit first (maxdsiz is the 32-bit limit, that looks reasonable to me.. these are fencepost limits, see http://docs.hp.com/en/TKP-90202/re34.html ) simply because it is easy and free. If that doesn't fit your needs, you'll need to look at PRM.

[siva0123]

Don,

I have a question here ...?

For restricting the application to use certain amount of memory cant (Rather than can't will it be possible )we use ulimit -m to specify the limit for the memory to be used by the application.

Thanks,
Siva

[Don Morris_1]

Sure... that's setting the soft limit rather than the hard limit (the tunable sets the hard limit). The down side to setting the soft limit is that an application can always raise it right back up to the hard limit using the setrlimit() system call. Setting the tunable ensures that you've limited the app.

[Bill Hassell]

As mentioned, you can use ulimit to temporarily reduce the maximum data size but unless the application explicitly handles failed memory allocation requests, they usually just crash (dumb, I know). And of course the DBAs will blame the sysadmin for the problem.

As far as users of root not being able to login, you just have to be patient. The system is probably swapping like crazy so give it 15-20 minutes to get a system prompt and you should be OK.

So you have several solutions:

1. Drastically increase the swap area. The programs will run and extra memory will be handled with swap space. Oh, did I forget to tell you that applications may slow down by 20x to 1000x slower? The DBAs might not like that...

2. Drastically increase your RAM -- just order another 64 Gb of RAM and that should solve the problem for a while. Might be tricky to get approval for the expense...

3. Find a new developer...while it is true that loading most of the database in RAM will definitely improve performance, management may have a different view of the cost of such a design. There are a lot of DBAs that would love to have a 20 Gb SGA but they are stuck with a 3 Gb SGA because they only have 6Gb of RAM.

It does not make any sense at all to stuff dozens of Gb of data into RAM to improve performance, only to have to wait for paging to take place. Paging (using swap space) is the MOST inefficient method there is to maximize memory. If you want performance, resize the applications to not force swapping to take place (in Glance, it is the VVVVV indicator in disk usage, or in vmstat, it is the po or page out line).

By the way, your kernel is more than 6 GB. You might want to review massively large parameters such as ninode (should be about 2048 to 4096). Glance will give you a summary of usage and high-water marks (use the t command). A more normal kernel is about 2-3 Gb, but if you really need to run 100,000 processes or open 2 million files at the same time, the larger kernel may be normal.

[yunardi]

This forum is GREAT...
I'd prefer asking my problem here, rather than open case to my HP local support and after days they follow to global support or backline support.

Thanks all, I will apply those kernel parameters (ofcourse if i get a time for reboot as those [arameters are static :) ).

I summarize your suggestion :
1. set maxdsiz64 to lower value than physical memory (need to reboot)
2. ulimit -m (soft limit), don't set it to UNLIMITED.
3. add swap to be minimum 2x physical memory.
4. review kernel settings for ninode (as i reviewed this value is set to 38912 on my server)
5. Buy PRM and limit each process memoty util.

So be patient and wait for some minutes, the login prompt will come up after paging out finished :)

Btw are some of you an HP global support ;)

Thanks,
EKO