Operating System - HP-UX
1826398 Members
3473 Online
109692 Solutions
New Discussion

Re: Managing Swapinfo & real memory

 
Ankit Grover
Frequent Advisor

Managing Swapinfo & real memory

Dear experts,

I am facing a problem regards to optimal use of swap info on my HP IA64 server. Almost every alternate day, we have to restart the server because real memory gets 100% full. The swapinfo usage usually go max till 70%.

We have kept swap memory more than 2 times our real memory. Is there anything special things which I co do to better utilize my swap space and avoid real memory problems?

[root]/#swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 2835988 1358316 68% 0 - 1 /dev/vg00/lvol2
dev 4194304 2833308 1360996 68% 0 - 1 /dev/vg00/lvol11
dev 4194304 2855980 1338324 68% 0 - 1 /dev/vg00/lvol4
dev 4096000 2841284 1254716 69% 0 - 1 /dev/vg00/lvol15
dev 4194304 2843416 1350888 68% 0 - 1 /dev/vg00/lvol12
reserve - 6617408 -6617408
memory 8376328 8375876 452 100%

Cheers,
Ankit
13 REPLIES 13
whiteknight
Honored Contributor

Re: Managing Swapinfo & real memory


ankit,

Your swapspace > 70% meaning, you have a memory pressure, may be increase more memory and swap space.

Let me attached a documents for your reference on introduction to Performance tuning

WK
please assign points
Problem never ends, you must know how to fix it
whiteknight
Honored Contributor

Re: Managing Swapinfo & real memory


ankit,

Your swapspace > 70% meaning, you have a memory pressure, may be increase more memory and swap space.

Let me attached a documents for your reference on introduction to Performance tuning

WK
please assign points
Problem never ends, you must know how to fix it
Steven E. Protter
Exalted Contributor

Re: Managing Swapinfo & real memory

Shalom Ankit,

1) Don't increase swap, your system will slow down very badly.
2) The demand for memory on this system obviously exceeds supply. The answer is to either decrease memory demand or increase memory supply (add RAM).

Ways to decrease memory demand.
1) Check oracle SGA and similar configurations see about reducing the memory footprint of the database.
2) Lower dbc_max_pct to between 5-10 in the kernel. This decreases the size of the buffer pool but can harm performance depending on application use.

Tools:
swapinfo -tam

Provides a better picture than swapinfo.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Prasanth V Aravind
Trusted Contributor

Re: Managing Swapinfo & real memory

You can add more physical memory to your server and solve this issue.
Otherwise create new vg (vg swap) and may add some more swap. If you are creating new vg for swap it will decrease the disk/io of vg00 disks.


Get kmeminfo utility and check which of your application is utilizing more memory. Remove some application will also help you!!!


Regards
Prasanth
Bill Hassell
Honored Contributor

Re: Managing Swapinfo & real memory

You do NOT want to utilize swap space if you care anything about performance. Your RAM is far, far too small for the processes that you are currently running. You need to at least double the amount of RAM you have. If that is not possible, you have to live with very bad performance, or reduce the size and/or number of processes running on your system.

When you use swap space, the performance can drop 50 to 200 times slower, a massive impact. Now if all your applications are interactive and users only talk to the programs once every few minutes, you won't see an impact. But if you have several programs running for long periods at the same, then the penalty for small RAM is enormous. You can see the page out rate by running vmstat -- look at the po column. If this is 1 digit (0-9), it is OK, 2 digits (10-99) means processes are starting to slow down, and more than 100 means your system is starved and cannot perform at all.

Here's what happens when two programs run at the same time but they won't fit into RAM:

1. process 1 is running and consuming 80% of RAM
2. process 2 has been waiting because it will not fit into the rest of memory. Process 1 is getting less important (because it is not allowing other processes to run) until finally, process 1 is deactivated.
3. The swap daemon starts removing memory pages from process 1 until there is enough room for process 2.
4. when process 2 has enough pages to run, it will start (or continue) running.

Now during the time page out is occurring, an enormous amount of time is wasted where neither process is running (paging is occurring). During this time, the kernel has to spinlock other processors because memory is being reorganized and this holds up kernel services.

You need a lot more RAM.


Bill Hassell, sysadmin
Ankit Grover
Frequent Advisor

Re: Managing Swapinfo & real memory

I will get back on this on Monday.
Apologies for not responding to your suggestions.

Cheers,
Ankit
Ankit Grover
Frequent Advisor

Re: Managing Swapinfo & real memory

Thanks a lot Whiteknight for the document. Its a really knowledgable doc. We are studying it currently.

It is not possible to increase RAM of the server and Swap space is already twice the RAM. The only thing which I am concentrating on is to identify the processes putting pressure on the memory (as sugggested by SEP).

I am not able to download kmeminfo from the FTP site of HP. Can someone send it through this forum (by renaming it, executables are blocked)?

Bill, here is vmstat output:

[root]/#vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
5 39 0 1123800 11632 173 40 51 90 9 0 760 6112 199957 8655 50 9 41

po value is quite high (90).

Any other way to monitor memory usage will be appreciated.

Thanks a ton,
Ankit
Bill Hassell
Honored Contributor

Re: Managing Swapinfo & real memory

po=90 is very high. If you cannot add RAM, you have to:

1. Stop running so many processes at the same time
or
2. Live with the poor performance.

You can sort the processes by size to get a rough idea on memory usage:

UNIX95=1 ps -e -o vsz,ruser,pid,args | sort -rn

Now this will not show shared mempory. Use ipcs -bmop to look at shared memory usage. If you are running a database, check with your DBA to see how to reduce memory usage. Naturally, less memory will cause the database to run slower.


Bill Hassell, sysadmin
Tim Nelson
Honored Contributor

Re: Managing Swapinfo & real memory

If you cannot purchase additional memory for some monetary reason or isolate/reduce the amount of RAM used by processes then the only other tuning possibility is to tune your swap space.

As the swap space has now become an IO issue make sure the device swap lvols are spread around different spindles or even on raid devices. This would be just like configuring the disk for the best application performance.

Your performance will still be incredibly poor but I guess less poor is better than nothing.

Steven E. Protter
Exalted Contributor

Re: Managing Swapinfo & real memory

Shalom,

A tool that uses a similar command to Bill's that will help identify the heavy memory use processes.

http://www.hpuxconsulting.com/mem.mon

Man pages
http://www.hpuxconsulting.com/mem.mon.manpage.pdf
http://www.hpuxconsulting.com/mem.mon.manpage.html

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Geoff Wild
Honored Contributor

Re: Managing Swapinfo & real memory

And here's yet another really usefull tool for checking memory and swap.

It's a c program called memdetail

I've attached the source which just compiles with the native HP-UX cc compiler.

Output like:

# memdetail
Memory Stat total used avail %used
physical 16128.0 15689.6 438.4 97%
active virtual 19886.7 7184.7 12702.0 36%
active real 13222.3 3905.6 9316.7 30%
memory swap 12647.5 2293.0 10354.6 18%
device swap 26528.0 18852.7 7675.3 71%


Rgds...Geoff
Proverbs 3:5,6 Trust in the Lord with all your heart and lean not on your own understanding; in all your ways acknowledge him, and he will make all your paths straight.
Ankit Grover
Frequent Advisor

Re: Managing Swapinfo & real memory

Hi Geoff,

I tried your C code. The following is O/P:
Memory Stat total used avail %used
physical 8180.0 8140.7 39.3 100%
active virtual 20854.8 4076.4 16778.4 20%
active real 5220.1 1453.5 3766.7 28%
memory swap 8180.0 8064.2 115.8 99%
device swap 20384.0 20384.0 0.0 100%

I am not able to understand what does active virtual & active real stand for? and how are they impacting my machine?

Ankit.
Geoff Wild
Honored Contributor

Re: Managing Swapinfo & real memory

Well, first off...your server either needs some more ram, or some apps need to be tuned to use less (IE - is Oracle is on here - reduce the size of the SGA).

Is this running 11.31 or 11.23?

If 11.23, what is the value of your buffer cache?

# kctune |grep dbc
dbc_max_pct 5 5 Immed
dbc_min_pct 5 Default Immed


On a system with 8 GB of ram - I would set to 8 (could go as low as say 6).

You need more device swap - not that it means you are swapping - but the reason you need to reboot is because programs can't allocated any more space - every program needs to be able to allocate swap space - even if it doesn't use it.

Have a look at this whitepaper:

http://www.docs.hp.com/en/5965-4641/index.html

Add some more device swap ASAP else you won't be able to fork!

Rgds...Geoff

Proverbs 3:5,6 Trust in the Lord with all your heart and lean not on your own understanding; in all your ways acknowledge him, and he will make all your paths straight.