Need help with high swap usage

[sbrews]

Disclaimer - I have been away from HPUX for a long time...

I have a system that appears to be experiencing high swap usage ( >90%).  I am trying to determine what is causing it and/or how to resolve it.  Any help/pointers would be appreciated.

System memory is: Memory = 8162 MB (7.970703 GB)

output from swapinfo -atm:

             Mb      Mb      Mb   PCT  START/      Mb
TYPE      AVAIL    USED    FREE  USED   LIMIT RESERVE  PRI  NAME
dev        4096       0    4096    0%       0       -    1  /dev/vg00/lvol2
dev        8192    3079    5113   38%       0       -    0  /dev/vg00/lvswap
reserve       -    9209   -9209
memory     8163    7919     244   97%
total     20451   20207     244   99%       -       0    -

cut & paste from glance:

Cpu  Util   S   SU                                                                                               U |100%  100%  100%
Disk Util   F    F                                                                                                 |  6%   18%   42%
Mem  Util   S                 SU                                                                       UB     B    | 98%   98%   98%
Swap Util   U                                                  UR                                            R     | 96%   96%   97%
------------------------------------------------------------------------------------------------------------------------------------
                                                             SWAP SPACE                                                 Users=    2

Swap Device                     Type        Avail       Used     Priority
--------------------------------------------------------------------------------
/dev/vg00/lvol2                device       4.0gb        0mb        1
/dev/vg00/lvswap               device       8.0gb      3.0gb        0
pseudo-swap                    memory       8.0gb      7.2gb       -1

<empy lines deleted>

Swap Available: 20451m  Swap Used: 10481m  Swap Util (%):  96  Reserved: 19691m

[Ken Grabowski]

Utilized does not mean a memory pressure problem by itself. Depending on what is running on a server, this could be a normal state.  Run vmstat -s and look at your swap out and pages swapped out numbers. If they are low or zero, your not having an issue.

The GBL_MEM_PAGEOUT_RATE kernel counter is also an indicator of memory pressure. You can see that from glance in the X-Windows (gpm) version, Global Info report, Pg Out Rate. Again, if this is low or preferably zero, then you are not swapping to disk and are probably not having a memory pressure problem.

[Dennis Handly]

>I am trying to determine what is causing it and/or how to resolve it.

You are trying to run too many processes.  (Or you have a memory leak.)

Since you only have 244 M free, you can't really run any more.

If you aren't actively swapping, you could add more device swap but you probably need more RAM.

You could look a top's output to find piggy processes.  Or use ps:

UNIX95=EXTENDED_PS ps -e -o vsz=Kbytes -o ruser -o pid,args=Command-Line | sort -rnk1

Similar threads and you can search for more:

http://h30499.www3.hp.com/t5/System-Administration/system-crawling-high-swapped-processes/m-p/3184078

http://h30499.www3.hp.com/t5/System-Administration/Process-memory-and-swap-reservation/m-p/5244827

http://h30499.www3.hp.com/t5/System-Administration/Some-idea-why-pagedevice-swap-can-be-100-full/m-p/3693084/

http://h30499.www3.hp.com/t5/System-Administration/memory-constantly-on-96-swap-on-51/m-p/3760403

[sbrews]

Thanks for the info and links.  Based on the page out rates in gpm (0), there doesnt appear to be any paging to disk.  So I guess that part is good.  I will research the other links and ps info to try an identify the processes that are sucking up resources.

[Ken Grabowski]

What kind of applications are your host(s) supporting?  I see this swap usage on systems hosting Sybase all the time.  Their memory model will reserve as much shared memory as you give it, and it normally has a high swap reserved number even if it has zero disk swap utilized.

[sbrews]

The system is running oracle.

[Ken Grabowski]

Oracle works a little different, but the outcome at the OS can be the same.  Since your memory and swap are both up in the high 90's% I would say your DBA's have maxed out the memory settings on the databases on this host.  That doesn't mean the databases are actually actively using that memory, it could just be locked by them.  Your DBA's can look at the actual usage inside the Database. If they have a lot of unused memory, they may be able to back off some of their configuration.

But as I said earlier, utilized does not mean problem.  You can sit at those high % of usage and as long as your not swapping, and your not having issues starting/running other processes, then you are OK.  But you will want to monitor swapping queues and make sure it's staying at or near zero.

[Ken Grabowski]

One other observation.  Your CPU is also showing 100%.  Take a look at your Glance Global report again and make sure that "Pri Queue" is low or zero. The Pri Queue is the average number of processes WAITing to run on the CPU's.

If the "CPU_%_CUM" (GBL_CPU_TOTAL_UTIL_CUM) is near 100% and Pri Queue is 3 or greater, your CPU is bottelnecked. If Pri Queue is zero or less than 3 then you are just utilized.

You may want to look for hung processes eating up CPU and/or memory.  Orphaned telnet sessions use to be a big consumer of CPU back when we use to let people use it.

[Dennis Handly]

>You may want to look for hung processes eating up CPU

I think you meant looping processes eating up CPU.

[Chirag Parikh]

Hi,

As you said that this server have the Oracle installaed and running, you may check for ipcs -m command which will give you the active shared memory segments which are consuming swap memory. And to release (kill) those if not required use ipcrm.

Or probably if you can get the Oracle downtime then bounc the Oracle services once and see the swap utilization.

Regards,

Chirag Parikh.