system crawling - high swapped processes

[Anthony deRito]

Here is the output of vmstat:
0 0 337 732472 15704 2 6 8 36 112 0 184 0 9 9 9 625 330 131 0 2 98
0 0 337 732472 15704 2 3 0 8 12 0 14 0 0 0 0 320 309 95 0 0 100
0 0 337 732472 15704 0 0 0 4 4 0 0 0 0 0 0 324 450 119 0 0 100
0 0 337 732472 15704 0 0 0 0 0 0 0 0 0 0 0 314 362 103 0 0 100
0 0 337 732472 15688 3 2 0 588 584 0 56 0 0 0 0 1214 808 582 0 5 94
0 0 337 732472 15704 6 7 12 116 132 0 94 0 0 0 0 505 424 184 0 1 99
2 0 337 731608 14968 228 1325 500 348 864 0 1861 0 40 20 20 709 8313 1864 36 25 40
0 0 337 732456 16456 108 700 20 548 720 0 536 0 0 0 0 1061 5768 1555 17 23 60
0 0 337 732456 16384 3 4 4 0 0 0 0 0 0 0 0 328 359 105 0 0 100

Note the 3rd column "w" which is the number of swapped out processes. The system was crawling with "out of memory" messages. I am sure we will find the culprit. However, what I would like from this group is more information on what you think could cause the number of swapped out processes to be so high. THere were only 370 processes listed with ps command and out of those 337 were swapped out! Please advise.

Tony

[Stefan Farrelly]

Sounds like you have massive massive memory pressure - causing the swapouts.

What does; swapinfo -mt
show ?

And what does this show (listing processes by memory usage);
UNIX95= ps -e -o vsz=Kbytes -o ruser -o pid,args=Command-Line | sort -rnk1 | more

[A. Clay Stephenson]

The only column of any real use in vmstat is the po (page out) column. Anything over 10 for any length of time is considered bad -- 0 is desirable. You have values > 500!. A few large processes could do exactly this. This is a case where small maxdsiz settings make sense. It would prevent any process from grabbing all the resourses. This is just a gusess but I suspect you have a process with a serious memory leak that starts out small and grows very large over time.

[Anthony deRito]

Stefan, unfortunately the problem has passed since I rebooted the system and so I donâ t have any further details for you.

Clay, the value I am concerned with is the third column (337). Almost all the processes were pushed out to swap. The problem was not slowly occurring. It happened within 3 seconds based on looking at vmstat output collected every couple of seconds by a cron job. One minute this value was 10, within the next few seconds it was 337 and stayed that way until we rebooted the box. I guess what I am looking for is understanding how and why the system reacted the way it did not so much what caused it

[Steven E. Protter]

swapinfo -tam would be useful.

Some long term sar based script collection of performance data would be useful.

I'm attaching a good script for that.

SEP

[Chris Vail]

This output from vmstat is confusing and inadequate. In my experience, the PO column (column 7) in more indicative of long-term memory issues than the short sleeper column (column 3). What kind of system is this, what version of the OS and what is the hardware like? What application is running?

Try "ps -ef|sort -k 4nr,4|pg" to see which processes are using the most CPU time. You can use "top" as well as glance plus for the same information. The process with the most CPU time is most likely the one that is using the most memory.


Chris

[Stefan Farrelly]

Hi Tony,

my guess is the reason this happened so quickly is some program started which requested a massive amount of memory. For example ive got a C prog which can grab a GB or 2 of memory within a couple of seconds of starting. This sort of thing would cause so many other processes to be swapped out so fast. Its hard to speculate further without more info. I would set up a lot of monitoring so at least if it happens again you will find the culprit.

Do you have measureware installed ? if so you should have historical info including processes running which can be viewed using PerfView.