System swapping but not paging

[Joe Krutenat]

I have a system that is pretty active and very sluggish at times, more than it should be. I have gone through the forums and the HP-UX Performance Cookbook and I have a set of metrics that I am unable to interpret. I think the problem is with memory, but I am not certain and would really appreciate any help I could get from the experts in the community.

* The system is an N4000 8-CPU with 8GB of RAM running a Sybase DB on HP-UX 11.11.

The perf metrics were taken with measureware over the busiest day this host experiences.

Data:

Cache Peak Memory Pg Out Swap Pri Run DskSubsys Memory

Time Hit % CPU % Disk % % Rate Out Rt Queue Queue Queue Queue

8:00 92.19 80.68 51.27 99.14 0 0.9 0 0.8 0 0

8:05 95.18 80.47 49.53 99.14 0 0.9 0 0.77 0 0

8:10 99.82 83.03 45.97 99.14 0 1 0 0.79 0 0

8:15 97.51 79.33 23.29 99.14 0 1 0 0.77 0 0

8:20 71.17 82.52 69.14 99.14 0 0.9 0 0.84 0 0

8:25 99.66 86.21 55.23 99.14 0 0.9 0 0.88 0 0

8:30 87.02 71.69 33.05 99.14 0 1 0 0.79 0 0

8:35 82.07 70.19 26.18 99.14 0 0.9 0 0.64 0 0

8:40 96.83 72.12 23.65 99.14 0 1 0 0.73 0 0


So from the above data, the CPU(s) are active, but the load is never above 1.

The buffer cache hits are consistently above 90% so the buffer cache probably could be reduced to 400MB (from the dynamic 5-50% current level). (The buffer cache sat at around 850MB during the above activity.)

The disks are pretty active, but there are rarely, if ever, queue'd requests there.

The last one is memory, which is used to 99% the entire time the host is up. Swap is only used to 71%, and there are no page-outs, but there is swapping occurring at a rate of 1/sec throughout the day. What would cause swapping when swap is not 100% used?

Also, pseudo-swap is not turned on.

I have attached a file with the data in case it was not clearly viewable from above.

[Joe Krutenat]

Just to be clear, my question is:

What would cause a system to swap when it is not paging?

Thanks.

[Patrick Wallek]

2 things I would like to see to help out.

1) output of 'swapinfo -tam'

2) output of 'vmstat 60 5' to get 5 minutes worth of activity.

[Joe Krutenat]

I have retrieved the additional data from swapinfo and vmstat. Although this data was from today and not from the day he original data was taken, I did verify that the GBL_SWAPOUT_RATE is is still 1/sec during the swapinfo and vmstat.

Please find the data attached.

[Bill Hassell]

The swapinfo command and vmstat are reporting things differently. swapinfo shows that 70% of the swap area is either reserved or occupied. But swapinfo does not report actual read/write to the swap area. And if you don't add any additional processes or none of the processes grows larger, then swapinfo will remain static.

But vmstat is monitoring all memory including virtual memory and po (page outs) are individual pages. These pages may be pages that have just been changed in RAM but because they cannot fit, they are being written to the reserved area. Since the reserved area for each process is large enough for the current size, a page will go back to its slot on disk and swapinfo won't report any growth.

There is another use for swap space: memory mapped files. A small amount of the swap space is used for managing memory mapped files and this is likely the cause of the page per second. NOTE: Anything less than 10 pages per second is really not meaningful.

However, the 5%-50% are way out of line for an 8Gb system. When your system boots, the buffer cache is allowed to grow to 4000 megs!! Now this will continue until enough processes push RAM to 100% usage and the buffer cache starts backing down. This reorganization for the buffer cache is highly invasive for 11.11 and earlier. The kernel must sweep through the cache to evaluate the entries and start flushing any changed pages to disk. So during startup, the system will be quite sluggish as the cache starts shrinking.

Once the cache gets close to 5% (400 megs), then processes will start paging out. If your system has long think-time processes, these are the first to be deactivated and then paged to the swap area. If the system is fairly stable (no new users, no one signing off) then the buffer cache will stabilize and won't grow or shrink much. This is fine except that it wastes a lot of CPU and disk I/O during the shrink-down period.

Your recommended buffer cache settings would be 2% and 10%. This means that you immediately get to use 3200 megs of RAM during startup as the buffer cache starts at 800 megs and will shrink down when necessary. Note that reducing the buffer cache size below about 600-900 megs affects filesystem performance. I believe that Sybase uses raw disk volumes so this is will not be a significant impact.

Now tracking down the sluggish periods may be tricky and will likely require Sybase metrics to see if excessive scans or other SQL faults and inefficiencies can be uncovered.

[Joe Krutenat]

Bill, thank you for your reply.

When referring to the page out (po) rate in your post, does this also apply to the swap out rate? In all of the metrics that I have taken, using Measureware and vmstat, there is no paging (po is always 0) occuring, yet there is swapping, where using vmstat reports si/so of > 0, and Measureware reports GBL_SWAPOUT_RATE > 0.

So my understanding is that paging is ok as long as it stays under 10/sec, but swapping is terribly bad and I am confused as to why swapping would occur when paging is not occurring.

Thanks!

[John Guster]

Having large buffer cache configured can have a negtive impact on performance, due to the time the lower level routines that check on free memory in the cache take(dbc_max_pct=50%).
Your swapinfo -map indicates at certain point of time your system pagging out. Did you add the 2nd device swap after device swap started paging out? ( a reboot will clear this 1% in device swap used column).

check share memory area to see if there is any unreleased memory.

[Joe Krutenat]

Hi John,

Thanks for your reply.

None of the swap was configured after the most recent reboot.

The concern here is not paging out, but the swapping out that is occurring.

[Joshua M. Miller]

This situation was resolved through reducing the memory allocation of Sybase. This reduced the swapping to 0 and through enough reduction did not begin paging at all. The sybase reservation was at 5.7 GB before and was reduce to 4.6 GB and now the performance is outstanding.