swap problems

[Scott Van Kalken]

I have a system that at the moment has an extremely high reclaim rate when I use the vmstat command - basically at any given time this machine can have a reclaim rate as high a 900. It usually "idles" around 500.

I realise this is WAY TOO HIGH.

The weird thing is that the swap space is only being half utilised.

At any given point there would be only 55% of swap space used. This is device swap, not filesystem.

Anyone got any ideas?

[Edward Sedgemore · Accepted solution]

If you have device swap actually USED then this means you have processes paged/swapped out to disc, which explains fully why vmstat shows such a high reclaim rate - pages are constantly being swapped in and out. Ideally you want to keep device swap USED to 0%. We do. This means you have sufficient ram for the applications you are running. We dont let any of our servers here ever get to device swap USED >0%.

For example;

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2

This means everything is in memory, great. vmstat shows no page reclaims at all.

As against;

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 512 27 485 5% 0 - 1 /dev/vg00/lvol2

On this server (not one of ours!) we have 27MB of processes swapped/paged out to disc, which means were really 27MB of RAM short! As a result vmstat shows page reclaims going on.

[Scott Van Kalken]

Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1048576 24256 1024320 2% 0 - 1 /dev/vg00/lvol2
dev 1024000 24188 999812 2% 0 - 1 /dev/vg00/lvol10
reserve - 330692 -330692
memory 2384868 1313136 1071732 55%


weird thing is that device swap is only about 2% - which from what you're saying is too high anyway.

Weird is all I can say.

[james strutz]

I have an 11.0 host running Oracle and AutoSys experiencing the same issue with vmstat showing average reclaim rates between 300-500 with spikes of 900, swapinfo (PCT USED) is at 1%. Although mentioned in the thread that this is never a "good" thing is it necessarily just lacking memory ... I'm in the process of installing tools (Glance, MeasureWare) to further analyze. Thx

[Bill Hassell]

Check the vmstat metric po (page out). This is the most important as it indicates actual pages written to the swap area. HP-UX starts every program with pi (page ins) but these come from the program's executable file. But paged outs are real writes to swap. Single digits are OK, double digits are questionable and more than 2 digits is a real shortage of RAM. It means that some processes have been deactivated and one or more pages have been written out to swap.

Note that swap space is utilizied in several ways. The first is reservation, which involves no I/O, just bookeeping. Swap reservation is often confused with swap usage. The second is space for memory mapped files. And the third is real page outs. vmstat and swapinfo are too crude to give a good picture of RAM usage. You really need GlancePlus to get an accurate (and meaningful) picture of RAM.

Standard incantation: check dbc_max_pct in the kernel. If it is 50, that's where a massive portion of your memory has gone. The DBC is supposed to move down towards dbc_min_pct when processes are threatened with deactivation, but depending on patches and OS version, it isn't too reliable. Because it is a percentage of RAM, the amount changes with RAM changes, it is a pain to manage. Instead, change bufpages to a value between 204800 and 409600 which is 4k pages. Now the dynmaic buffer cache will be fixed at a reasonable value of 800 to 1600 megs regardless of RAM size.