confused about swap activity

[Jdamian]

Hi

I'm using Glance for monitoring paging in my box. Glance shows some sustained 'VM Wr Rt' (Virtual Memory Write Rate) on two disks that are used for swap space... Then I suspect system is paging on those disks but 'swapinfo' shows 'PCT USED' is !!! 0% !!! What does mean 'VM Wr Rt' ? I'm confused.

other issue: vmstat shows sustained values on 'sr' column, but null on 'po' column.

other issue: some PerfView metrics related to SWAPING, as GBL_MEM_SWAPOUT_BYTE_RATE, don't show the real swaped-out to disk. They only count REACTIVATION or DEACTIVATION, but a process may be deactivated and reactivated immediately without transfering bytes to disk. Then it is counted as a REACTIVATION and a DEACTIVATION but no byte has been writen to disk...

Can anybody help me ?

[Zeev Schultz]

Please post swapinfo -mt,vmstat.

[Michael Steele_2]

Yes, need 'swapinfo -tam', 'vmstat 5 5' and 'sar -b 5 5' during slow performance periods. May as well evaluate the whole system and include 'sar -d 5 5', 'sar -v 5 5' and 'sar -u 5 5.

Also, its normal for the system to constantly be loading in new pages of memory, its when processes are interrupted before they've completed their work and then swapped out that's the problem.

[Zeev Schultz]

Btw,couldn't that be some memory mapped IO activity there?

[Jdamian]

vmstat output (My script adds a timestamp in the left side in order to know the time where it ocurrs -- this script will run for several days)

2003/06/18 18:38:00 79 0 0 77808 23425 676 216 0 0 247 0 0 3184 34008 36973 18 36 46
2003/06/18 18:38:05 procs memory page faults cpu
2003/06/18 18:38:05 r b w avm free re at pi po fr de sr in sy cs us sy id
2003/06/18 18:38:05 0 2 0 100320 2988 667 92 0 2 119 0 3 4245 25836 40099 12 47 42
2003/06/18 18:38:10 0 10 0 100067 3400 550 30 0 14 56 0 3631 4454 18766 29876 9 24 67
2003/06/18 18:38:15 229 1 0 101122 3091 589 9 0 6 35 0 7320 5013 27741 44482 16 43 41
2003/06/18 18:38:20 90 0 0 79248 24993 513 2 1 3 27 0 3484 4145 32357 50699 17 42 41
2003/06/18 18:38:25 159 0 0 108654 24821 534 0 0 1 29 0 1140 3419 30831 38762 14 32 54
2003/06/18 18:38:30 181 0 0 79247 24634 641 0 0 1 31 0 560 3461 33656 49303 16 43 41
2003/06/18 18:38:35 226 0 0 79498 24505 602 0 0 0 33 0 182 3036 27170 37550 12 27 61
2003/06/18 18:38:40 0 0 0 78933 24532 447 0 0 0 23 0 292 2637 20186 27650 7 18 75
2003/06/18 18:38:45 53 0 0 78778 24378 556 0 0 0 29 0 94 2859 22674 35938 12 31 56
2003/06/18 18:38:50 413 0 0 79599 24536 523 0 0 1 29 0 320 3072 25527 41365 12 33 55
2003/06/18 18:38:55 129 0 0 78225 24815 590 0 0 0 31 0 104 2860 22392 35812 11 27 62
2003/06/18 18:39:00 130 0 0 78583 24816 480 0 0 0 21 0 32 2847 21982 36034 10 31 59
2003/06/18 18:39:05 22 1 0 78726 24779 732 16 1 2 56 0 9 2743 21394 29305 11 26 63
2003/06/18 18:39:10 393 0 0 79167 24590 589 4 0 1 30 0 2 2588 17592 29831 7 23 70

[Michael Tully]

What is your current buffer cache settings?

They should be somewhere in the vicinity of 300-400Mb

# kmtune -l -q dbc_max_pct
If the return is 50 (50percent) then you will need to lower it.

WHat is your current RAM usage?

[Jdamian]

Yes, it is 50.

Buffer-cache is using near 45% physical memory...

I suspect that buffer-cache pages are freed when more memory pages is needed. This makes 'sr' column in 'vmstat' output is high however 'po' column is low because there is no swap-out to disk...

[Zeev Schultz]

Looks like Michael is pretty close here.
It has a high values of SR (scanning rate) low
(relatively) of PO (page outs),PR (reclaims) are also high,that can be due to buffer cache dynamics...I guess.Also a high numbers of processes in running queue (sometimes) and (in the beginning) many are blocked (due to paging?).I'd check buffer cache settings (nbuf etc).

Zeev