Why am I paging

[Wayne Green]

Got a box runing out of memory during peak times but now is not peak time and glance memory window shows

--------------------------------------------------------------------------------
CPU Util S SN NRU U | 44% 17% 55%
Disk Util F F | 80% 92% 100%
Mem Util S SU UB B | 74% 74% 74%
Swap Util U UR R | 35% 35% 35%
--------------------------------------------------------------------------------
MEMORY REPORT Users= 17
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 34724 49774 6808.6 1885.3 6808.6
Page In 9824 14533 1926.2 550.4 1926.2
Page Out 33 56 6.4 2.1 6.4
KB Paged In 24kb 28kb 4.7 1.0 4.7
KB Paged Out 132kb 224kb 25.8 8.4 25.8
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 3 4 0.5 0.1 0.5
VM Writes 72 104 14.1 3.9 14.1

Total VM : 1.25gb Sys Mem : 306.0mb User Mem: 2.94gb Phys Mem: 6.00gb
Active VM: 663.4mb Buf Cache: 1.20gb Free Mem: 1.56gb

vmstat still shows low number of page outs. swapinfo -tm shows
swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 55 4041 1% 0 - 1 /dev/vg00/lvol2
dev 2000 0 2000 0% 0 - 2 /dev/vg01/lvol1
dev 2000 0 2000 0% 0 - 2 /dev/vg01/lvol2
reserve - 3105 -3105
memory 3838 994 2844 26%
total 11934 4154 7780 35% - 0 -

I thought it didn't page until freemem got below lotsfree
lotsfree: 24576

For background box has several oracle DBs and financials running. Financials users starting to complain about performance. Several more DBs loaded recently and had to use memory windows to accommodate shared memory requirements. 6GB RAM 4GB swap swapmem_on=1 got process unable to fork message in application log file. Added extra swap devices and now dont get those messages.

Been reading up on pseudo-swap so I now dont understand why the application couldn't fork and I dont understand why the box is still paging when glance says there is 1.5GB free.

One to ponder until tomorrow.

[Paula J Frazer-Campbell]

Hi

What are you dbc_man_pct and dbc_min_pct ?


Paula

[Paula J Frazer-Campbell]

That should be
dbc_max_pct

Paula

[Paula J Frazer-Campbell]

Hi
Looking at sysmem I would suggest you reduce dbc_max_pct to 10% and dbc_min to 2%.

A kernel rebuild and a reboot is required.

Paula

[Jeff Schussele]

Hi Wayne,

Page-outs with plenty of free mem can simply be shared memory driven & totally normal.

You'll know when "true" page-outs are happening because you'll be really short on free mem & the PO rate will be hot & heavy & you'll start to see deactivations/reactivations & these are the killers.

The addition of swap is what stopped the forking msgs as the system has to have reservation space for every process that starts whether it will actually be swapped out or not.

Pseudo swap & swapmem_on are only really needed when you have more swap than memory & in your case it doesn't apply so there's no need to have it on, but it doesn't hurt a thing to have it on.

RGds,
Jeff

[Wayne Green]

Thanks both for the replies, dbc_min and dbc_max are both locked at 20%. I to would reduce this, in fact I would probably use bufpages on this system but I get out voted. Reason for the high value is the benefit the customer gets when they copy the DBs every day as a backup. Taking dbc down more than doubles these times and they are time sensitive on their batchwork. Of course their interactive response goes through the floor during the copies but thats another story. When I explained you can tune for predominantly sequential or random I/O but not both the reaction was HP should rewrite their O/S. Alternatives to the copy have also been suggested ...

I think I am misunderstanding the page outs figures. I thought this only gets reported when out of memory i.e. below lotsfree but from looking at other systems this is not true.

For reference this is a normal load from glance
--------------------------------------------------------------------------------
CPU Util S SN NRU U | 75% 47% 100%
Disk Util F F | 17% 62% 100%
Mem Util S SU UB B | 99% 98% 99%
Swap Util U UR R | 54% 53% 54%
--------------------------------------------------------------------------------
MEMORY REPORT Users= 214
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 12417 2187837 2483.4 2629.2 13509.4
Page In 4217 711706 843.4 855.3 3396.1
Page Out 14 2441 2.8 2.9 9.1
KB Paged In 0kb 45.0mb 0.0 55.4 953250.0
KB Paged Out 56kb 9.5mb 11.2 11.7 36.6
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 2940 0.0 3.5 68.8
VM Writes 28 4882 5.6 5.8 18.3

Total VM : 2.81gb Sys Mem : 306.0mb User Mem: 4.42gb Phys Mem: 6.00gb
Active VM: 865.2mb Buf Cache: 1.20gb Free Mem: 85.8mb

I think they need some more meory, even though they've just plugged in 2GB.

[Jeff Schussele · Accepted solution]

Hi Wayne,

Well, looking at those last reports the bad news is that you appear to definitely be experiencing a memory bottleneck. With Mem util locked in @ 98/99% & page-outs at a significant rate, I don't think there's any doubt. The only good news is that you aren't suffering any de/reactivations. If you were you'd be getting all kinds of calls RE performance.

So you really have two options:
A) Reduce memory demand - via kernel tuning or putting the apps on a memory diet - preferrably both.
B) Add more physical memory.

It would be interesting to see a swapinfo -tam output to see how much swap is actually being used as opposed to just reserved.

Rgds,
Jeff

[keith persons]

Wayne,

First, let me answer a couple questions you asked:

>swapmem_on=1 got process unable to fork message in application log file. Added extra swap devices and now dont get those messages

since the snapshots provided are not during peak the relevant info in in swap output, which means you had:

dev 4096 55 4041 1% 0 - 1 /dev/vg00/lvol2
memory 3838 994 2844 26%

or similar. The simple answer is of the 4gb of dev swap configured, much of it was already allocated as represented by the memory line. Assuming higher memory demands during peak would start a potential cascade of events where the inability to fork would be at least an early indicator. Adding dev swap resolved this, it is still a good rule to try to config dev swap equal to phys memory.

> reading up on pseudo-swap so I now dont understand why the application couldn't fork and I dont understand why the box is still paging when glance says there is 1.5GB free.

Well, there is and there isn't 1.5gb free (it's a virtual thing...hehehe). It's really just a raw calculation that means of the 6gb installed, approximately 1.5gb remain on the kernel free list. It's not an actual count of free pages on the free page list. It's most appropriate use is aiding in establishing norms for your particular config.

>When I explained you can tune for predominantly sequential or random I/O but not both the reaction was HP should rewrite their O/S.

ALL operating systems are a balance of processing vs. IO, perhaps they should attain a better understanding of a demand paged, virtual memory operating system and quit trying to force a "...square peg into a round hole..."

>I think I am misunderstanding the page outs figures

Pageouts are inherent to a virtual memory os. On a smoothly running system they're never noticed unless one looks for them or they occur with such frequency they impact throughput. This is part of the reason for pseudo-swap. If only using device swap, raw IO activity is a function of pageouts, if the pageouts can only go to device swap, the more pageouts, the more IO, the more IO, the greater the demand for kernel scheduling, the greater the demand, the more time system overhead is required, the more time.....and on and on. When using pseudo-swap even tho the page may be 'paged out', it still remains in memory until or unless page demand increases to the point physical pageouts/swapouts are required.

Whew, long time getting here but a couple general suggestion:

check your alt dev swap lvols, make sure the do not reside on the same physical spindle(s) - if they do try to combine them into a single swap device, less abusive to the disks themselves. Also, change the priority of all swap devices to 0 - take any swap load off the root IO channel for as long as possible.

Based on your last snapshot:

CPU Util S SN NRU U | 75% 47% 100%
Disk Util F F | 17% 62% 100%
Mem Util S SU UB B | 99% 98% 99%
Swap Util U UR R | 54% 53% 54%

one could easily conclude memory use is the constraint however, you could easily get led astray in this scenario. What I find more intriguing is the cpu use and distribtion - there's not a lot of user time for cpu util at 75%! A lot of nicing seems to be going ontoo, between that and the system mode looks like you're getting less than 40% user on the processor. So, the most important and probably the most difficult question becomes why? This is where rule #1 of performance comes most play -- It depends....

Keith

[Wayne Green]

Jeff, I agree. at the moment swapinfo is showing 8% on the first device, I've seen it into the teens during peak periods.

I could cut down the buffer cache but then the application support team complains about the copy times, or a possibiltiy is asking the DBA about oracles requirements. There are 22 DBs running.

But on the bright side, an extra 2GB was added to 4 servers including this one as preparation for the extra DBs. One of these boxes isn't used much so hopefully we can add another 2GB to my problem machine and it will go away.

Keith, thanks for the info although I thought the memory line in swapinfo indicated pseudo swap and this is never used but makes the O/S think it has swap space available so that processes can eb started.

So if I have a 4GB swap device and nearly 4GB pseudo swap - albeit 1GB is reserved - and 6GB of physical RAM then even if I used up my 6GB of memory new processes would start cause paging would start else whats to point of pseudo swap.

As for the sequential vs random I/O limitations. I've been banging my head against that brick wall for several years. The response was funny the first time, now I just shrug my shoulders.

Thanks all.

[Jeff Schussele]

If the response to requests for more physical memory is shot down & probably even before the request is made, you should consult with the DBAs & find out what size they are defining for the SGAs for these DBs.
They may be going overboard & not realizing the impact now that they have multiple DBs on these systems.

Rgds,
Jeff