topic Re: How to decrease page out ? in Operating System - HP-UX

How to decrease page out ?

Leonard C. — Tue, 16 Feb 2010 05:47:02 GMT

Dear all,

There is HP-UX 11.23 including Oracle RAC 10g with 8 core cpu and 24GB Physical Memory.
Swap area is 32GB:

vmstat

procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
2 4 0 3326871 19354 184 44 36 31 13 0 689 3025 56135 1523 2 2 96
2 4 0 3326871 19458 173 25 503 471 242 0 13397 6300 119120 7360 6 1 93
2 4 0 3326871 19440 153 21 516 445 272 0 10717 6471 123346 6654 9 3 88
2 4 0 3326871 19680 145 18 512 435 273 0 8578 6622 124787 6040 8 1 91
2 4 0 3326871 19450 132 16 526 407 314 0 6871 6436 116391 5350 7 3 90
2 2 0 3308955 20763 120 14 575 340 260 0 5496 6536 119604 5014 10 1 89
2 2 0 3308955 19803 158 53 590 272 208 0 4396 6366 113985 4606 7 4 89
2 2 0 3308955 18576 467 115 516 716 336 0 3516 6446 149100 4426 9 7 84
2 2 0 3308955 19436 486 93 533 894 323 0 2812 6649 207864 4176 7 5 88
2 2 0 3308955 21537 425 87 558 803 260 0 2249 6561 184938 3939 16 1 83
2 2 0 3308955 22551 523 129 549 642 208 0 1799 6422 170900 4200 9 8 83
2 2 0 3308955 22911 531 105 537 513 166 0 1439 7313 246192 19173 13 24 62
2 2 0 3308955 21205 531 85 496 410 132 0 1151 6918 211116 15911 7 1 92
2 2 0 3308955 20584 441 69 465 328 105 0 2913 6724 183983 13279 6 2 92
2 2 0 3308955 19573 390 56 502 262 84 0 10175 6508 163114 11202 11 2 87
2 5 0 3400256 20820 329 46 521 408 568 0 9601 6552 153610 9634 10 3 87
2 5 0 3400256 20650 400 39 687 535 788 0 7827 6868 166082 8611 11 5 84

sar:
sar09:13:05 %usr %sys %wio %idle
09:13:06 2 1 37 60
09:13:07 4 2 46 48
09:13:08 2 0 47 51
09:13:09 3 1 44 52
09:13:10 1 1 37 61
09:13:11 2 1 44 53
09:13:12 2 0 50 48
09:13:13 1 2 37 60
09:13:14 2 3 26 69
09:13:15 1 1 27 71
Average 2 1 40 57

sar -w
sar -w09:13:36 swpin/s bswin/s swpot/s bswot/s pswch/s
09:13:37 0.00 0.0 0.00 0.0 1506
09:13:38 0.00 0.0 0.00 0.0 68911
09:13:39 0.00 0.0 0.00 0.0 14055
09:13:40 0.00 0.0 0.00 0.0 1614
09:13:41 0.00 0.0 0.00 0.0 3313
09:13:42 0.00 0.0 0.00 0.0 4018
09:13:43 0.00 0.0 0.00 0.0 4567
09:13:44 0.00 0.0 0.00 0.0 1793
09:13:45 0.00 0.0 0.00 0.0 1249
09:13:46 0.00 0.0 0.00 0.0 1507
Average 0.00 0.0 0.00 0.0 10309

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16384 4190 12194 26% 0 - 1 /dev/vg00/lvol2
dev 16384 3990 12394 24% 0 - 1 /dev/vg00/lvol_swap01
reserve - 19346 -19346
memory 20450 4650 15800 23%
total 53218 32176 21042 60% - 0 -

---------------------------------------------
Any way to decrease page outs ?

Re: How to decrease page out ?

Leonard C. — Tue, 16 Feb 2010 05:56:48 GMT

Oracle SGA+PGA : 15GB

dbc_max_pct is 50 but current usage is 4 .

Re: How to decrease page out ?

Johnson Punniyalingam — Tue, 16 Feb 2010 06:00:02 GMT

The more physical memory there is, the more processes can be run and/or the larger a process (or processes) can be, without the system having to "page out". When the system has no more room in memory and begins to page out, performance will begin to degrade.

Not all physical memory is available to user processes. The kernel occupies some main memory and it is never paged. The amount of main memory not reserved for the kernel is termed available memory. Available memory is used by the system for executing processes.

The amount of memory available to applications is determined by the amount of swap configured plus physical memory. The amount of physical memory available will directly affect when or if paging will occur. Paging imposes a serious performance penalty. There is a critical threshold for physical memory, below which the system spends much of its CPU time paging. When it begins to spend most, if not all of the cpu time paging, it is known as thrashing. Thrashing is evident by the fact that system performance virtually comes to a standstill.

In a best case scenario, paging would never occur. However, not everyone has the luxury of enormous amounts of memory. Understanding how memory size affects performance is important. HP's performance monitoring tools (Glance/GlancePlus/MeasureWare/PerfView) can help in tracking memory and associated bottlenecks/problems.

Re: How to decrease page out ?

Johnson Punniyalingam — Tue, 16 Feb 2010 06:02:30 GMT

you could either increasing you "Psyical memory" or try decreasing work load of the server ,
Just thought .:)

Re: How to decrease page out ?

Leonard C. — Tue, 16 Feb 2010 06:30:43 GMT

Hi Johnson ,
Do you feel any misconfiguration?

Re: How to decrease page out ?

Venkatesh BL — Tue, 16 Feb 2010 06:42:43 GMT

What is dbc_min_pct? You could try reducing that value based on your file caching needs.

Re: How to decrease page out ?

Leonard C. — Tue, 16 Feb 2010 06:45:56 GMT

dbc_max_pct 50
dbc_min_pct 5

Re: How to decrease page out ?

Leonard C. — Tue, 16 Feb 2010 06:47:04 GMT

shmmax 25769803776
shmmni 1024
shmseg 1024

Any idea?

Re: How to decrease page out ?

Horia Chirculescu — Tue, 16 Feb 2010 07:33:29 GMT

Hello Leonard,

>Oracle SGA+PGA : 15GB

How much will be SGA?

You should try to decrease the shmmax to the value of SGA. (should be large enough to load the entire SGA).

Horia.

Re: How to decrease page out ?

Leonard C. — Tue, 16 Feb 2010 07:49:52 GMT

Sorry:
SGA : 15GB
PGA : 2.5GB

What do you recommend for SHMMAX , SHMMNI and SHMSEG

Re: How to decrease page out ?

Horia Chirculescu — Tue, 16 Feb 2010 08:10:24 GMT

>What do you recommend for SHMMAX , SHMMNI and SHMSEG

There is one great HP document (HP3KOracle) regarding HP_UX and ORACLE.

Please read this thread:

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1266307685682+28353475&threadId=1297266

See the ppt from the link that Dennis Handly provided.

Horia.

Re: How to decrease page out ?

Johnson Punniyalingam — Tue, 16 Feb 2010 09:50:36 GMT

>>>What do you recommend for SHMMAX , SHMMNI and SHMSEG
<<<

shmmax :-

Oracle recommends that you set the shmmax parameter value to the amount of available physical memory on the system. Doing this ensures that the entire shared memory for a single Oracle instance is assigned to one shared memory segment and the instance requires only one protection key

As per Oracle Recommdation :-

shmmni Max segments/system 512
shmseg Max segments/proc 120 per Oracle database

you shoulld always caluculate to contain SGA in 1 SHM segment .

Thanks,
Johnson

Re: How to decrease page out ?

Rita C Workman — Tue, 16 Feb 2010 14:53:34 GMT

dbc_max
dbc_min

How many times must this be answer I wonder sometimes........????

You have a really nice sized box - really!

Set your dbc_max%/dbc_min% to something around:
10-15 for max
5-10 for min

Next - SGA at 15Gb, your DBA is an idiot. Tell them to bring that down to reality. I don't care that you have 24 or 32Gb of memory, unless your running something to build the plans and calculate populating Mars they don't need that much. If you don't know about Oracle and databases you better get a book and learn more, or suffer the consequences of DBA's blaming every bad oracle query on the O/S. Although with your parms - for now they'd be right!

Now...parms

Run sar -v 1 20

For now, just look at inod-sz column. It shows /ninode parm setting.
How far different are they?

Likely you can reduce that setting DOWN and you should! ninode parm creates a table that takes up space. It supplies inodes, but since the table will recycle those inodes it doesn't need to be over large. So look at the usage and adjust accordingly. I generally do it increments of 2K.
For your big box the highest you may need is 8192, but you take a look and decide.

I don't care what you make your shmmax. I generally give it all I possibly can.

Now, that's a start for you. And just a start! There are plenty more parms you need to adjust...like maybe nproc, your semaphore settings and another one I adjust manually vx_ninode. But take small steps, tune one or two parms, monitor, see if you need more adjustments. Tune, monitor...until you are happy with performance.

Stepping off my soap box....cause once you change those first few you should see some improvement.

Oh...and don't forget to reset your counters so you see if the pageouts stop.
vmstat -z

Rgrds,
Rita

Re: How to decrease page out ?

Rita C Workman — Tue, 16 Feb 2010 14:56:45 GMT

Oh forgot to mention, once you adjust some parms, like maybe nproc, you will find that other parms will auto-adjust their sizing.
Like maybe ninode will get changed.
Well change it back.....

Remember, default settings on parms, DON'T make them right.
Case in point - dbc_max & dbc_min....very bad default setting!

/rcw

Re: How to decrease page out ?

Viktor Balogh — Wed, 17 Feb 2010 11:28:48 GMT

Hi Leonard,

In addition to Rita's comment: if you set the suggested parameters to a baseline, for finetuning only change a single parameter at a time so the effects will be clearly visible.