High paging rate - why and what do I need to do?

[Mott Given]

On an N4000 with 4 cpu's and 6144 MB memory, the paging rate is around 250 to 300 pages/sec. There are 3 paging volumes:
c19t1d1, c1t6d0, and c23t3d1

Can anyone offer advice on why the paging rate is so high?

During prime shift, c1t6d0 is 48-54% busy while c19t1d1 is 6-19% busy while disk c23t3di is less than 1% busy.

The Glance memory report shows:
system 316.4
user 1.19 GB
buf cache 614 MB
Free Mem 3.90 GB
Total VM 451 MB
Active VM 220 MB
The overall report shows Mem Util= 35%.

$ /usr/sbin/sysdef
NAME VALUE BOOT MIN-MAX UNITS FLAGS
acctresume 4 - -100-100 -
acctsuspend 2 - -100-100 -
allocate_fs_swapmap 0 - - -
bufpages 157286 - 0- Pages -
create_fastlinks 0 - - -
dbc_max_pct 10 - - -
dbc_min_pct 5 - - -
default_disk_ir 0 - - -
dskless_node 0 - 0-1 -
eisa_io_estimate 768 - - -
eqmemsize 39 - - -
file_pad 10 - 0- -
fs_async 0 - 0-1 -
hpux_aes_override 0 - - -
maxdsiz 503866 - 0-655360 Pages -
maxdsiz_64bit 1048576 - 256-1048576 Pages -
maxfiles 1024 - 30-2048 -
maxfiles_lim 2048 - 30-2048 -
maxssiz 32768 - 0-655360 Pages -
maxssiz_64bit 262144 - 256-1048576 Pages -
maxswapchunks 16384 - 1-16384 -
maxtsiz 16384 - 0-655360 Pages -
maxtsiz_64bit 262144 - 256-1048576 Pages -
maxuprc 4092 - 3- -
maxvgs 30 - - -
msgmap 268369920 - 3- -
nbuf 117005 - 0- -
ncallout 4132 - 6- -
ncdnode 150 - - -
ndilbuffers 30 - 1- -
netisr_priority -1 - -1-127 -
netmemmax 0 - - -
nfile 63798 - 14- -
nflocks 4096 - 2- -
ninode 8192 - 14- -
no_lvm_disks 0 - - -
nproc 4116 - 10- -
npty 256 - 1- -
nstrpty 60 - - -
nswapdev 10 - 1-25 -
nswapfs 10 - 1-25 -
public_shlibs 1 - - -
remote_nfs_swap 0 - - -
rtsched_numpri 32 - - -
sema 0 - 0-1 -
semmap 536805376 - 4- -
shmem 158 - 0-1 -
shmmni 1024 - 3-1024 -
streampipes 0 - 0- -
swapmem_on 1 - - -
swchunk 2048 - 2048-16384 kBytes -
timeslice 10 - -1-2147483648 Ticks -
unlockable_mem 24576 - 0- Pages -

$ UNIX95= ps -e -o vsz=Kbytes -o ruser -o pid,args=Command-Line | sort -rnk1 | more
KBYTES USER PID command
74304 oracle 25955 oracledebx (LOCAL=NO)
74048 oracle 15486 ora_lgwr_debx
70208 oracle 15488 ora_ckpt_debx
69952 oracle 15484 ora_dbw0_debx
69696 oracle 15494 ora_d000_debx
69696 oracle 15482 ora_pmon_debx
68160 oracle 15492 ora_reco_debx
68160 oracle 15490 ora_smon_debx
36132 root 5794 /opt/Navisphere/bin/naviagent -f /etc/Navisphere/agent.confi
28196 mqm 6649 amqzlaa0 -mQMTH0P05 -fip0
28116 patrol 7859 PatrolAgent -p 3181
26740 debx 13714 httpd -DSSL -d /h/data/global/EC/www
25200 best1 5264 /usr/adm/best1_6.6/bgs/bin/bgsagent -b /usr/adm/best1_6.6 -a
21556 debx 25673 httpd -DSSL -d /h/data/global/EC/www
13068 debx 25730 trans_inject
12300 debx 25744 reporter
12300 debx 7363 reporter
12276 debx 25668 httpd -DSSL -d /h/data/global/EC/www
11892 debx 16119 httpd -DSSL -d /h/data/global/EC/www
10748 root 1143 /opt/dce/sbin/rpcd
10100 debx 27492 httpd -DSSL -d /h/data/global/EC/www
9468 root 5930 /opt/perf/bin/midaemon
9332 debx 25672 httpd -DSSL -d /h/data/global/EC/www
8756 debx 22442 httpd -DSSL -d /h/data/global/EC/www

Mott Given

[Jose Mosquera]

Hi, In the past we have a similar issue, in our case I noticed that **swapper** presence was always on top 1st page, then meant that our system handles activation and reactivation of processes when free memory falls bellow minfree, or when the system appears to be **trashing**.

We solved this adapting swap space and dbc_max_pct Kernel parameter. The recomendations are from 5% to 10% depends on the total amount of memory in your system. The minimun value must be 300MB, then you can try to put a low percentage, I sugest you that first try with 25% and look at the backup performance in order to know if this is a good percentage. After you do this and reboot your dont want to see the USED line for the device lines from swapinfo -mt > 0. This means no memory pressure and no paging. This will increase performance considerably.

In your case this parameter looks fine. Have you enough swap space?

Pls find attached a useful doc about HP-UX memory management.

Rgds.

[Mott Given]

Thanks for the info, Jose.

I'm not sure about our swap space. This system running HP-UX v11.0 has 833 MB being used for pseudo-swap and has swapmem_on =1.

$ /usr/sbin/swapinfo -mt
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
dev 2000 0 2000 0% 0 - 0 /dev/vg01/swap2
reserve - 1056 -1056
memory 4764 833 3931 17%
total 7788 1889 5899 24% - 0 -

Mott Given

[twang]

In my opinion, it is suitable to take measures to tune the application. The high I/O actions are most likely caused by some problem SQLs. I would suggest to apply STATSPACK to find out what is the buttleneck in oracle, what are the problem SQL, and how the I/O loads are distributed. You may have to redistribute the datafiles to reduce I/O.

[Ted Buis]

I assume you are focused on page outs rather than page ins. Make sure that swap space is larger than RAM and than swap_chunks is large enough to cover that space. Once this is verified, disable pseudo-swap (swapmem_on=0), as it is only useful if you have more RAM than addressable disk swap space.

[Ted Buis]

I should add that Chapter 7 or "hp-ux 11i internals" by Chris Cooper and Chris Moore, and specifically page 195, should provide you some good background. While only more RAM can really help in some cases to reduce page outs, eliminating psuedo-swap can help make the process a little more more fair by not overpenalizing those application that don't happen to be in the psuedo-swap area of RAM.

[Mott Given]

Twang, the I/O rate averages 500 I/O /sec, with peaks of 1300 I/O per second. Most of the I/Os are coming from an Oracle DB, v9.2.0.1. The disks with the highest I/O rates are c19t1d1 with 300 I/O/sec and c1t6d0 with 120 I/O/second. The page out rate is 140 to 180 pages/second.

[Francisco Plaza G.]

Hi Mott, we have same problem and we have detected that PatrolAgent produce it.
Patrol begun to "eat Memory" even start a heavy swapping, leave'em without access to server.
The patch Bundle installed is March/2004 and this especific patches PHKL_31867, PHKL_30553, PHKL30578, PHNE_29473, PHNE_29530. This bundle and patches works better but the symptom persist.
We choose to shutdown patrolagent.
Can you tell me what version do you have of Patrol?
Sorry for my english

Greetings

[G. Vrijhoeven]

HI Mott,

What is the behavour of you application. Does it start new processes frequently. Since you have Free Mem 3.90 GB and swap space usage of 0% i do not think tuning swap/adding memory will help.
Your primary memory user is Oracle Check your SGA settings, may be increasing that will help.

HTH,

Gideon

[Steven E. Protter]

collect peformance data. See attached script.

SEP

[Sridhar Bhaskarla]

Hi Mott,

How are you determining that you are having high 'pageouts'? If they are simply pageins, you don't need to worry too much about it.

Your 'swapinfo' shows no paging activity at all. If the system is low on memory and if is paging, you should have seen some usage in 'swapinfo -dt' output.

I don't believe your system is low on memory. Post the outputs of the following 'when' you think that the system is paging out.

sar 2 5
vmstat 2 5
sar -d 2 5
sar -q 2 5

-Sri

[Mott Given]

Thanks for all your help.

Mott