Memory and Swap issues

[Rafael Gutierrez]

Hi.

I have a SD32 server with 16 processors and 24GB RAM. It have 4 swap devices of 12GB each one and parameter swapmem_on is equal to 1. Besides, that server is hosting 1 Oracle 9i DB instance.

The issue is the server is showing high level of paging when it still have physical memory available and use of swap devices on disk is pretty high.

Any idea ? how could I understand and tune that behavior ?

[RAC_1]

What are your settings for buffer cache?? Dynamic or static??
kmtune -lq dbc_max_ct
kmtune -lq dbc_min_pct
kmtune -lq nbuf
kmtune -lq bufpages

vmstat 5 5
swapinfo -mat

[Pete Randall]

How are you measuring this high level of paging? Are they page-ins or page-outs? Run "vmstat 1 10" and look at the "po" column. Activity here is indicative of a problem. Activity in the "pi" column is not a cause for concern.


Pete

[A. Clay Stephenson]

Are you seeing high pageout rates? Pagein rates don't matter. Please post your swapinfo -tam output and either vmstat or glance output showing paging. It would also be good to see you ipcs -ma output.


One possible explanation for device swap showing as used is that memory-mapped files are in use. You may also have huge chunks of shared memory allocated from defunct oracle instances.

[A. Clay Stephenson]

It's probably a good idea to also post your kmtune output.

[Rafael Gutierrez]

Hi guys, answers for all of you...

ECARV10D - root # kmtune -lq dbc_max_pct
Parameter:dbc_max_pct
Current:15
Planned:15 Default:50 Dynamic:No
ECARV10D - root # kmtune -lq dbc_min_pct
Parameter:dbc_min_pct
Current:5
Planned:5 Default:5 Dynamic:No
ECARV10D - root # kmtune -lq nbuf
Parameter:nbuf
Current:0
Planned:0 Default:0
Dynamic:No
ECARV10D - root # kmtune -lq bufpages
Parameter:bufpages
Current:0
Planned:(NBUF*2)
Default:(NBUF*2)
Dynamic:No

ECARV10D - root # vmstat 5 5
procs memory page
faults cpu
r b w avm free re at pi po fr de sr in
sy cs us sy id
8 2 0 2454053 219459 461 15 25 35 1 0 644 7461
81073 3492 45 4 51
8 2 0 2454053 214918 507 12 5 0 0 0 0 9630
82118 3271 42 4 54
7 3 0 2414161 209127 767 12 8 0 0 0 0 8897
70750 2991 37 3 60
7 3 0 2414161 207413 529 12 5 0 0 0 0 8784
73989 2985 39 3 58
7 3 0 2436193 209589 348 25 3 0 0 0 0 7980
65597 2655 42 3 54

ECARV10D - root # swapinfo -mat
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 12288 351 11937 3% 0 - 1 /dev/vg00/lvol2
dev 12288 347 11941 3% 0 - 1 /dev/vg00/lvol9
dev 12272 358 11914 3% 0 - 1 /dev/vg02/lvol1
dev 12272 361 11911 3% 0 - 1 /dev/vg10/lvol1
reserve - 20503 -20503
memory 18455 4749 13706 26%
total 67575 26669 40906 39% - 0 -

[Rafael Gutierrez]

ECARV10D - root # ipcs -ma
IPC status from /dev/kmem as of Fri Mar 4 16:33:24 2005
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATT
CH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x41180262 --rw-rw-rw- root root root root
0 348 856 856 22:47:33 22:47:33 22:47:27
m 1 0x4e0c0002 --rw-rw-rw- root root root root
1 61760 856 856 22:47:29 22:47:33 22:47:27
m 2 0x41306b32 --rw-rw-rw- root root root root
1 8192 856 868 22:47:29 22:47:27 22:47:27
m 11267 0x0c6629c9 --rw-r----- root root root root
3 59227784 2202 28504 18:52:35 14:18:17 22:48:47
m 4 0x06347849 --rw-rw-rw- root root root root
1 77384 2202 2325 22:48:53 22:48:47 22:48:47
m 4101 0x4910364e --rw-r--r-- root root root root
0 22908 2198 7683 16:33:12 16:33:12 22:48:51
m 64518 0x5e141613 --rw------- root root root root
1 512 2836 2836 22:49:29 no-entry 22:49:29
m 86023 0x00000000 --rw-r----- ora500 dba ora500 dba 12
17 6442450944 27524 3213 16:33:24 16:33:24 4:11:33
m 7176 0x3c4f4c74 --rw-r----- ora500 dba ora500 dba 12
17 1991241728 27524 3213 16:33:24 16:33:24 4:11:33

Why I'm getting po=35 at the first line of vmstat, but the other lines po is equal to zero (0) ?

Thanks, Rafael.

[Rafael Gutierrez]

I'm uploading kmtune output.

[RAC_1]

What makes you think that your server has swap problem?? What does glance -m says?? Do you experience performance problems. Looking at your figures, everything looks normal except po of 35. Try resetting vmstat conter (-z option) and watch vmstat.

Anil

[A. Clay Stephenson · Accepted solution]

Your pageout rates are so low as to be insignificant. You didn't do an ipcs -ma output so I suspect you have large chunks of shared memory allocated. One thing to note is that your dbc_max_pct (even at 15% - 3.6GB) is too large. I would set bufpages to a non-zero value to pin the buffer cache at a fixed level. That makes much more sense on a system with large amounts of memory. About the absolute maximum buffer cache size on an 11.11 box is 1600MB (bufpages=409600) and that is quite generous. 1200MB would work quite well. Because you have more swap than physical memory, swapmem_on=1 serves no purpose. I would turn it off. The small amount of used device swap could very easily be memory-mapped files and if so is no problem at all. Although not related make sure that your timeslice value is not set to 1 but rather is set to 10 or very nearly so.

[Rafael Gutierrez]

Hi.

Clay, you can see ipcs output in an earlier reply.

After vmstat -z command, I haven't found po greater than zero anymore (I have executed vmstat several times).

I have another question: bufpages=nbuf*2 and nbuf=0 currently, how can I guess nbuf value according to server configuration ?

Thanks, Rafael.

[RAC_1]

My question for Clay.

If nbuf and bufpages both are set non-zero, then how cache is calculated??

[A. Clay Stephenson]

In almost every case, the default behavior with nbuf set to zero is fine. I would set bufpages as I mentioned earlier and leave nbuf=0.

[MAUCCI_2]

RAC,

compute it as mentionned here
http://docs.hp.com/en/939/KCParms/KCparam.Nbuf.html
http://docs.hp.com/en/939/KCParms/KCparam.BufPages.html

++Cyrille

[Rafael Gutierrez]

Thank you everybody. The memory and swap usage is really hard to understand in HP-UX, but it's more clear for me now.

Regards,

Rafael GZ.