Maxing out of swap, twice

[Michael Steele_2]

Hi Everyone:


A) I'm a little concerned about how much swap is being utilized on a L3000, HP-UX 11.11, running SAP and Oracle. Yesterday total swap was 98% so I added 8208 mb to 10240 mb and today after running a monthly order report its again at 98%.

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 10240 6193 4047 60% 0 - 1 /dev/vg00/lvol2
dev 8208 6181 2027 75% 0 - 1 /dev/vg00/swap
reserve - 6074 -6074
memory 6471 6091 380 94%
total 24919 24539 380 98% - 0 -

maxswapchunks = 16384 (max)
swchunk = 2048

B) Can't find UNIX95 on this box. Where's it suppose to be? What's best way to isolate memory consumming process without UNIX95?

C) NATTACH looks find in ipcs. No zero entries.

[Steven E. Protter]

Shalom Michael,

I'm a little concerned about the general health of this box. Oracle and SAP surely can consume a lot of resources but if they really needed all that extra swap they would not have run properly before that swap.

So.

A) I think this system needs patching or the OS done over. Maybe tuning, but it seems quite sick.

B) UNIX95 has always just been there. I've never had to install it. I'll assume then that it can be found on the Core OS disk, because I had it after doing a fresh system install this week. The fact that its missing points to a possibly catastrophic problem with this system.

swverify \*
report back with errors.

ipcs command?

I share your concern.

[Geoff Wild]

An SAP system needs swap space - but shouldn't be using it like yours.

Mine is:

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 100 3996 2% 0 - 1 /dev/vg00/lvol2
dev 22432 0 22432 0% 0 - 2 /dev/vg01/swap
reserve - 14174 -14174
memory 11140 1930 9210 17%
total 37668 16204 21464 43% - 0 -


Check out the following for memory:
ftp://hprc.external.hp.com/memory.htm


Does this not work on your system?

UNIX95= ps -e -o 'vsz pid ruser args' |sort -nr|head -30


Rgds...Geoff

[Michael Steele_2]

A) # swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 10240 6193 4047 60% 0 - 1 /dev/vg00/lvol2
dev 8208 6181 2027 75% 0 - 1 /dev/vg00/swap
reserve - 6074 -6074
memory 6471 6091 380 94%
total 24919 24539 380 98% - 0 -

B) Here's swverify \*


* Beginning Analysis
* Session selections have been saved in the file
"/root/.sw/sessions/swverify.last".
ERROR: "rodan:/": 1 software selection(s) are corrupt.
WARNING: "rodan:/": 1 check scripts had warnings.
ERROR: "rodan:/": 10 files had errors during this operation.
WARNING: "rodan:/": 417 files had warnings during this operation.
* The analysis phase failed for "rodan:/".
* Verification had errors.


NOTE: More information may be found in the agent logfile using the
command "swjob -a log rodan-0097 @ rodan:/".

======= 07/11/06 09:43:27 EDT END swverify SESSION (non-interactive)
(jobid=rodan-0097)

C) Here's ipcs:

# ipcs -moba
IPC status from /dev/kmem as of Tue Jul 11 09:46:07 2006
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 1792 0x0c6629c9 --rw-r----- root root root root 2 19058208 1753 14991 9:38:09 9:43:37 19:47:17
m 1 0x06347849 --rw-rw-rw- root root root root 1 77384 1753 1879 19:47:23 19:47:17 19:47:17
m 1282 0xffffffff --rw-r--rw- root root root root 0 22908 1749 1749 19:47:21 19:47:21 19:47:21
m 9731 0x5e180008 --rw------- root root root root 1 512 1972 1972 19:47:43 no-entry 19:47:43
m 77316 0x00004dbe --rw-rw-rw- root sapsys root sapsys 0 503848 2829 3475 9:42:21 9:42:21 19:50:44
m 13061 0xa237aff8 --rw-r----- orabwp dba orabwp dba 126 1579130880 3314 14708 9:26:19 9:26:25 19:51:14
m 12038 0x00002b95 --rw-r----- bwpadm sapsys bwpadm sapsys 32 551452 3452 9839 8:36:26 8:36:26 19:51:36
m 7 0x00000000 --rw-r----- bwpadm sapsys bwpadm sapsys 1 1024 3420 3420 19:51:32 no-entry 19:51:32
m 8 0x0382be8f --rw-rw-rw- bwpadm sapsys bwpadm sapsys 35 4096 3424 9839 8:36:26 no-entry 19:51:32
m 1289 0x00002b66 --rw-r----- bwpadm sapsys bwpadm sapsys 32 124000000 3452 9839 8:36:26 no-entry 19:51:36
m 522 0x00002b5d --rw-r----- bwpadm sapsys bwpadm sapsys 32 312 3452 9839 8:36:26 8:36:26 19:51:38
m 523 0x00002b5e --rw-r----- bwpadm sapsys bwpadm sapsys 32 32381312 3452 9839 8:36:26 8:36:26 19:51:38
m 524 0x00002b5f --rw-r----- bwpadm sapsys bwpadm sapsys 32 100944000 3452 9839 8:36:26 8:36:26 19:51:38
m 525 0x00002b90 --rw-r----- bwpadm sapsys bwpadm sapsys 32 53752 3452 9839 8:36:26 8:36:26 19:51:38
m 526 0x00002b7b --rw-r----- bwpadm sapsys bwpadm sapsys 32 3922420 3452 9839 8:36:26 8:36:26 19:51:38
m 527 0x00002b8f --rw-r----- bwpadm sapsys bwpadm sapsys 31 10034420 3452 8752 7:46:28 7:46:28 19:51:38
m 528 0x00000000 D-rw-rw-rw- bwpadm sapsys bwpadm sapsys 31 2147483648 3452 3452 19:51:38 no-entry 19:51:38
m 273 0x00002b9a --rw-r----- bwpadm sapsys bwpadm sapsys 31 85716680 3452 8752 7:46:28 7:46:28 19:51:38
m 2066 0x00002b65 --rw-r----- bwpadm sapsys bwpadm sapsys 30 134217728 3475 8752 7:46:28 7:46:28 19:51:40
m 787 0x00002b64 --rw-r----- bwpadm sapsys bwpadm sapsys 30 67108864 3475 8752 7:46:28 7:46:28 19:51:40
m 788 0x00002b6e --rw-r----- bwpadm sapsys bwpadm sapsys 30 656416 3475 8752 7:46:28 7:46:28 19:51:40
m 789 0x00002b84 --rw-r----- bwpadm sapsys bwpadm sapsys 30 210000000 3475 8752 7:46:28 no-entry 19:51:40
m 790 0x00002b63 --rw-r----- bwpadm sapsys bwpadm sapsys 30 7440 3475 8752 7:46:28 7:46:28 19:51:40
m 791 0x00002b60 --rw-r----- bwpadm sapsys bwpadm sapsys 30 537856 3475 8752 7:46:28 7:46:28 19:51:40
m 2584 0x00002b71 --rw-r----- bwpadm sapsys bwpadm sapsys 1 2258192 9839 9839 8:36:26 8:36:26 8:36:26
m 1049 0x00002b85 --rw-r----- bwpadm sapsys bwpadm sapsys 30 9974144 3475 8752 7:46:28 7:46:28 19:51:41
m 794 0x00002b6f --rw-r----- bwpadm sapsys bwpadm sapsys 30 64000128 3475 8752 7:46:28 7:46:28 19:51:42
m 795 0x00002b7d --rw-r----- bwpadm sapsys bwpadm sapsys 30 102400112 3475 8752 7:46:28 7:46:28 19:51:42
m 796 0x00002b92 --rw-r----- bwpadm sapsys bwpadm sapsys 30 126976192 3475 8752 7:46:28 7:46:28 19:51:43
m 797 0x00002b9d --rw-r----- bwpadm sapsys bwpadm sapsys 30 40960192 3475 8752 7:46:28 7:46:28 19:51:43
m 798 0x00002b9c --rw-r----- bwpadm sapsys bwpadm sapsys 30 4194512 3475 8752 7:46:28 7:46:28 19:51:43
m 29727 0x00002b62 --rw-r----- bwpadm sapsys bwpadm sapsys 30 1024000000 3480 8752 7:46:28 7:46:28 19:51:50
m 21792 0x00002b7a --rw-r----- bwpadm sapsys bwpadm sapsys 30 31276 3477 8752 7:46:28 7:46:28 19:51:51
m 50209 0x00002b96 --rw-r----- bwpadm sapsys bwpadm sapsys 19 2076 3475 8752 7:49:33 7:49:33 19:51:56
m 2082 0x00002b7e --rw-r----- bwpadm sapsys bwpadm sapsys 20 10240000 3475 3527 9:09:53 9:09:53 19:51:55
m 121379 0x00002b94 --rw-r----- bwpadm sapsys bwpadm sapsys 6 519920 3476 3511 9:17:40 9:17:40 21:05:16
#



D) What CD is UNIX95 going to be located on? Or, can you whence or where and provide a path? I'll ftp it from another 11.11 box.

[Pete Randall]

UNIX95 is not located anywhere - it is a variable, that, when set, makes certain commands, particularly ps, behave in XPG4 fashion, rather than standard HP-UX fashion.

Cut and paste the command as given by Geoff and see if it doesn't work on your system:

UNIX95= ps -e -o 'vsz pid ruser args' |sort -nr|head -30


Pete

[James R. Ferguson]

Hi Michael:

Are you seeing page-outs with 'vmstat' or 'glance'?

Are you having ENOMEM (errno=12) errors, or 'can't fork' or 'not enough space' messages?

What is the size of your UNIX buffer cache? Is it too large?

As far as UNIX95, this is an environmental variable that controls the behavior of certain commands, most usefully 'ps':

# UNIX95= ps -ef -o pid,ppid,vsz,args|sort -k3nr|more

...will create a list of processes in descending order of memory footprint.

Note the blank (whitespace) after the equal sign before the 'ps' command. This sets UNIX95 *only* for the duration of the command line.

Regards!

...JRF...

[Geoff Wild]

Your ipcs output looks normal for a SAP system - I just compared it to mine.

How much ram is in this system?

Did Oracle DBA increase SGA size and not tell you?

Rgds...Geoff

[Steven E. Protter]

swjob -a log rodan-0097 @ rodan:/

This system has significant problems.

I thought UNIX95 was just there and now thats been validated.

What is corrupt will determine further recommendations on my part.

SEP

[Michael Steele_2]

A) Here is UNIX95

# UNIX95= ps -e -o 'vsz sz pid ppid ruser comm ruser args' |sort -nr|head -30
123704 15582 2853 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
114808 16613 2852 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
113976 16622 5124 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
109400 16572 3475 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
109112 18708 8752 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
108760 20597 3476 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
108344 16615 3576 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
106584 20859 3477 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
104108 797 9839 3452 bwpadm gwrd bwpadm gwrd -dp -apptrc pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_roda
101080 20150 3480 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
95832 20189 3485 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
93048 16590 3573 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
93016 19574 3511 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
91384 20058 3513 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
91192 16612 3543 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
90632 18121 3515 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
87704 16604 3518 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
87672 16612 3521 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
87672 16604 3524 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
87160 12524 3527 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
86808 16604 3530 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
86648 16604 3535 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
86376 16612 3540 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
86136 14246 3579 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
82664 16590 3546 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
82240 19255 3338 1 bwpadm ora_lgwr_BWP bwpadm ora_lgwr_BWP
82240 18586 3335 1 bwpadm ora_dbw0_BWP bwpadm ora_dbw0_BWP
80296 16628 3557 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
80296 16590 3585 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r
80296 16590 3580 3452 bwpadm dw.sapBWP_DVEBMGS11 bwpadm dw.sapBWP_DVEBMGS11 pf=/usr/sap/BWP/SYS/profile/BWP_DVEBMGS11_r


B) Here is 'sar -b' and 'sar -v'

10:21:41 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
10:21:46 0 9 100 2 2 0 1088 1
10:21:51 0 2 100 2 3 21 989 0
10:21:56 0 4 100 1 1 33 837 0
10:22:01 0 2 100 3 5 43 957 4
10:22:06 0 1 100 1 1 0 794 0

Average 0 4 100 2 2 25 933 1
# sar -v 5 5

HP-UX rodan B.11.11 U 9000/800 07/11/06

10:22:11 text-sz ov proc-sz ov inod-sz ov file-sz ov
10:22:16 N/A N/A 246/2560 0 1042/16384 0 2633/32778 0
10:22:21 N/A N/A 246/2560 0 1042/16384 0 2633/32778 0
10:22:26 N/A N/A 246/2560 0 1042/16384 0 2633/32778 0
10:22:31 N/A N/A 246/2560 0 1042/16384 0 2634/32778 0
10:22:36 N/A N/A 246/2560 0 1042/16384 0 2634/32778 0


C) SGA = 2.6 gb

D) Total RAM = 8 GB

E) Working on locating corrupt patch / module.

[Michael Steele_2]

Stephen: The only corrupt fileset is sudo. Which is only invoked from the keyboard and not embedded in any application.

# swlist -l fileset -a state | grep -i corrupt
sudo.sudo-RUN corrupt

[Michael Steele_2]

Yes, there are page outs and this is accumulated over 12 hours.

# vmstat 5 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
2 2 0 3984049 7273 47 15 171 270 13 0 7096 1836 35317 1251 19 5 76
1 3 0 4143899 7251 5 0 541 405 88 0 6917 1775 6184 1010 16 4 80
1 3 0 4143899 7730 3 0 542 534 28 0 24495 1887 10460 1340 19 4 77
2 2 0 3920688 7419 3 0 551 504 89 0 29932 1778 11805 1456 21 5 74
2 2 0 3920688 7330 2 0 523 543 41 0 10889 1795 9846 1317 21 3 75

[TwoProc]

Well, totalling up your shared memory, it comes to somewhere between 4.5 and 5 Gig. Since you stated you added 8G and went to 10G, it means you originally ran in 2G before yesterday. This means someone MUST have allocated a LOT more memory to Oracle and SAP b/c before yesterday your current configuration wouldn't have fit in the box - unless I've misunderstood what " I added 8208 mb to 10240 mb" means...

Anyways, since you've got 10G of memory, and shared memory is taking something less than half of that, then you probably getting wiped out in the data and heap spaces of running program(s), and probably not in the running of the server based programs which use shared memory.

I'd look at the monthly order report and see what has changed. Are you somehow processing a lot more data? Did someone patch the report program possibly and sucessfully invite new memory bugs over to live in your server? If you're using the canned report writer for SAP (whatever that uses I'm not sure), my guess is that this is your problem. Maybe your report is 100% custom using sqlplus or Pro-C, if so the bug lives in there.

Also, just for review, recheck your max_dbc_pct setting in the kernel. I don't think this is your problem, b/c I don't see a large shared memory chunk sitting in your ipcs report, but you should review this after adding memory as a matter of course regardless.

[Pete Randall]

Your vmstat posting shows significant po (page out) activity - definitely not a good sign - you need to either free up some memory or provide more. What is your DBC set to?


Pete

[Michael Steele_2]

# grep -i dbc system
dbc_max_pct 3
dbc_min_pct 2

And this is a recent change from June 4:

# grep -i dbc system.prev
dbc_max_pct 5
dbc_min_pct 3

what do you think 5 and 3?

[TwoProc]

5 and 3 are fine.

Review the running month end report. Run glance and pull up the process list and sort by memory size from the top (don't forget to turn OFF filtering) - your problem(s) should be shown at the top of the list.

[Pete Randall]

Well, 3 and 2 should be using less memory than 5 and 3, so it would seem that DBC is not part of the problem.


Pete

[Steven E. Protter]

Well then, the system is not so bad off. No corruption.

SAP is eating your system for breakfast, lunch and dinner. Perhaps some SAP patches are in order.

It may be a problem with SAP, but the next step would be an increase in RAM. How much is hard to tell at this point.

SEP

[Geoff Wild]

Actually - I'd say your dbc's are too low - with only 8GB of ram @ 2% - that means only 164 MB is allocated for buffer cache - you really want this to be between 500 and 800 mb.

That said - comparing apples to apples, my BW system has 10GB of ram - not much more then yours...

Here's some info from my BWP system:

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4080 2640 1440 65% 0 - 1 /dev/vg00/lvol2
dev 24000 0 24000 0% 0 - 2 /dev/vg02/lvol1
reserve - 6556 -6556
memory 7708 1670 6038 22%
total 35788 10866 24922 30% - 0 -

root@pc1002 [ / ]
# memdetail
Memory Stat total used avail %used
physical 10080.0 7590.6 2489.4 75%
active virtual 9539.0 2408.3 7130.7 25%
active real 5297.3 1186.2 4111.1 22%
memory swap 7708.2 1670.5 6037.7 22%
device swap 28080.0 9195.8 18884.2 33%

root@pc1002 [ / ]
# hpmem
HP-UX pc1002 B.11.11 U 9000/800 1753409998 unlimited-user license
CPU Count: 6
CPU Speed: 750 MHz
CPU HW Support: 64-bit
Kernel Support: 64-bit
RAM Size: 10080 MB
bufpages: 806 MB
maxuprc: 800
maxvgs: 128
maxfiles: 2048
max_thread_proc: 256
nfile: 32778
nflock: 1200
nproc: 2560
ninode: 16384
shmmax: 1073741824
shmmni: 256
dbc_max_pct: 8

Rgds...Geoff

[Pete Randall]

I'm not really familiar with SAP. Is it normal for there to be so many bwpadm processes? I count somewhere around 30 of them - seems like a lot.


Pete

[Rita C Workman]

Hello Michael (and Stephen,Jim & Pete)

Well...let me throw in my thought. I didn't see anywhere when you describe your hardware. How much memory is on this box please ?
...Now I notice in your sar -v that the ninode value is a bit high. This value gets auto set by other parms and it invariably is way too high. You can easily reset yours to 2048 and have room to spare. I'd also consider changing my vx_ninode (after hearing Stephen Ciullo-how do you spell his name...), when this parm is left at 0, and you have 8Gb of memory or more, this parm will create a huge unnecessary table taking up memory. On my 8Gb boxes I set mine around 20-40m and multiple instances of Oracle 10G hasn't run along smooth.
So my guess is you have a number of parms that need fixing. Setting dbc_max and dbc_min down may be good, if you had enough memory to justify the reduction to what you set 5/3. You may have it too low for your boxes.....but can't tell without knowing the memory you have in there.

So....I'd like to know how much memory you have...and I'd like to know see a clean txt file of your parms. I think folks here could give you a hand on your parms at least. I'm not a SAP shop...but hey, tuning is tuning.

Kindest regards to all & see ya in Houston,
Rita

[Geoff Wild]

Pete - yep - that's normal - I have 32 on my BW system.

Rgds...Geoff

[A. Clay Stephenson]

First of all, you have more actual swap space than memory so pseudoswap is doing nothing for you other than confusing the output of swapinfo. I would set swapmem_on=0.

I haven't seen anything in all of this where you show actual pageouts although you are probably on the hairy edge. Note that you could be seeing device swap usage indicated by swapinfo and still not actually swapping if memory-mapped files are in use.

Examine your buffer cache settings and I would suggest a static buffer cache by setting bufpages to a non-zero value.

One minor point (although it's rearranging deck chairs on the Titanic) if both of your swap areas are located on the same physical disk, you should set then at different priorities. Primary swap is at priority 1 so your other swap should be set to other than 1.

[Michael Steele_2]

Hi again all.

A) You have no idea how much I appreciate this.

Geoff is coming very, very close and appears to be correct about increasing dbc_max_pct and dbc_min_pct to stop the paging. From what I can figure out, dbc_max_pct and dbc_min_pct were changed to 3 and 2 on June 4 but the server was never rebooted. I am concluding this because 'uptime' displayed 71 days since last reboot but its 36 days back to the June 4 timestamp on system.prev and vmunix.prev. So the 3 and 2 dbc's appear to have gone active at yesterday's reboot and exacerbated the page outs of +2000 pos' in 12 hours after reboot verses ~1000 pos' in the 71 days before reboot.

dbc_max_pct and dbc_min_pct are back to 5 and 3 now. The month end application is running for 4+ hours and no paging seen. Total swap is constant at 41%.

Follow up:

A Here's some current kernel params for Rita:

CURR bufpages 0
CURR CPU 4
CURR maxuprc 800
CURR maxvgs 16
CURR maxfiles 1024
CURR max_thread_proc 250
CURR nfile 32768

CURR nproc 2560
CURR ninode 16384
CURR shmmax 17179869184
CURR shmmni 256
CURR dbc_max_pct5
CURR dbc_min_pct 3

B) Total swap = 18 GB

C) Total RAM = 8 GB

Gimme 'five' Geoff and I'll give you ten.

[Chan 007]

Hi Mic,

There is a command SAP based at /sapmnt//exe/memlimit(s), using which SAP can configure your process. (I guess using this they do some config for the SAP process & their threads)

When I had a sililar problem, SAP sorted out on my BWP instance.

If you need more info, I can get it tomorrow from my SAP team & my BWP Ker. Param, SWAP and MEM info.

Chan

[A. Clay Stephenson]

I missed your vmstat po's earlier; you are definitely under memory pressure; swap is the least of your problems. You really need to install more memory and almost certainly, your buffer cache is now too small for good performance but anything that reduces swapping will be a good thing.

You need to reduce the size of shared memory structure to the point where the pageout/sec rate is very nearly zero because any extra trips to the disk to fetch database rows, for example, because these structures are now smaller have less impact (by a factor of ~100) than does swapping.

You don't have a kernel tuning / swap size problem; you have a lack of physical memory problem (or too much load for the present configuration) (or unrealistic expectations that critical tasks need to be completed in a timely manner) -- tack your pick.