Maxing out of swap, twice

[Michael Steele_2]

Clay: thanks, they've been told to buy more RAM. (budget thing).

Chan: yes please, I'll also research this on the internet.

[Bill Hassell]

The dynamic buffer cache parameters are a very painful set of parameters, always set wrong out of the box and set wrong whenever you add more RAM. The buffer cache improves filesystem performance in an asymptotic manner where performance increases rapidly as more memory is used but then levels out and eventually begins to drop. The rise means that needed file records come from memory but drops off as massively large numbers of records need to be managed.

I would recommend setting bufpages=250000 and now you can forget the dbc_ parameters. That will give you 1000 megs of buffer cache no matter how much memory you add.

The vmstst po (page out) metric is way too high and indicates that you need a LOT more RAM or the SAP and Oracle DBAs need to change their shared memory usage. DBAs will read that a larger SGA means better performance (and it can) but they fail to account for the severe performance impact that paging will cause with small RAM systems. Reduce the shared memory usage and keep the po values below 15-25 or less and performance should be a lot better. Page outs in high 2 digits or more will erase any gains made by increasing shared memory.

AS mentioned, UNIX95 is a trigger for XPG4 behavior, part of the Unix branding standards. There are several commands that have XPG4 extensions (or differences) and these go into effect when the UNIX95 variable is defined (null or any value, doesn't matter). Out of all the man pages with XPG4 notes, only ps is important.

As far as finding processes that are consuming memory, the ps command will show you local data areas but shared memory like SGA cannot be displayed by ps because, well, it's shared -- so no one process can count this area. So you need to use ipcs (ipcs -bmop is a good start), but it's much easier to ask the DBAs to report on their memory usage. The Oracle statspak is very important in looking at Oracle.

[James R. Ferguson]

Hi (again):

Be advised, that 'bufpages', is deprecated as of 11.23. The recommendation is to use an appropriate percentage of physical memory, bounded by 'dbc_min_pct' and 'dbc_max_pct'. To obtain a static value for the buffer cache, simply set dbc_min_pct = dbc_max_pct.

http://www.docs.hp.com/en/B2355-60105/bufpages.5.html

Regards!

...JRF...

[Chan 007]

Hi Mic,

Please find the details below, Check KP results, you may need to change a few.

Chan

SWAP : 18GB
RAM : 6GB
# of CPU : 4
CPU Speed : 550

OS : 11.11

bwpsap:/# model
9000/800/L3000-5x

bwpsap:/# kmtune
Parameter Current Dyn Planned Module Version
===============================================================================
NSTRBLKSCHED - - 2
NSTREVENT 50 - 50
NSTRPUSH 16 - 16
NSTRSCHED 0 - 0
STRCTLSZ 0 - 0
STRMSGSZ 65535 - 65535
acctresume 4 - 4
acctsuspend 2 - 2
aio_listio_max 256 - 256
aio_max_ops 2048 - 2048
aio_physmem_pct 10 - 10
aio_prio_delta_max 20 - 20
allocate_fs_swapmap 0 - 0
alwaysdump 1 - 1
bootspinlocks - - 256
bufcache_hash_locks 128 - 128
bufpages 0 - (NBUF*2)
chanq_hash_locks 256 - 256
core_addshmem_read 0 Y 0
core_addshmem_write 0 Y 0
create_fastlinks 0 - 0
dbc_max_pct 8 - 8
dbc_min_pct 5 - 5
default_disk_ir 0 - 0
desfree - - 0
disksort_seconds 0 - 0
dnlc_hash_locks 512 - 512
dontdump 0 - 0
dskless_node - - 0
dst 1 - 1
effective_maxpid - - ((NPROC<=30000)?30000:(NPROC*5/4))
eisa_io_estimate - - 0x300
enable_idds 0 - 0
eqmemsize 15 - 15
executable_stack 1 - 1
fcp_large_config 0 - 0
file_pad - - 10
fs_async 0 - 0
ftable_hash_locks 64 - 64
hdlpreg_hash_locks 128 - 128
hfs_max_ra_blocks 8 - 8
hfs_max_revra_blocks 8 - 8
hfs_ra_per_disk 64 - 64
hfs_revra_per_disk 64 - 64
hp_hfs_mtra_enabled 1 - 1
hpux_aes_override - - 0
initmodmax 50 - 50
io_ports_hash_locks 64 - 64
iomemsize - - 40000
ksi_alloc_max 84000 - (NPROC*8)
ksi_send_max 32 - 32
lotsfree - - 0
max_async_ports 50 - 50
max_fcp_reqs 512 - 512
max_mem_window 0 - 0
max_thread_proc 64 - 64
maxdsiz 990056448 - 990056448
maxdsiz_64bit 4294967296 - 4294967296
maxfiles 2048 - 2048
maxfiles_lim 2048 Y 2048
maxqueuetime - - 0
maxssiz 0x800000 - 0X800000
maxssiz_64bit 0x800000 - 0X800000
maxswapchunks 16384 - 16384
maxtsiz 0x4000000 Y 0X4000000
maxtsiz_64bit 0x40000000 Y 0X40000000
maxuprc 1000 Y 1000
maxusers 1050 - 1050
maxvgs 128 - 128
mesg 1 - 1
minfree - - 0
modstrmax 500 - 500
msgmap 32142 - (2+MSGTQL)
msgmax 8192 Y 8192
msgmnb 65535 Y 65535
msgmni 50 - 50
msgseg 32767 - 32767
msgssz 128 - 128
msgtql 32140 - 32140
nbuf 0 - 0
ncallout 18407 - (16+NKTHREAD)
ncdnode 150 - 150
nclist 16900 - (100+16*MAXUSERS)
ncsize 21504 - (NINODE+VX_NCSIZE)+(8*DNLC_HASH_LOCKS)
ndilbuffers 30 - 30
netisr_priority - - -1
netmemmax - - 0
nfile 159548 - (15*NPROC+2048)
nflocks 10500 - (NPROC)
nhtbl_scale 0 - 0
ninode 16384 - 16384
nkthread 18391 - (((NPROC*7)/4)+16)
nni - - 2
no_lvm_disks 0 - 0
nproc 10500 - (10*MAXUSERS)
npty 60 - 60
nstrpty 60 - 60
nstrtel 60 - 60
nswapdev 12 - 12
nswapfs 10 - 10
nsysmap 21000 - ((NPROC)>800?2*(NPROC):800)
nsysmap64 21000 - ((NPROC)>800?2*(NPROC):800)
o_sync_is_o_dsync 0 - 0
page_text_to_local - - 0
pfdat_hash_locks 128 - 128
public_shlibs 1 - 1
region_hash_locks 128 - 128
remote_nfs_swap 0 - 0
rtsched_numpri 32 - 32
scroll_lines 100 - 100
scsi_max_qdepth 8 Y 8
scsi_maxphys 1048576 - 1048576
sema 1 - 1
semaem 16384 - 16384
semmap 21002 - (SEMMNI+2)
semmni 21000 - (NPROC*2)
semmns 32767 - 32767
semmnu 10496 - ((SEMMNI/2)-4)
semmsl 2048 Y 2048
semume 100 - 100
semvmx 32767 - 32767
sendfile_max 0 - 0
shmem 1 - 1
shmmax 17179869184 Y 17179869184
shmmni 512 - 512
shmseg 200 Y 200
st_ats_enabled 0 - 0
st_fail_overruns 0 - 0
st_large_recs 0 - 0
streampipes 0 - 0
swapmem_on 1 - 1
swchunk 2048 - 2048
sysv_hash_locks 128 - 128
tcphashsz 0 - 0
timeslice 10 - (100/10)
timezone 420 - 420
unlockable_mem 0 - 0
vas_hash_locks 128 - 128
vnode_cd_hash_locks 128 - 128
vnode_hash_locks 128 - 128
vps_ceiling 16 - 16
vps_chatr_ceiling 1048576 - 1048576
vps_pagesize 4 - 4
vx_fancyra_enable 0 - 0
vx_maxlink 32767 - 32767
vx_ncsize 1024 - 1024
vxfs_max_ra_kbytes 1024 - 1024
vxfs_ra_per_disk 1024 - 1024

[Geoff Wild · Accepted solution]

How big is your DB?

I was talking to one of our DBA's - he thinks your SGA is too large - depending on the size of your db - if you decrease that you will relieve the memory pressure.

Our BW system is 500GB and the SGA is only 1GB.

Sort_area_size = 10,485,760 bytes = memory outside of SGA needed by each connecting session. Max # of sessions connected was 60 => we need at least 600 MB available.

Find out the max sessions used and go from there.

Remember - for a BW system both Oracle and SAP run on same server - so they need to share the available ram.

Adding more ram will also aleviate the pressure - just don't allow dba's to increase SGA - unless you really add a lot more ram...

If possible, borrow some ram from another server - and see how the system behaves.

IE try going to 10GB of ram....then maybe 12 and so on.

As far as the buffer cache goes - I would set dbc_max_pct and dbc_min_pct to be the same - that will stop the system from trying to free of ram when the pressure is on - can be very costly in performance...

Oracle uses it's own buffer cache.

Rgds...Geoff

PS - while typing this, my mp3 player happened to be playing "Under Pressure" by Queen :)

[Michael Steele_2]

Thanks. I'll start a thread about SaP maxlimit.