Perfornance tuning : memory bottleneck

[Nathalie Pecot]

Hi,
I have a typical memory bottleneck issue on a SAP BW production server :
- high page out rate
- disk bottleneck on the 2 swap lvols (100% used on internal disks)
- large CPU time given to the vhand process
- swapinfo -t returns a total used at 66%
- user processes are slow
- the load is not high (0.17) but system seems to do nothing
Physical memory is 8G, 2 swap lvol (pri=1) are 4 and 16G.
I think to kernel parameters tuning :
dbc_max_pct is 8, dbc_min_pct is 5
Any idea to help?
Thanks,
Nat

[Joseph Loo]

hi,

u may like to refer to the guide given in this doc, KBRC00000947:

http://www1.itrc.hp.com/service/cki/search.do?searchString=KBRC00000947&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchCrit=exactphrase&search.x=12&category=c0&mode=id&search.y=10

regards.

[Nathalie Pecot]

Already read this usefull technical doc.
Used to identify the problem.
Need more details on the kind of effects of the parameters modifications and may be proposed values or at least advices for the change (increase, decrease, rule?) : it is a production server, not possible to iterate a lot! nbuf=0, %rcache is around 99%, but %wcache is more around 40%.

[Joseph Loo]

hi nathalie,

not exactly familiar with SAP but i believe it should have some recommended kernel parameters settings. have u follow it accordingly?

if it is a memory problem, what is the output like for vmstat:

# vmstat 3 20

# sar -b 3 20

# sar -d 3 20

regards.

[CAS_2]

Please, Nathalie, would you mind post the output of following commands ?

swapinfo -tam
ipcs -mob
/usr/contrib/Q4/bin/kmeminfo

[Nathalie Pecot]

Application people just asked for a sanity reboot :(
Will wait tomorrow morning for the system to be strongly working again for further investigations.
Thanks for now.

[Unix Team]

Nathalie,

In an SAP environment, the Dialog Work Processes are launched when SAP is started. In SAP, these processes service many jobs. The larger the job - the more memory these processes acquire. At the UNIX level, these process are represent by dw.sap... processs. Since they are not terminated at the UNIX level until SAP is shutdown, they will eventually consume all available system memory and force the system to page out to disk. In a BW system where you're often running large reports - if you cannot upgrade system memory - regular application shutdowns/restarts could go some way to keeping your application from exhausting system memory. How long was your system running before you rebooted?

This problem is frequently exaggerated in 32-Bit systems..... I'm assuming you guys are running a 64Bit environment (i.e. HP-UX, Oracle & SAP), otherwise you would not really be able to make use of the 8GB of system RAM.

It is a good idea to tune dbc_max_pct and dbc_min_pct to prevent wasted valuable memory being used as buffer cache. Using tools like glance should give you some idea of how much memory you're consuming in this area. I think those figures you've quoted are reasonable for a BW system.

You can also use transaction sm51 within SAP to recycle these dialog work processes on an individual basis - but this is a messy way of managing this problem.

Sorry to state the obvious, but bottom line however is that if you are frequently running out of system memory and you have no provision for regular application restarts, then consider adding more RAM. Ensure you have the dbc_max/min_pct tuned first as this will only increase the amount of memory wasted in execess buffer cache otherwise!!

[Geoff Wild]

I have a SAP BW server as well - it also was experiencing a memory bottleneck - until we upped the ram to 10GB....

I attached my kernel parms...

Your swap is a bit low - SAP recommends atleast 20GB - I have 28GB.

How many CPU's?

I have a RP7410 with 6 cpu's.

Rgds...Geoff