Bad performance SAP-machine on a RPG5400 : memory problem ?

[Hugo Wilhelmus]

We currently experience a bad performance on
our RPG5400 machine with 4 Gb memory, running
SAP. I'm thinking of a lack of memory :
TOP command gives 6000K or less free memory
then SAP is locked, mosty idle time go up to 100 % and with vmstat i see that pi and po number are non zero. Sometimes the vhand process consumes some CPU-time.

[Sanjay Kumar Suri]

What is the viewpoint of Basis guy supporting SAP?

Has the system been evaluated using SAP standard transaction codes?

sks

[Hugo Wilhelmus]

From a SAP point of view every possible check is done, SAP configuration is according to different Notes ok.
There might be some reports/ABAP that could consume memory, our programmers are checking.

[Steven E. Protter]

Sounds like a lot of memory unless the box is really being stressed.

Perhaps collect some data which will lead to other problems. Kernel settings and tuning comes to mind.

SEP

[Volker Borowski]

Hello,

I think we will require some additional input on this.

Could you please download the screens from
ST02 / ST04 / ST06 while bad system response time and attach them.

Please post your SAP-Workprocess distribution in addition.

Ist your overall impression slow, or is just a single business process slow.

While the application is slow, watch transaction SM50 for colored semaphore fields or anything in state "wait".

What is your SAP-/OS-/DB- Release.

In general I'd suspect some kind of locking problem if the application is slow while CPU is idle. Even if you have memory problems one should expect swapping (SAP requires a lot of it) and this shoul pull at least some system-cpu-time.

Consider to configure your System for EarlyWatchAlert to get some Info from SAP about this.

Waiting for data
Volker

[Stuart Abramson_2]

what does glance say?

what does sar say? sar doesn't show much for memory, but it shows cpu, disk, tables.

Stuart

[Sridhar Bhaskarla]

Hi Hugo,

Little more information will help.

swapinfo -t will show you if you paged out already.

What's your buffer cache parameters?. If you are using default of 50% dbc_max_pct, then it will be a over kill. What are your max_dbc_pct, nbuf and bufpages settings? You can use 'kmtune -l -q ' to get this information.

-Sri

[Hugo Wilhelmus]

I'm now checking memory segments with
ipcs -mbob and see 1 deleted (D) segment
of 4Gb. Whit shminfo-script i see that
the SAP-processes disp+work are attatched
to it, but these processes are also attatched
to an other segment. In /usr/sap/PRD/DEVEBMGS00/work there are
also dev_w.old files together with
the dev_w files (these are the disp+work processes). Are the old files(processes) holding a memorysegment of 4 Gb and why is it
deleted ?

[Hugo Wilhelmus]

swapinfo -t (hope you can read it)

dev 4194304 3228840 965464 77% 0 - 1 /dev/vg00/lvol2

dev 6144000 44 6143956 0% 0 - 2 /dev/vg00/lvswap2

dev 10240000 0 10240000 0% 0 - 3 /dev/vg00/lvswap3

reserve - 5062504 -5062504
memory 3373788 508472 2865316 15%
total 23952092 8799860 15152232 37% - 0 -

max_dbc_pct = 50
nbuf = 0
bufpages = 10000

[Hugo Wilhelmus]

Here's my output from ST02/ST04/ST06

We have oracle 8.1.1.7 on HPUX 11.i
Machine RPG5400 with 4 Gb.

Performance is total bad. Nobody can
work, not even on OS-level, have to wait
a few sec. before my command appears.

[Volker Borowski]

Hello,

these screenshots look entirely normal.
Oracle shared pool might be increased to 450MB (several notes concerning activation performance problems on this in OSS).
But this has nothing to do with your current problem.

The dev_wXX.old are just the tracefiles from the last restart. They should not have an active connection (check with fuser) beside someone has it in a viewer.

I guess you have to take down the SAP and either reboot, or get rid of the shared-mem trouble first. Check OSS for the "cleanipc" command of your current SAP release.

Volker

[Sanjay Kumar Suri]

Apart from other inputs:

ST02 shows swapping on: Program, CUA,
Screen.

This can be due to:
Buffer size is too small.
Fequently changed programs. Check with ABAP guys whether programs are changed quite often.
Allocated memory is fragmented.

St04 - Normal. Good.

ST06 - Show excessive swapping. CPU performance is good.

sks

[Saurav_1]

Hi,

Tune the kernel parametres tO Hp standards.
sam-> kernel config -> tunable parametres-> Action-> tuned parametre set-> see if OLTP Database server is there apply & collect the chages in a file... confirm with DBA abt. changes. In every new HP installation it is supposed to be done.

Saurav

[Hein van den Heuvel]

>>> SAP configuration is according to different Notes ok.

I doubt very much that SAP configuration would approve of a 4GB extended memory section for a 4GB physical system. Furthermore, your Sap tune summary over 4 week days shows 'only' 1.6 Max use. So you should urgently re-start Sap 2 or 3GB ininitial_extent

The 'marked delete' section might be the normal situtation. I don't have a life systems just now to check, but it woudl make sense. It just means that as soon as the last attached process is gone then the memory segment will be gone.

What i am a little uneasy with is how to translate this all to even OS commands being slow: "Performance is total bad. Nobody can
work, not even on OS-level, have to wait
a few sec. before my command appears. "
Still, imho this setup is asking for trouble, and you are in trouble, so may the paging induced by this setup explains all.

Met vriendlijke groetjes,
Hein.

[Geoff Wild]

How big is the database?

I'd drop dbc_mac_pct to 10 - that will free up some ram for Oracle/SAP.

Rgds...Geoff

[Hein van den Heuvel]

I got my test system up... if does show the 'D' attribute just for the extended memory section:
(ipcs -ma > x; vi x; sort -k 10 x | tail )

m 147477 0x00002719 --rw-r----- advadm sapsys advadm sapsys 7 134217728 770 769 9:11:32 9:11:32 9:11:23
m 458783 0x00002716 --rw-r----- advadm sapsys advadm sapsys 7 153600000 770 769 9:11:34 9:11:34 9:11:32
m 3588101 0x78c50468 --rw-r----- oraadv dba oraadv dba 48 400785408 656 786 9:11:23 9:11:23 9:11:03
m 65555 0x00000000 D-rw-rw-rw- advadm sapsys advadm sapsys 8 788529152 753 753 9:11:22 no-entry 9:11:22
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
root> su - advadm


advadm> cdpro
advadm> grep init *
ADV_DVEBMGS00_n20:em/initial_size_MB = 750
advadm> bc
750*1024*1024
786432000

hth,
Hein.

[Sridhar Bhaskarla]

Hi Hugo,

//
dev 4194304 3228840 965464 77% 0 - 1 /dev/vg00/lvol2
//

The above indicates that more than 3 GB is sitting on your swap. This is almost 80% of your memory.

//
We have oracle 8.1.1.7 on HPUX 11.i
Machine RPG5400 with 4 Gb.

Performance is total bad. Nobody can
work, not even on OS-level, have to wait
a few sec. before my command appears. //

This may be an effect of paging/swapping. What are your typical 'po' values in vmstat?.

I don't know anything about SAP and how it works.

I believe that the system needs more memory. Unless you have memory leaks on your servers and|or your application is incorrectly configured, I would add another 4GB to it.

-Sri

[Bill Hassell]

Ignore the pi entry in vmstat since it counts program startup as well as return from swap. HOWEVER, if po is more than 1 digit (greater than 9) you need a lot more memory. If po is in the range of 10-50, another 2Gb may help. If po is greater than 50, especially for long periods of time, you must double your RAM to 8Gb or more. Ignore the free memory information as it gives you no useful information. SAP and Oracle can creep along with a light load at 4Gb but as you've seen, it is unusable based on current performance. Normally, a very heavy use of disk by applications will not cause login or shell prompt delays, but paging (swapping) will indeed cause delays as this is a major memory management task. When memory is being rearranged, a spinlock is needed to stop all the processors from making mistakes and this impacts everything.

The dbc_max_pct value is probably not being hit since you are paging out. The buffer cache is reduced when programs need more RAM. Oracle SGA probably needs to be increased to 800 megs or more to avoid delays with sorting temp files and large row insertions. The database may need additional indexes created. vhand usually starts consuming CPU time when memory is too small. If you are running 32bit Oracle, see what you can do to switch to 64bit so SGA will not be limited.

Bottom line is that there is nothing you can do to fix performance delays caused by lack of memory except: add more memory. And in the case of Oracle, give it more memory to reduce disk I/O. If this system will grow (most do) then plan ahead and order 8Gb additional RAM (for a total of 12Gb) and get your DBAs to take advantage of a large SGA.

[Hugo Wilhelmus]

To answer some questions :

- SAP-db is about 35 Gb.

- We have 3 swapareas :

1 of 4 Gb /dev/vg00/lvol2
1 of 6 GB /dev/vg00/lvswap2
1 of 10 Gb /dev/vg00/lvswap3

Only the first one is 80 % full,it
has priority 1 so will be used first.
THe other 2 are empty.

There's also 1 of 3,37 GB (Called memory)
which is about 15% full (is this my 4Gb
memory ??)

So together : 23 Gb, about 37 % used ...

I guess the D in the memory-segment is normal, because we stopped and started R3 and after that it was there again. We changed
the this :

reduced :
_the size of the R/3 Extended Memory (it's the one of the 4GB segment).
_some R/3 buffer too large that was not swapping.
_the number of workprocess (-1 btc, -2 dia)

enlarged:
_the program buffer that was doing too displacements

performance is better but still TOP says that there's about 20 to 10 Mb free space ...

[Geoff Wild]

One of my "smaller" SAP installs is on a 2 CPU RP7410 with 4GB of ram.

We don't have any performance issues.

The DB is 54GB allocated and 40 GB used.

The oracle SGA is only 358 MB.

Here's some of my main info:

# memdetail
Memory Stat total used avail %used
physical 4088.0 3274.8 813.2 80%
active virtual 5469.0 1303.2 4165.8 24%
active real 2288.3 538.1 1750.2 24%
device swap 20480.0 5213.6 15266.4 25%

# hpmem
HP-UX svr007 B.11.11 U 9000/800 54719995 unlimited-user license
CPU Count: 2
CPU Speed: 750 MHz
CPU HW Support: 64-bit
Kernel Support: 64-bit
RAM Size: 4088 MB
bufpages: 408 MB
maxuprc: 800
maxvgs: 256
maxfiles: 2048
max_thread_proc: 1024
nfile: 189100
nflock: 4096
nproc: 4096
ninode: 23072
shmmax: 1073741824
shmmni: 512
dbc_max_pct: 10

Rgds...Geoff

[Hein van den Heuvel]

Hugo wrote
"reduced :
_the size of the R/3 Extended Memory (it's the one of the 4GB segment).
_some R/3 buffer too large that was not swapping."

By how much? Judging only by your prior output something like 2GB or 2.2GB should be fine.


Bill (and Sri) wrote:
"Bottom line is that there is nothing you can do to fix performance delays caused by lack of memory except: add more memory"

UNLESS your allow a component in the system to allocate (and touch) a piece of memory that is not strictly needed. As per my earlier reply, Hugo posted evidence that this is the case, not just by a little bit, but by 2GB! So while adding memory may be needed, and will be a simple 'mere money' brute force solution, it woudl be (IMHO) irresponsible to do so without fixing that application problem first.

Hugo> " performance is better but still TOP says that there's about 20 to 10 Mb free space ... "

That could be the dbc eating up memory because 'it is allowed to' and 'noone else needs it'. You should monitor your memory usage by the cache. And if on your system dbc_max_pct is dynamic, then maybe just force it down in the 5 - 15% range as in a typical SAP/Oracle application any more will not help. Get out the big hammer.

Cheers,

[Bill Hassell]

Correct. Adding memory won't automatically fix everything, it will reduce or eliminate paging which is a huge drain. Beyond that, then the programs (Oracle and SAP) must be reconfigured to take advantage of the additional memory. Note also that for filesystem-based Oracle, a small buffer cache (less than 300 megs) can cause performance drops. The easy answer is that there is no magic solution in fixing performance issues.