topic Re: Bad performance SAP-machine on a RPG5400 : memory problem ? in Operating System - HP-UX

Bad performance SAP-machine on a RPG5400 : memory problem ?

Hugo Wilhelmus — Thu, 19 Feb 2004 07:43:39 GMT

We currently experience a bad performance on
our RPG5400 machine with 4 Gb memory, running
SAP. I'm thinking of a lack of memory :
TOP command gives 6000K or less free memory
then SAP is locked, mosty idle time go up to 100 % and with vmstat i see that pi and po number are non zero. Sometimes the vhand process consumes some CPU-time.

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Sanjay Kumar Suri — Thu, 19 Feb 2004 07:55:32 GMT

What is the viewpoint of Basis guy supporting SAP?

Has the system been evaluated using SAP standard transaction codes?

sks

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Hugo Wilhelmus — Thu, 19 Feb 2004 08:03:33 GMT

From a SAP point of view every possible check is done, SAP configuration is according to different Notes ok.
There might be some reports/ABAP that could consume memory, our programmers are checking.

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Steven E. Protter — Thu, 19 Feb 2004 08:11:28 GMT

Sounds like a lot of memory unless the box is really being stressed.

Perhaps collect some data which will lead to other problems. Kernel settings and tuning comes to mind.

SEP

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Volker Borowski — Thu, 19 Feb 2004 15:37:18 GMT

Hello,

I think we will require some additional input on this.

Could you please download the screens from
ST02 / ST04 / ST06 while bad system response time and attach them.

Please post your SAP-Workprocess distribution in addition.

Ist your overall impression slow, or is just a single business process slow.

While the application is slow, watch transaction SM50 for colored semaphore fields or anything in state "wait".

What is your SAP-/OS-/DB- Release.

In general I'd suspect some kind of locking problem if the application is slow while CPU is idle. Even if you have memory problems one should expect swapping (SAP requires a lot of it) and this shoul pull at least some system-cpu-time.

Consider to configure your System for EarlyWatchAlert to get some Info from SAP about this.

Waiting for data
Volker

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Stuart Abramson_2 — Thu, 19 Feb 2004 17:19:28 GMT

what does glance say?

what does sar say? sar doesn't show much for memory, but it shows cpu, disk, tables.

Stuart

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Sridhar Bhaskarla — Thu, 19 Feb 2004 18:39:20 GMT

Hi Hugo,

Little more information will help.

swapinfo -t will show you if you paged out already.

What's your buffer cache parameters?. If you are using default of 50% dbc_max_pct, then it will be a over kill. What are your max_dbc_pct, nbuf and bufpages settings? You can use 'kmtune -l -q ' to get this information.

-Sri

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Hugo Wilhelmus — Fri, 20 Feb 2004 03:18:10 GMT

I'm now checking memory segments with
ipcs -mbob and see 1 deleted (D) segment
of 4Gb. Whit shminfo-script i see that
the SAP-processes disp+work are attatched
to it, but these processes are also attatched
to an other segment. In /usr/sap/PRD/DEVEBMGS00/work there are
also dev_w.old files together with
the dev_w files (these are the disp+work processes). Are the old files(processes) holding a memorysegment of 4 Gb and why is it
deleted ?

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Hugo Wilhelmus — Fri, 20 Feb 2004 03:35:00 GMT

swapinfo -t (hope you can read it)

dev 4194304 3228840 965464 77% 0 - 1 /dev/vg00/lvol2

dev 6144000 44 6143956 0% 0 - 2 /dev/vg00/lvswap2

dev 10240000 0 10240000 0% 0 - 3 /dev/vg00/lvswap3

reserve - 5062504 -5062504
memory 3373788 508472 2865316 15%
total 23952092 8799860 15152232 37% - 0 -

max_dbc_pct = 50
nbuf = 0
bufpages = 10000

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Hugo Wilhelmus — Fri, 20 Feb 2004 03:53:33 GMT

Here's my output from ST02/ST04/ST06

We have oracle 8.1.1.7 on HPUX 11.i
Machine RPG5400 with 4 Gb.

Performance is total bad. Nobody can
work, not even on OS-level, have to wait
a few sec. before my command appears.

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Volker Borowski — Fri, 20 Feb 2004 04:16:07 GMT

Hello,

these screenshots look entirely normal.
Oracle shared pool might be increased to 450MB (several notes concerning activation performance problems on this in OSS).
But this has nothing to do with your current problem.

The dev_wXX.old are just the tracefiles from the last restart. They should not have an active connection (check with fuser) beside someone has it in a viewer.

I guess you have to take down the SAP and either reboot, or get rid of the shared-mem trouble first. Check OSS for the "cleanipc" command of your current SAP release.

Volker

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Sanjay Kumar Suri — Fri, 20 Feb 2004 04:22:26 GMT

Apart from other inputs:

ST02 shows swapping on: Program, CUA,
Screen.

This can be due to:
Buffer size is too small.
Fequently changed programs. Check with ABAP guys whether programs are changed quite often.
Allocated memory is fragmented.

St04 - Normal. Good.

ST06 - Show excessive swapping. CPU performance is good.

sks

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Saurav_1 — Fri, 20 Feb 2004 04:34:30 GMT

Hi,

Tune the kernel parametres tO Hp standards.
sam-> kernel config -> tunable parametres-> Action-> tuned parametre set-> see if OLTP Database server is there apply & collect the chages in a file... confirm with DBA abt. changes. In every new HP installation it is supposed to be done.

Saurav

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Hein van den Heuvel — Fri, 20 Feb 2004 08:38:52 GMT

>>> SAP configuration is according to different Notes ok.

I doubt very much that SAP configuration would approve of a 4GB extended memory section for a 4GB physical system. Furthermore, your Sap tune summary over 4 week days shows 'only' 1.6 Max use. So you should urgently re-start Sap 2 or 3GB ininitial_extent

The 'marked delete' section might be the normal situtation. I don't have a life systems just now to check, but it woudl make sense. It just means that as soon as the last attached process is gone then the memory segment will be gone.

What i am a little uneasy with is how to translate this all to even OS commands being slow: "Performance is total bad. Nobody can
work, not even on OS-level, have to wait
a few sec. before my command appears. "
Still, imho this setup is asking for trouble, and you are in trouble, so may the paging induced by this setup explains all.

Met vriendlijke groetjes,
Hein.

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Geoff Wild — Fri, 20 Feb 2004 08:39:55 GMT

How big is the database?

I'd drop dbc_mac_pct to 10 - that will free up some ram for Oracle/SAP.

Rgds...Geoff

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Hein van den Heuvel — Fri, 20 Feb 2004 09:51:52 GMT

I got my test system up... if does show the 'D' attribute just for the extended memory section:
(ipcs -ma > x; vi x; sort -k 10 x | tail )

m 147477 0x00002719 --rw-r----- advadm sapsys advadm sapsys 7 134217728 770 769 9:11:32 9:11:32 9:11:23
m 458783 0x00002716 --rw-r----- advadm sapsys advadm sapsys 7 153600000 770 769 9:11:34 9:11:34 9:11:32
m 3588101 0x78c50468 --rw-r----- oraadv dba oraadv dba 48 400785408 656 786 9:11:23 9:11:23 9:11:03
m 65555 0x00000000 D-rw-rw-rw- advadm sapsys advadm sapsys 8 788529152 753 753 9:11:22 no-entry 9:11:22
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
root> su - advadm

advadm> cdpro
advadm> grep init *
ADV_DVEBMGS00_n20:em/initial_size_MB = 750
advadm> bc
750*1024*1024
786432000

hth,
Hein.

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Sridhar Bhaskarla — Fri, 20 Feb 2004 10:46:13 GMT

Hi Hugo,

//
dev 4194304 3228840 965464 77% 0 - 1 /dev/vg00/lvol2
//

The above indicates that more than 3 GB is sitting on your swap. This is almost 80% of your memory.

//
We have oracle 8.1.1.7 on HPUX 11.i
Machine RPG5400 with 4 Gb.

Performance is total bad. Nobody can
work, not even on OS-level, have to wait
a few sec. before my command appears. //

This may be an effect of paging/swapping. What are your typical 'po' values in vmstat?.

I don't know anything about SAP and how it works.

I believe that the system needs more memory. Unless you have memory leaks on your servers and|or your application is incorrectly configured, I would add another 4GB to it.

-Sri

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Bill Hassell — Fri, 20 Feb 2004 10:53:56 GMT

Ignore the pi entry in vmstat since it counts program startup as well as return from swap. HOWEVER, if po is more than 1 digit (greater than 9) you need a lot more memory. If po is in the range of 10-50, another 2Gb may help. If po is greater than 50, especially for long periods of time, you must double your RAM to 8Gb or more. Ignore the free memory information as it gives you no useful information. SAP and Oracle can creep along with a light load at 4Gb but as you've seen, it is unusable based on current performance. Normally, a very heavy use of disk by applications will not cause login or shell prompt delays, but paging (swapping) will indeed cause delays as this is a major memory management task. When memory is being rearranged, a spinlock is needed to stop all the processors from making mistakes and this impacts everything.

The dbc_max_pct value is probably not being hit since you are paging out. The buffer cache is reduced when programs need more RAM. Oracle SGA probably needs to be increased to 800 megs or more to avoid delays with sorting temp files and large row insertions. The database may need additional indexes created. vhand usually starts consuming CPU time when memory is too small. If you are running 32bit Oracle, see what you can do to switch to 64bit so SGA will not be limited.

Bottom line is that there is nothing you can do to fix performance delays caused by lack of memory except: add more memory. And in the case of Oracle, give it more memory to reduce disk I/O. If this system will grow (most do) then plan ahead and order 8Gb additional RAM (for a total of 12Gb) and get your DBAs to take advantage of a large SGA.

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Hugo Wilhelmus — Fri, 20 Feb 2004 10:59:08 GMT

To answer some questions :

- SAP-db is about 35 Gb.

- We have 3 swapareas :

1 of 4 Gb /dev/vg00/lvol2
1 of 6 GB /dev/vg00/lvswap2
1 of 10 Gb /dev/vg00/lvswap3

Only the first one is 80 % full,it
has priority 1 so will be used first.
THe other 2 are empty.

There's also 1 of 3,37 GB (Called memory)
which is about 15% full (is this my 4Gb
memory ??)

So together : 23 Gb, about 37 % used ...

I guess the D in the memory-segment is normal, because we stopped and started R3 and after that it was there again. We changed
the this :

reduced :
_the size of the R/3 Extended Memory (it's the one of the 4GB segment).
_some R/3 buffer too large that was not swapping.
_the number of workprocess (-1 btc, -2 dia)

enlarged:
_the program buffer that was doing too displacements

performance is better but still TOP says that there's about 20 to 10 Mb free space ...

Re: Bad performance SAP-machine on a RPG5400 : memory problem ?

Geoff Wild — Fri, 20 Feb 2004 11:58:59 GMT

One of my "smaller" SAP installs is on a 2 CPU RP7410 with 4GB of ram.

We don't have any performance issues.

The DB is 54GB allocated and 40 GB used.

The oracle SGA is only 358 MB.

Here's some of my main info:

# memdetail
Memory Stat total used avail %used
physical 4088.0 3274.8 813.2 80%
active virtual 5469.0 1303.2 4165.8 24%
active real 2288.3 538.1 1750.2 24%
device swap 20480.0 5213.6 15266.4 25%

# hpmem
HP-UX svr007 B.11.11 U 9000/800 54719995 unlimited-user license
CPU Count: 2
CPU Speed: 750 MHz
CPU HW Support: 64-bit
Kernel Support: 64-bit
RAM Size: 4088 MB
bufpages: 408 MB
maxuprc: 800
maxvgs: 256
maxfiles: 2048
max_thread_proc: 1024
nfile: 189100
nflock: 4096
nproc: 4096
ninode: 23072
shmmax: 1073741824
shmmni: 512
dbc_max_pct: 10

Rgds...Geoff