Strange SAP Performance Issue

[Geoff Wild]

Basis has just finished doing a kernel upgrade to our DEV landscape. They have their own test box - in which we just refreshed the data from DEV to it. They did the kernel upgrade on their box first. Then, after testing, they did it to our DEV server.
Both servers are HP-UX 11i with same patches.

Here's the strange thing - a job that takes 2 hours to run on their test server - takes 4 hours and 15 minutes on our DEV server.

What is different -well, their test server is a lowly J5600 with 2 552 Mhz cpus and 4 GB of ram and single ended scsi disks.

The DEV server - N4000-44 - with 4 x 440 Mhz cpus, 10 GB of ram and Fibre attached disks on DMX1000.

No swapping on either box.

The DEV server is a "shared" box - but SAP does most of the work - other APPS barely tax the system - so much so - we tend to wonder if they are used at all...

The oracle db is just over 100GB...

I've checked glance and perview data - see nothing to indicate a performance problem...

Any ideas?

Rgds...Geoff

[TwoProc]

Geoff, would this new kernel have changed the kernel parameter shmmax ? If the shared memory is cut up into more than 6 segments (from what I've been told), you'll get delays as the database will handle only 6 segments actively at a time, the rest of the requests queue up. Just a thought as to what kernel parameter could drastically change the performance of an Oracle database - but, I'm sure there are others. You should see a large amount of buffer_waits if that is the case.

While I'm thinking of it - have you run statspack yet to see if the large delays can be identified?

Just a thought...

[Geoff Wild]

Sorry - unclear - the kernel upgrade is a SAP kernel - not OS kernel.

Rgds...Geoff

[TwoProc]

:-)

Oh... Well Then... Nevermind...
- Rosanne Rosanna Danna

[Geoff Wild]

No statspack - don't have it...

See attached txt of O/S kernel parms...

Rgds...Geoff

[TwoProc]

Statspack - You've more than likely already have it - it's just not installed.
Look in $ORACLE_HOME/rdbms/admin/spdoc.txt to get the instructions. All of the scripts to install it/run it, etc are in the same directory. Suggest creating a separate user with a separate tablespace for this purpose as it can get sizeable after a while (there are purge tools though, so just purge more often if this is an issue).

[Geoff Wild]

ping...

[Kenneth Platz]

Geoff,

Do you see any "weird" behavior while viewing the systems in glance? Are the systems using higher-than-expected amounts of time in system mode or context switching?

[Geoff Wild]

Nothing weird in glance...

The N-class server is using a bit more system time - but we are talking an average of 3.1 % with a high of 9.8 % and a context switch rate average of 0.2 and a high of 0.8 whereas the J5600 has an average system time of 0.8 % and a high of 3.4% and a context switch rate average of 0 with a high of only 0.2...

Rgds...Geoff

[Tony Scully_2]

Geoff,

What is this job - is it a single process that attachs to the database, gets data, crunches it up and writes it back?

If its all in one process and it doesn't have lots of threads of execution then you may not be getting much benefit from the additional CPUs, and the J5600 CPUs are a bit faster.

Would not account for such a massive difference though.

Tony

[Tony Scully_2]

Geoff,

Just wondering if this is resolved? I'd be interested in what happened if it is.

Tony

[TwoProc]

Sorry Geoff, I don't understand "ping..." - but I gather that you didn't want to hear more about statspack - please excuse...

[Tim D Fulford]

super late...

so If I get this right your test server has 2 550MHz CPUs and your DEV server has 4x 440 MHz CPUs...

I'm guessing here, & this does not account for the whole difference... you have configured the dev as a 2 CPU system?? this would be about 20% performance dfference.

The next thing, the disks, with a job that takes that long sometimes a dub JBOD disks works faster than cleaver fibre/cache because it avoids doing read cache misses. (I'm guessing the job is read intensive). Also if the DMX has less spindles or slower spindles this too can cause slow down

regards

tim

[Sanjay Kumar Suri]

SAP-Basis itself offers some very useful transaction codes such as:

ST04
ST02
ST06
ST03

which provide vital clue on performance.

Has the Basis team taken care of requred kernel, SAP, Database parameters as per the SAP notes.

sks

[Geoff Wild]

I think Tony was close - the job is single threaded - Basis is happy with that answer.

DEV is a 4 CPU system...with all cpu's running.

Basis has taken care of requred kernel, SAP, Database parameters as per the SAP notes.

Thanks for the help.

Rgds...Geoff

[Geoff Wild]

Closing...