topic Re: PERFORMANCE HELP!!!! in Operating System - HP-UX

PERFORMANCE HELP!!!!

Brian_161 — Mon, 03 Mar 2003 15:06:09 GMT

We have installed 9.2.0.2 on an rp5470 server that 8G of memory and 2 CPU's.

The customer is doing some load testing, and here is what I am seeing from iostat and vmstat.

From iostat, I have not seen ONE seek that is greater than one millisecond, so I am assuming IO is fine.

From vmstat, I see a run queue of anywhere from 15 to 19, but this is a load test, so it looks fine.

Scan rate is 0, so I know i am not seeing any memory issues, and the cpu time is generally around 75 % in user time and 25 % in system time.

These sound like pretty good numbers to me.

The Oracle database stats look fine also.

But from the clients homegrown app, they are having trouble obtaining some locks.

The thing is, based on what I have seen, this looks like an application issue, but the server looks like it performing fine.

If anyone has any thoughts or suggestions, I would really appreciate them.

If you would like to see the vmstat, iostat output, I can put that out here also.

Thanks in advance,

Brian

Re: PERFORMANCE HELP!!!!

Ross Zubritski — Mon, 03 Mar 2003 15:08:26 GMT

Brian,

How many procs per user process (maxuprc) do you have configured? This could cause locking issues with oracle.

Regards,

RZ

Re: PERFORMANCE HELP!!!!

John Payne_2 — Mon, 03 Mar 2003 15:10:13 GMT

Brian,

"They are having trouble obtaining some locks"

Do you mean file locks? Do you have glance? If so, run it and hit 't' for the system tables report, and look at the percent used for nflocks. If you are at 100%, you need to increase that kernel parameter.

Hope it helps

John

Re: PERFORMANCE HELP!!!!

John Palmer — Mon, 03 Mar 2003 15:10:50 GMT

A runqueue of 15+ sounds like you're short of CPU resources to me. Either that or you've got a lot of rogue processes.

Suggest you post the info.

Regards,
John

Re: PERFORMANCE HELP!!!!

Brian_161 — Mon, 03 Mar 2003 15:12:13 GMT

maxuprc is set to 3686.

The type of lock is a row level transaction lock in oracle.

not familiar with glance. where do i get it? or is it part of the o/s?

thanks,

brian

Re: PERFORMANCE HELP!!!!

Sridhar Bhaskarla — Mon, 03 Mar 2003 15:12:59 GMT

Hi Brian,

Just to make sure, please post the following while the loadtest is going on.

sar 2 5
sar -d 2 5
sar -b 2 5
vmstat 2 5
swapinfo -t
sar -v 2 5
sar -q 2 5

Kernel parameters (kmtune -l q paramter)
timeslice
dbc_max_pct (or bufpages/nbuf)

-Sri

Re: PERFORMANCE HELP!!!!

Ross Zubritski — Mon, 03 Mar 2003 15:13:47 GMT

Glance + is an add-on perf tool. It is essential part of your admin "tool-bag". It costs a bit, but well worth the money. Download the tril and check it out.

Regards,

RZ

Re: PERFORMANCE HELP!!!!

Brian_161 — Mon, 03 Mar 2003 15:15:50 GMT

# kmtune -q dbc_min_pct
Parameter Current Dyn Planned Module Version
===============================================================================
dbc_min_pct 5 - 5
# kmtune -q dbc_max_pct
Parameter Current Dyn Planned Module Version
===============================================================================
dbc_max_pct 50 - 50
# kmtune -q timeslice
Parameter Current Dyn Planned Module Version
===============================================================================
timeslice 10 - (100/10)

Re: PERFORMANCE HELP!!!!

Brian_161 — Mon, 03 Mar 2003 15:17:57 GMT

here is some more output:

sar -q
HP-UX HPUX1 B.11.11 U 9000/800 03/03/03

10:28:24 runq-sz %runocc swpq-sz %swpocc
10:28:26 4.8 100 0.0 0
10:28:28 5.5 100 0.0 0
10:28:30 4.5 100 0.0 0
10:28:32 5.8 100 0.0 0
10:28:34 7.2 100 0.0 0

Average 5.6 100 0.0 0

Re: PERFORMANCE HELP!!!!

Sridhar Bhaskarla — Mon, 03 Mar 2003 15:25:50 GMT

Hi Brian,

50% of dbc_max_pct is not good. Depending on the memory and the processing power on your system, you may get into intermittent problems while syncer tries to flush the buffers.

Start with 400MB and increase it in the intervals of 100MB upto 700 MB and see which value gives you better performance.

Your sar -q does indicate that you are utilizing your cpu. Post the other sar outputs to get a better understanding.

-Sri

Re: PERFORMANCE HELP!!!!

Brian_161 — Mon, 03 Mar 2003 15:58:17 GMT

Sorry,

I was only collecting iostat/vmstat data during the run, I will get the rest of the sar output to you on our next run.

Here is the vmstat output.

Re: PERFORMANCE HELP!!!!

Brian_161 — Mon, 03 Mar 2003 16:10:02 GMT

Sorry, attachment didnt work.

Here is the vmstat output, my next post will have the iostat output.

Thanks,

Brian

Re: PERFORMANCE HELP!!!!

Brian_161 — Mon, 03 Mar 2003 16:28:52 GMT

Last post had iostat, here is the vmstat.

Gettig ready to kick off another test, i will collect sar data for that one.

Also, I realize that 15 procs in the run queue is cpu bound, its a load test, i expect it. I am trying to prove to some people that it is cpu, not memory or io..

Thanks,

Brian

Re: PERFORMANCE HELP!!!!

Graham Cameron_1 — Tue, 04 Mar 2003 08:26:42 GMT

If you are suffering from Oracle lock contention caused by the way your application works, then all the cpu and all the memory and disk in the world will not help you.
You need to use one of the Oracle tools to examine this, but I don't think there are any that are free.
Here we use OEM (Oracle Enterprise Manager), which has a Tuning Pack option which can show locks and other potential problems.
http://otn.oracle.com/products/oem/content.html
There are 3rd party tools also - spotlight from Quest for example.
http://www.quest.com/quest_central/qco/performance_diagnostics/
If your application is bought in as opposed to home grown, then contact the vendor.
If you don't want to do any of these, then all the info you need is available within Oracle's dictionary tables and views, but getting it out and interpreting it is a job for an expert.

--Graham