topic Re: Server Performance in Operating System - HP-UX

Server Performance

Shehan — Fri, 06 Jun 2008 13:07:15 GMT

Hi

We have rp4440 server running hp-ux 11i v1. Server spec as below.

2 CPU (dual core)
32 GB memory
Oracle 10g is running heavily.
Two storages are connected through fiber(EVA8K and EVA3K)
OS is patched up to last December 2007

We are getting some performance sometime. Anyway how can we find swap-out/In details in order to check swap usage. In additiona to that how we can find the read/write hits to the disk groups.

Appreciate your kindly response.

Regards
Niru

Re: Server Performance

Duncan Edmonstone — Fri, 06 Jun 2008 13:26:52 GMT

Niru,

Don't woory too much about swap utilisation first. First thing would be to look at page-ins/pahe-outs:

vmstat 2 10

pay attention to the po column - if you see page-outs then you are under memory pressure.

for disk use sar:

sar -d 2 10

look at the avque and avserv column - if avque is over 2 and avserv over 20 you may have an IO problem.

Post some output here.

HTH

Duncan

Re: Server Performance

Avinash20 — Sat, 07 Jun 2008 02:57:15 GMT

Hi. I would suggest you to install Glance software

https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-15-28^9637_4000_100__

Re: Server Performance

Yogeeraj_1 — Sat, 07 Jun 2008 09:24:30 GMT

hi Niru,

Since you are already on Oracle 10g, you can start your Oracle Enterprise Manager Database control and be able to monitor your server resources more effectively.

In case, you are fearing any abnormally high utilisation of any of the resources by the Oracle database, the Performance section of OEM will be able to give you a global picture on Instance I/O, Instance Throughput, Average Active Session and Host runnable processes over an interval of time.

kind regards,
yogeeraj

Re: Server Performance

Shehan — Mon, 09 Jun 2008 03:39:05 GMT

Hi Duncan

I have attached some "sar" output. Please go though it and let me know any bottleneck is there.

regards
Niru

Re: Server Performance

Shehan — Mon, 09 Jun 2008 03:44:37 GMT

Hi Yogeeraj

We are not handling Oracle. Another Vender is responsible for maintaining Oracle. Anyway I will discuss with the Oracle vender about this.

Regards
Niru

Re: Server Performance

Duncan Edmonstone — Mon, 09 Jun 2008 05:54:14 GMT

Niru,

Were these stats gathered when you were actually having a performance issue?

Cos I see nothing concerning in any of them. CPU util is quite high, but never actually seems to be a significant bottleneck.

Disk IO seems OK, and memory usage as well... what is the (percieved) problem with performance?

HTH

Duncan

Re: Server Performance

Shehan — Mon, 09 Jun 2008 07:26:30 GMT

Hi

When I am taking this there was some level of performance issue. In addition to that , I attached another output when they are running query in the server and also felt some performance degardtion. Please go though it and let me know status.

Regards
Niru

Re: Server Performance

Duncan Edmonstone — Mon, 09 Jun 2008 10:37:21 GMT

Niru,

Nope, there's nothing in there that raises too many alarm bells - again CPU util is quite high and does top out at 100% on 1 CPU for a short period of time - however looking at the mix of user vs. system CPU time, it looks like all the time is spent in user space (presumably in Oracle somewhere), so I'd be looking at the application and Oracle team to 'prove' there is a system level problem before doing any more work.

HTH

Duncan

Re: Server Performance

Rita C Workman — Mon, 09 Jun 2008 11:23:10 GMT

Couple thoughts.......

1. Your parm ninode is too high. Reduce to around 4096 [See your sar -v output and you will see what I mean]
2. Since ninode is high, I'll guess that vx_ninode is set to "0". If it is, then the system will auto create a huge table based on your physical memory, which becomes a waste of memory. Try setting it around 20m to 40m, and monitor for any tweeking you might need to do.
3. Swapdisk - I don't see you have any set up from your output. Set up a couple disks for swap and enable them. I'd set up my swap lvols to 4Gb each based on the output you supplied.

NEXT:
> What is your dbc_max & dbc_min% parm ?
> What are your semm* parms set at?
Run command sar -mS 1 20 and compare the output with your semm* parms setting. Are they sufficient to cover your needs? If yes, good - if not, adjust.

That's just some thoughts I had.

Rgrds,
Rita

Re: Server Performance

Duncan Edmonstone — Mon, 09 Jun 2008 12:19:19 GMT

Rita,

Looks like he has 4GB of swap setup;

# swapinfo -tam

Mb Mb Mb PCT START/ Mb

TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME

dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2

reserve - 4096 -4096

memory 25506 18047 7459 71%

total 29602 22143 7459 75% - 0 -

with 32GB of memory, thats not v much, but I guess he must have pseudo swap enabled or he'd be seeing more than perf issues... still its not really enough, so Niru I'd say you really need at least 25% (8GB), ideally 50% (16GB) of real memory as swap.

HTH

Duncan

Re: Server Performance

Rita C Workman — Mon, 09 Jun 2008 12:31:41 GMT

Hello Duncan,

I agree. I think the amount of physical memory is what has saved him so far. Not sure about the Oracle instances (? how many, SGA ?).
What caught my eye were that his parms look to be the result of parm algorithims, so I'm wondering what other default parms are out there...

Niru,

Let us know how things go.

Also....I couldn't download your last attachment.

Rgrds again,
Rita

Re: Server Performance

Yogeeraj_1 — Mon, 09 Jun 2008 15:15:38 GMT

hi Niru,

Based on the top output,

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
3 ? 7243 oracle 234 20 10658M 9684K run 1904:55 100.03 99.85 oracledwh
2 ? 7223 oracle 234 20 10786M 151M run 148:53 100.00 99.83 oracledwh
0 ? 13711 oracle 234 20 10688M 53140K run 1098:00 99.62 99.44 oracledwh

I would first look into processes 7243, 7223 and 13711.

The following SQL statement will give you more information on the Oracle sessions associated with these processes.

select b.sid SID,b.serial# "Serial#", c.spid "srvPID", b.osuser, b.username, b.
status, b.client_info, machine from v$session b, v$process c where b.paddr = c.a
ddr and c.sPID = &OSPID
/

hope this helps!
kind regards
yogeeraj

Re: Server Performance

Yogeeraj_1 — Mon, 09 Jun 2008 15:15:57 GMT

hi Niru,

Based on the top output,

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
3 ? 7243 oracle 234 20 10658M 9684K run 1904:55 100.03 99.85 oracledwh
2 ? 7223 oracle 234 20 10786M 151M run 148:53 100.00 99.83 oracledwh
0 ? 13711 oracle 234 20 10688M 53140K run 1098:00 99.62 99.44 oracledwh

I would first look into processes 7243, 7223 and 13711.

The following SQL statement will give you more information on the Oracle sessions associated with these processes.

select b.sid SID,b.serial# "Serial#", c.spid "srvPID", b.osuser, b.username, b.status, b.client_info, machine from v$session b, v$process c where b.paddr = c.addr and c.sPID = &OSPID
/

hope this helps!
kind regards
yogeeraj

Re: Server Performance

Dennis Handly — Tue, 10 Jun 2008 01:45:24 GMT

>Duncan: # swapinfo -tam
>dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
>memory 25506 18047 7459 71%
>but I guess he must have pseudo swap enabled

We know he has pseudoswap because of the "memory" line in swapinfo. Since device swap isn't being used, he may not need more unless the load is heavier.

Re: Server Performance

Shehan — Tue, 10 Jun 2008 03:49:01 GMT

Hi Handly

I also thought it.If there is no swap utilization, we don't need increase the no of partition for swap. If there is a some what utilization , we would have to increase it & see. wouldn't it?

regards
Nirukshitha

Re: Server Performance

Dennis Handly — Tue, 10 Jun 2008 06:11:26 GMT

>I also thought it. If there is no swap utilization, we don't need increase the no of partition for swap.

You only need to do that if you are planning to increase your load from your current 75% to higher.

>If there is a some what utilization, we would have to increase it & see. wouldn't it?

It also depends on how accurate and when you measured the swapinfo.

Re: Server Performance

Shehan — Tue, 10 Jun 2008 06:31:34 GMT

Hi Handly/Duncan

Anybody can let me know what are the reference levels of the sar -d 5 10 outputs?

Basically I want to know reference level of following.

%busy avque r+w/s blks/s avwait avserv

I believe there is no performance bottleneck when I am taking this output.If I have the reference level, Then I can come to a conclusion regrading the I/O utilization in Volume group comparing with the output which I am going to take when there is a performance Issue.

It is better if you can send all reference levels in all "sar" output.

Regards
Nirukshitha

Re: Server Performance

Shehan — Wed, 11 Jun 2008 07:37:49 GMT

Hi All

Any update on this?

Regards
Niru

Re: Server Performance

Ilkka Seittenranta — Wed, 18 Jun 2008 09:14:23 GMT

Hi Niru,

You machine uses about 70% memory to swap (see swapinfo), so I quest the SGA size is between 5-10Gb.

You must disable pseudoswap if you like find what kind performance problems you have. After pseudoswap uses all free memory you can't get any error messages in the syslog-file, there is not enough resources to open file handles!

After pseudoswap uses all free memory the system change to "idle-process"-state, the state is something "waiting until there is more recourses free". The nonsystem-process are that time to "halt"-state and that time load is something like 0.02

The HP-UX not ever use device based swap if you have pseudoswap enabled, there is problem how OS handles swap device priority.

Best Regard
Ilkka

"It take years to quest what is behind there"