Server Performance

[Shehan]

Hi

We have rp4440 server running hp-ux 11i v1. Server spec as below.

2 CPU (dual core)
32 GB memory
Oracle 10g is running heavily.
Two storages are connected through fiber(EVA8K and EVA3K)
OS is patched up to last December 2007

We are getting some performance sometime. Anyway how can we find swap-out/In details in order to check swap usage. In additiona to that how we can find the read/write hits to the disk groups.

Appreciate your kindly response.

Regards
Niru

[Duncan Edmonstone]

Niru,

Don't woory too much about swap utilisation first. First thing would be to look at page-ins/pahe-outs:

vmstat 2 10

pay attention to the po column - if you see page-outs then you are under memory pressure.

for disk use sar:

sar -d 2 10

look at the avque and avserv column - if avque is over 2 and avserv over 20 you may have an IO problem.

Post some output here.

HTH

Duncan

[Avinash20]

Hi. I would suggest you to install Glance software

https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-11-15-28^9637_4000_100__

[Yogeeraj_1]

hi Niru,

Since you are already on Oracle 10g, you can start your Oracle Enterprise Manager Database control and be able to monitor your server resources more effectively.

In case, you are fearing any abnormally high utilisation of any of the resources by the Oracle database, the Performance section of OEM will be able to give you a global picture on Instance I/O, Instance Throughput, Average Active Session and Host runnable processes over an interval of time.


kind regards,
yogeeraj

[Shehan]

Hi Duncan

I have attached some "sar" output. Please go though it and let me know any bottleneck is there.

regards
Niru

[Shehan]

Hi Yogeeraj

We are not handling Oracle. Another Vender is responsible for maintaining Oracle. Anyway I will discuss with the Oracle vender about this.

Regards
Niru

[Duncan Edmonstone]

Niru,

Were these stats gathered when you were actually having a performance issue?

Cos I see nothing concerning in any of them. CPU util is quite high, but never actually seems to be a significant bottleneck.

Disk IO seems OK, and memory usage as well... what is the (percieved) problem with performance?

HTH

Duncan

[Shehan]

Hi

When I am taking this there was some level of performance issue. In addition to that , I attached another output when they are running query in the server and also felt some performance degardtion. Please go though it and let me know status.



Regards
Niru

[Duncan Edmonstone]

Niru,

Nope, there's nothing in there that raises too many alarm bells - again CPU util is quite high and does top out at 100% on 1 CPU for a short period of time - however looking at the mix of user vs. system CPU time, it looks like all the time is spent in user space (presumably in Oracle somewhere), so I'd be looking at the application and Oracle team to 'prove' there is a system level problem before doing any more work.

HTH

Duncan

[Rita C Workman]

Couple thoughts.......

1. Your parm ninode is too high. Reduce to around 4096 [See your sar -v output and you will see what I mean]
2. Since ninode is high, I'll guess that vx_ninode is set to "0". If it is, then the system will auto create a huge table based on your physical memory, which becomes a waste of memory. Try setting it around 20m to 40m, and monitor for any tweeking you might need to do.
3. Swapdisk - I don't see you have any set up from your output. Set up a couple disks for swap and enable them. I'd set up my swap lvols to 4Gb each based on the output you supplied.

NEXT:
> What is your dbc_max & dbc_min% parm ?
> What are your semm* parms set at?
Run command sar -mS 1 20 and compare the output with your semm* parms setting. Are they sufficient to cover your needs? If yes, good - if not, adjust.

That's just some thoughts I had.

Rgrds,
Rita

[Duncan Edmonstone]

Rita,

Looks like he has 4GB of swap setup;

# swapinfo -tam

Mb Mb Mb PCT START/ Mb

TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME

dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2

reserve - 4096 -4096

memory 25506 18047 7459 71%

total 29602 22143 7459 75% - 0 -



with 32GB of memory, thats not v much, but I guess he must have pseudo swap enabled or he'd be seeing more than perf issues... still its not really enough, so Niru I'd say you really need at least 25% (8GB), ideally 50% (16GB) of real memory as swap.

HTH

Duncan

[Rita C Workman]

Hello Duncan,

I agree. I think the amount of physical memory is what has saved him so far. Not sure about the Oracle instances (? how many, SGA ?).
What caught my eye were that his parms look to be the result of parm algorithims, so I'm wondering what other default parms are out there...

Niru,

Let us know how things go.

Also....I couldn't download your last attachment.

Rgrds again,
Rita

[Yogeeraj_1]

hi Niru,

Based on the top output,

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
3 ? 7243 oracle 234 20 10658M 9684K run 1904:55 100.03 99.85 oracledwh
2 ? 7223 oracle 234 20 10786M 151M run 148:53 100.00 99.83 oracledwh
0 ? 13711 oracle 234 20 10688M 53140K run 1098:00 99.62 99.44 oracledwh


I would first look into processes 7243, 7223 and 13711.


The following SQL statement will give you more information on the Oracle sessions associated with these processes.

select b.sid SID,b.serial# "Serial#", c.spid "srvPID", b.osuser, b.username, b.
status, b.client_info, machine from v$session b, v$process c where b.paddr = c.a
ddr and c.sPID = &OSPID
/


hope this helps!
kind regards
yogeeraj

[Yogeeraj_1]

hi Niru,

Based on the top output,

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
3 ? 7243 oracle 234 20 10658M 9684K run 1904:55 100.03 99.85 oracledwh
2 ? 7223 oracle 234 20 10786M 151M run 148:53 100.00 99.83 oracledwh
0 ? 13711 oracle 234 20 10688M 53140K run 1098:00 99.62 99.44 oracledwh


I would first look into processes 7243, 7223 and 13711.


The following SQL statement will give you more information on the Oracle sessions associated with these processes.

select b.sid SID,b.serial# "Serial#", c.spid "srvPID", b.osuser, b.username, b.status, b.client_info, machine from v$session b, v$process c where b.paddr = c.addr and c.sPID = &OSPID
/


hope this helps!
kind regards
yogeeraj

[Dennis Handly]

>Duncan: # swapinfo -tam
>dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
>memory 25506 18047 7459 71%
>but I guess he must have pseudo swap enabled

We know he has pseudoswap because of the "memory" line in swapinfo. Since device swap isn't being used, he may not need more unless the load is heavier.

[Shehan]

Hi Handly

I also thought it.If there is no swap utilization, we don't need increase the no of partition for swap. If there is a some what utilization , we would have to increase it & see. wouldn't it?



regards
Nirukshitha

[Dennis Handly]

>I also thought it. If there is no swap utilization, we don't need increase the no of partition for swap.

You only need to do that if you are planning to increase your load from your current 75% to higher.

>If there is a some what utilization, we would have to increase it & see. wouldn't it?

It also depends on how accurate and when you measured the swapinfo.

[Shehan]

Hi Handly/Duncan

Anybody can let me know what are the reference levels of the sar -d 5 10 outputs?

Basically I want to know reference level of following.

%busy avque r+w/s blks/s avwait avserv

I believe there is no performance bottleneck when I am taking this output.If I have the reference level, Then I can come to a conclusion regrading the I/O utilization in Volume group comparing with the output which I am going to take when there is a performance Issue.

It is better if you can send all reference levels in all "sar" output.

Regards
Nirukshitha

[Shehan]

Hi All

Any update on this?

Regards
Niru

[Ilkka Seittenranta]

Hi Niru,

You machine uses about 70% memory to swap (see swapinfo), so I quest the SGA size is between 5-10Gb.

You must disable pseudoswap if you like find what kind performance problems you have. After pseudoswap uses all free memory you can't get any error messages in the syslog-file, there is not enough resources to open file handles!

After pseudoswap uses all free memory the system change to "idle-process"-state, the state is something "waiting until there is more recourses free". The nonsystem-process are that time to "halt"-state and that time load is something like 0.02

The HP-UX not ever use device based swap if you have pseudoswap enabled, there is problem how OS handles swap device priority.

Best Regard
Ilkka

"It take years to quest what is behind there"

[Don Morris_1]

I think you've been dramatically misinformed.

Disk swap is used *before* pseudo-swap in the default reservation algorithm. The only time that changes is when you're using the mlock interfaces, and for kernel dynamic allocations [which are implicitly mlock'd since HP-UX doesn't swap out kernel pages].

Now if you're running Oracle, it wouldn't surprise me if you don't realize this -- since Oracle usually mlock's the SGA if it can. So you can get a big memory swap reservation that matches physical consumption of the SGA. But that doesn't mean everything _else_ on the system will skip the device swap for reservation.

Not to mention, pseudo-swap doesn't "consume free memory" in any way shape or form. Pseudo-swap resources are independent of free RAM. You can completely exhaust one without touching the other.

If you completely run out of kernel memory (which would likely exhaust memory swap due to the stealing... unless you've got a bunch of Ejectable memory on v3, granted) then yes -- you can start to get hangs. Kernel memory exhaustion will do that.

[Ilkka Seittenranta]

Disk swap newer used first if pseudoswap is enabled, see any of you HP-UX. The swapinfo show how memory and disk is used, like this one:

Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8388608 364 8386836 0% 0 - 1 /dev/vg00/lvol2
reserve - 1558364 -1558364
memory 4163584 1775200 2388384 43%

The memory is pseudoswap-area and it used first. There is now way to change priority of swap-areas, because pseudoswap-area is not in /etc/fstab like others!

If you disable pseudoswap then there is only dev-types in the swapinfo.

[Don Morris_1]

Listen to what I said. Disk swap is preferred over pseudo-swap in the *reservation* algorithm unless the pages are locked (mlock or kernel memory).

Swap is first reserved to back virtual objects on HP-UX (and most Unix OSs) by default. This keeps the kernel from having to do random process killing if there isn't enough swap when actual paging occurs. If an object has disk swap reservations and is chosen for paging, *then* swap will be allocated at the time of paging and in the amount actually paged. This is the "Used" part of swap.

In the swapinfo you showed, a little is Used -- a bunch more is Reserved from disk swap (as it should be). Yes, pseudo-swap also shows use -- from kernel/memory locked objects. All normal. It doesn't mean that disk swap won't be preferred on new non-mlock reservations...

You can't change the priority of pseudo-swap because there's no selection to it. Priority isn't used at reservation time, only at allocation time. Pseudo-swap isn't allocated because there's no point in "paging" a RAM page from itself... to itself.

Seriously -- please read the Memory Management white paper if you haven't before at:

http://docs.hp.com/en/1218/mem_mgt.html

It is a bit out of date being aimed at 11.11 original release (so most of the locks and lists it refers to directly don't exist as such), but the concepts are solid.

[Ilkka Seittenranta]

That's right it work that way.

But I find that there is something under, when pseudoswap is enabled and applications stop respond and maybe some time continue "without" problems.

Then one way to find what is wrong is disable pseudoswap. What I see is after this change real messages arriving like "not enough semaphores", "out of memory" and so on.

The answer is change parameters, increase swapspace or really by more memory.

It's not easy calculate some kernel parameter or how must memory needed if pseudoswap is enabled.

[Don Morris_1]

What you've described is oversubscribing the box. If you're going to cause extreme physical memory pressure and have the virtual address space to create the object to do so (i.e. have enough total swap, disk + pseudoswap) you can cause the system to thrash in paging [the perceived hangs]. Removing virtual address space from the system will cause allocation failures earlier -- as malloc(), fork(), etc. fail before getting underlying physical memory.

You'd get the same effect with adding disk swap identical to the pseudoswap you removed -- it has nothing to do with pseudoswap itself. What _will_ be different in that case is that the thrashing (may, depends a bit on workload) may be gentler since vhand will find many objects backed by device swap and hence can page them out. Target rich environment and all.

So yes -- reducing the application load or increasing physical memory if you're thrashing the box is the right answer.

Not sure what you're referring to with the kernel parameter thing, sorry.