topic Re: Performance problem in Operating System - HP-UX

Performance problem

Alberto Hurtado — Fri, 11 Apr 2003 00:41:14 GMT

i have a bad performance and possible swap problems. The
database server it's running HP-UX 11.0 64 bit, 16 cpus, 8188 MB. INFORMIX 7.31.FC7XS also i have 4 aplication server 3584, 3584, 3072, 3072 MB Memory running SAP/R3 Release 40B.
the swapinfo of database server is in attachment:

Re: Performance problem

Michael Steele_2 — Fri, 11 Apr 2003 01:05:25 GMT

Change the priority level to 1 instead of 0 for mvi909. This should be in /etc/fstab.

Since your total pct utilized is less than 85% you have no problem with lack of swap. In fact, you could probably remove two or three logical volumes with no problem. Why don't you just comment out the entries in /etc/fstab, reboot and see.

Since you're running Informix I assume you're file system structure is raw and not vxfs.

What problems are you seeing that led you to conclude that you have performance problems?

Please attach:

sar -v 5 5
sar -u 5 5
vmstat 5 5
sar -d 5 5

Re: Performance problem

Sunil Sharma_1 — Fri, 11 Apr 2003 02:31:12 GMT

Hi,
if you see carefully in database server swap is not even utilising so it can not be problem related to swap.

what kind of problem you are facing ?
just give some more info regarding problem like
sar -u
sar -d
sar -r
etc.

Sunil

Re: Performance problem

A. Clay Stephenson — Fri, 11 Apr 2003 03:10:23 GMT

THe problem doesn't appear to be with your database server but your application servers are all swapping. Those boxes need more memory.
I suspect that you would actually be better off to shutdown one of the App servers; steal the memory and put it in the remaining boxes. You might be able to reduce buffer cache to help some but you really need more memory.

Re: Performance problem

Alberto Hurtado — Fri, 11 Apr 2003 17:33:25 GMT

Here ara the information of :
sar -u 5 5
vmstat 5 5
sar -d 5 5

Re: Performance problem

Zafar A. Mohammed_1 — Fri, 11 Apr 2003 19:01:06 GMT

Will you send these information also.

onstat -g seg
onstat -d
onstat -D
onstat -p

These commands will show the Informix Database stats and then we can able to dig more.

Thanks
Zafar

Re: Performance problem

Bill Hassell — Fri, 11 Apr 2003 23:49:19 GMT

From a purely swap-and-performaqnce perspective, mvi905 is in SERIOUS need of a lot of RAM, a minimum of 8 to 16 Gb is badly needed. The reason is found in vmstat which shows 400 pages being swapped out, indicating severe memory shortages.

The rest of the machine stats show areas that need help but there will be no simple solution as certain disks are very busy with high queue lengths and servicve times. There are no magic settings to fix this set of servers.

Re: Performance problem

Michael Steele_2 — Sat, 12 Apr 2003 00:43:44 GMT

Regarding the results of sar -v 5 5:

mvi904 /-> sar -v 5 5

HP-UX mvi904 B.11.00 U 9000/800 04/11/03

10:35:15 text-sz ov proc-sz ov inod-sz ov file-sz ov
10:35:20 N/A N/A 617/2560 0 5088/16384 0 2445/64010 0

Your using 25% of your process table - nproc 617/2560 = 24.10%

Your using 31% of your inode table - ninode: 5088/16384.

Your using 4% of your file table - nfile : 2445/64010 = 3.819.

There are similar under utilized numbers for mvi909, mvi905, mvi914 and mvi912.

What does this mean? If the percentages approach something smaller, i.e., 10-12%, the over time your system memory would become fragmented and hang the server. As it is, you can easily reduce these kernel parameters if over time you see same kind of percentages.

Regarding the results of vmstat 5 5:

po is the metric to note here especially for mvi905: This server is swapping or paging out processes before there work is completed. Add some RAM. All of the other servers except mvi912 were also paging, but no where near as bad as mvi905.

Regarding the results of sar -d 5 5:

Disk c1t6d0 on mvi905 was borderline bottleneck for one second. Probably because swap is located here. If over time %busy continues to approach and exceed the 50% mark you'll have identified a bottleneck.

In conclusion, I'd revisit the kernel parameters indicated and add some RAM to mvi905.

Re: Performance problem

Steven E. Protter — Mon, 14 Apr 2003 02:10:06 GMT

My first question would be why do you have swap spread all over the place in little pieces?

If swap is sharing a disk with other things like database files, there could be a lot of disk arm movement thats hurting disk performance.

You are however utilizing a fair percentage of your swap, so it appears the amount in total you've chosen is appropriate.

I would use the attached shell script to measure performance. I'd modify it and set the measurement time to an hour. Take a look at which disks are bottlenecking and consider moving swap off or other things off those physical disks.

There are lots of other data points you can get from this script.

SEP

Re: Performance problem

Steven E. Protter — Mon, 14 Apr 2003 02:19:22 GMT

Actually, had I read the thread more carefully I'd have had a recommendation. File away my script for future use.

See whats on those disks that are getting hammered on the sar tests. You might want to use a different swap layout strategy. I have two local physical disks totally devoted to swap on my older systems. Thats kind of hard now with the huge disks they put in the big boxes now.

Still, swap and application layout should be done in a way that keeps swap and application data away from each other.

SEP

Re: Performance problem

Alberto Hurtado — Tue, 22 Apr 2003 14:55:02 GMT

i send the information of Informix "onstat"

Re: Performance problem

Tim Sanko — Tue, 22 Apr 2003 15:25:07 GMT

There is little problem with the DB server from a tuning perspective for the DB itself.

The IO LOAD could be distributed better. I would move the swap to a fast large space and watch it. This box doesn't look too bad,

You may need to tune the query's hitting the server. They should be using indexes and avoid outer joins and full table scans.

Tim

Re: Performance problem

Michael Steele_2 — Tue, 22 Apr 2003 15:32:25 GMT

non spreckenzie informix