topic Re: High %wio in Operating System - HP-UX

High %wio

Anoopkumar — Mon, 24 May 2010 11:53:25 GMT

Hi support,

I am facing a strange issue in our production box.sar out showing high wio%

P-UX sgen03 B.11.11 U 9000/800 05/24/10

16:41:39 %usr %sys %wio %idle
16:41:44 24 8 44 24
16:41:49 18 3 50 30
16:41:54 22 4 49 25
16:41:59 23 3 50 25
16:42:04 23 3 52 22

Average 22 4 49 25

I did collected the sar -d and sar -q (attached) But I couldnt see any disk bottleneck.But in oracle performnce montior shows i/o thruput is less than expected and apps working too slow.

Please help me to understand the issue.

Thanks
Anoop

Re: High %wio

Michael Steele_2 — Mon, 24 May 2010 13:55:37 GMT

Hi:

Reviewing your sar -d report, no where did a see avwait > avserv, (* definition of a disk bottleneck when using sar *), so you don't have a disk issue.

More so, your avwait is always zero, so your %wio is not disk I/O related and probably a good thing since its working harder to keep up with the disk array. Therefore I would say it was I/O related elsewhere, like I/O to RAM or perhaps disk controller, or even networking.

Question: Are you running raw volumes? Raw vols I think also runs higher than file system vols.

Re: High %wio

TwoProc — Mon, 24 May 2010 14:04:09 GMT

Amazing - I posted an answer to this, and while it took a long time, it updated like it was sucessful - and now, it's not there.
Oh yeah, the internet is perfect for hosting everything in cloud...

Re: High %wio

Steven E. Protter — Mon, 24 May 2010 14:07:55 GMT

Shalom,

Please post oracle software versions and patch levels. The sar output was legible, but the graphic part of your document I could not read.

You say>>
oracle performance monitor shows i/o throughput is less than expected
<<
I've corrected the typos.

I assume that screen I could not read may provide the information I seek, but I'd like to know how oracle is measuring performance and determining there is a disk problem.

sar indicates no such problem.

So the source could be oracle software needing a patch or bad application software developed for your oracle application.

Could also be caused by inefficient sql calls within your application. This last issue is normally handled by DBA's monitoring sql performance and identifying inefficient queries.

SEP

Re: High %wio

Anoopkumar — Tue, 25 May 2010 06:07:14 GMT

Hi all,

Thanks for the replies,

Hi Michael,
We are not using raw volumes and Async i/o also. Planning to enable the Async driver
John,
Can I have the answer please? ï

Steven,
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production

Here am attaching the message in oracle performance tool and report.
As per them its waiting for buffer time is more expected in some table space. This can happen because of buffer is not flushing out to disk properly. My doubt is OS capable of taking care this much i/o operation. Do anyone have any such document on HPUx 11.11 Mass storage i/o performance.

OS: HPUX 11.11(QPK Dec 2008)

Re: High %wio

Aneesh Mohan — Tue, 25 May 2010 06:30:26 GMT

Anoop,

How about the memory utilization and also your DBA should check number of dbwr configured.

Aneesh

Re: High %wio

Prasanth V Aravind — Tue, 25 May 2010 06:43:37 GMT

Yes.. OS capable of taking care this much i/o operation .. we have some servers which having up to 5TB database file systems. I am not sure what is the size of your database server.

you can check.

1. increase dbc_max_pct if you have ample free memory in your server.

2. in which file system you have redo-log files ?? is it distributed over all file systems ?? can you create a separate file system of redologs which is stripped over maximum disks as possible ?

Gudluck
Prasanth

Re: High %wio

Vijaya Ragavan_1 — Tue, 25 May 2010 07:12:56 GMT

Hi Anoop
from where the disk is presented is it from storage?
if it is from EVA(storage)Kindly check the controller load for particular vdisk.

other wise change the controller path and observe & kindly send me the autopath display.

Re: High %wio

Anoopkumar — Tue, 25 May 2010 09:42:13 GMT

Hi,
Aneesh,
Mem Utilization is 80% and we have 63GB mem. Attached kmeminfo o/p.
DBWR is 4.
Prashanth,
dbc_max_pct is 10 and min is 5.
I think I will increase to 20 and 10 .As per my understanding this will reserve 20% of total mem.
Is there anyways to find out buffer cache utilization. I donâ t have glance.
We are now using separate file system for redo.
Vijaya,
As per the SAR output and Storage essential reports there is no write latency and load between the controllers is very much balanced. So I donâ t think something to do with disk subsystem.

# ./kmeminfo
tool: kmeminfo 5.19
unix: /stand/vmunix 11.11 64bit PA2.0 on "sgen03"
core: /dev/kmem live
link: Fri Apr 9 03:41:10 oman 2010
boot: Sat May 8 02:52:15 2010
time: Tue May 25 14:40:47 2010
nbpg: 4096 bytes

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 16239872 62.0g 100%
Free memory = 2720693 10.4g 17%
User processes = 10351798 39.5g 64% details with -user
System = 3130699 11.9g 19%
Kernel = 1506712 5.7g 9% kernel text and data
Dynamic Arenas = 571822 2.2g 4% details with -arena
M_TEMP = 394559 1.5g 2%
ALLOCB_MBLK_LM = 38050 148.6m 0%
M_SPINLOCK = 35885 140.2m 0%
VFD_BT_NODE = 29777 116.3m 0%
M_SWAP = 14046 54.9m 0%
Other arenas = 59505 232.4m 0% details with -arena
Super page pool = 8094 31.6m 0% details with -kas
Static Tables = 776105 3.0g 5% details with -static
pfdat = 372610 1.4g 2%
nbuf = 143216 559.4m 1% bufcache headers
htbl2_0 = 131072 512.0m 1%
pfn_to_virt = 62101 242.6m 0%
bufhash = 40960 160.0m 0% bufcache hash headers
Other tables = 26145 102.1m 0% details with -static
Buffer cache = 1623987 6.2g 10% details with -bufcache
# ./kmeminfo -bufcache
tool: kmeminfo 5.19
unix: /stand/vmunix 11.11 64bit PA2.0 on "sgen03"
core: /dev/kmem live
link: Fri Apr 9 03:41:10 oman 2010
boot: Sat May 8 02:52:15 2010
time: Tue May 25 14:40:57 2010
nbpg: 4096 bytes

----------------------------------------------------------------------
Buffer Cache usage summary (in pages):

Processing buffer cache (913002 headers) ...
Processing buffer cache page pool...

Nbuf = 913002 File-system buffer cache headers
Bufpages = 1623987 File-system buffer cache pages:
Physical = 1623973 Allocated (mapped, b_bufsize's)
B_inode = 4876 4862 bufs, BCACHE
B_dir = 19674 10026 bufs, BCACHE
B_cylgrp = 402 203 bufs, FSDATA
B_sblock = 2 1 bufs, FSDATA
B_indbk = 760 376 bufs, FSDATA
B_data = 1598259 896789 bufs, BCACHE
Pagepool = 0 Available (not mapped, bhead free list)
Virtual = 1862848 Virtual pages from bufmap

Re: High %wio

chris huys_4 — Tue, 25 May 2010 23:06:39 GMT

Hi Anoop,

You should decrease the amount of filesystem buffercache, not increase it. You have way to much filesystem buffercache specified.

I never seen any benefit on HP-UX 11.11, to have a filesystem buffercache with a size of more then 800Mbyte. (unless its something very specific ) Thus decreasing dbc_max_pct and dbc_min_pct, instead of increasing dbc_max_pct/dbc_min_pct, to 2%, which still equals to 1.2Gbyte of filesystem buffercache, should be the way forward.

Its also a oracle database, databases dont benifit from filesystem buffercache, they have there own buffercache mechanism..

For the rest open a support call with HP. Investigation with kitrace should be able to catch the system calls, were most time is lost on.

Offcourse, this done after the system is patched up to the latest.

My personal guess. Looks like you are using oracle database files on a vxfs filesystem.
Filesystems which hold database files, are known for filesystem lock contention of the database files, thats why on older HP-UX, most oracle databases are on raw devices, to get around the "filesystem metadata" lock contention were "databases on filesystems" suffer from.

With vxvm in combination with vxfs, on HP-UX 11.23 and more recent versions, a database accelerator as odm (oracle disk manager), part of the HP Serviceguard storage management suite for oracle, can give the oracle database files "raw volume speed", when part of a vxfs filesystem.

Greetz,
Chris