topic Re: sar -u - high wio% in Operating System - HP-UX

sar -u - high wio%

Deepak Seth_1 — Thu, 27 Mar 2003 21:31:42 GMT

We have V-2200(16 GB RAM and 10 CPU) used as a database server with 3 fiber cards . We use EMC disks and also have EMC powerpath . For the last few days we have been getting complaints on slow performance. There are 2 things whch bother me .
- sar -b - % write cache remain very low - avg 30-40 only (buffer cache = 1 gig)

- sar -u - the avgv %wio remain as high as 30 .

what should i look exactly in glance to help me find the bottleneck ?

Important - how do interprate the glance result and find the reason of bottleneck .

I know this question is asked many times and i already gone through few of the old archives. But may be over a period of time somebody find a better way to diag this type of issue.

Any help appreciated .

Re: sar -u - high wio%

John Poff — Thu, 27 Mar 2003 21:36:25 GMT

Hi,

Have you checked your powerpath to make sure there are no problems? Try 'powermt display', I think that is the command that will show you the paths.

Also, has anything changed on the box since it started running slow? Been rebooted or patched? Any runaway processes? We've seen similar problems when our Oracle DBAs were running an analyze program that went nuts.

JP

Re: sar -u - high wio%

Steven E. Protter — Thu, 27 Mar 2003 21:40:48 GMT

Here is a data collection script that might help.

Attached.

SEP

Re: sar -u - high wio%

Sridhar Bhaskarla — Thu, 27 Mar 2003 21:40:53 GMT

Hi,

Can you also post your sar -d 2 5
You will get the response time and average wait time here along with utilization.

-Sri

Re: sar -u - high wio%

John Dvorchak — Thu, 27 Mar 2003 21:45:34 GMT

This kind of sounds like a primary/alternate path issue to the EMC frame. I think maybe sar -d would uncover a problem with either the primary or alternate path to the EMC attached drives.

As far as glance goes, you could use "s" to select one of the top oracle processes and look for a lot of voluntary context switches and very little I/O, with very few involuntary switches. But that would only be a hint as to the problem. Check the second to the bottom line as it tells the reason the pid is waiting. i.e. I/O, streams, etc.

Re: sar -u - high wio%

Deepak Seth_1 — Thu, 27 Mar 2003 21:50:35 GMT

just remember i increased the nproc from 16000 to 20000 last week. I have read it some where that setting its value is not recommended.

I already checked the powermt watch command and maximum q-IO's i see is 1or 2 under q-iO's plus all paths are optimal.

Also attached the sar -d 2 5 output .

Re: sar -u - high wio%

John Poff — Thu, 27 Mar 2003 22:02:04 GMT

Did you change any other kernel parameters when you increased nproc? What are your mount options for the filesystems [mount -v]?

JP

Re: sar -u - high wio%

Deepak Seth_1 — Thu, 27 Mar 2003 22:13:58 GMT

sorry , it is nfile not nproc .

Re: sar -u - high wio%

Deepak Seth_1 — Thu, 27 Mar 2003 22:16:51 GMT

attached is the mount -v output. We use online JFS mount options .

Re: sar -u - high wio%

Deepak Seth_1 — Thu, 27 Mar 2003 22:32:53 GMT

i think before my last reboot i changed the /etc/fstab file added online JFS mount option and then rebooted the machine .

Re: sar -u - high wio%

Sridhar Bhaskarla — Thu, 27 Mar 2003 22:56:27 GMT

Hi Deepak,

Sorry for the delayed reply.

Looking at your sar -d, your disks are looking perfectly normal. I neither see utilization high nor abnormal response times.

I would rule out the possibility of disk subsystem here.

It is interesting to see that you enabled onlineJFS options for almost all the filesystems. While it is a general perception that these options help, they are to be used only case by case basis. Some applications benefit from the cache and some do not.

I don't know if your application likes to bypass the buffer cache. The lower write cache hit and increased %wio may be corresponding to your bypassing buffer cache.

Try to remount the filesystem without these options (mincache and convosync) and see if it helps. You can use 'remount' option of mount to do it online without having to unmount the filesystems.

-Sri

Re: sar -u - high wio%

Deepak Seth_1 — Thu, 27 Mar 2003 23:16:16 GMT

thanx for the reply . looks a good idea to try re-mounting the file system without the option . If i change my /etc/fstab file and use mount -a . does it means it remount it in normal way .

Re: sar -u - high wio%

Sridhar Bhaskarla — Thu, 27 Mar 2003 23:27:45 GMT

Hi Deepak,

mount -F vxfs -o delaylog,remount /mount_point

should reset it back.

You can enable these options back using the same remount option.

-Sri

Re: sar -u - high wio%

Deepak Seth_1 — Thu, 27 Mar 2003 23:44:07 GMT

thanx . i re-mounted them. i check with mount -v command . let me see how it goes tomorrow.

Re: sar -u - high wio%

Deepak Seth_1 — Fri, 28 Mar 2003 00:09:27 GMT

one more thing , i used on-line JFS option only for oracle databases and keeping in mind only for datafiles and index , nothing for archive and redo logs .
any way , let see how it works with those option taken out .

Re: sar -u - high wio%

Deepak Seth_1 — Fri, 28 Mar 2003 17:54:51 GMT

So far looks good . I havn't heard any compliants from users today . It will be good investigation for me to find why online JFS mount option impacted one database and other remain fine.
Just one last question . When we look at sar -u output , do we need to get worried if we see high wio%. in my case it remains around 30% . Second , sar -d output , should i compare avg service time v/s avg wait time . Should i need to be worried if i see avg wait time more than avg serv time.
Sridhar - u have seen my sar -d output , few of the disks were showing avg wait time more than serv time . Is this means i have some kind of bottleneck .

Any pointers.