sar -u - high wio%

[Deepak Seth_1]

We have V-2200(16 GB RAM and 10 CPU) used as a database server with 3 fiber cards . We use EMC disks and also have EMC powerpath . For the last few days we have been getting complaints on slow performance. There are 2 things whch bother me .
- sar -b - % write cache remain very low - avg 30-40 only (buffer cache = 1 gig)

- sar -u - the avgv %wio remain as high as 30 .

what should i look exactly in glance to help me find the bottleneck ?

Important - how do interprate the glance result and find the reason of bottleneck .

I know this question is asked many times and i already gone through few of the old archives. But may be over a period of time somebody find a better way to diag this type of issue.

Any help appreciated .

[John Poff]

Hi,

Have you checked your powerpath to make sure there are no problems? Try 'powermt display', I think that is the command that will show you the paths.

Also, has anything changed on the box since it started running slow? Been rebooted or patched? Any runaway processes? We've seen similar problems when our Oracle DBAs were running an analyze program that went nuts.

JP

[Steven E. Protter]

Here is a data collection script that might help.

Attached.

SEP

[Sridhar Bhaskarla]

Hi,

Can you also post your sar -d 2 5
You will get the response time and average wait time here along with utilization.

-Sri

[John Dvorchak]

This kind of sounds like a primary/alternate path issue to the EMC frame. I think maybe sar -d would uncover a problem with either the primary or alternate path to the EMC attached drives.

As far as glance goes, you could use "s" to select one of the top oracle processes and look for a lot of voluntary context switches and very little I/O, with very few involuntary switches. But that would only be a hint as to the problem. Check the second to the bottom line as it tells the reason the pid is waiting. i.e. I/O, streams, etc.

[Deepak Seth_1]

just remember i increased the nproc from 16000 to 20000 last week. I have read it some where that setting its value is not recommended.

I already checked the powermt watch command and maximum q-IO's i see is 1or 2 under q-iO's plus all paths are optimal.

Also attached the sar -d 2 5 output .

[John Poff]

Did you change any other kernel parameters when you increased nproc? What are your mount options for the filesystems [mount -v]?

JP

[Deepak Seth_1]

sorry , it is nfile not nproc .

[Deepak Seth_1]

attached is the mount -v output. We use online JFS mount options .

[Deepak Seth_1]

i think before my last reboot i changed the /etc/fstab file added online JFS mount option and then rebooted the machine .

[Sridhar Bhaskarla]

Hi Deepak,

Sorry for the delayed reply.

Looking at your sar -d, your disks are looking perfectly normal. I neither see utilization high nor abnormal response times.

I would rule out the possibility of disk subsystem here.

It is interesting to see that you enabled onlineJFS options for almost all the filesystems. While it is a general perception that these options help, they are to be used only case by case basis. Some applications benefit from the cache and some do not.

I don't know if your application likes to bypass the buffer cache. The lower write cache hit and increased %wio may be corresponding to your bypassing buffer cache.

Try to remount the filesystem without these options (mincache and convosync) and see if it helps. You can use 'remount' option of mount to do it online without having to unmount the filesystems.


-Sri

[Deepak Seth_1]

thanx for the reply . looks a good idea to try re-mounting the file system without the option . If i change my /etc/fstab file and use mount -a . does it means it remount it in normal way .

[Sridhar Bhaskarla]

Hi Deepak,

mount -F vxfs -o delaylog,remount /mount_point

should reset it back.

You can enable these options back using the same remount option.

-Sri

[Deepak Seth_1]

thanx . i re-mounted them. i check with mount -v command . let me see how it goes tomorrow.

[Deepak Seth_1]

one more thing , i used on-line JFS option only for oracle databases and keeping in mind only for datafiles and index , nothing for archive and redo logs .
any way , let see how it works with those option taken out .

[Deepak Seth_1]

So far looks good . I havn't heard any compliants from users today . It will be good investigation for me to find why online JFS mount option impacted one database and other remain fine.
Just one last question . When we look at sar -u output , do we need to get worried if we see high wio%. in my case it remains around 30% . Second , sar -d output , should i compare avg service time v/s avg wait time . Should i need to be worried if i see avg wait time more than avg serv time.
Sridhar - u have seen my sar -d output , few of the disks were showing avg wait time more than serv time . Is this means i have some kind of bottleneck .

Any pointers.