sar -d heavy load on root disk

[nancy rippey]

sar -d is showing heavy load on my root disk. I am wondering what would cause this. The load is higher when xpath is pulling data from the database and transfering it to a mainframe. Could memory be an issue? Output from sar -d
Average c0t5d0 2.63 86.62 4 32 249.27 16.71

Any help would be appreciated.

[Victor_5]

Hi Nancy:

Sometimes memory is the critical section for perfermance, however, only according to sar -d, we can not 100 percent say it is memory problem.

Try to use sar -du 5 5
see 'avque', '%wio'

avque average number of requests waiting to access the device. A disk queue with a length greater than three often mean I/O requests will spend more time waiting in the queue than actually being serviced

%wio idle with some process waiting for I/O(Only block I/O, raw I/O, or VM pageins/swapins indicated), the performance will slow if the number is over 30, possibly it is I/O bottleneck

vmstat
if the 'po'>0 and 'free'*4>2M, that means it is a memory bottleneck.

[nancy rippey]

Shawn,
Thanks for the info.
sar -du output is
Average c0t5d0 2.02 64.63 3 26 187.06 17.05
avq wio of 64.63 looks pretty bad.
from vmstat po=0 and free=109560
From the sar -du I guess I can figure on a disk bottle neck but with this being my root drive I am at a lose as how redistrubute resources.

[Victor_5]

Hi Nancy:

It looks like it is your I/O problem. Yes, regarding performance tuning, it is hard for us to do something on running systems for I/O bottleneck, especially for 24*7 shops. The logic is try to balance the work load on different channnels and different disks, however, you will do a lot of work and those changes will definitely affect your business.

Two ideas for about it:
1. add memory
Generally, it is the easiest way although memory is not your main bottleneck, you will see a big performance improvement after that.

2. banlance your work load
I mean, try to banlance the I/O on your root disk, move those non-critical tasks to another disk, it is varied deponds on your working environment.

Hope it can helps.

[David Allen]

Hi Nancy,

I found a few documents on HP's Knowledge Base that were quite helpful in determining where bottlenecks were occuring on a system. If you search the knowledge base, look for the following document IDs.

S3100002312A
S3100002312B
S3100002312C

The document is titled "Sys Adm: determining the cause of system performance problems".

If you can, try moving non-root specific data or applications from your root disks. This could include users home directories. If you have any secondary swap devices configured, you may want to move them to another disk.

Regards,
Dave

[Tim D Fulford]

Hi

Just a couple of things

1/ do you have glance or MewasuerWare? If so you can look at the IO's per filesystem. In glance it is

glance -i

For MeasureWare there are metrics beginning with LV_ (say LV_READ_RATE, LV_WRITE_RATE, LV_BYTE_RATE)

Also What is the average time taken for each IO. I assume your disk utilisation is 100% so all you need to do is find out the IO rate for the root disk.

IOtime[ms] = disk_util[%]*10/IO_Rt[IO/s]

So say you have an IO rate of 100 IO/s then the average time per IO would be 10ms/IO (which is not too good). This would suggest thrashing.

2/ are there only the standard filesystems on vg00? If there are any others or any database raw LV's look at them

3/ Do you have more than one swap area in vg00? If so, & the priorties are wrong, you could also get thrashing.

Phew

Good hunting

Tim

[Bill McNAMARA_1]

You may want to see if you're swapping much.. swapinfo.

You mich also consider rearranging the distribution of your vg00 lvols to balance the data across more than one disk, on ideally more than one scsi bus.

Make sure your root disk has a high scsi id relative to the scsi bus elements that its sharing with.

Analyse your
ioscan -fnk
strings /etc/lvmtab
and vgdisplay -v vg00

Later,
Bill

[Tim D Fulford]

Nancy

As Lt Columbo says, "Just one more thing!"

you supplied a sar -du 5 5 output but forgot the bottom line.

It is my suspicion that the bottom line will have a low %usr & high %sys and %idle will be zero (I expect %wio will also be high). If this is the case then you are having.

But according to your blk/s (32|26)you have an average time per IO of about 30+ ms/IO(depending on which sar -du you take, & assuming %idle is 0). If this is the case
Your disk IS THRASHING.

Just thought you you might like to know

Tim

[nancy rippey]

Thanks for all the input. My root volume contains only the standard filesystems, thrashing is occuring occasionally, I only have my primary swap and I have not had any problems w/swapping My current plan is to have additional memory installed next weekend. I will let you know how performance increases. These performance issues are always so fun to figure out!