I/O intensive on root disk

[Camel_1]

Hi there,

I found the root disk has intensive IO (always util 100%). Especially on root & var FS. Does anyone has idea what causing the problem.

The attached is the capture from glance.

Thanks,

Simon

[A. Clay Stephenson]

Your attachment is all but worthless because you make no mention of the processes running on the box. You need to find the processes associated with this disk activity. Glance should be able to pinpoint this for you.

[Denver Osborn]

anything logged in the syslog, dmesg?
sar -d output and what's top showing?

-denver

[Adisuria Wangsadinata_1]

Hi Simon,

Could you send us :
# sar -d
# sar -b
# sar -u
# iostat
# vmstat 5 30
# ioscan -nfCdisk

Also send the file below :
/var/opt/resmon/log/event.log
/var/adm/syslog/syslog.log

Wait for your update.

Cheers,
AW

[Ermin Borovac]

Here's how to find disk IO hogs with glance.

Run glance
Type 'o'
Select 1 from 'Go to 1) Process; 2) Transaction; 3) Thread option screen(1) :'
Use arrow keys to scroll down to 'Sort key (name/cpu/disk/rss):'
Type 'disk' and press
Answer 'y' to 'Set your user defaults to these values (y/n)?'

This will sort processes by 'Disk IO Rate', which will identify disk IO intensive processes at the top of the list.

[Camel_1]

thank you for your suggestion/questions. I attached the sar output and iostat for your reference.

- root disks are c0t2d0 & c56t2d0 (mirrored)
- IO hitting / and /var
- I don't think it is caused by swapping, swap usage is low.
- have sorted the top IO processes, they are oracle processes and hitting oracle datafiles only (raw devices @EMC frame).
- no error reported on syslog and ems log.

[Alzhy]

Can you send us a copy of "bdf -k" OR do you know if your Oracle software is located on your vg00 diskgroup? Also how do you know that "/" and "/var" are your busiest?

I am theorizing you probably have big system disks (73Giggers +) and you have Oracle software on the same disk. It may also be you have logs (active) that are being heavily hit and it also resides on your root/boot/os disks.

Your "sar -d" and "sar -b" logs do appear "healthy" to me.

[Camel_1]

Nelson, if you open my first attachment in the 1st forum thread, you will see / & /var have heavy IO. We are running oracle 9 and the bin is located @EMC disk.

Thanks

[Alzhy]

Did not notice the attach.. the icon does not appear at times.

If that's the case, do you have any non-OS/system stuff that resides on / and that's possibly using /var intensely?

What processes do glance show you as the heaviest I/O users?

[Andy Torres]

You might try fuser -u on the / & /var filesystems to see who's using them. Maybe you can track down usage from there.