Extensive Constant I/O around 100%

[Muhammad Ahmad]

Hi Performance/Tunning Gurus,

We are facing I/O Issues on one of our production servers.

No matter, oracle database is running or down, extensive constant I/Os were observed (around 100% disk utilization) on the Physical Volumes presented through SAN.

Please find the attached logs.

The PV c14t0d1 & c14t0d3 were presented through SAN.


Regards,

Muhammad Ahmad

[Steven E. Protter · Accepted solution]

Shalom Muhammad,

I would say this:
what is on those two disks? What processes are performing I/O?

Lets say there is a filesystem called /clutz on the disk.

fuser -cu /clutz

This will identify the processes with open file handles on the disk.

Lets say there is a process number 4433 on the disk.

ps -ef | grep 4433

That will get you process information.

It could be a number of things, but I suspect either swap because the system is busy, or a poorly written application causes the problem.

SEP

[Duncan Edmonstone]

Muhammad,

I see nothing of concern in these outputs. The %busy column in sar -d simply shows the % of time during the interval that the disk was doing an IO operation. This doesn't necessarily imply an IO issue (particularly if the LUN in question is in fact on a disk array and is made up of many physical spindles).

The important numbers to look at on your sar -d output are avque, avwait and avserv. In your case these do not look partilcularly concerning. Typically if we see avque values of >2 and avserv values of >10ms we might have soemthing to be concerned about. Looking at your output I'm guessing that c0t6d0 and c3t6d0 are your system disks, and even though they have service times just over 10ms they aren't doing a significant amount of IO anyway. Your other (presumably database) disks are doing a significant amount of IO but queue lengths and service times are not anything to be concerned about.

Do you actually have a performance issue reported to you, or are you just thinking there is cos of the 100% util number?

HTH

Duncan

[Yogeeraj_1]

hi Muhammad Ahmad,

if you run "glance -i", you may also get some indication about the "Top disk user" and thereby investigate further.


e.g. (extract of glance output)

--------------------------------------------------------------------------------
IO BY FILE SYSTEM Users= 2
Idx File System Device Type Logl IO Phys IO
--------------------------------------------------------------------------------
1 / /dev/vg00/lvol3 vxfs 0.1/ 3.6 1.1/ 1.6
2 /stand /dev/vg00/lvol1 hfs 0.0/ 0.0 0.0/ 0.0
3 /var /dev/vg00/lvol8 vxfs 0.7/ 0.6 2.6/ 2.1
4 /usr /dev/vg00/lvol7 vxfs 0.0/ 6.4 0.0/ 0.8
5 /tmp /dev/vg00/lvol4 vxfs 0.0/ 0.0 0.7/ 0.4
6 /prod */vgcmtapp/lv_cmtapp vxfs 0.0/ 0.0 0.0/ 0.0
7 /opt /dev/vg00/lvol6 vxfs 0.0/ 2.0 0.0/ 1.8
8 /home /dev/vg00/lvol5 vxfs 0.0/ 0.0 0.0/ 0.0
9 /d10 */lv_pfsdb01 vxfs 0.0/ 0.0 0.0/ 0.0
10 /d02 /dev/vg02/lvol1 vxfs 0.0/ 0.1 0.0/ 0.0

Top disk user: PID 0, swapper 2.1 IOs/sec S - Select a Disk



hope this helps too!
kind regards
yogeeraj

[Muhammad Ahmad]

Hi,

first of all thanks a log to all.

Actually, Its an insert-intensive application (bulk inserts from 8 to 10 sources in parallel) in conjunction with fetch/select queries.

Hi SEP,
its the oracle user and oracle application processes,which are doing I/O.

Hi Duncan,
yes, the performance issue was observed when select statements are given while the parallel bulk inserts are running.

Hi Yougeeraj,

glance shows disk utilization around 100%, and memory plus cpu around 55-80%.


Any other areas to investigate?

Thank you.
Regards,

Muhammad Ahmad.

[piyush mathiya]

Muhammad,
have you use dd command to check disk utilization. the command is . . .

# dd if=/dev/rdsk/c93t9d5 of=/dev/null bs=64k
44132+0 records in
44132+0 records out

If this command return any error, that meanse the disk have a problem( effected disk).
Second thing, are you using striping or not? if not then you should implement it to reduce this type of problem.

Regards,
Piyush Mathiya

[Duncan Edmonstone]

Muhammad,

And was your sar output actually taken when this problem was occurring?

Because as I said I see *no* issue with the performance of these two disks.

HTH

Duncan

[Tim Nelson]

Your last statement conflicts with your opening remarks.
>>No matter, oracle database is running or down, extensive constant I/Os were observed (around 100% disk utilization) on the Physical Volumes presented through SAN.


You responded to SEP stating Oracle was up and running, using these disks.

Your opening statement leads us to believe that 100% util is seen even when Oracle has been stopped.

so, are we looking to anylyze the performance of these disks or attempt to find out whey they are executing IOs even when the app is down ???

[Tim Nelson]

Your last statement conflicts with your opening remarks.
>>No matter, oracle database is running or down, extensive constant I/Os were observed (around 100% disk utilization) on the Physical Volumes presented through SAN.


You responded to SEP stating Oracle was up and running, using these disks.

Your opening statement leads us to believe that 100% util is seen even when Oracle has been stopped.

so, are we looking to analyze the performance of these disks or attempt to find out whey they are executing IOs even when the app is down ???