Higheset disk utilizaing users

[Spark_2]

Hi,

I have done quite a research on various ITRC threads. With help of those, I can find the disk's utilization using sar -d, iostat and glance -u. In my box, there are two internal disks only which have partitions like /orahome
and /oraclient mounted on it. And the utilization is shooting to as high as 100%.

Is it possible to identify the exact highest users please?

[Aneesh Mohan]

Hi,

You need to use glance for this.

Please follow Pete advice on this in the below thread.
http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1012505&admit=109447627+1258002064211+28353475


Regards,
Aneesh

[Michael Steele_2]

Hi

Use sar -d for this and

%busy
Portion of time device was busy servicing a request;

avque
Average number of requests outstanding for the device;

r+w/s
Number of data transfers per second (read and writes) from and to the device;

blks/s
Number of bytes transferred (in 512-byte units) from and to the device;

avwait
Average time (in milliseconds) that transfer requests waited idly on queue for the device;

avserv
Average time (in milliseconds) to service each transfer request (includes seek, rotational latency, and data transfer times) for the device.

When avwait is > than avserv that disk is bottlenecked

[Spark_2]

Thank you for your replies.

I am able to identify the utilization per disk using glance, sar and iostat
Now I want to know who exactly is using. How to find the processes, users who are utilizing the disk highest

[Johnson Punniyalingam]

>>How to find the processes, users who are utilizing the disk highest<<

its not possible, as per my knowledge,

but you can narrow down,

using lsof & fuser command to check which process & userid hold file system

fuser -c /

man lsof and man fuser

[Aneesh Mohan]

You can sort the i/o utilisation list based on the process.

I dont have a system with glance installed now..please try Pete advice to get the same.

From the GPM Main window, select Reports > Process List. In the Process List window, select Configure > Sort Fields. That brings up a window where you highlight the field you want to sort on and move that field to the leftmost side - the field are sorted from left to right so moving Phys IO Rate all the way to the left will show you your most I/O intensive process.



Aneesh

[Hakki Aydin Ucar]

Hi,
Your question remind me of a script I know; spacehogs: Try it:

#!/bin/sh

echo "Enter FS or Path i.e. /tmp "
read patika

find $patika -type f -exec /bin/ls -ls {} ';' | awk '
{ using[$10] += $1 }
END { for (name in using)
{ print using[name], name } }' | sort -nr | head |tee /tmp/trash_go
exit 0

[Michael Steele_2 · Accepted solution]

Hi

As stated your request can only be narrowed down.

a) Using sar -d and based upon the test avwait > avserv note the disk.
pvdisplay -v disk |more - Note the file systems.

# cd filesystem
# touch 04230000 mfs
# ll
..... 0 Apr 23 00:00 mfs

Example of using the find on filesystem ..
# find / -type f -newer /mfs -exec ll {} \;

This will give you a report of all data files currently opened any updated after

..... 0 Apr 23 00:00

Or you can use lsof and fuser.

[Spark_2]

Hakki,

The script seems to be pulling biggest files.

I was trying to find which user/process is making disk utilization high.

I have orahome in vg00. Now how to justify that its the oracle and nothing st system level

[Spark_2]

As it is the internal disks which have 7 standard and oracle mount points too...so becoming little difficult...fuser and touch also not helping much.

Although the above two can help in a little different scenario. Thank you

Still if someone has faced similar situation, please do guide me a bit

/dev/vg00/lvol1 1025617 166195 756860 18% /stand
/dev/vg00/lvol7 20971520 12365408 8606112 59% /var
/dev/vg00/lvol6 8290304 2660592 5585776 32% /usr
/dev/vg00/lvol5 10485760 590600 9819576 6% /tmp
/dev/vg00/orahome 6160384 4498245 1558257 74% /orahome
/dev/vg00/oraclient
6160384 19568 5757983 0% /oraclient
/dev/vg00/lvol4 10485760 5630688 4817168 54% /opt
/dev/vg00/lvhome1 2097152 1132020 905657 56% /home1
/dev/vg00/lvol8 3145728 1205880 1924712 39% /home

[Michael Steele_2]

Sigh.

The problem is you only have a couple of disks and everything is on them.

I can only offer you my experience at this point, the O/S file systems will always create the greatest amount of I/O traffic on any disk. For example, you have 10 disks and the O/S, vg00, only resides on the first two. These two will have more I/O traffic than any of the other 8.

Regardless, the way to fix this problem hasn't changed in a decade, add more disks and move your applications and database out of vg00 and off of the O/S disks.

The O/S should always have their own disks and always be the only thing in vg00.

[Spark_2]

Yeah Michael...I agree

so, till the movement, there doesnt seem anyway to write to Oracle team with some proofs...that u r the one utilizing highest :-(

[Hakki Aydin Ucar]

>Spark: The script seems to be pulling biggest files.

OK, then you can try this partition based version :

#!/bin/sh
# spacehogs-bypartition HP-
# For Linux use /bin/mount | awk '$5 != "nfs" { print $3 }'.

for p in `/usr/bin/cat /etc/fstab | awk '$3 != "nfs" { print $2 }'`
do
echo "Usage on partition $p"
find $p -xdev -type f -exec /bin/ls -ls {} ';' | awk '
{ using[$4] += $1 }
END { for (name in using)
{ print using[name], name }}' |
sort -nr |
head
done
exit 0

[Spark_2]

But we can identify highest CPU and Memory users....correct?...there are those unix95 commands...can someone please paste them

and also something about disk issue

[Johnson Punniyalingam]

Hi Spark,

please check attached file


# UNIX95= ps -ef -o pid,sz,vsz,args

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1353245


# sar -d 5 50
-- What is the disk I/O load?

[Johnson Punniyalingam]

$ UNIX95= ps -efo vsz,ruser,pid,ppid,args | sort -rn | more

[Michael Steele_2]

UNIX95 is an excellent tool for analyzing processes. However, it relys on the -o options found in the 'ps' command and the 'ps' command neither measures I/O or disk utilization.

I do think its worth your while using the vsz and pcpu, state, wchan, args, pid, ppid arguements though. And becomming familar with the biggest users of virtual memory on CPU time. State will be run, sleep, wait, etc. Whan will till you what pid is holding up a waiting process. All are good metrics.

[Bill Hassell]

Because there are so few disks on this system, the issue is very likely an Oracle configuration issue and possibly some very poor database procedures. For good performance, you need a lot of RAM and have Oracle configured to use a large SGA to cache disk activities. Setting up SGA requires a good Oracle DBA with performance tuning skills. The DBA should also be able to identify inefficient SQL statements as well as queries that use sequential searches (which means very busy disks).