high disk usage -need to identify cause-how to identify open files

[Karl oliver]

Can any one help here please
we have high disk busy on 1 disk see sar below
Average c2t0d0 100.00 67.32 8 95 3.09 6.96
this is a mirror disk but only one disk showing high.
Can identify cause from glance ,no errors in syslog.
NO runaway progs.
file sys on this disk are /var /usr.
The prob has just happened in last few days.Is there anyway to find out what fil is in use on file system.
Fuser only seems to show proc id using file sys not file it has open

[Rodney Hills]

lsof is a must tool for system administration. It can tell you what files are open and what seek position they are currently at.

You can get it from the porting and archive centre-
http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.76/

HTH

-- Rod Hills

[baiju_3]

FS are /usr and /var so disk is a part of root VG and is internal disk .

Use lvdisplay on the logical volume and make sure no stale PE is existing .

If not a critical system try rebooting .


also can use lsof and search for the process using these file systems .Observer the process size in the o/p , if you see any process abnormal kill it .

lsof is a third party tool , you may have to down load it , but is a good tool.


thx,
bl.

[Sandman!]

Get lsof if you don't have it from...

http://hpux.connect.org.uk

usage:

# lsof /var /usr

cheers!

[Karl oliver]

we are currently trying lsof - is it a possible hardware error thats not showing in syslog

[Michael Jorgensen]

Karl,

If only one side of the mirror is getting hammered you might have something hardware wise going on. Are the disks exactly the same specifications? What is the other disk in the mirror performing like at the exact same time as this disk is performing badly?

Lsof is pretty good for listing open files. Another way of figuring out all the files that are being created is to do something like:

find / -type f -mtime -1 -exec ls -la {} \;

This will show you all files modified in the last day and you can check the timestamps to see when they're created. Or store the output away and then run it again a few minutes later during peak times and do a diff on the output...

[morganelan]

Disk failures typically get noticed in one of three ways:
-I/O errors in the system log Often an error message in the system log file is your first indication that thereâ s a disk problem. In /var/adm/syslog/syslog.log, you may see an error like this:
SCSI: Request Timeout -- lbolt: 329741615, dev: 1f022000

-Disk failure notification messages from diagnostics If you have EMS hardware monitors installed on your system, and youâ ve enabled the disk monitor disk_em, a failing disk may trigger an event to the Event Monitoring Service (EMS). Depending on how youâ ve configured EMS, you can get an electronic mail message, information in /var/adm/syslog/syslog.log, or messages in another log file. EMS error messages identify a hardware problem, what caused it, and what must be done to correct it.

-LVM command errors Sometimes the LVM commands, like the vgdisplay command, will return an error suggesting that a disk has problems. Here are some examples:
# vgdisplay â v | more
--- Physical volumes --- PV Name /dev/dsk/c0t3d0 PV Status unavailable Total PE 1023 Free PE 173 â ¦ The physical volume status of â unavailableâ indicates that LVM is having problems with the disk. You can get the same status information from pvdisplay. The next two examples are warnings from vgdisplay or vgchange indicating that LVM has no contact with a disk:

[morganelan]

Confirming Disk Failure:
-Use the ioscan command to check the diskâ s S/W state. Only disks in state CLAIMED are currently accessible by the system. Disks in other states like NO_HW or disks that are completely missing from the ioscan output are suspicious. If the disk is marked as CLAIMED, then at least its controller is responding.

-If the disk responds to an ioscan, test it with the diskinfo command. The reported size must be nonzero, otherwise the device is not ready for some reason. For example:
#diskinfo /dev/rdsk/c2t0d0
If the size is 0, so the disk is malfunctioning.

-If both ioscan and diskinfo succeed, the disk may still be suspect. As a final test, try to read from the disk using the dd command. Depending on the size of the disk, a comprehensive read may be time-consuming, so you may only want to read a portion of the disk. No I/O errors should be reported. For example, read the first 64 megabytes:
# dd if=/dev/rdsk/c2t0d0 of=/dev/null bs=1024k count=64
64+0 records in
64+0 records out

Read the whole disk:
# dd if=/dev/rdsk/c2t0d0 of=/dev/null bs=1024k
dd read error: I/O error
0+0 records in
0+0 records out

[David de Beer]

We use 'dmesg' to identify these type of errors. You'll see SCSI errors if there is something wrong with your disk/controller.

Have a script that runs a 'dmesg -' and mails me the output every day, if a hardware error occurs it's usually easily identified. You could even have your script check for a blank message and only mail you when there's output from 'dmesg -'.

Regards,
David.

[Karl oliver]

Thanks for your suggestions people. We have had an HP Tech in and he cant find any errors in disk. We suppect sar might be telling lies about the 100 % dis busy
The q at 65 is right though - it agrees with glance . So we are going to reboot system and see what happens

[Karl oliver]

Reboot of system fixed this problem ,I guess we will never know the cause