monitor the i/o of a disk

[Ratzie]

I recently had a failure in a SCSI controller that I feel may be software related. Or as I was told, the I/o requested bombarded the controller and it servered the connection. This is the error in syslog...
Jul 30 15:56:00 server vmunix: SCSI: Ultra160 Controller at 0/10/0/0: Error: The domain validation test for target 1 determined that communication may not be possible to this target. Verify the hardware configuration.

Now I got no alarms from the hardware monitoring, nor was there any lights on the server.

It was suggested that I increase the I/O requests on the drive from the default 30 msec to 180 msec.

What I want to do is run some tests. Is the a way to monitor the I/O requests. If it does fail again I want to be able to go back and check if the I/O was high at that point.

Just to mention that I did not have a ts99 file that showed any errors. It was created after the reboot, and the ts98 file had all zeros.
pvdisplay displayed no errors, all claimed. no stale extents.

So basically there is no record of any errors except what was in dmesg and syslog.
cstm did record that there were over 600 i/o error entries, but the funny thing is that line where it states last entry is dated last month so 1 month before our failure.

[Brian M Rawlings · Accepted solution]

If you get too granular, you can kind of kill yourself with this, but an interval of 300 (seconds) is generally about as frequent as you'd want to monitor most of the time (a snapshot every five minutes). Too often, and you start to affect what you're trying to measure, never a good idea.

sar -A


DD is the 2 digits of the day of the month, 01-31.

See the 'sar' man page for details, and for other more specific sar switches that you might care to employ.

If you have frequent problems, you might want to go ahead and make the monitor interval shorter, like maybe 60 (seconds), if you're trying to capture events right before an error occurs, but you won't want to run this way too long, and be cognizant that some of the CPU stats may be skewed by your 'sar' processing.

Regards, --bmr

[Con O'Kelly]

Hi

You can check historical disk i/o using sar.
# sar -d
(shows i/o data for today.)
# sar -d -f sa29
(shows historical data for 29th of month - in this case july 29.)

To collect hitorical sar data you need an entry in crontab as follows:
0,15,30,45 * * * * /usr/lbin/sa/sa1

This will collect sar data every 15mins. You also need to create the /var/adm/sa directory.

Cheers
Con

[Steven E. Protter]

Attached is a full suite of scripts to collect data over time. Low overhead, tested with years of data collection.

SEP