sar works, sar -d doesn't

[Jared Middleton]

I have a strange situation on one of my three Linux servers... sar -d doesn't work anymore... it used to.

Here's the problem:
$ sar -d
Requested activities not available in file <-- why getting this message?

On all 3 Linux machines, the sa data collection stuff is running every 10 minutes (via cron):

$ cat /etc/cron.d/sysstat
# run system activity accounting tool every 10 minutes */10 *
* * * root /usr/lib/sa/sa1 1 1 # generate a daily summary of
process accounting at 23:53
53 23 * * * root /usr/lib/sa/sa2 -A

Plain "sar" (or sar -A) is correctly reporting current stats for CPU, network, etc. on all 3 machines. However, on the one server, "sa" appears to be capturing everything expected EXCEPT the disk/block device (-d) stats. I can't figure out why.

Just for fun, I even tried the MS-Windows-all-purpose-problem-resolver (reboot) and that didn't help.

What do I need to do to get this server to resume capturing "sar" block device data?

$ sar -V
sysstat version 4.0.3

Red Hat Linux 7.3 Professional
Kernel: 2.4.20-20.7bigmem

Also...

"iostat -d" also comes up blank on that server:
$ iostat -d
Linux 2.4.20-20.7bigmem (test) 07/06/2004

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

Whereas, on the Production server:
$ iostat -d
Linux 2.4.20-20.7bigmem (prod) 07/06/2004

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev3-0 0.00 0.01 0.00 101360 0


ADDITIONAL INFO:
I just went to the console of the problem server and noticed the following displayed 6 times:
cciss: cmd f7780000 has CHECK CONDITION, sense key = 0x3

Note:
- It's been about *6* days since I rebooted (probably not a coincidence)
- I believe "cciss" is the Compaq SCSI driver

Even though there has been no real run-time problems reported, maybe the machine just recently started experiencing some problem with a SCSI device or controller?

HP/Compaq ProLiant DL580 G2
Compaq Smart Array 5312

The equipment should still be under 3-year warranty. Unfortunately, I don't have much experience diagnosing low-level hardware errors. A quick Google didn't reveal much. Anyone have a clue as to what to look at next?

Jared

[K.C. Chan]

Jared,
I ran into this problem as well. It was resolved when I recompiled/get a kernel version which has this option turn on.

sar -d and iostat -d are utilities which hooks into your kernel. There is an option in the kernel to turn this on. Is this a custome kernel?

[Ragu_3]

My guess is that there might be some startup script which has to be enabled, to get sadc working. Doing an `sar -d' may not be enough.

[Jared Middleton]

The kernel is: 2.4.20-20.7bigmem on both the Production server and the Test server (that has the "sar -d" problem). The machines are nearly identically configured with minor differences that I'm sure are of no significance here. Besides, they have all been working fine for two years, and then this one just stopped.

I'm 99% sure the problem has to do with the console error message "cciss: cmd f7780000 has CHECK CONDITION, sense key = 0x3" which seems to relate to SCSI. This implies that the "sar" issue is just a secondary symptom.

I would still appreciate any advice dealing with that error though.

Jared