CSTM output

[Asif Chaudhary]

Hi,

I am getting this error in CSTM. Can any one help me about it. Actully the main problem is 100% disk utlization and performance is dead slow.

# sar -d 5 5

HP-UX hpux B.11.23 U 9000/800 12/06/06
04:21:18 device %busy avque r+w/s blks/s avwait avserv

04:21:23 c2t0d0 95.41 0.50 1 319 6.74 2111.54

04:21:28 c2t0d0 100.00 0.50 22 8698 0.00 70.93

04:21:33 c2t0d0 85.03 0.50 3 339 0.00 992.50

04:21:38 c2t0d0 94.00 0.50 3 342 0.00 1029.58

04:21:43 c2t0d0 89.58 0.72 17 574 15.23 262.12

++++++++++++++++++++++++++++++++++++++++++++++
CSTM OUT PUT ERROR
++++++++++++++++++++++++++++++++++++++++++++++
------------ I/O Module Error Log Information ------------

IO Subsystem Log Entries

Found 1 IOC error
------------------------------------------------

Detail display of IO subsystem log entries
------------------------------------------

IOC Error information

IOC Error 1
--- Section Header ---
GUID
data1 0xe429faf7
data2 0x3cb7
data3 0x11d4
data4 0xbc a7 0 80 c7 3c 88 81
REVISION 0x0200
ERROR_RECOVERY_INFO 0x80
SECTION_LENGTH 0x000000b8
VALIDATION_BITS 0x0000000000000047
ERROR_STATUS 0x00000000005a0000
REQUESTOR_ID 0xfffffffffed00000
RESPONDER_ID 0x00000000fed2e000
TARGET_ID 0x0000000000000000
BUS_DATA 0x0000000000000000
OEM_COMPONENT_ID 0x000000000105b270
HP_DEV_PATH 0x0000000000000000

HP IOC Information
CEC Header:
--- Section Header ---
GUID
data1 0x13276c76
data2 0x37de
data3 0x42e9
data4 0x a5 2f 41 89 ba 10 dd ed
SECTION_LENGTH 0x00000058
CELL_NUMBER 0
SBA_NUMBER 0
ROPE_NUMBER 7
CEC Data:
CEC Pluto Data:
ROPE_CONFIG 0x0000000000000400
ROPE7_ERROR 0x0000000000000210
ROPE7_REQ_ERR_LOG 0x0000000000000000
ROPE7_IBF_ERR_LOG 0x0000000000000000
ROPE7_DID_LOG 0x0000000000101000
ROPE7_ENABLE 0x00000000000006f7
ROPE7_CONFIG 0x001200055ffff8a4
HP Mercury Rope Data
CEC Mercury Data:
ROPE_ERROR_LOG 0x0000000000000400


End of IOC Error Information for Error 1

End of IOC Error Information

Thanks.
AFA

[Sameer_Nirmal]

The error information points to the malfunctioning hardware, maybe a LBA or the PCI card in rope 7. The ROPE 7 would point to a PCI slot.

You haven't mentioned the server model. Run logtool as given below and take a look at its output. If you want you can post it here.

#Print the formatted logtool to /tmp directory
#
# Syntax: 'cstm -f logtool.txt'
#
ru
logtool
rs
saveas
/tmp/logtool.summary
done
fl
/var/stm/logs/os
saveas
/tmp/logtool.formatted
done
quit
ok
quit
ok

[Michael Steele_2]

Hi AFA:

Regarding disk c2t0d0 in your 'sar -d' report and the extremely high avserv numbers along with virtually no avwait numbers, High numbers of block reads and writes per second ( r+w/s ). For example the first line:

95.41 = %busy

0.50 = avque ( not high, not blocking )

1 = r+w/operations / second ( one operation )

319 = r+w/blocks / second ( high, one operation is writing 319 blocks )

6.74 = avwait ( small compared to avserv, no disk bottlenec )

2111.54 = avserv ( high, slow responding disk )

One disk operation on a slow disk is writing 319 blocks. The device is 95% consumed by this operation.

Can you "pvdisplay -v /dev/dsk/c2t0d0 | more" and note the file systems on this disk? Can you also provide a "diskinfo -v /dev/rdsk/c2t0d0"? And finally, the logtool report.

/usr/sbin/cstm<<-EOF
runutil logtool
rs
EOF

The IOC error is odd. IOC is your Input Output Controller. Is this a linux PA-Risc box?

%busy: This is the percentage of time the device is being read from or written to.

avque: This is the average depth of the queue that is used to serialize disk activity. The higher the avque value, the more blocking is occurring.

r+w/s, blks/s: This is disk activity per second in terms of read or write operations and disk blocks, respectively.

avwait: This is the average time (in milliseconds) that a disk read or write operation waits before it is performed.

avserv: This is the average time (in milliseconds) that a disk read or write operation takes to execute.

[Andrew Merritt_2]

AFA,
Just a comment, you've got the same message posted in two places, it's normally enough to post one (duplicates will usually be removed).

How is your investigation going? Several questions have been asked in both threads, can you give any of the requested data that might help the posters here suggest answers or further lines of enquiry?


> The IOC error is odd. IOC is your Input
> Output Controller. Is this a linux PA-Risc box?

The evidence in the output above is that this is an HP-UX box (running CSTM, for a start, and "HP-UX hpux B.11.23" in the sar output). Is it a PA or IA box?

Andrew