I/O error entries

[jeena_2]

Hi,

syslog given the below error message,please help on this......

Oct 5 13:39:58 phxdev7 vmunix: DIAGNOSTIC SYSTEM WARNING:
Oct 5 13:39:58 phxdev7 vmunix: The diagnostic logging facility is no longer receiving excessive
Oct 5 13:39:58 phxdev7 vmunix: errors from the I/O subsystem. 8 I/O error entries were lost.
Oct 5 13:39:58 phxdev7 vmunix: The diagnostic logging facility has started receiving excessive
Oct 5 13:39:58 phxdev7 vmunix: errors from the I/O subsystem. I/O error entries will be lost
Oct 5 13:39:58 phxdev7 vmunix: until the cause of the excessive I/O logging is corrected.
Oct 5 13:39:58 phxdev7 vmunix: DIAGNOSTIC SYSTEM WARNING:

Thanks in Adv.

[Steven E. Protter]

Shalom,

I think you have lost a disk or controller.

dmesg

grep lbolt /var/adm/syslog/syslog.log

SEP

[jeena_2]

Hi Steven,

see the details below,

/bin/ksh> dmesg

Oct 5 14:46
...
stic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 5 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 5 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 4 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 3 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 5 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 4 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
root@phxdev7GB.fmi.com:/home/pkeeriya
/bin/ksh> dmesg

Oct 5 14:46
...
ging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 1 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 1 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 2 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 5 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 15 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 3 I/O error entries were lost.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem. I/O error entries will be lost
until the cause of the excessive I/O logging is corrected.
If the diaglogd daemon is not active, use the Daemon Startup command
in stm to start it.
If the diaglogd daemon is active, use the logtool utility in stm
to determine which I/O subsystem is logging excessive errors.
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility is no longer receiving excessive
errors from the I/O subsystem. 11 I/O error entries were lost.


/bin/ksh> grep lbolt /var/adm/syslog/syslog.log
root@phxdev7GB.fmi.com:/home/pkeeriya

[chris huys_4]

Hi,

"format" the binary "current" disk log, i.e. /var/stm/logs/os/logX.raw.cur, with the cstm utility, to a readable ascii file.

Procedure can be found in :
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=809057

NOTE: in the above thread its the "t3" file that is the ascii readable text that can then be vi'd to check what is going on..

Greetz,
Chris

[jeena_2]

Hi Chris,

i couldn't able to find out which IO device has problem..please help....

.... phxdev7.fmi.com : 10.180.32.104 ....

-- Logtool Utility: View Formatted Summary --

Summary of: /var/stm/logs/os/log2267.fmt2
Formatted from: /var/stm/logs/os/log2267.raw.cur


Date/time of first entry: Wed Jan 6 17:42:43 2010

Date/time of last entry: Tue Oct 5 14:15:11 2010



Number of LPMC entries: 0
Number of System Overtemp entries: 0
Number of LVM entries: 0
Number of Logger Event entries: 23

Number of I/O Error entries: 669


Device paths for which entries exist:
Device paths for which entries exist:

(2) 0/5/1/0
(2) 0/4/1/0
(24) 0/5/1/0.2.16.0.0.0.2
(23) 0/5/1/0.2.16.0.0.0.1
(21) 0/5/1/0.2.0.0.0.0.6
(18) 0/4/1/0.1.0.0.0.0.6
(10) 0/4/1/0.1.16.0.0.0.6
(18) 0/5/1/0.2.16.0.0.0.6
(22) 0/5/1/0.2.16.0.0.0.5
(26) 0/4/1/0.1.0.0.0.0.2
(29) 0/5/1/0.2.0.0.0.0.2
(31) 0/4/1/0.1.16.0.0.0.2
(36) 0/5/1/0.2.0.0.0.0.1
(38) 0/4/1/0.1.16.0.0.0.1
(38) 0/4/1/0.1.0.0.0.0.1
(21) 0/5/1/0.2.16.0.0.0.4
(12) 0/4/1/0.1.0.0.0.0.5
(22) 0/5/1/0.2.0.0.0.0.5
(32) 0/4/1/0.1.16.0.0.0.5
(15) 0/4/1/0.1.16.0.0.0.4
(21) 0/5/1/0.2.0.0.0.0.4
(20) 0/4/1/0.1.0.0.0.0.4
(44) 0/4/1/0.1.16.0.0.0.3
(38) 0/5/1/0.2.0.0.0.0.3
(59) 0/5/1/0.2.16.0.0.0.3
(47) 0/4/1/0.1.0.0.0.0.3

Products for which entries exist:

(4) Fibre Channel Interface
(665) SCSI Disk

Product Qualifiers for which entries exist:

(4) HP6795A_Tachyon_XL2
(665) DGCCX4-480WDR5

Logger Events for which entries exist:

(4) td
(665) sdisk

Device Types for which entries exist:

(4) Interface
(665) Disk

[chris huys_4]

Hi,

>-- Logtool Utility: View Formatted Summary --

This is the output of the formatted summary, instead of the output of the formatted log, I was hinting on.

In "steno"

run cstm
cstm>ru
2 (for logtool)

fr (format raw)
[RETURN] select default directory
q (quit)
Done

fl (View formatted log)
q (quit)
SA (Save As) /var/tmp/t3.out (this file will be ascii viewable)
Done

quit logtool/cstm
quit
OK
quit
OK

And then a "#tail -500 /var/tmp/t3.out" to see what is going on.

NOTE: From the formatted summary output, looks like 1 of the emc clariion diskarray luns has problems..

Greetz,
Chris