topic Re: I/O Error Message in Operating System - HP-UX

I/O Error Message

Dineshkumar Surpur — Thu, 19 Feb 2004 13:14:39 GMT

I am getting the following error message
in the the syslog

--------------------------------------------
vmunix: DIAGNOSTIC SYSTEM WARNING:
vmunix: The diagnostic logging facility has started receiving excessive
vmunix: errors from the I/O subsystem. I/O error entries will be lost
vmunix: until the cause of the excessive I/O logging is corrected.
vmunix: If the diaglogd daemon is not active, use the Daemon Startup command
vmunix: in stm to start it.
vmunix: If the diaglogd daemon is active, use the logtool utility in stm
vmunix: to determine which I/O subsystem is logging excessive errors.

--------------------------------------------

When i use the stm tool which has the suboption
for logtool we get this data

Entry Type: I/O Error
Entry logged on Tue Feb 17 11:53:52 2004
Entry id: 0x4032634000000000

Device Path: 0/1/1/0.1.5.0.1.12.1
Product: SCSI Disk
Product Qualifier: Volume
Logger: sdisk

Entry Data:

0 1 2 3 4 5 6 7 8 9 ASCII
00000000: 00 00 00 10 00 00 00 04 00 00 ..........
00000010: 00 08 00 00 00 1e 00 00 00 28 .........(
00000020: 00 00 00 00 24 01 a5 05 00 00 ....$.....
00000030: 00 00 00 00 00 28 00 00 00 00 .....(....
00000040: 24 0c 01 00 cd 00 00 06 1b 00 $.........
00000050: 00 00 01 00 10 00 00 00 00 00 ..........
00000060: 00 b6 00 00 00 00 00 00 00 f6 ..........
00000070: 82 68 00 00 00 00 00 00 00 00 .h........
00000080: 00 00 00 00 00 b5 d8 00 ff ff ..........
00000090: ff ff ff fa 00 00 00 00 00 00 ..........
00000100: 00 00 00 00 00 00 00 00 0c b3 ..........
00000110: c4 00 00 00 00 00 0f ac b8 00 ..........
00000120: 00 00 00 00 00 03 4b fc 00 00 ......K...
00000130: 00 00 00 00 00 00 00 00 00 00 ..........
00000140: 41 35 ef 50 00 00 00 00 00 00 A5.P......
00000150: 00 00 00 00 00 00 00 5f 58 48 ......._XH
00000160: 00 00 00 00 40 16 f2 c0 00 00 ....@.....
00000170: 00 00 41 36 00 d8 00 00 00 00 ..A6......
00000180: 41 21 a0 00 00 00 00 00 00 00 A!........
00000190: 00 25 00 00 00 00 41 36 01 80 .%....A6..
00000200: 00 00 00 00 00 00 00 25 00 00 .......%..
00000210: 00 00 48 b6 a8 00 00 00 00 00 ..H.......
00000220: 00 00 00 7f 00 00 00 00 00 00 ..........
00000230: 00 04 00 00 00 00 50 6f 79 e0 ......Poy.
00000240: 00 00 00 00 00 00 00 04 00 00 ..........
00000250: 00 00 00 00 00 01 00 00 00 00 ..........
00000260: 00 b5 b8 00 00 00 00 00 00 b6 ..........
00000270: 00 00 00 00 00 00 00 f6 82 68 .........h
00000280: 00 00 00 00 00 00 00 00 00 00 ..........
00000290: 00 00 00 b5 d8 00 00 00 00 00 ..........
00000300: 00 16 52 ec 00 00 00 00 00 00 ..R.......
00000310: 00 01 3f 62 9e 41 29 e4 12 9e ..?b.A)...
00000320: 00 00 00 00 29 e4 12 9e 00 00 ....).....
00000330: 00 00 00 00 00 00 00 00 00 00 ..........
00000340: 00 00 00 00 00 00 00 00 00 99 ..........
00000350: be c0 00 00 00 00 00 00 00 00 ..........
00000360: 00 00 00 00 00 00 00 00 00 00 ..........
00000370: 00 00 00 00 00 00 00 00 00 00 ..........
00000380: 00 00 00 00 00 00 00 00 00 00 ..........
00000390: 00 00 00 00 00 00 00 00 00 00 ..........
00000400: 00 00 00 00 00 00 00 00 3f 62 ........?b
00000410: 9e 41 29 e4 12 9e 40 51 2a aa .A)...@Q*.
00000420: aa aa aa b0 00 00 00 00 00 00 ..........
00000430: 00 44 00 00 00 00 00 ff 10 00 .D........
00000440: 3f 90 ec db f5 25 75 00 3f f5 ?....%u.?.
00000450: c3 4d 80 90 60 e5 3f f2 b2 17 .M..`.?...
00000460: 4a 43 7f 57 00 00 00 00 00 00 JC.W......
00000470: 00 08 00 00 00 00 00 00 00 00 ..........
00000480: 41 10 03 7a cc cc cc d2 00 00 A..z......
00000490: 00 00 00 1a dd d0 00 00 00 00 ..........
00000500: 00 99 bf 58 00 00 00 00 00 16 ...X......
00000510: 52 ec 00 00 00 00 00 f6 50 00 R.......P.
00000520: 00 00 00 00 00 00 00 00 ........

Can anybody tell from this raw data what sort of error this is ?

Re: I/O Error Message

Steven E. Protter — Thu, 19 Feb 2004 13:20:06 GMT

if you run dmesg or check /var/adm/syslog/syslog.log you may see some scsi reset or timeouts.

If you see an lbolt, you are about to lose a disk.

I am attaching a checker script.

SEP

Re: I/O Error Message

Dineshkumar Surpur — Thu, 19 Feb 2004 14:38:05 GMT

In the dmesg and syslog.log i do not get any reset or lbolt. We are using vxvm volume
manager and have around 256 disks admitted
to VxVM (Veritas) volume manager. It was seen
that when system is coming up and when VxVM is initializing the disks we see this message
once only but we do not loose the disk,
I/O activity is fine . Veritas volume manager does inquiry and reads of the disks at initialization time.It is possible that I/O subsystems thinks this as invalid I/O operation and logs as error. Is there possible way from the given output data what sort of I/O operation is been performed.

Re: I/O Error Message

Andrew Merritt_2 — Fri, 20 Feb 2004 05:45:35 GMT

You can get more information by if instead of selecting to View the Raw Log in logtool, you select View Formatted Log. That should interpret the event for you, giving some more detail about what the event is caused by.

Andrew

Re: I/O Error Message

T G Manikandan — Fri, 20 Feb 2004 06:06:33 GMT

The disk on the hardware path Device Path: 0/1/1/0.1.5.0.1.12.1
has some problem.

Just do a
ioscan -fnC disk
to find the disk in that path.

Doing a ioscan should log messages into syslog.log file

Re: I/O Error Message

Paula J Frazer-Campbell — Fri, 20 Feb 2004 06:07:25 GMT

Hi

The disk 0/1/1/0.1.5.0.1.12.1 is reporting I/O errors and should not be trusted.

The:-
Entry Data:

0 1 2 3 4 5 6 7 8 9 ASCII
00000000: 00 00 00 10 00 00 00 04 00 00 ..........
00000010: 00 08 00 00 00 1e 00 00 00 28 .........(
00000020: 00 00 00 00 24 01 a5 05 00 00 ....$.....
00000030: 00 00 00 00 00 28 00 00 00 00 .....(....
00000040: 24 0c 01 00 cd 00 00 06 1b 00 $.........
00000050: 00 00 01 00 10 00 00 00 00 00 ..........
00000060: 00 b6 00 00 00 00 00 00 00 f6 ..........
00000070: 82 68 00 00 00 00 00 00 00 00 .h........
00000080: 00 00 00 00 00 b5 d8 00 ff ff ..........
00000090: ff ff ff fa 00 00 00 00 00 00 ..........
00000100: 00 00 00 00 00 00 00 00 0c b3 ..........
00000110: c4 00 00 00 00 00 0f ac b8 00 ..........
00000120: 00 00 00 00 00 03 4b fc 00 00 ......K...
00000130: 00 00 00 00 00 00 00 00 00 00 ..........

Is not the error info but the data that was being moved when the error occured.

1. Backup this disks data.
2. Watch it very closly.
3. Have a disk ready to replace it.

Paula

Re: I/O Error Message

Dineshkumar Surpur — Fri, 20 Feb 2004 17:31:09 GMT

Here are some of my repies.

1. The ioscan shows the disk as claimed
so no problem accessing the disks.

2. The fomatted log doesn't tell
what sort of error it is
ex:

Entry Type: I/O Error
Entry logged on Wed Feb 18 09:56:04 2004
Entry id: 0x4033a7340000001a

Device Path: 1/0/0/0.2.16.0.0.9.2
Product: SCSI Disk
Product Qualifier: Storage Volume Name
Logger: sdisk
Device Type: Disk
Device Qualifier: Hard

3. I kind of agree with Paula that it is somewhat data since VxVM does some read
operation during initilization and i don't
understand why the OS see this data operations as error.

It is possible that I/O subsystem thinks
data operations as bad since this is not
user data but read data by VxVM or what under circumstances does hpux I/O susbsystem thinks this data as error ?

Also paula can you tell if the data is for read scsi command