vmunix: SCSI: Read error -- dev: b 31 0x060100, errno: 126, resid: 2048,

[OLIVA_1]

Hello,

We have one L3000 and two L2000 connected to a SAN EMC, and we have the errors below in /var/adm/syslog/syslog.log file on 3 servers...
Someone can help me ?


Thanks,

Dec 7 12:51:28 cvdpar1 vmunix: SCSI: Write error -- dev: b 31 0x010000, errno: 126, resid: 2048,
Dec 7 12:51:28 cvdpar1 vmunix: blkno: 16265282, sectno: 32530564, offset: -524220416, bcount: 2048.
Dec 7 12:51:28 cvdpar1 vmunix: blkno: 16273496, sectno: 32546992, offset: -515809280, bcount: 2048.
Dec 7 12:51:28 cvdpar1 vmunix: blkno: 14929446, sectno: 29858892, offset: -1892116480, bcount: 2048.
Dec 7 12:51:28 cvdpar1 vmunix: blkno: 14844472, sectno: 29688944, offset: -1979129856, bcount: 2048.
Dec 7 12:51:28 cvdpar1 vmunix: blkno: 14683404, sectno: 29366808, offset: -2144063488, bcount: 2048.
Dec 7 12:51:28 cvdpar1 vmunix: SCSI: Read error -- dev: b 31 0x010000, errno: 126, resid: 2048,
Dec 7 12:51:28 cvdpar1 vmunix: SCSI: Write error -- dev: b 31 0x010000, errno: 126, resid: 2048,
Dec 7 12:51:29 cvdpar1 above message repeats 4 times
Dec 7 12:51:28 cvdpar1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.
Dec 7 12:51:28 cvdpar1 vmunix: LVM: vg[1]: pvnum=0 (dev_t=0x1f010000) is POWERFAILED
Dec 7 12:51:28 cvdpar1 vmunix:
Dec 7 12:51:33 cvdpar1 above message repeats 5 times
Dec 7 12:51:33 cvdpar1 vmunix: LVM: Recovered Path (device 0x1f010000) to PV 0 in VG 1.
Dec 7 12:51:34 cvdpar1 vmunix: LVM: Restored PV 0 to VG 1.
Dec 7 12:52:02 cvdpar1 vmunix: LVM: Recovered Path (device 0x1f010000) to PV 0 in VG 1.
Dec 7 12:52:03 cvdpar1 vmunix: LVM: Restored PV 0 to VG 1.
Dec 7 12:52:59 cvdpar1 vmunix:
Dec 7 12:52:59 cvdpar1 vmunix: SCSI: Read error -- dev: b 31 0x010000, errno: 126, resid: 2048,
Dec 7 12:52:59 cvdpar1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.
Dec 7 12:52:59 cvdpar1 vmunix: LVM: vg[1]: pvnum=0 (dev_t=0x1f010000) is POWERFAILED
Dec 7 12:53:04 cvdpar1 vmunix: LVM: Recovered Path (device 0x1f010000) to PV 0 in VG 1.
Dec 7 12:53:04 cvdpar1 vmunix: LVM: Restored PV 0 to VG 1.
Dec 7 12:53:35 cvdpar1 vmunix:
Dec 7 12:53:35 cvdpar1 vmunix: SCSI: Write error -- dev: b 31 0x010000, errno: 126, resid: 2048,
Dec 7 12:53:35 cvdpar1 vmunix: blkno: 15888174, sectno: 31776348, offset: -910379008, bcount: 2048.
Dec 7 12:53:35 cvdpar1 vmunix: SCSI: Read error -- dev: b 31 0x010000, errno: 126, resid: 2048,
Dec 7 12:53:35 cvdpar1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.
Dec 7 12:53:35 cvdpar1 vmunix: LVM: vg[1]: pvnum=0 (dev_t=0x1f010000) is POWERFAILED

[Steven E. Protter]

Is vg01 on the SAN or local.

Either way a disk has failed or been made unavailable by the SAN.

This can happen when you swap out fiber cards and don't update the san/array with the new worldwide name WWN available from fcmsutil

ex

fcmsutil /dev/td0

Get a good backup and proceed. If its local disk, get a replacement in.

SEP

[Sanjay_6]

Hi,

Looks like disk c0t1d0 has got problems,

0x010000 --> c0t1d0

do a dd on the disk and check and see if this disk has problems.

dd if=/dev/rdsk/c0t1d0 of=/dev/null bs=1024k

Hope this helps.

Regds

[OLIVA_1]

Ok, ok,

But on another server I have this error message:

Dec 7 17:03:05 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060100, errno: 126, resid: 2048,
Dec 7 17:03:05 cvppar1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.
Dec 7 17:03:10 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060200, errno: 126, resid: 2048,
Dec 7 17:03:30 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060200, errno: 126, resid: 2048,
Dec 7 17:03:40 cvppar1 above message repeats 2 times
Dec 7 17:03:40 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060200, errno: 126, resid: 2048,
Dec 7 17:03:35 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060100, errno: 126, resid: 2048,
Dec 7 17:03:41 cvppar1 above message repeats 3 times
Dec 7 17:03:45 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060100, errno: 126, resid: 2048,
Dec 7 17:03:45 cvppar1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.
Dec 7 17:03:50 cvppar1 above message repeats 8 times
Dec 7 17:03:50 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060200, errno: 126, resid: 2048,
Dec 7 17:03:50 cvppar1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.
Dec 7 17:03:55 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060100, errno: 126, resid: 2048,
Dec 7 17:05:25 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060100, errno: 126, resid: 2048,
Dec 7 17:05:32 cvppar1 above message repeats 9 times
Dec 7 17:05:35 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060100, errno: 126, resid: 2048,
Dec 7 17:05:45 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060100, errno: 126, resid: 2048,
Dec 7 17:06:00 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060200, errno: 126, resid: 2048,
Dec 7 17:06:06 cvppar1 above message repeats 13 times
Dec 7 17:06:06 cvppar1 syslog: INFO : nb indicators from concord : 158846 nb files: 5 nb elements: 27193
Dec 7 17:06:11 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060200, errno: 126, resid: 2048,
Dec 7 17:06:36 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060100, errno: 126, resid: 2048,
Dec 7 17:06:45 cvppar1 above message repeats 5 times
Dec 7 17:06:41 cvppar1 vmunix: SCSI: Read error -- dev: b 31 0x060200, errno: 126, resid: 2048,
Dec 7 17:06:45 cvppar1 above message repeats 3 times
Dec 7 17:06:41 cvppar1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.
Dec 7 17:06:45 cvppar1 above message repeats 34 times

[Sanjay_6]

Hi,

Device 0x060100 --> c6t0d1

Try a dd on this too like the previous one. change the lun id

Hope this helps.

Regds

[OLIVA_1]

Ok thank you.

These 2 disks /dev/dsk/c6t0d1 and /dev/dsk/c6t0d2 are SAN disks...

[Sanjay_6]

Hi,

You may want to check on the JFS patches you have on the system. Check for JFS patches for your version of OS and JFS.

http://www2.itrc.hp.com/service/patch/search.do?BC=patch.breadcrumb.main|&pageContextName=hpux:::

Hope this helps.

Regds

[Stuart Abramson]

Here is how you decode these addresses. You get different addresses on different servers because the same device (or a device on the same switch port or FA) will have different addresses on each machine.

Once you identify the disk, do a syminq or inq on the disk, get his serial number, and find out the common elements.

1. Get the "dev:" entry from the lbolt:

# dmesg | grep lbolt | grep dev:

SCSI: Abort -- lbolt: 18346341, dev: e7015000, io_id: 122e9a3
SCSI: Request Timeout -- lbolt: 18351441, dev: e7015000
SCSI: Abort -- lbolt: 18351441, dev: e7015000, io_id: 122e9be
SCSI: Request Timeout -- lbolt: 18356641, dev: e7015000
SCSI: Abort -- lbolt: 18356641, dev: e7015000, io_id: 122e9cf
SCSI: Request Timeout -- lbolt: 18362141, dev: e7015000
SCSI: Abort -- lbolt: 18362141, dev: e7015000, io_id: 122e9e0
SCSI: Request Timeout -- lbolt: 74105435, dev: 1f000000
SCSI: Abort Tag -- lbolt: 74105435, dev: 1f000000, io_id: 4ead34

Here we have two:

1f
e7

2. This is the major number of the device in question. Convert the first
two digits of the device from hex to decimal:

# printf "%#d\n" 0x1f
31

3. find out what driver this major number is. It tells us the type of
device:

# lsdev 31

Character Block Driver Class
188 31 sdisk disk

So, this is probably a disk !


4. Find the device file entry from the remainder of the lbolt error:

SCSI: Abort Tag -- lbolt: 74105435, dev: 1f000000, io_id: 4ead34

This is the minor number for the device that is failing.

a. Block device:

# ll -R /dev/ | grep 31 | grep 0x000000

brw-r----- 1 bin sys 31 0x000000 Jul 15 16:25 c0t0d0

Or:

b. Character Device:

# ll -R /dev/ | grep 188 | grep 0x000000
crw-r----- 1 bin sys 188 0x000000 Oct 11 07:15 c0t0d0

5. Find the Hardware Address:

# lssf /dev/dsk/c0t0d0
sdisk card instance 0 SCSI target 0 SCSI LUN 0 section 0
at address 0/0/0.0.0 /dev/dsk/c0t0d0


6. Find the type of device:

# diskinfo /dev/rdsk/c0t0d0# diskinfo /dev/rdsk/c0t0d0
SCSI describe of /dev/rdsk/c0t0d0:
vendor: DGC
product id: C2300WDR1
type: direct access
size: 4102875 Kbytes
bytes per sector: 512


So, we have a Nike disk at hardware address 0/0/0.0.0, device file
/dev/dsk/c0t0d0