SCSI: Request Timeout; Abort Tag -- lbolt

[Indrajit Bhagat]

Hi All

I am facing the below issue:
hpkhs-root> vgsync /dev/hpkhsvg01
Resynchronized logical volume "/dev/hpkhsvg01/khs_home".
Resynchronized logical volume "/dev/hpkhsvg01/khs_links".
Resynchronized logical volume "/dev/hpkhsvg01/khs_log1".
Resynchronized logical volume "/dev/hpkhsvg01/khs_log".
vgsync: Couldn't re-synchronize stale partitions of the logical volume:
Device offline/Powerfailed

vgsync: Couldn't resynchronize logical volume "/dev/hpkhsvg01/khs_opt".
Resynchronized logical volume "/dev/hpkhsvg01/khs_opt_base".
Resynchronized logical volume "/dev/hpkhsvg01/khs_samba".
Resynchronized logical volume "/dev/hpkhsvg01/khs_tmpi".
Resynchronized logical volume "/dev/hpkhsvg01/khs_var".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_datadbs1".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_datadbs2".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_histdbs2".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_histdbs3".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_histdbs4".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_histdbs5".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_histdbs".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_logdbs".
vgsync: Couldn't re-synchronize stale partitions of the logical volume:
Device offline/Powerfailed

vgsync: Couldn't resynchronize logical volume "/dev/hpkhsvg01/lvhpkhs_physdbs".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_rootdbs".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_tempdbs1".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_tempdbs2".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_basedbs".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_baselogdbs".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_histdbs6".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_logdbs2".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_histdbs7".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_histdbs8".
Resynchronized logical volume "/dev/hpkhsvg01/lvhpkhs_histdbs9".
vgsync: Couldn't resynchronize volume group "/dev/hpkhsvg01".

Also we are getting lbolt error message on one of the disk:
SCSI: Request Timeout; Abort Tag -- lbolt: 567420609, dev: 1f048000, io_id: 4b479ec
LVM: Failed to automatically resync PV 1f058000 error: 5

SCSI: Request Timeout; Abort Tag -- lbolt: 567503018, dev: 1f048000, io_id: 4b5d61b
LVM: Failed to automatically resync PV 1f058000 error: 5

SCSI: Request Timeout; Abort Tag -- lbolt: 567506418, dev: 1f048000, io_id: 4b5d6ac
LVM: Failed to automatically resync PV 1f058000 error: 5

SCSI: Request Timeout; Abort Tag -- lbolt: 567514318, dev: 1f048000, io_id: 4b5d78e
LVM: Failed to automatically resync PV 1f058000 error: 5
LVM: Failed to automatically resync PV 1f058000 error: 5
LVM: Failed to automatically resync PV 1f058000 error: 5

SCSI: Request Timeout; Abort Tag -- lbolt: 567538918, dev: 1f048000, io_id: 4b5dc09
LVM: Failed to automatically resync PV 1f058000 error: 5

SCSI: Request Timeout; Abort Tag -- lbolt: 567551444, dev: 1f048000, io_id: 4b5dde7
LVM: Failed to automatically resync PV 1f058000 error: 5

SCSI: Request Timeout; Abort Tag -- lbolt: 567554555, dev: 1f048000, io_id: 4b5dea4
LVM: Failed to automatically resync PV 1f058000 error: 5

SCSI: Request Timeout; Abort Tag -- lbolt: 567562473, dev: 1f048000, io_id: 4b5dff6

SCSI: Request Timeout; Abort Tag -- lbolt: 567565473, dev: cb048000, io_id: 4b5e078
LVM: Failed to automatically resync PV 1f058000 error: 5

SCSI: Request Timeout; Abort Tag -- lbolt: 567570573, dev: 1f048000, io_id: 4b5e0d7
LVM: Failed to automatically resync PV 1f058000 error: 5

SCSI: Request Timeout; Abort Tag -- lbolt: 567578373, dev: 1f048000, io_id: 4b5e1c6

hpkhs-root> ll
total 0
brw-r----- 1 bin sys 31 0x000000 Aug 11 2005 c0t0d0
brw-r----- 1 bin sys 31 0x020000 Aug 11 2005 c2t0d0
brw-r----- 1 bin sys 31 0x021000 Aug 11 2005 c2t1d0
brw-r----- 1 bin sys 31 0x04a000 Aug 25 2005 c4t10d0
brw-r----- 1 bin sys 31 0x04c000 Aug 25 2005 c4t12d0
brw-r----- 1 bin sys 31 0x048000 Aug 25 2005 c4t8d0
brw-r----- 1 bin sys 31 0x05a000 Aug 25 2005 c5t10d0
brw-r----- 1 bin sys 31 0x05c000 Aug 25 2005 c5t12d0
brw-r----- 1 bin sys 31 0x058000 Aug 25 2005 c5t8d0

lbolt: 567420609, dev: 1f04800 c4t8d0
1f058000 c5t8d0

CRITICAL - (1 errors in messages-log.protocol-2010-11-29-20-50-39) - Nov 29 20:48:07 hpkhs EMS [3250]: ------ EMS Event Notification ------ Value: CRITICAL (5) for Resource: /storage/events/disks/default/0_3_1_0.8.0 (Threshold: = 3) Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 212992014 -r /storage/events/disks/default/0_3_1_0.8.0 -n 212992048 -a

CRITICAL - (2 errors, 2 warnings in messages-log.protocol-2010-11-29-19-20-39) - Nov 29 19:17:56 hpkhs EMS [3250]: ------ EMS Event Notification ------ Value: MAJORWARNING (3) for Resource: /storage/events/disks/default/0_4_1_0.12.0 (Threshold: = 3) Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 212992044 -r /storage/events/disks/default/0_4_1_0.12.0 -n 212992043 -a ...

CRITICAL - (2 errors, 1 warnings in messages-log.protocol-2010-11-26-14-55-34) - Nov 26 14:51:39 hpkhs EMS [3250]: ------ EMS Event Notification ------ Value: SERIOUS (4) for Resource: /storage/events/disks/default/0_4_1_0.8.0 (Threshold: = 3) Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 212992032 -r /storage/events/disks/default/0_4_1_0.8.0 -n 212992041 -a ...

hpkhs-root> ioscan -kfnC disk
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DW-224E-B
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GMAU3073NC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 2 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 73.4GMAU3073NC
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
disk 3 0/3/1/0.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c4t8d0 /dev/rdsk/c4t8d0
disk 4 0/3/1/0.10.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c4t10d0 /dev/rdsk/c4t10d0
disk 5 0/3/1/0.12.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c4t12d0 /dev/rdsk/c4t12d0
disk 6 0/4/1/0.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0
disk 7 0/4/1/0.10.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t10d0 /dev/rdsk/c5t10d0
disk 8 0/4/1/0.12.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t12d0 /dev/rdsk/c5t12d0

hpkhs-root> lvdisplay /dev/hpkhsvg01/khs_log1
--- Logical volumes ---
LV Name /dev/hpkhsvg01/khs_log1
VG Name /dev/hpkhsvg01
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 8192
Current LE 2048
Allocated PE 4096
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

hpkhs-root> cat /etc/lvmpvg
VG /dev/hpkhsvg01
PVG hpkhspvg01
/dev/dsk/c4t8d0
/dev/dsk/c4t10d0
/dev/dsk/c4t12d0

PVG hpkhspvg02
/dev/dsk/c5t8d0
/dev/dsk/c5t10d0
/dev/dsk/c5t12d0


hpkhs-root> strings /etc/lvmtab
/dev/vg00
/dev/dsk/c2t0d0
/dev/dsk/c2t1d0
/dev/hpkhsvg01
/dev/dsk/c4t8d0
/dev/dsk/c4t10d0
/dev/dsk/c4t12d0
/dev/dsk/c5t8d0
/dev/dsk/c5t10d0
/dev/dsk/c5t12d0


Request to take a look into the error message and suggest what next can be done, to resolve the issue.

[Jayakrishnan G Naik]

Hi

For me this looks like one or more of the Physical volume is having issue on it and it may not be disk failure but a bad disk blocks etc.

/opt/resmon/bin/resdata -R 212992014 -r /storage/events/disks/default/0_3_1_0.8.0 -n 212992048 -a

Please check the resdata command output for each event and see whether the info have any details for you to identify defective pv.
see the lvdisplay -v output for each lv that failed to sync. This may help us to identify the failed pv

Can you copy these outputs here. Recomment to copy it in txt file or word file rather than pasting it directly.

Thanks & Regards
Jayakrishnan G Naik

[Indrajit Bhagat]

Hi Jaykrishna

As requested Please find the attachement.

[Jayakrishnan G Naik]

Hi

Have you checked all the 3 resmon events?

I can see only one in the output and I can understand there is something wrong with a disk as in resmon message.

Component Data:
Physical Device Path...: 0/3/1/0.8.0
Device Class...........: Disk
Inquiry Vendor ID......: HP 73.4G
Inquiry Product ID.....: MAT3073NC
Firmware Version.......: HPC2
Serial Number..........: XX004503 0508

Summary:
Disk at hardware path : I/O request failed.


Description of Error:

As part of the polling functionality, the monitor periodically requests
data from the device. The monitor's I/O request failed in this case. The
monitor was requesting data for Inquiry command.

Probable Cause / Recommended Action:

The monitor could not finish the requested I/O operation to the device.
Check /etc/opt/resmon/log/api.log file for an entry logged by
tl_scsi_dev_io request.

Similary check others also. I just copied one of them from your initial post. Check all and see where is the mistake confirm whether the issue is from only one device.

/opt/resmon/bin/resdata -R 212992044 -r /storage/events/disks/default/0_4_1_0.12.0 -n 212992043 -a

/opt/resmon/bin/resdata -R 212992032 -r /storage/events/disks/default/0_4_1_0.8.0 -n 212992041 -a

See whether all events pointing to same device? , You can copy the outputs again.

Thanks & Regards
Jayakrishnan G Naik