autopath failed

[irshad ali]

I got below error on my syslog file. my bdf was also hanged. I rebooted the server. issue resolved. now I want to know root cause for this issue----
PLZ HELP

Jun 21 04:54:18 hostname vmunix: AUTOPATH : Path 0xbc1e0100 failed! Rerouting to alternate path
Jun 21 04:54:18 hostname vmunix: DIAGNOSTIC SYSTEM WARNING:
Jun 21 04:54:18 hostname vmunix: The diagnostic logging facility has started receiving excessive
Jun 21 04:54:18 hostname vmunix: errors from the I/O subsystem. I/O error entries will be lost
Jun 21 04:54:18 hostname vmunix: until the cause of the excessive I/O logging is corrected.
Jun 21 04:54:18 hostname vmunix: If the diaglogd daemon is not active, use the Daemon Startup command
Jun 21 04:54:18 hostname vmunix: in stm to start it.
Jun 21 04:54:18 hostname vmunix: If the diaglogd daemon is active, use the logtool utility in stm
Jun 21 04:54:18 hostname vmunix: to determine which I/O subsystem is logging excessive errors.
Jun 21 04:54:18 hostname vmunix: DIAGNOSTIC SYSTEM WARNING:
Jun 21 04:54:18 hostname vmunix: The diagnostic logging facility is no longer receiving excessive
Jun 21 04:54:18 hostname vmunix: errors from the I/O subsystem. 44 I/O error entries were lost.
Jun 21 04:54:21 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:21 hostname vmunix:
Jun 21 04:54:21 hostname vmunix: AUTOPATH : Path 0xbc200100 failed! Rerouting to alternate path
Jun 21 04:54:21 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:21 hostname vmunix: AUTOPATH : Path 0xbc040100 failed! Rerouting to alternate path
Jun 21 04:54:21 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:21 hostname vmunix: AUTOPATH : Path 0xbc060100 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc1e0400 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc200400 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc040400 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc060400 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc1e0500 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc200500 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc040500 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc060500 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc1e0700 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc200700 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc040700 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc060700 failed! Rerouting to alternate path
Jun 21 04:54:30 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f1e0500 failed!
Jun 21 04:54:33 hostname vmunix: LVM: WARNING: VG 64 0x060000: LV 41: Some I/O requests to this LV are waiting
Jun 21 04:54:33 hostname vmunix: indefinitely for an unavailable PV. These requests will be queued until
Jun 21 04:54:30 hostname vmunix:
Jun 21 04:54:33 hostname above message repeats 15 times
Jun 21 04:54:33 hostname vmunix: the PV becomes available (or a timeout is specified for the LV).
Jun 21 04:54:33 hostname vmunix:
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f1e0100 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f200100 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Jun 21 04:54:33 hostname vmunix: LVM: WARNING: VG 64 0x060000: LV 1: Some I/O requests to this LV are waiting
Jun 21 04:54:33 hostname vmunix: indefinitely for an unavailable PV. These requests will be queued until
Jun 21 04:54:33 hostname vmunix:
Jun 21 04:54:33 hostname above message repeats 2 times
Jun 21 04:54:33 hostname vmunix: the PV becomes available (or a timeout is specified for the LV).
Jun 21 04:54:33 hostname vmunix:
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f1e0500 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f200500 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f040500 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f060500 failed!
Jun 21 04:54:34 hostname vmunix: SCSI: Read error -- dev: b 31 0x1e0500, errno: 126, resid: 1024,
Jun 21 04:54:34 hostname vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Jun 21 04:54:39 hostname vmunix: LVM: WARNING: VG 64 0x090000: LV 1: Some I/O requests to this LV are waiting
Jun 21 04:54:34 hostname vmunix:
Jun 21 04:54:39 hostname above message repeats 5 times
Jun 21 04:54:39 hostname vmunix: indefinitely for an unavailable PV. These requests will be queued until
Jun 21 04:54:39 hostname vmunix: the PV becomes available (or a timeout is specified for the LV).
Jun 21 04:54:39 hostname vmunix:
Jun 21 04:54:39 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f1e0400 failed!
Jun 21 04:54:39 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f200400 failed!
Jun 21 04:54:39 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f040400 failed!
Jun 21 04:54:39 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f060400 failed!
Jun 21 04:54:42 hostname vmunix: SCSI: Read error -- dev: b 31 0x1e0400, errno: 126, resid: 1024,
Jun 21 04:54:42 hostname vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Jun 21 04:54:46 hostname vmunix: LVM: WARNING: VG 64 0x060000: LV 17: Some I/O requests to this LV are waiting
Jun 21 04:54:42 hostname vmunix:
Jun 21 04:54:46 hostname above message repeats 4 times

[Torsten.]

>> Jun 21 04:54:39 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f060400 failed!


All paths failed.

Investigate the reason (array down, SAN down, LUN deleted, ...).

[irshad ali]

Thanks Torsten,

Storage team says there was no error from their end. no alert or log on SAN and EVA.
Is there any issue from server side.
each LUN is comming from 4 paths from storage (EVA)

[S. Ney]

Which OS version are you at? If at 11.31 try running some scsimgr commands for more information. (man scsimgr)scsimgr get_stat and get_info can be pretty helpful. Is there any other information in dmesg?

check ioscan and fcmsutil /dev/fcd# as well as /var/opt/resmon/log/event.log

[AnthonySN]

what is the OS version?
if 11.31 you do not need autopath.
check the online status by
fcmsutil /dev/fcd?
any changes done on san switches like zoning

[irshad ali]

hi,

the os version is 11.23.
No event noticed in event.log.
checked the fc status by fcmsutil it also showed online.
after that problem i rebooted the server then problem (bdf hanging) resolved. but now i want to root cause of this issue.
plz help

[AnthonySN]

ok if OS is 11.23 what is the secure path version you are using ?
what is the SAN model ?
need to check the xcs/vcs version.

[irshad ali]

the secure version is below-
HPswsp A.3.0F.03F.00F HP StorageWorks Secure Path Device Driver and utilities for Active-Active Disk Arrays

san switch 4/24 brocade san switch
eva 6000

[AnthonySN · Accepted solution]

well if your san team is sure that there is no problem from the EVA/Switch side
then you can plan to upgrade the Secure Path version to sp4 now there is patch also for sp4
below is the link
go thru the release notes before updating.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=1825336&prodTypeId=18964&prodSeriesId=471465&swLang=8&taskId=135&swEnvOID=7

[irshad ali]

thanks, but I want to know,
did this issue occure due to secure path? or there may be something else............

plz reply