cancel
Showing results for 
Search instead for 
Did you mean: 

autopath failed

SOLVED
Go to solution
irshad ali
Frequent Advisor

autopath failed

I got below error on my syslog file. my bdf was also hanged. I rebooted the server. issue resolved. now I want to know root cause for this issue----
PLZ HELP

Jun 21 04:54:18 hostname vmunix: AUTOPATH : Path 0xbc1e0100 failed! Rerouting to alternate path
Jun 21 04:54:18 hostname vmunix: DIAGNOSTIC SYSTEM WARNING:
Jun 21 04:54:18 hostname vmunix: The diagnostic logging facility has started receiving excessive
Jun 21 04:54:18 hostname vmunix: errors from the I/O subsystem. I/O error entries will be lost
Jun 21 04:54:18 hostname vmunix: until the cause of the excessive I/O logging is corrected.
Jun 21 04:54:18 hostname vmunix: If the diaglogd daemon is not active, use the Daemon Startup command
Jun 21 04:54:18 hostname vmunix: in stm to start it.
Jun 21 04:54:18 hostname vmunix: If the diaglogd daemon is active, use the logtool utility in stm
Jun 21 04:54:18 hostname vmunix: to determine which I/O subsystem is logging excessive errors.
Jun 21 04:54:18 hostname vmunix: DIAGNOSTIC SYSTEM WARNING:
Jun 21 04:54:18 hostname vmunix: The diagnostic logging facility is no longer receiving excessive
Jun 21 04:54:18 hostname vmunix: errors from the I/O subsystem. 44 I/O error entries were lost.
Jun 21 04:54:21 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:21 hostname vmunix:
Jun 21 04:54:21 hostname vmunix: AUTOPATH : Path 0xbc200100 failed! Rerouting to alternate path
Jun 21 04:54:21 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:21 hostname vmunix: AUTOPATH : Path 0xbc040100 failed! Rerouting to alternate path
Jun 21 04:54:21 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:21 hostname vmunix: AUTOPATH : Path 0xbc060100 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc1e0400 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc200400 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc040400 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc060400 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc1e0500 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc200500 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc040500 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc060500 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc1e0700 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc200700 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc040700 failed! Rerouting to alternate path
Jun 21 04:54:22 hostname vmunix: AUTOPATH : PPM Failing path error returned <5>
Jun 21 04:54:22 hostname vmunix: AUTOPATH : Path 0xbc060700 failed! Rerouting to alternate path
Jun 21 04:54:30 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f1e0500 failed!
Jun 21 04:54:33 hostname vmunix: LVM: WARNING: VG 64 0x060000: LV 41: Some I/O requests to this LV are waiting
Jun 21 04:54:33 hostname vmunix: indefinitely for an unavailable PV. These requests will be queued until
Jun 21 04:54:30 hostname vmunix:
Jun 21 04:54:33 hostname above message repeats 15 times
Jun 21 04:54:33 hostname vmunix: the PV becomes available (or a timeout is specified for the LV).
Jun 21 04:54:33 hostname vmunix:
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f1e0100 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f200100 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Jun 21 04:54:33 hostname vmunix: LVM: WARNING: VG 64 0x060000: LV 1: Some I/O requests to this LV are waiting
Jun 21 04:54:33 hostname vmunix: indefinitely for an unavailable PV. These requests will be queued until
Jun 21 04:54:33 hostname vmunix:
Jun 21 04:54:33 hostname above message repeats 2 times
Jun 21 04:54:33 hostname vmunix: the PV becomes available (or a timeout is specified for the LV).
Jun 21 04:54:33 hostname vmunix:
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f1e0500 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f200500 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f040500 failed!
Jun 21 04:54:33 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f060500 failed!
Jun 21 04:54:34 hostname vmunix: SCSI: Read error -- dev: b 31 0x1e0500, errno: 126, resid: 1024,
Jun 21 04:54:34 hostname vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Jun 21 04:54:39 hostname vmunix: LVM: WARNING: VG 64 0x090000: LV 1: Some I/O requests to this LV are waiting
Jun 21 04:54:34 hostname vmunix:
Jun 21 04:54:39 hostname above message repeats 5 times
Jun 21 04:54:39 hostname vmunix: indefinitely for an unavailable PV. These requests will be queued until
Jun 21 04:54:39 hostname vmunix: the PV becomes available (or a timeout is specified for the LV).
Jun 21 04:54:39 hostname vmunix:
Jun 21 04:54:39 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f1e0400 failed!
Jun 21 04:54:39 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f200400 failed!
Jun 21 04:54:39 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f040400 failed!
Jun 21 04:54:39 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f060400 failed!
Jun 21 04:54:42 hostname vmunix: SCSI: Read error -- dev: b 31 0x1e0400, errno: 126, resid: 1024,
Jun 21 04:54:42 hostname vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Jun 21 04:54:46 hostname vmunix: LVM: WARNING: VG 64 0x060000: LV 17: Some I/O requests to this LV are waiting
Jun 21 04:54:42 hostname vmunix:
Jun 21 04:54:46 hostname above message repeats 4 times
14 REPLIES
Torsten.
Acclaimed Contributor

Re: autopath failed

>> Jun 21 04:54:39 hostname vmunix: AUTOPATH : All the paths to the end device 0x1f060400 failed!


All paths failed.

Investigate the reason (array down, SAN down, LUN deleted, ...).

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
irshad ali
Frequent Advisor

Re: autopath failed

Thanks Torsten,

Storage team says there was no error from their end. no alert or log on SAN and EVA.
Is there any issue from server side.
each LUN is comming from 4 paths from storage (EVA)
S. Ney
Trusted Contributor

Re: autopath failed

Which OS version are you at? If at 11.31 try running some scsimgr commands for more information. (man scsimgr)scsimgr get_stat and get_info can be pretty helpful. Is there any other information in dmesg?

check ioscan and fcmsutil /dev/fcd# as well as /var/opt/resmon/log/event.log


AnthonySN
Respected Contributor

Re: autopath failed

what is the OS version?
if 11.31 you do not need autopath.
check the online status by
fcmsutil /dev/fcd?
any changes done on san switches like zoning
irshad ali
Frequent Advisor

Re: autopath failed

hi,

the os version is 11.23.
No event noticed in event.log.
checked the fc status by fcmsutil it also showed online.
after that problem i rebooted the server then problem (bdf hanging) resolved. but now i want to root cause of this issue.
plz help
AnthonySN
Respected Contributor

Re: autopath failed

ok if OS is 11.23 what is the secure path version you are using ?
what is the SAN model ?
need to check the xcs/vcs version.
irshad ali
Frequent Advisor

Re: autopath failed

the secure version is below-
HPswsp A.3.0F.03F.00F HP StorageWorks Secure Path Device Driver and utilities for Active-Active Disk Arrays

san switch 4/24 brocade san switch
eva 6000
AnthonySN
Respected Contributor
Solution

Re: autopath failed

well if your san team is sure that there is no problem from the EVA/Switch side
then you can plan to upgrade the Secure Path version to sp4 now there is patch also for sp4
below is the link
go thru the release notes before updating.

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=1825336&prodTypeId=18964&prodSeriesId=471465&swLang=8&taskId=135&swEnvOID=7
irshad ali
Frequent Advisor

Re: autopath failed

thanks, but I want to know,
did this issue occure due to secure path? or there may be something else............

plz reply
AnthonySN
Respected Contributor

Re: autopath failed

i mostly suspect a zoning change was done but since ur san team is sure of no change then the logs you provided somewhat indicate towards SP.
btw did you experience this prob only one time or frequently ?
irshad ali
Frequent Advisor

Re: autopath failed

thanks, we received this problem first time.
let me share more about this server-
this is cell manager of DP. every night few FS mount from a server and then backup starts. after backup completion these fs unmount automatically. This is all done by RSM. storage team has checked the RSM logs but did n't find anything. all their script went well. the error occured at 4:54AM and i noticed at 10 AM that bdf was hanged. after that I rebooted the server
AnthonySN
Respected Contributor

Re: autopath failed

hmm..
if its first time then i think we should wait. anyways if its backup server i think there is no harm in updating the OS patches,firmwares and software patches to the latest level.
S. Ney
Trusted Contributor

Re: autopath failed

Just a shot in the dark (because your errors still look like a storage hiccup) does your DP do nfs or remote mounts? If the nfs mount hung and became stale it would cause bdf to hang. Is there anything in the DP logs to indicate timeouts or some kind of errors with the filesystem mounts?
irshad ali
Frequent Advisor

Re: autopath failed

Hi,

I have upgraded the secure path version.
problem seems to be resolved.

Thanks All