FCD Error

[rana sarkar]

Hi I am unable to edit any file in certain FS in my HPUX box.

Syslog errors are :
Mar 20 00:43:47 tcobad03 vmunix: 0/3/1/0/4/0: Fibre Channel Driver received Link Dead Notification.
Mar 20 00:43:47 tcobad03 vmunix:
Mar 20 00:44:06 tcobad03 vmunix: AUTOPATH : Path 0xbc040200 failed! Rerouting to alternate path
Mar 20 00:44:14 tcobad03 vmunix: AUTOPATH : Path 0xbc080100 failed! Rerouting to alternate path
Mar 20 00:44:14 tcobad03 vmunix: AUTOPATH : Path 0xbc0a0100 failed! Rerouting to alternate path
Mar 20 00:44:15 tcobad03 vmunix: AUTOPATH : Path 0xbc060100 failed! Rerouting to alternate path
Mar 20 00:44:16 tcobad03 vmunix: AUTOPATH : Path 0xbc040100 failed! Rerouting to alternate path
Mar 20 00:44:16 tcobad03 vmunix: LVM: VG 64 0x010000: Lost quorum.
Mar 20 00:44:16 tcobad03 vmunix: This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of
the following PVs (represented by current link) must become available:
Mar 20 00:44:16 tcobad03 vmunix:
Mar 20 00:44:16 tcobad03 above message repeats 5 times
Mar 20 00:44:16 tcobad03 vmunix: <31 0x040100>
Mar 20 00:44:16 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x040100 Failed! The PV is not accessible.
Mar 20 00:44:16 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x060100 Failed! The PV is not accessible.
Mar 20 00:44:16 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x080100 Failed! The PV is not accessible.
Mar 20 00:44:16 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x0a0100 Failed! The PV is not accessible.
Mar 20 00:44:16 tcobad03 vmunix:
Mar 20 00:44:16 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Mar 20 00:44:16 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Mar 20 00:44:16 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080100 failed!
Mar 20 00:44:16 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0100 failed!
Mar 20 00:44:22 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Mar 20 00:44:22 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Mar 20 00:44:22 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080100 failed!
Mar 20 00:44:22 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0100 failed!
Mar 20 00:44:26 tcobad03 vmunix: AUTOPATH : Path 0xbc060200 failed! Rerouting to alternate path
Mar 20 00:44:28 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Mar 20 00:44:28 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Mar 20 00:44:28 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080100 failed!
Mar 20 00:44:28 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0100 failed!
Mar 20 00:44:33 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Mar 20 00:44:33 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Mar 20 00:44:33 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080100 failed!
Mar 20 00:44:33 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0100 failed!
Mar 20 00:44:35 tcobad03 vmunix: AUTOPATH : Path 0xbc080200 failed! Rerouting to alternate path
Mar 20 00:44:35 tcobad03 vmunix: AUTOPATH : Path 0xbc0a0200 failed! Rerouting to alternate path
Mar 20 00:44:35 tcobad03 vmunix: LVM: VG 64 0x020000: Lost quorum.
Mar 20 00:44:35 tcobad03 vmunix: This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of
the following PVs (represented by current link) must become available:
Mar 20 00:44:35 tcobad03 vmunix:
Mar 20 00:44:35 tcobad03 above message repeats 18 times
Mar 20 00:44:35 tcobad03 vmunix: <31 0x040200>
Mar 20 00:44:35 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x040200 Failed! The PV is not accessible.
Mar 20 00:44:35 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x060200 Failed! The PV is not accessible.
Mar 20 00:44:35 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x080200 Failed! The PV is not accessible.
Mar 20 00:44:35 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x0a0200 Failed! The PV is not accessible.
Mar 20 00:44:38 tcobad03 vmunix:
Mar 20 00:44:38 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Mar 20 00:44:38 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Mar 20 00:44:38 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080100 failed!
Mar 20 00:44:38 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0100 failed!
Mar 20 00:44:40 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040200 failed!
Mar 20 00:44:40 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060200 failed!
Mar 20 00:44:40 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080200 failed!
Mar 20 00:44:40 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0200 failed!
Mar 20 00:44:43 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Mar 20 00:44:43 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Mar 20 00:44:43 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080100 failed!
Mar 20 00:44:43 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0100 failed!
Mar 20 00:44:45 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040200 failed!
Mar 20 00:44:45 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060200 failed!
Mar 20 00:44:45 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080200 failed!
Mar 20 00:44:45 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0200 failed!
Mar 20 00:44:45 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040200 failed!
Mar 20 00:44:45 tcobad03 vmunix: LVM: VG 64 0x020000: Reestablished quorum.
Mar 20 00:44:45 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x040200 Recovered.
Mar 20 00:44:45 tcobad03 vmunix:
Mar 20 00:44:45 tcobad03 above message repeats 19 times
Mar 20 00:44:45 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x060200 Recovered.
Mar 20 00:44:45 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040200 failed!
Mar 20 00:44:45 tcobad03 above message repeats 3 times
Mar 20 00:44:45 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x080200 Recovered.
Mar 20 00:44:45 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x0a0200 Recovered.
Mar 20 00:44:45 tcobad03 vmunix:
Mar 20 00:44:45 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040200 failed!
Mar 20 00:44:46 tcobad03 vmunix: AUTOPATH : Path 0xbc080100 recovered
Mar 20 00:44:46 tcobad03 vmunix: AUTOPATH : Path 0xbc0a0100 recovered
Mar 20 00:44:45 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040200 failed!
Mar 20 00:44:46 tcobad03 above message repeats 4 times
Mar 20 00:44:46 tcobad03 vmunix: AUTOPATH : Path 0xbc060100 recovered
Mar 20 00:44:46 tcobad03 vmunix: AUTOPATH : Path 0xbc040100 recovered
Mar 20 00:44:46 tcobad03 vmunix: AUTOPATH : Path 0xbc040200 recovered
Mar 20 00:44:46 tcobad03 vmunix: AUTOPATH : Path 0xbc060200 recovered
Mar 20 00:44:46 tcobad03 vmunix: AUTOPATH : Path 0xbc080200 recovered
Mar 20 00:44:46 tcobad03 vmunix: AUTOPATH : Path 0xbc0a0200 recovered
Mar 20 00:44:48 tcobad03 vmunix: LVM: VG 64 0x010000: Reestablished quorum.
Mar 20 00:44:48 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x040100 Recovered.
Mar 20 00:44:46 tcobad03 vmunix:
Mar 20 00:44:48 tcobad03 above message repeats 12 times
Mar 20 00:44:48 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x060100 Recovered.
Mar 20 00:44:48 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x080100 Recovered.
Mar 20 00:44:48 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x0a0100 Recovered.
Mar 20 00:45:40 tcobad03 vmunix: 0/3/1/0/4/0: Fibre Channel Driver received Link Dead Notification.
Mar 20 00:45:40 tcobad03 vmunix:
Mar 20 00:45:59 tcobad03 vmunix: AUTOPATH : Path 0xbc080200 failed! Rerouting to alternate path
Mar 20 00:46:11 tcobad03 vmunix: AUTOPATH : Path 0xbc0a0200 failed! Rerouting to alternate path
Mar 20 00:46:23 tcobad03 vmunix: AUTOPATH : Path 0xbc080100 failed! Rerouting to alternate path
Mar 20 00:46:23 tcobad03 vmunix: AUTOPATH : Path 0xbc0a0100 failed! Rerouting to alternate path
Mar 20 00:46:23 tcobad03 vmunix: AUTOPATH : Path 0xbc060100 failed! Rerouting to alternate path
Mar 20 00:46:23 tcobad03 vmunix: AUTOPATH : Path 0xbc040100 failed! Rerouting to alternate path
Mar 20 00:46:23 tcobad03 vmunix: LVM: VG 64 0x010000: Lost quorum.
Mar 20 00:46:23 tcobad03 vmunix: This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of
the following PVs (represented by current link) must become available:
Mar 20 00:46:23 tcobad03 vmunix:
Mar 20 00:46:23 tcobad03 above message repeats 6 times
Mar 20 00:46:23 tcobad03 vmunix: <31 0x040100>
Mar 20 00:46:23 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x040100 Failed! The PV is not accessible.
Mar 20 00:46:23 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x060100 Failed! The PV is not accessible.
Mar 20 00:46:23 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x080100 Failed! The PV is not accessible.
Mar 20 00:46:23 tcobad03 vmunix: LVM: VG 64 0x010000: PVLink 31 0x0a0100 Failed! The PV is not accessible.
Mar 20 00:46:23 tcobad03 vmunix:
Mar 20 00:46:23 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Mar 20 00:46:23 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Mar 20 00:46:23 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080100 failed!
Mar 20 00:46:23 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0100 failed!
Mar 20 00:46:27 tcobad03 vmunix: AUTOPATH : Path 0xbc060200 failed! Rerouting to alternate path
Mar 20 00:46:27 tcobad03 vmunix: AUTOPATH : Path 0xbc040200 failed! Rerouting to alternate path
Mar 20 00:46:27 tcobad03 vmunix: LVM: VG 64 0x020000: Lost quorum.
Mar 20 00:46:27 tcobad03 vmunix: This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of
the following PVs (represented by current link) must become available:
Mar 20 00:46:27 tcobad03 vmunix:
Mar 20 00:46:27 tcobad03 above message repeats 5 times
Mar 20 00:46:27 tcobad03 vmunix: <31 0x040200>
Mar 20 00:46:27 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x040200 Failed! The PV is not accessible.
Mar 20 00:46:27 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x060200 Failed! The PV is not accessible.
Mar 20 00:46:27 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x080200 Failed! The PV is not accessible.
Mar 20 00:46:27 tcobad03 vmunix: LVM: VG 64 0x020000: PVLink 31 0x0a0200 Failed! The PV is not accessible.
Mar 20 00:46:29 tcobad03 vmunix:
Mar 20 00:46:29 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Mar 20 00:46:29 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Mar 20 00:46:29 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080100 failed!
Mar 20 00:46:29 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0100 failed!
Mar 20 00:46:32 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040200 failed!
Mar 20 00:46:32 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060200 failed!
Mar 20 00:46:32 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080200 failed!
Mar 20 00:46:32 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0200 failed!
Mar 20 00:46:35 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!
Mar 20 00:46:35 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f060100 failed!
Mar 20 00:46:35 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f080100 failed!
Mar 20 00:46:35 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f0a0100 failed!

[Ankitj_1]

Hello,

Cause: The Fibre Channel driver has declared the link to be dead because it has been down for more than 10 seconds.

Action: Please check the following:


1) Verify that the fiber optic cable is functional and securely attached to the Fibre Channel Mass Storage Adapter.

2) Verify that the device which is connected to the adapter port is functioning correctly.

3) Verify that the adapter port is functioning correctly by performing internal and external loopback tests. Please attach a loopback hood to the adapter port when executing a loopback test.

Let me know if yiu an alternate path to the disk.

Regards

Ankit

[SUDHAKAR_18]

can you check NO HARDWARE notification in ioscan?

#ioscan -fn |grep -i no

[rana sarkar]

Yes I am getting no_hw for the VD routed through fcd. The fcdutil is giving sometimes claimed and sometimes no_hw for the fcd0

[SUDHAKAR_18]

There is a connectivity disturbance in your SAN connectivity. Check your SAN switch logs. If possible check SAN cabling.

[Johnson Punniyalingam]

>>Mar 20 00:43:47 tcobad03 vmunix: 0/3/1/0/4/0: Fibre Channel Driver received Link Dead Notification.<<

You check "FC link" from switch and your external storage,

could be FC cable problem..

Thanks,
Johnson

[Torsten.]

>> Mar 20 00:44:16 tcobad03 vmunix: AUTOPATH : All the paths to the end device 0x1f040100 failed!

Assuming you have a redundant SAN, the reason for all pathes failed is likely the array itself (rebootet or shutdown). Check this or ask your SAN admin.

[rana sarkar]

the problem sorted out after changing the OFC Cable.