Syslog.log errors: Mar 27 08:00:19 urrtw41 vmunix: fclp driver at 1/0/0/7/0/0/0 (/dev/fclp2) : has lost a device with id 0x32b700, PWWN 0x50060e801534c10e. Mar 27 08:00:29 urrtw41 vmunix: Target path (class=tgtpath, instance=45) has gone offline. The target path h/w path is 1/0/0/7/0/0/0.0x50060e801534c10e Mar 27 08:00:39 urrtw41 vmunix: Target path (class=tgtpath, instance=45) has gone online. The target path h/w path is 1/0/0/7/0/0/0.0x50060e801534c10e urrtw41:[root::/var/adm/syslog]# And also getting the same error in dmesg also: fclp driver at 1/0/0/7/0/0/0 (/dev/fclp2) : has lost a device with id 0x32b700, PWWN 0x50060e801534c10e. This device has disappeared from the Name Server database, LUNs behind this port will not be seen. class : tgtpath, instance 45 Target path (class=tgtpath, instance=45) has gone offline. The target path h/w path is 1/0/0/7/0/0/0.0x50060e801534c10e class : tgtpath, instance 45 Target path (class=tgtpath, instance=45) has gone online. The target path h/w path is 1/0/0/7/0/0/0.0x50060e801534c10e urrtw41:[root::/home/sverma55]# ioscan -fnC fc Class I H/W Path Driver S/W State H/W Type Description ====================================================================== fc 2 1/0/0/7/0/0/0 fclp CLAIMED INTERFACE HP 456972-B21 8Gb PCIe 2-port LPe1205 FC Mezzanine Adapter /dev/fclp2 fc 3 1/0/0/7/0/0/1 fclp CLAIMED INTERFACE HP 456972-B21 8Gb PCIe 2-port LPe1205 FC Mezzanine Adapter /dev/fclp3 urrtw41:[root::/home/sverma55]# ioscan -fnNC tgtpath |grep -i 1/0/0/7/0/0/1 tgtpath 43 1/0/0/7/0/0/1.0x100000e002432c51 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x285000 tgtpath 44 1/0/0/7/0/0/1.0x100000e002433eff estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x28c480 tgtpath 41 1/0/0/7/0/0/1.0x203d001321440b73 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x3cbc00 tgtpath 42 1/0/0/7/0/0/1.0x2043001321440b73 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x3cbd00 tgtpath 46 1/0/0/7/0/0/1.0x50060e801534c11e estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x3cb700 <--Issue was with only urrtw41:[root::/home/sverma55]# ioscan -fnNC tgtpath Class I H/W Path Driver S/W State H/W Type Description ================================================================= tgtpath 0 0/0/0/2/0/0/0.0x0 estp CLAIMED TGT_PATH sas target served by ciss driver, target port id 0x0 tgtpath 1 1/0/0/2/0/0/0.0x0 estp CLAIMED TGT_PATH sas target served by ciss driver, target port id 0x0 tgtpath 39 1/0/0/7/0/0/0.0x100000e00222fcbd estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x1ec880 tgtpath 40 1/0/0/7/0/0/0.0x100000e002232c51 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x1ec480 tgtpath 38 1/0/0/7/0/0/0.0x100000e002233eff estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x1ecf00 tgtpath 33 1/0/0/7/0/0/0.0x2001001321440b73 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x320800 tgtpath 32 1/0/0/7/0/0/0.0x2004001321440b73 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x320000 tgtpath 35 1/0/0/7/0/0/0.0x2007001321440b73 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x321800 tgtpath 34 1/0/0/7/0/0/0.0x200a001321440b73 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x321000 tgtpath 37 1/0/0/7/0/0/0.0x2040001321440b73 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x32bc00 tgtpath 36 1/0/0/7/0/0/0.0x2046001321440b73 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x32bd00 tgtpath 45 1/0/0/7/0/0/0.0x50060e801534c10e estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x32b700 tgtpath 43 1/0/0/7/0/0/1.0x100000e002432c51 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x285000 tgtpath 44 1/0/0/7/0/0/1.0x100000e002433eff estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x28c480 tgtpath 41 1/0/0/7/0/0/1.0x203d001321440b73 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x3cbc00 tgtpath 42 1/0/0/7/0/0/1.0x2043001321440b73 estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x3cbd00 tgtpath 46 1/0/0/7/0/0/1.0x50060e801534c11e estp CLAIMED TGT_PATH fibre_channel target served by fclp driver, target port id 0x3cb700 urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp2 Vendor ID is = 0x10df Device ID is = 0xf100 PCI Sub-system Vendor ID is = 0x103c PCI Sub-system ID is = 0x1719 Chip version = 3 Firmware Version = 1.11A5 SLI-3 (U3F1.11A5) EFI Version = UU6.03A8 EFI Boot = ENABLED Driver-Firmware Dump Available = NO Driver-Firmware Dump Timestamp = N/A Topology = PTTOPT_FABRIC Link Speed = 2Gb Local N_Port_id is = 0x720800 Previous N_Port_id is = None N_Port Node World Wide Name = 0x20000000c9f1e916 N_Port Port World Wide Name = 0x10000000c9f1e916 Switch Port World Wide Name = 0x200800051ecc3d2a Switch Node World Wide Name = 0x100000051ecc3d2a Driver state = ONLINE Hardware Path is = 1/0/0/7/0/0/0 Maximum Frame Size = 2048 Driver Version = @(#) FCLP: PCIe Fibre Channel driver (FibrChanl-02), B.11.31.1003.02, Feb 24 2010, FCLP_IFC (3,2) urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp3 Vendor ID is = 0x10df Device ID is = 0xf100 PCI Sub-system Vendor ID is = 0x103c PCI Sub-system ID is = 0x1719 Chip version = 3 Firmware Version = 1.11A5 SLI-3 (U3F1.11A5) EFI Version = UU6.03A8 EFI Boot = ENABLED Driver-Firmware Dump Available = NO Driver-Firmware Dump Timestamp = N/A Topology = PTTOPT_FABRIC Link Speed = 2Gb Local N_Port_id is = 0xd60800 Previous N_Port_id is = None N_Port Node World Wide Name = 0x20000000c9f1e917 N_Port Port World Wide Name = 0x10000000c9f1e917 Switch Port World Wide Name = 0x200800051ecc3d40 Switch Node World Wide Name = 0x100000051ecc3d40 Driver state = ONLINE Hardware Path is = 1/0/0/7/0/0/1 Maximum Frame Size = 2048 Driver Version = @(#) FCLP: PCIe Fibre Channel driver (FibrChanl-02), B.11.31.1003.02, Feb 24 2010, FCLP_IFC (3,2) urrtw41:[root::/home/sverma55]# ioscan -kfnNC lunpath |grep -i 1/0/0/7/0/0/0.0x50060e801534c10e |wc -l ============= ========================== urrtw41:[root::/home/sverma55]# ll total 27152 -r--r--r-- 1 sverma55 offshr_uni 832 Mar 28 08:41 .cshrc -r--r--r-- 1 sverma55 offshr_uni 347 Mar 28 08:41 .exrc -r--r--r-- 1 sverma55 offshr_uni 334 Mar 28 08:41 .login -r--r--r-- 1 sverma55 offshr_uni 700 Mar 28 08:41 .profile -rw------- 1 sverma55 offshr_uni 96 Mar 28 12:46 .sh_history -rw------- 1 root sys 369 Mar 28 11:18 Procedure.txt -rw-r--r-- 1 root sys 1294 Mar 28 08:43 disk -rw-r--r-- 1 root sys 13186 Mar 28 08:48 disk_alert -rwxr-xr-x 1 root sys 38476 Mar 28 11:18 getsysinfo.sh -rw-r--r-- 1 root sys 13781504 Mar 28 11:08 sysinfo_urrtw41_201303281106.tar urrtw41:[root::/home/sverma55]# cat disk I have had a response on the Oracle SR to request that other LOS perform the same checks as they did for the EBUSPROD issue, I've pasted their update at the bottom. For reference EBUSHA is single instance standby database sat on urrtw41, it reported it could not contact multiple disks from 07:48 today. I list a few but attach the relevant part of the alert log for exact details and timestamps. On a daily basis at 7pm it is shutdown, its disk unmounted from thr rtw prod nodes and mounted on urrtw01a/01b and an rman duplicate activity is performed to created our daily clone. EBUSHA is normally back on urrtw41 at about 2am each day and catches up with EBUSPROD by 4am. WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk088 disk:0x8d.0xf0d708b4 WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk100 disk:0x63.0xf0d708c2 Oracle request... + Consider all the checks we discussed so far for the issue on Prod case, on the standby machines as well. Starting with identifying the disks, and checking OS logs and storage logs, kernel parameter setting, Let me know the feedback from the other LOS and I can update the SR with anything you feel relevant Sarah urrtw41:[root::/home/sverma55]# more disk_alert Archived Log entry 22823 added for thread 4 sequence 5470 ID 0xfeb3f00b dest 1: Media Recovery Log +EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_4_seq_5470.2583.811151269 Media Recovery Log +EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_1_seq_5204.452.811150987 Wed Mar 27 07:48:02 2013 Archived Log entry 22824 added for thread 1 sequence 5205 ID 0xfeb3f00b dest 1: Wed Mar 27 07:48:14 2013 Media Recovery Log +EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_1_seq_5205.802.811151273 Wed Mar 27 07:48:32 2013 Media Recovery Waiting for thread 1 sequence 5206 (in transit) Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-27063: number of bytes read/written is incorrect Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc: ORA-27063: number of bytes read/written is incorrect Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3053.trc: ORA-27072: File I/O error HPUX-ia64 Error: 11: Resource temporarily unavailable Additional information: 4 Additional information: 3088 Additional information: -1 WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 au:14404 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3057.trc: ORA-27063: number of bytes read/written is incorrect WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk088 disk:0x8d.0xf0d708b4 au:24962Errors in file /apps/EBUSHA/ora cle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3059.trc: ORA-27072: File I/O error HPUX-ia64 Error: 11: Resource temporarily unavailable Wed Mar 27 07:48:14 2013 Media Recovery Log +EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_1_seq_5205.802.811151273 Wed Mar 27 07:48:32 2013 Media Recovery Waiting for thread 1 sequence 5206 (in transit) Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-27063: number of bytes read/written is incorrect Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc: ORA-27063: number of bytes read/written is incorrect Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3053.trc: ORA-27072: File I/O error HPUX-ia64 Error: 11: Resource temporarily unavailable Additional information: 4 Additional information: 3088 Additional information: -1 WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 au:14404 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3057.trc: ORA-27063: number of bytes read/written is incorrect WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk088 disk:0x8d.0xf0d708b4 au:24962Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3059.trc: ORA-27072: File I/O error HPUX-ia64 Error: 11: Resource temporarily unavailable Additional information: 4 Additional information: 3088 Additional information: -1 Wed Mar 27 08:00:48 2013 WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk100 disk:0x63.0xf0d708c2 au:3 iop:0x9fffffffbefbd320 bufp:0x9fffffffbf40fe00 offset(bytes):15103973376 iosz:1024 operation: 2(Write) synchronous:0 WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk100 disk:0x63.0xf0d708c2 au:3 iop:0x9fffffffbef9c880 bufp:0x9fffffffbf5f7e00 offset(bytes):3162112 iosz:16384 operation:1(Read) synchronous:1 iop:0x9fffffffbefbcff0 bufp:0x9fffffffbf40fe00 offset(bytes):26175529984 iosz:57344 operation:2(Write) synchronous:0Wed Mar 27 08:00:48 2013 iop:0x9fffffffbef9d320 bufp:0x9fffffffbf5f7e00 offset(bytes):3162112 iosz:16384 operation:1(Read) synchronous:1 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3055.trc: ORA-27072: File I/O error HPUX-ia64 Error: 11: Resource temporarily unavailable Additional information: 4 Additional information: 14352 Additional information: -1 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc: ORA-27063: number of bytes read/written is incorrect WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 au:10449 result: 4 osderr:0x0 osderr1:0x0 pid:3059 result: 4 osderr:0x0 osderr1:0x0 pid:3044 result: 4 osderr:0x0 osderr1:0x0 pid:3053Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc: ORA-15080: synchronous I/O operation to a disk failed WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk024 disk:0x2.0xf0d708d4 au:14 WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk083 disk:0x95.0xf0d708aa au:25034 iop:0x9fffffffbef9d210 bufp:0x9fffffffbc722e00 offset(bytes):10956650496 iosz:968704 operatio n:1(Read) synchronous:1 result: 4 osderr:0x0 osderr1:0x0 pid:3049 iop:0x9fffffffbef9d430 bufp:0x9fffffffbf5f7e00 offset(bytes):14696448 iosz:16384 operation:1(Read) synchronous:1WARNING: failed to read mirro r side 1 of virtual extent 0 logical extent 0 of file 484 in group 5 from disk EBUSHA_DATA_0099 allocation unit 3 reason error; if possible,will try another mirror side result: 4 osderr:0x0 osderr1:0x0 pid:3057 result: 4 osderr:0x0 osderr1:0x0 pid:3055 iop:0x9fffffffbefbccc0 bufp:0x9fffffffbf40fe00 offset(bytes):26250451968 iosz:1024 operation:2(Write) synchronous:0 WARNING: failed to write mirror side 1 of virtual extent 231 logical extent 0 of file 892 in group 4 on disk 0 allocation unit 14404 WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 484 in group 5 from disk EBUSHA_DATA_0099 allocation unit 3 reason error; if possible,will try another mirror side result: 4 osderr:0x0 osderr1:0x0 pid:3046WARNING: failed to read mirror side 1 of virtual extent 95 logical extent 0 of file 1795 in group 4 from disk EBUSHA_FRA_0022 allocation unit 10449 reason error; if possible,will try another mirror side Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-27063: number of bytes read/written is incorrect Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc: ORA-27063: number of bytes read/written is incorrect NOTE: successfully read mirror side 1 of virtual extent 0 logical extent 0 of file 484 in group [5.4046851930] from disk EBUSHA_DATA_0099 allocation unit 3 WARNING: IO Failed. subsys:System dg:4, di skname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 au:10449NOTE: successfully read mirror side 1 of virtual extent 0 logical extent 0 of file 484 in group [5.4046851930] from disk EBUSHA_DATA_0099 al location unit 3 WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 909 in group 4 from disk EBUSHA_FRA_0024 allocation unit 14 reason error; if possible,will try another mirror side Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc: ORA-00345: redo log write error block 236822 count 1 ORA-00312: online log 73 thread 4: '+EBUSHA_FRA/ebusha/onlinelog/group_73.892.808572333' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk024 disk:0x2.0xf0d708d4 au:14402Wed Mar 27 08:00:48 2013 ERROR: cannot read disk header of disk EBUSHA_DATA_0099 (99:4040624322) iop:0x9fffffffbefbccc0 bufp:0x9fffffffbee0fe00 offset(bytes):10957609984 iosz:9216 operation:2(Write) synchronous:0 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3051.trc: ORA-27063: number of bytes read/written is incorrect Wed Mar 27 08:00:48 2013 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc: ORA-27063: number of bytes read/written is incorrect WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk024 disk:0x2.0xf0d708d4 au:10902 result: 4 osderr:0x0 osderr1:0x0 pid:3049 iop:0x9fffffffbefbc440 bufp:0x9fffffffbf00fe0 0 offset(bytes):15101993984 iosz:1024 operation:2(Write) synchronous:0 NOTE: process 3059 initiating offline of disk 99.4040624322 (EBUSHA_DATA_0099) with mask 0x7e in group 5 result: 4 osderr:0x0 osderr1:0x0 pid:3046 iop:0x9fffffffbefbcaa0 bufp:0x9fffffffbf40fe00 offset(bytes):11432335360 iosz:2048 operation:2(Write) synchronous:0 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-15080: synchronous I/O operation to a disk failed result: 4 osderr:0x0 osderr1:0x0 pid:3051 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc: ORA-15080: synchronous I/O operation to a disk failed WARNING: failed to write mirror side 1 of virtual extent 95 logical extent 0 of file 1795 in group 4 on disk 0 allocation unit 10449 WARNING: failed to write mirror side 1 of virtual extent 79 logical extent 0 of file 515 in group 4 on disk 2 allocation unit 14402 NOTE: successfully read mirror side 1 of virtual extent 0 logical extent 0 of file 909 in group [4.2119569248] from disk EBUSHA_FRA_0024 allocation unit 14 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-15080: synchronous I/O operation to a disk failed Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3051.trc: ORA-15080: synchronous I/O operation to a disk failed WARNING: failed to write mirror side 1 of virtual extent 95 logical extent 0 of file 746 in group 5 on disk 141 allocation unit 24962 WARNING: failed to write mirror side 1 of virtual extent 270 logical extent 0 of file 1264 in group 4 on disk 2 allocation unit 10902 ERROR: cannot read disk header of disk EBUSHA_FRA_0024 (2:4040624340) NOTE: process 3055 initiating offline of disk 2.4040624340 (EBUSHA_FRA_0024) with mask 0x7e in group 4 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-00345: redo log write error block 98233 count 56 ORA-00312: online log 53 thread 2: '+EBUSHA_DATA/ebusha/onlinelog/group_53.746.808572245' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3051.trc: ORA-00345: redo log write error block 277222 count 2 ORA-00312: online log 11 thread 1: '+EBUSHA_FRA/ebusha/onlinelog/group_11.1264.808572261' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk NOTE: successfully read mirror side 1 of virtual extent 95 logical extent 0 of file 1795 in group [4.2119569248] from disk EBUSHA_FRA_0022 allocation unit 10449 WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/xpdisk148 disk:0x3.0xf0d708d1 au:4 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc: ORA-15080: synchronous I/O operation to a disk failed iop:0x9ffffffffd40c000 bufp:0x9ffffffffd5e7e00 offset(bytes):4489216 iosz:16384 operation:2(Write) synchronous:0 WARNING: failed to write mirror side 1 of virtual extent 79 logical extent 0 of file 769 in group 5 on disk 149 allocation unit 25034 result: 4 osderr:0x0 osderr1:0x0 pid:2837 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc: ORA-15080: synchronous I/O operation to a disk failed WARNING: failed to write mirror side 1 of virtual extent 104 logical extent 0 of file 909 in group 4 on disk 3 allocation unit 4 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc: ORA-00345: redo log write error block 81287 count 1 ORA-00312: online log 63 thread 3: '+EBUSHA_DATA/ebusha/onlinelog/group_63.769.808572319' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc: ORA-00206: error in writing (block 6786, # blocks 1) of control file ORA-00202: control file: '+EBUSHA_FRA/ebusha/control02.dbf' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc: ORA-00221: error on write to control file ORA-00206: error in writing (block 6786, # blocks 1) of control file ORA-00202: control file: '+EBUSHA_FRA/ebusha/control02.dbf' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk CKPT (ospid: 2837): terminating the instance due to error 221 Wed Mar 27 08:00:48 2013 System state dump is made for local instance System State dumped to trace file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_diag_2803.trc Instance terminated by CKPT, pid = 2837 urrtw41:[root::/home/sverma55]# urrtw41:[root::/home/sverma55]# urrtw41:[root::/home/sverma55]# pwd /home/sverma55 urrtw41:[root::/home/sverma55]# ll /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc -rw------- 1 ebusha oinstall 2358 Mar 27 08:00 /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc urrtw41:[root::/home/sverma55]# ioscan -fnC fc Class I H/W Path Driver S/W State H/W Type Description ====================================================================== fc 0 0/0/0/9/0/0/0 fclp CLAIMED INTERFACE HP 456972-B21 8Gb PCIe 2-port LPe1205 FC Mezzanine Adapter /dev/fclp0 fc 1 0/0/0/9/0/0/1 fclp CLAIMED INTERFACE HP 456972-B21 8Gb PCIe 2-port LPe1205 FC Mezzanine Adapter /dev/fclp1 fc 2 1/0/0/7/0/0/0 fclp CLAIMED INTERFACE HP 456972-B21 8Gb PCIe 2-port LPe1205 FC Mezzanine Adapter /dev/fclp2 fc 3 1/0/0/7/0/0/1 fclp CLAIMED INTERFACE HP 456972-B21 8Gb PCIe 2-port LPe1205 FC Mezzanine Adapter /dev/fclp3 urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp2 vpd V I T A L P R O D U C T D A T A --------- ------------- ------- Product Description : "HP BLc Emulex LPe1205-HP 8Gb FC" Part number : "456973-001" Part Serial number : "THC0290TC6" Engineering Date Code : "A-4927" Mfd. Date : "5029" Misc. Information : "PW=10W; PCIE X8 GEN 2" ROM Firmware version : "1.11A5 SLI-3 (U3F1.11A5)" EFI version : "UU6.03A8" urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp3 vpd V I T A L P R O D U C T D A T A --------- ------------- ------- Product Description : "HP BLc Emulex LPe1205-HP 8Gb FC" Part number : "456973-001" Part Serial number : "THC0290TC6" Engineering Date Code : "A-4927" Mfd. Date : "5029" Misc. Information : "PW=10W; PCIE X8 GEN 2" ROM Firmware version : "1.11A5 SLI-3 (U3F1.11A5)" EFI version : "UU6.03A8" urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp3 Vendor ID is = 0x10df Device ID is = 0xf100 PCI Sub-system Vendor ID is = 0x103c PCI Sub-system ID is = 0x1719 Chip version = 3 Firmware Version = 1.11A5 SLI-3 (U3F1.11A5) EFI Version = UU6.03A8 EFI Boot = ENABLED Driver-Firmware Dump Available = NO Driver-Firmware Dump Timestamp = N/A Topology = PTTOPT_FABRIC Link Speed = 2Gb Local N_Port_id is = 0xd60800 Previous N_Port_id is = None N_Port Node World Wide Name = 0x20000000c9f1e917 N_Port Port World Wide Name = 0x10000000c9f1e917 Switch Port World Wide Name = 0x200800051ecc3d40 Switch Node World Wide Name = 0x100000051ecc3d40 Driver state = ONLINE Hardware Path is = 1/0/0/7/0/0/1 Maximum Frame Size = 2048 Driver Version = @(#) FCLP: PCIe Fibre Channel driver (FibrChanl-02), B.11.31.1003.02, Feb 24 2010, FCLP_IFC (3,2) urrtw41:[root::/home/sverma55]# man fcmsutil fcmsutil(1M) fcmsutil(1M) NAME fcmsutil - Fibre Channel Mass Storage Utility Command for Fibre Channel Host Bus Adapters SYNOPSIS /opt/fcms/bin/fcmsutil device_file [options] The device_file specifies the Fibre Channel device special file associated with the Fibre Channel HBA port. The device file has the format /dev/FC_driverX, where X is the instance number of the Fibre Channel HBA port, as reported by the ioscan output. The [options] that follow the device file are the ones listed in the fcmsutil display usage for each type of Fibre Channel driver. The usage of fcmsutil can be seen by running the command alone. Options listed under the fcmsutil usage for different HBAs are supported for that HBA. In case an invalid device file is specified, that is, if the device file does not belong to the fc class, the fcmsutil command will return an error and display the usage options of fcmsutil. Note: For all options that take remote-N-Port-ID as an argument, remote-N-Port-ID can be replaced with -l loop_id (in Private Loop), or -w wwn. SECURITY RESTRICTIONS The usage of fcmsutil command is restricted to processes having super user privileges. Some of the options require detailed knowledge of the device specific adapter. See privileges(5) for more information about privileged access on systems that support fine-grained privileges. DESCRIPTION For a list of all supported Fibre Channel HBAs, please refer to the HP-UX Fibre Channel HBA Support Matrix available at: http://docs.hp.com/en/FCI-SM/FCI-SM.htm The fcmsutil command is a common diagnostic tool used for all Fibre Channel Host Bus Adapters supported on HP-UX. This command provides the ability to perform Fibre Channel Test and Echo functionality, read the card's registers, and so on. This command requires the use of a device file to indicate the interface over which the requested command needs to be performed. Options fcmsutil options are described below. Some options are HBA-specific and, therefore, are not supported by all Fibre Channel HBAs. Hewlett-Packard Company - 1 - HP-UX 11i Version 3: Aug 2007 fcmsutil(1M) fcmsutil(1M) The list of options supported by each HBA may be obtained by running fcmsutil without specifying any arguments. All keywords are case-insensitive and are position dependent. device_file Can be used alone or with other options. When used without any options it provides information such as the N_Port ID, Node World Wide Name and Port World Wide Name of the HBA and the switch (when applicable), Topology, the negotiated Link Speed, Vendor ID, Device ID, the Driver State, Hardware Path, Maximum Frame Size, and Driver Version. As per the FC protocol, Node WWN will be numerically one more than the Port WWN. The following topologies are defined: UNINITIALIZED. There is no connectivity to the host bus adapter port or the topology could not be determined. UNKNOWN. The Fibre Channel driver has returned a topology code that the utility does not understand. PRIVATE_LOOP. The host bus adapter is attached to a loop/loopback hood. PUBLIC_LOOP. The host adapter is attached to a loop connected to a Fibre Channel switch. IN_PTTOPT_NPORT/PTTOPT_NPORT. The host bus adapter has come up in a point to point topology. This topology can be an error if the card was expected to come up in loop topology. Not legal in most cases. PTTOPT_FABRIC. The host bus adapter has come up in a point to point topology when connected through a Fibre Channel switch. This topology can be an error if the card was expected to come up in loop topology. The following driver states are defined: LOOPBACK_STATE. The host bus adapter is in the loop back test Hewlett-Packard Company - 2 - HP-UX 11i Version 3: Aug 2007 fcmsutil(1M) fcmsutil(1M) phase. OFFLINE/DISABLED. The host bus adapter is not participating on the loop. This state is the result of user disabling the card through the disable option in fcmsutil or the Fibre Channel driver being unable to recover from an error. READY/ONLINE. The driver is up and functional. RESETTING. The host bus adapter is being reset. SUSPENDED. The driver has been suspended by the user. AWAITING_LINK_UP. The driver is waiting for the Fibre Channel link to come up. There is no connectivity to the HBA port. All other states are only transient and should not continue for long. If the transient state persists, there might be a problem in the hardware connectivity or configuration. The following link speeds are defined: UNINITIALIZED or UNKNOWN. The host bus adapter not could converge to a common link speed or adapter is not connected. 1Gb. The link is online and the operating speed is 1 gigabits per second. 2Gb. The link is online and the operating speed is 2 gigabits per second. 4Gb. The link is online and the operating speed is 4 gigabits per second. vpd This option is used to display "Vital Product Data" information of the HBA. It includes information such as the product description, part number, engineering date code, part serial number, and so on. This option is not supported by the TACHYON TL HBAs. echo remote-N-Port-ID data_size [count] This option is used to send an ECHO ELS frame on the Hewlett-Packard Company - 3 - HP-UX 11i Version 3: Aug 2007 fcmsutil(1M) fcmsutil(1M) wire. It requires two parameters, the remote-N-Port-ID and data-size (size of packet to send). An optional third argument (count) can be specified for the number of echo packets to be sent. If the count option is not specified, one packet will be sent. Fibre Channel Echo packets of the specified size are sent to the remote node. The command completes successfully when an echo response is received from the remote node and matches the data sent, for all packets sent. The command times out if a response is not received in twice RA_TOV time. Echo packets cannot be sent in a PUBLIC_LOOP topology. Note: Packet size specified must be a multiple of 4. rls remote-N-Port-ID This option is used to send an RLS (Request Link Status) ELS frame on the wire. It requires one parameter, the remote-N-Port-ID. The ELS is sent to this remote-N-Port-ID and the response data is displayed. test remote-N-Port-ID data_size [count] This option is used to send a TEST ELS on the wire. It requires two parameters, the remote-N-Port-ID and data-size (size of packet to send). An optional third argument (count) can be specified for the number of echo packets to be sent. If the count option is not specified, one packet will be sent. The command completes successfully and immediately on sending all the test packets. Note: Packet size specified must be a multiple of 4. read offset [pci] This option is used to read from HBA's internal registers. It requires one parameter, the offset of the register to read from. The offset can be specified in either hex or in decimal format. The offset specified is an offset from the base of the Memory Map. The user of this command is therefore expected to have internal knowledge of the chip. Reading from the TACHYON frame manager status register (0x01c8) is restricted. An optional second argument (pci) can be specified for Fibre Channel HBAs, to read from the PCI config space. If no second argument is specified, it reads from the chip register space. Hewlett-Packard Company - 4 - HP-UX 11i Version 3: Aug 2007 fcmsutil(1M) fcmsutil(1M) write offset value [pci] This option is used to write into HBA's registers. It requires two parameters, the offset of the register to write to and the value to be written. An optional third argument (pci) can be specified for the Fibre Channel HBAs, to write into the PCI config space. If no third argument is specified, it writes into the chip register space. [-f] lb This option is used to perform loopback tests on the port. Warning: This is a DESTRUCTIVE test, and DATA LOSS during the execution of this test may occur. The -f option can be used to suppress the warning message displayed by the Fibre Channel driver utility. For TL and XL2 HBAs, this option requires one parameter and an optional count: [-f] lb {plm|crpat|cjtpat} [count] Here plm refers to physical link module or gigabit link module, which builds the default payload for the loopback frame. If either crpat or cjtpat is used, then the card builds specific payloads based on the recommendations in Fibre Channel - Methodologies for Jitter Specifications. These patterns are designed to generate bit patterns which stress the transmit and receive channels of the card. The self test then involves sending a packet and receiving back the packet within the adapter and checking its integrity. Since this self test is at the adapter level, no packet goes on the fibre link. All Fibre Channel HBAs (except TL and XL2) need to specify two parameters. Here is the syntax: [-f] lb {ext|int} {crpat|cjpat} [count] The first parameter should be either ext or int to specify whether the loopback should be external or internal, respectively. The second parameter specifies the loopback pattern. Only crpat and cjtpat options are supported for these cards. Frames are looped back at the single bit interface in the Internal loopback mode. For external loopback, frames are sent out and received from the wire. External loopback mode is Hewlett-Packard Company - 5 - HP-UX 11i Version 3: Aug 2007 fcmsutil(1M) fcmsutil(1M) supported only in Loop topology. NOTE: In the internal loopback mode, frames are also sent out on the wire even though they are internally looped back at the 1 Bit interface. The receiver, however, is turned off during this operation. Therefore, it is not safe to run Internal loopback tests when the fiber is connected to a Switch or Hub, as the transmitted loopback frames can disrupt operation on the SAN. An optional third argument (count) can be specified for the number of loopback packets to be sent. If the count option is not specified, one packet will be sent. get local get fabric get remote {all|remote-N-Port-ID} This option is used to obtain Fibre Channel login parameters of either the local port, the fabric port, or a remote port. The Fibre Channel HBAs do not support the local option. If the all argument is specified for the remote option, login parameters and current states of all N_Ports that the initiator is aware of, are displayed. [-f] reset This option is used to reset the HBA (or a single FC port in case of multi-port HBAs). WARNING: This is a DESTRUCTIVE test. The reset operation will result in aborting communication to all nodes till the process is completed. The -f option can be used to suppress the warning message displayed by the Fibre Channel driver utility. [-f] bdr target-device_file This option is used to issue a Bus Device Reset to device. WARNING: This is a DESTRUCTIVE test. The -f option can be used to suppress the warning message displayed by the Fibre Channel driver utility. This option resets the target, clearing all commands, without doing any checks. Hewlett-Packard Company - 6 - HP-UX 11i Version 3: Aug 2007 fcmsutil(1M) fcmsutil(1M) read_cr This option can be used to read all of the readable registers on the card and format the detailed information. stat [-s] This option is used to obtain detailed statistics maintained by the driver. An optional argument -s can be specified to obtain a shortened version of the statistics maintained by the driver. Generally, the link statistics for the HBA port is displayed. clear_stat This option is used to clear the statistics maintained by the driver. nsstat This option is used to obtain detailed nameserver statistics maintained by the driver. clear_nsstat This option is used to clear the nameserver statistics maintained by the driver. devstat {all|remote-N-Port-ID} This option is used to obtain detailed statistics associated with each N_Port that this N_Port has communicated with. If the remote-N-Port-ID is specified, then the statistics associated with that N_Port are displayed. If the all option is specified, statistics associated with all N_Ports that the initiator has been able to communicate with are displayed. Along with the statistics for each N_Port, it also displays the loop_id (in Private Loop) and the nport_id (in Fabric). clear_devstat {all|remote-N-Port-ID} This option is used to clear the statistics associated with a target. If the remote-N-Port-ID is specified, then the statistics associated with that N_Port are cleared. If the all option is specified, statistics associated with all valid N_Port_IDs are cleared. replace_dsk OBSOLETED. Starting with HP-UX 11i Version 3, this option is no longer supported. On releases prior to HP-UX 11i Version 3, the Fibre Channel Tachyon TL, Tachyon TL2 and FCD drivers implemented an authentication mechanism to protect against accidental data corruption in case of the replacement of devices. This mechanism prevented I/O transfer when the target port Worldwide name (WWN) changed for the same remote N-Port-Id. The replace_dsk Hewlett-Packard Company - 7 - HP-UX 11i Version 3: Aug 2007 fcmsutil(1M) fcmsutil(1M) option was used to validate replacement of the disk by associating the new WWN with the remote N-Port-Id. Starting with HP-UX 11i Version 3, this authentication mechanism has been replaced by an enhanced mechanism based on the LUN Worldwide Identifier (WWID). Instead of the fcmsutil replace_dsk command, use the scsimgr commands, replace_wwid and replace_leg_dsf, to validate the replacement of devices. See the scsimgr(1M) manpage for more information. [-f] disable This option is used to disable the card. WARNING: This is a DESTRUCTIVE test and communication to all nodes will be terminated. The -f option can be used to suppress the warning message displayed by the Fibre Channel driver utility. This option is typically used when a hardware problem cannot be resolved and is interfering with system performance. enable This option is used to enable the card, typically when a previous hardware problem has been resolved. [-k] ns_query_ports This option is used to query the name server and get the list of nports for the Fibre Channel driver. The -k option is used to get the list of nports cached in the driver query buffer. The name server will not be queried in this case. [-f] dump_current_state This option is used to force the driver and firmware to dump their current state information and other data structures. WARNING: This is a DESTRUCTIVE operation. This might result in failure of current I/O requests. The -f option can be used to suppress the warning message displayed by the Fibre Channel driver utility. The dump data will be saved in the /tmp directory. The firmware dump will be stored in a file named FC- driverfw_date-timestamp.dmp and the driver dump will be saved in a file named FC-driverdrv_date-timestamp.dmp. dump_saved_state This option is used to retrieve firmware and driver Hewlett-Packard Company - 8 - HP-UX 11i Version 3: Aug 2007 fcmsutil(1M) fcmsutil(1M) dump saved in the driver memory. The driver initiates a dump when an internal error is encountered. Internal errors could be either due to firmware hang or to an irrecoverable error in the firmware or hardware. The dump files will be saved in the /tmp directory. The firmware dump will be stored in a file named FC- driverfw_date-timestamp.dmp and the driver dump will be saved in a file named FC-driverdrv_date-timestamp.dmp. These dumps should be sent to HP for further analysis of the problem. NOTE: The driver does not save any new dumps, until the previously saved dump is retrieved with this option. The availability of a saved dump can be checked by running fcmsutil device_file. dump_nvram This option is used to display the contents of NVRAM on the adapter. rom_fw_update ROM-firmware-file This option is used to update the ROM firmware stored in card's FLASH ROM. WARNING: This is a DESTRUCTIVE operation. Using this option may result in failure of current I/O requests. This option requires the name of a binary image file that contains the updated firmware. This operation should only be performed by qualified personnel. Failure to successfully complete the firmware update may result in adapter and/or system failure in case the boot disks are accessed through this card. efi_drv_update EFI-Driver-file This option is used to update the EFI driver stored in card's FLASH ROM. WARNING: This is a DESTRUCTIVE operation. Using this option may result in failure of current IO requests. This option requires the name of a binary image file that contains the EFI driver. This operation should only be performed by qualified personnel. Failure to successfully complete the EFI driver update may result in adapter and/or system boot up failure if the boot disks are accessed through this card. [-f] set_int_delay This option is used to set the interrupt delay mode and value, or to turn off interrupt delay. Settings made using this option are not persistent across reboots. Hewlett-Packard Company - 9 - HP-UX 11i Version 3: Aug 2007 fcmsutil(1M) fcmsutil(1M) WARNING: This is a DESTRUCTIVE operation and will abort communication to all target devices until the process is completed. The complete syntax for this command is: [-f] set_int_delay {off | [-z {5 | 6}] value} The -f option can be used to suppress the warning message. The off option turns off interrupt delay. The interrupt delay mode, also known as Zero Interrupt Operation (ZIO) mode, is set using the -z option. There are two interrupt delay modes available: 5 and 6. Mode 5 delays every interrupt by the interrupt delay period. Mode 6 delays an interrupt unless there are no active I/Os in the HBA port, in which case the interrupt is generated immediately. If the -z option is not specified, then mode 6 is used by default. The interrupt delay period is calculated from the interrupt delay value using the formula: value * 200 microseconds get_int_delay This option displays the current interrupt delay settings for the HBA port. sfp This option is used to display diagnostics information from the card's optical transceiver. It includes information from the SFF-8472 specification such as cable lengths, current temperature, voltage, transmit and receive power, TX bias, as well as other data. NOTE: This option is only supported by 4Gb/s capable Fibre Channel cards. EXAMPLES Print the remote port parameters using the get remote option if the driver is idle. In this example, /dev/td1 is the device file and /dev/rdsk/c27t0d0 is the respective raw disk file. fcmsutil /dev/td1 get remote 0x98 < /dev/rdsk/c27t0d0 Print a short listing of the statistics maintained by the driver, with /dev/td1 as the device file. fcmsutil /dev/td1 stat -s Hewlett-Packard Company - 10 - HP-UX 11i Version 3: Aug 2007 fcmsutil(1M) fcmsutil(1M) Send 5 echo packets of 200 bytes each to a remote N_Port with loop_id 4, with /dev/td1 as the device file. fcmsutil /dev/td1 echo -l 4 200 5 Print a short listing of the statistics of the device whose remote-N- Port-ID is 0x02ae4 and with /dev/td1 as the device file. fcmsutil /dev/td1 devstat 0x02ae4 Clear the device statistics of the device whose wwn is 0x100000e002219f45 and with /dev/fcd2 as the device file. fcmsutil /dev/fcd2 clear_devstat -w 0x100000e002219f45 Perform a Internal loopback test, sending 1000 packets with /dev/fcd2 as the device file. fcmsutil /dev/fcd2 lb int crpat 1000 Display diagnostics information from the HBA's optical transceiver with /dev/fcd2 as the device file. fcmsutil /dev/fcd2 sfp SEE ALSO ioscan(1M), privileges(5). AUTHOR /opt/fcms/bin/fcmsutil was developed by HP. Hewlett-Packard Company - 11 - HP-UX 11i Version 3: Aug 2007 urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp2 stat -s ****************************************************************************** **** Driver Statistics **** ****************************************************************************** Thu Mar 28 12:58:52 2013 --- Link Statistics ------------------------------------------------------ Link Failure count 0 Loss of Sync count 1 Loss of Signal count 0 Prim seq proto err count 0 Invalid Word count 118 Invalid CRC count 1 Prim seq Timeout count 0 Elstc Bufr Overrun count 11696 FC-AL Arb Timeout count 0 Advertised Recv B2B cred 20 Current Recv B2B cred 20 Advertised Xmit B2B cred 8 Current Xmit B2B cred 8 Received EOFa count 0 Received EOFdti count 0 Received EOFni count 0 Received SOFf count 0 urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp3 stat -s ****************************************************************************** **** Driver Statistics **** ****************************************************************************** Thu Mar 28 12:59:06 2013 --- Link Statistics ------------------------------------------------------ Link Failure count 0 Loss of Sync count 1 Loss of Signal count 0 Prim seq proto err count 0 Invalid Word count 89 Invalid CRC count 1 Prim seq Timeout count 0 Elstc Bufr Overrun count 20821 FC-AL Arb Timeout count 0 Advertised Recv B2B cred 20 Current Recv B2B cred 20 Advertised Xmit B2B cred 8 Current Xmit B2B cred 8 Received EOFa count 0 Received EOFdti count 0 Received EOFni count 0 Received SOFf count 0 urrtw41:[root::/home/sverma55]# sfp urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp2 sfp fcmsutil /dev/fclp2: Unable to read SFP information. urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp3 sfp fcmsutil /dev/fclp3: Unable to read SFP information. urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp2 Vendor ID is = 0x10df Device ID is = 0xf100 PCI Sub-system Vendor ID is = 0x103c PCI Sub-system ID is = 0x1719 Chip version = 3 Firmware Version = 1.11A5 SLI-3 (U3F1.11A5) EFI Version = UU6.03A8 EFI Boot = ENABLED Driver-Firmware Dump Available = NO Driver-Firmware Dump Timestamp = N/A Topology = PTTOPT_FABRIC Link Speed = 2Gb Local N_Port_id is = 0x720800 Previous N_Port_id is = None N_Port Node World Wide Name = 0x20000000c9f1e916 N_Port Port World Wide Name = 0x10000000c9f1e916 Switch Port World Wide Name = 0x200800051ecc3d2a Switch Node World Wide Name = 0x100000051ecc3d2a Driver state = ONLINE Hardware Path is = 1/0/0/7/0/0/0 Maximum Frame Size = 2048 Driver Version = @(#) FCLP: PCIe Fibre Channel driver (FibrChanl-02), B.11.31.1003.02, Feb 24 2010, FCLP_IFC (3,2) urrtw41:[root::/home/sverma55]# fcmsutil /dev/fclp1 Vendor ID is = 0x10df Device ID is = 0xf100 PCI Sub-system Vendor ID is = 0x103c PCI Sub-system ID is = 0x1719 Chip version = 3 Firmware Version = 1.11A5 SLI-3 (U3F1.11A5) EFI Version = UU6.03A8 EFI Boot = ENABLED Driver-Firmware Dump Available = NO Driver-Firmware Dump Timestamp = N/A Previous Topology = UNINITIALIZED Link Speed = UNKNOWN Local N_Port_id is = None Previous N_Port_id is = None N_Port Node World Wide Name = 0x20000000c9f1e805 N_Port Port World Wide Name = 0x10000000c9f1e805 Switch Port World Wide Name = 0x0000000000000000 Switch Node World Wide Name = 0x0000000000000000 Driver state = AWAITING_LINK_UP Hardware Path is = 0/0/0/9/0/0/1 Maximum Frame Size = 0 Driver Version = @(#) FCLP: PCIe Fibre Channel driver (FibrChanl-02), B.11.31.1003.02, Feb 24 2010, FCLP_IFC (3,2) 513