I have had a response on the Oracle SR to request that other LOS perform the same checks as they did for the EBUSPROD issue, I've pasted their update at the bottom. For reference EBUSHA is single instance standby database sat on urrtw41, it reported it could not contact multiple disks from 07:48 today. I list a few but attach the relevant part of the alert log for exact details and timestamps. On a daily basis at 7pm it is shutdown, its disk unmounted from thr rtw prod nodes and mounted on urrtw01a/01b and an rman duplicate activity is performed to created our daily clone. EBUSHA is normally back on urrtw41 at about 2am each day and catches up with EBUSPROD by 4am. WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk088 disk:0x8d.0xf0d708b4 WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk100 disk:0x63.0xf0d708c2 Oracle request... + Consider all the checks we discussed so far for the issue on Prod case, on the standby machines as well. Starting with identifying the disks, and checking OS logs and storage logs, kernel parameter setting, Let me know the feedback from the other LOS and I can update the SR with anything you feel relevant Sarah urrtw41:[root::/home/sverma55]# more disk_alert Archived Log entry 22823 added for thread 4 sequence 5470 ID 0xfeb3f00b dest 1: Media Recovery Log +EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_4_seq_5470.2583.811151269 Media Recovery Log +EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_1_seq_5204.452.811150987 Wed Mar 27 07:48:02 2013 Archived Log entry 22824 added for thread 1 sequence 5205 ID 0xfeb3f00b dest 1: Wed Mar 27 07:48:14 2013 Media Recovery Log +EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_1_seq_5205.802.811151273 Wed Mar 27 07:48:32 2013 Media Recovery Waiting for thread 1 sequence 5206 (in transit) Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-27063: number of bytes read/written is incorrect Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc: ORA-27063: number of bytes read/written is incorrect Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3053.trc: ORA-27072: File I/O error HPUX-ia64 Error: 11: Resource temporarily unavailable Additional information: 4 Additional information: 3088 Additional information: -1 WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 au:14404 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3057.trc: ORA-27063: number of bytes read/written is incorrect WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk088 disk:0x8d.0xf0d708b4 au:24962Errors in file /apps/EBUSHA/ora cle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3059.trc: ORA-27072: File I/O error HPUX-ia64 Error: 11: Resource temporarily unavailable Wed Mar 27 07:48:14 2013 Media Recovery Log +EBUSHA_FRA/ebusha/archivelog/2013_03_27/thread_1_seq_5205.802.811151273 Wed Mar 27 07:48:32 2013 Media Recovery Waiting for thread 1 sequence 5206 (in transit) Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Wed Mar 27 08:00:48 2013 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-27063: number of bytes read/written is incorrect Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc: ORA-27063: number of bytes read/written is incorrect Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3053.trc: ORA-27072: File I/O error HPUX-ia64 Error: 11: Resource temporarily unavailable Additional information: 4 Additional information: 3088 Additional information: -1 WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 au:14404 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3057.trc: ORA-27063: number of bytes read/written is incorrect WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk088 disk:0x8d.0xf0d708b4 au:24962Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3059.trc: ORA-27072: File I/O error HPUX-ia64 Error: 11: Resource temporarily unavailable Additional information: 4 Additional information: 3088 Additional information: -1 Wed Mar 27 08:00:48 2013 WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk100 disk:0x63.0xf0d708c2 au:3 iop:0x9fffffffbefbd320 bufp:0x9fffffffbf40fe00 offset(bytes):15103973376 iosz:1024 operation: 2(Write) synchronous:0 WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk100 disk:0x63.0xf0d708c2 au:3 iop:0x9fffffffbef9c880 bufp:0x9fffffffbf5f7e00 offset(bytes):3162112 iosz:16384 operation:1(Read) synchronous:1 iop:0x9fffffffbefbcff0 bufp:0x9fffffffbf40fe00 offset(bytes):26175529984 iosz:57344 operation:2(Write) synchronous:0Wed Mar 27 08:00:48 2013 iop:0x9fffffffbef9d320 bufp:0x9fffffffbf5f7e00 offset(bytes):3162112 iosz:16384 operation:1(Read) synchronous:1 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3055.trc: ORA-27072: File I/O error HPUX-ia64 Error: 11: Resource temporarily unavailable Additional information: 4 Additional information: 14352 Additional information: -1 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc: ORA-27063: number of bytes read/written is incorrect WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 au:10449 result: 4 osderr:0x0 osderr1:0x0 pid:3059 result: 4 osderr:0x0 osderr1:0x0 pid:3044 result: 4 osderr:0x0 osderr1:0x0 pid:3053Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc: ORA-15080: synchronous I/O operation to a disk failed WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk024 disk:0x2.0xf0d708d4 au:14 WARNING: IO Failed. subsys:System dg:5, diskname:/dev/oracle/xp12disk083 disk:0x95.0xf0d708aa au:25034 iop:0x9fffffffbef9d210 bufp:0x9fffffffbc722e00 offset(bytes):10956650496 iosz:968704 operatio n:1(Read) synchronous:1 result: 4 osderr:0x0 osderr1:0x0 pid:3049 iop:0x9fffffffbef9d430 bufp:0x9fffffffbf5f7e00 offset(bytes):14696448 iosz:16384 operation:1(Read) synchronous:1WARNING: failed to read mirro r side 1 of virtual extent 0 logical extent 0 of file 484 in group 5 from disk EBUSHA_DATA_0099 allocation unit 3 reason error; if possible,will try another mirror side result: 4 osderr:0x0 osderr1:0x0 pid:3057 result: 4 osderr:0x0 osderr1:0x0 pid:3055 iop:0x9fffffffbefbccc0 bufp:0x9fffffffbf40fe00 offset(bytes):26250451968 iosz:1024 operation:2(Write) synchronous:0 WARNING: failed to write mirror side 1 of virtual extent 231 logical extent 0 of file 892 in group 4 on disk 0 allocation unit 14404 WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 484 in group 5 from disk EBUSHA_DATA_0099 allocation unit 3 reason error; if possible,will try another mirror side result: 4 osderr:0x0 osderr1:0x0 pid:3046WARNING: failed to read mirror side 1 of virtual extent 95 logical extent 0 of file 1795 in group 4 from disk EBUSHA_FRA_0022 allocation unit 10449 reason error; if possible,will try another mirror side Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-27063: number of bytes read/written is incorrect Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc: ORA-27063: number of bytes read/written is incorrect NOTE: successfully read mirror side 1 of virtual extent 0 logical extent 0 of file 484 in group [5.4046851930] from disk EBUSHA_DATA_0099 allocation unit 3 WARNING: IO Failed. subsys:System dg:4, di skname:/dev/oracle/evadisk022 disk:0x0.0xf0d708d2 au:10449NOTE: successfully read mirror side 1 of virtual extent 0 logical extent 0 of file 484 in group [5.4046851930] from disk EBUSHA_DATA_0099 al location unit 3 WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 909 in group 4 from disk EBUSHA_FRA_0024 allocation unit 14 reason error; if possible,will try another mirror side Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3044.trc: ORA-00345: redo log write error block 236822 count 1 ORA-00312: online log 73 thread 4: '+EBUSHA_FRA/ebusha/onlinelog/group_73.892.808572333' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk024 disk:0x2.0xf0d708d4 au:14402Wed Mar 27 08:00:48 2013 ERROR: cannot read disk header of disk EBUSHA_DATA_0099 (99:4040624322) iop:0x9fffffffbefbccc0 bufp:0x9fffffffbee0fe00 offset(bytes):10957609984 iosz:9216 operation:2(Write) synchronous:0 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3051.trc: ORA-27063: number of bytes read/written is incorrect Wed Mar 27 08:00:48 2013 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc: ORA-27063: number of bytes read/written is incorrect WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/evadisk024 disk:0x2.0xf0d708d4 au:10902 result: 4 osderr:0x0 osderr1:0x0 pid:3049 iop:0x9fffffffbefbc440 bufp:0x9fffffffbf00fe0 0 offset(bytes):15101993984 iosz:1024 operation:2(Write) synchronous:0 NOTE: process 3059 initiating offline of disk 99.4040624322 (EBUSHA_DATA_0099) with mask 0x7e in group 5 result: 4 osderr:0x0 osderr1:0x0 pid:3046 iop:0x9fffffffbefbcaa0 bufp:0x9fffffffbf40fe00 offset(bytes):11432335360 iosz:2048 operation:2(Write) synchronous:0 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-15080: synchronous I/O operation to a disk failed result: 4 osderr:0x0 osderr1:0x0 pid:3051 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc: ORA-15080: synchronous I/O operation to a disk failed WARNING: failed to write mirror side 1 of virtual extent 95 logical extent 0 of file 1795 in group 4 on disk 0 allocation unit 10449 WARNING: failed to write mirror side 1 of virtual extent 79 logical extent 0 of file 515 in group 4 on disk 2 allocation unit 14402 NOTE: successfully read mirror side 1 of virtual extent 0 logical extent 0 of file 909 in group [4.2119569248] from disk EBUSHA_FRA_0024 allocation unit 14 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-15080: synchronous I/O operation to a disk failed Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3051.trc: ORA-15080: synchronous I/O operation to a disk failed WARNING: failed to write mirror side 1 of virtual extent 95 logical extent 0 of file 746 in group 5 on disk 141 allocation unit 24962 WARNING: failed to write mirror side 1 of virtual extent 270 logical extent 0 of file 1264 in group 4 on disk 2 allocation unit 10902 ERROR: cannot read disk header of disk EBUSHA_FRA_0024 (2:4040624340) NOTE: process 3055 initiating offline of disk 2.4040624340 (EBUSHA_FRA_0024) with mask 0x7e in group 4 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3049.trc: ORA-00345: redo log write error block 98233 count 56 ORA-00312: online log 53 thread 2: '+EBUSHA_DATA/ebusha/onlinelog/group_53.746.808572245' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3051.trc: ORA-00345: redo log write error block 277222 count 2 ORA-00312: online log 11 thread 1: '+EBUSHA_FRA/ebusha/onlinelog/group_11.1264.808572261' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk NOTE: successfully read mirror side 1 of virtual extent 95 logical extent 0 of file 1795 in group [4.2119569248] from disk EBUSHA_FRA_0022 allocation unit 10449 WARNING: IO Failed. subsys:System dg:4, diskname:/dev/oracle/xpdisk148 disk:0x3.0xf0d708d1 au:4 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc: ORA-15080: synchronous I/O operation to a disk failed iop:0x9ffffffffd40c000 bufp:0x9ffffffffd5e7e00 offset(bytes):4489216 iosz:16384 operation:2(Write) synchronous:0 WARNING: failed to write mirror side 1 of virtual extent 79 logical extent 0 of file 769 in group 5 on disk 149 allocation unit 25034 result: 4 osderr:0x0 osderr1:0x0 pid:2837 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc: ORA-15080: synchronous I/O operation to a disk failed WARNING: failed to write mirror side 1 of virtual extent 104 logical extent 0 of file 909 in group 4 on disk 3 allocation unit 4 Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_rfs_3046.trc: ORA-00345: redo log write error block 81287 count 1 ORA-00312: online log 63 thread 3: '+EBUSHA_DATA/ebusha/onlinelog/group_63.769.808572319' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc: ORA-00206: error in writing (block 6786, # blocks 1) of control file ORA-00202: control file: '+EBUSHA_FRA/ebusha/control02.dbf' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk Errors in file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_ckpt_2837.trc: ORA-00221: error on write to control file ORA-00206: error in writing (block 6786, # blocks 1) of control file ORA-00202: control file: '+EBUSHA_FRA/ebusha/control02.dbf' ORA-15081: failed to submit an I/O operation to a disk ORA-15081: failed to submit an I/O operation to a disk CKPT (ospid: 2837): terminating the instance due to error 221 Wed Mar 27 08:00:48 2013 System state dump is made for local instance System State dumped to trace file /apps/EBUSHA/oracle/diag/rdbms/ebusha/EBUSHA3/trace/EBUSHA3_diag_2803.trc Instance terminated by CKPT, pid = 2837