SQL Server having I/O issues with EVA6000 and high invalid CRC count

[Eric Domazlicky]

We have a SQL 2005 server connected to our EVA6000 SAN. It is having problems with slow I/O that seems to come in waves with no explanation. This is causing many Event Id 833s (SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file) and SQL query connection timeouts. I have tried taking performance measurements with EVAPerf but it doesn't seem to reveal anything useful. Everything seems to be within standard performance parameters even when the timeouts happen. The only thing I am seeing is on SANSurfer one of the FC ports has a very high Invalid CRC count of 85000+ while the other port has none. Could thses Invalid CRCs explain the slow I/O?

[Cmorrall]

You might be having a physically bad connection somewhere between the array and the host.
Assuming you're using Brocade switches, please post the output from "porterrshow" on the switch where the suspicious HBA is connected.

[Eric Domazlicky]

Good idea- the SQL Server is on this port- here is what a portshow says:

XXXX:admin> portshow #
portName: XXX
portHealth: No Fabric Watch License

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x24b03 PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT
portType: 10.0
portState: 1 Online
portPhys: 6 In_Sync
portScn: 32 F_Port
port generation number: 132
portId: 020700
portIfId: 43020004
portWwn: XXX
portWwn of device(s) connected:
XXX
Distance: normal
portSpeed: 2Gbps

LE domain: 0
Interrupts: 516805 Link_failure: 12 Frjt: 0
Unknown: 0 Loss_of_sync: 11 Fbsy: 0
Lli: 516789 Loss_of_sig: 2
Proc_rqrd: 437 Protocol_err: 0
Timed_out: 0 Invalid_word: 0
Rx_flushed: 0 Invalid_crc: 0
Tx_unavail: 0 Delim_err: 0
Free_buffer: 0 Address_err: 0
Overrun: 0 Lr_in: 258361
Suspended: 0 Lr_out: 14
Parity_err: 0 Ols_in: 10
2_parity_err: 0 Ols_out: 15
CMI_bus_err: 0

Port part of other ADs: No


But yet SANSurfer reports 85000+ Invalid CRC errors on this same port. So it seems the HBA sees errors but the switch doesn't. Any ideas?

[Uwe Zessin]

Yes, looks like the path from the switch to the adapter has a quality problem. Maybe its a bad laser in the SFP, a bad cable, a bad receiver or something simple as a connector that has not 'snapped in' properly.

[Steven Clementi]

I've seen weird tings with SAN Switches.

Try reseating the transceiver kit (gbic).

As stated, check for crimped/pinched cables, bad/loose connections, etc.

how long has this been going on?


Steven

[troyano]

Hi,

I have the same problem beetween an EVA8000 and a Oracle server...

Could you solve the problem? How?

Thanks.

[Sivakumar MJ._1]

Try replacing the GBIC Card, Checking the End to end connvectivity (host to switch to storage) should be good. Also check the Host HBA Card...