SQL Server Proformance issues BL685c g6

[larryb]

We are having an issue with performance on our Microsoft SQL server Cluster. We are getting two events in our application event log event 833 and 19019. On occasion the SQL service will fail over. The details on the events are below. The cluster is built on two BL685 g6 servers with 4x6 AMD CPUs and 64 GB of memory connected to a EMC CX4-20 SAN through two Brocade HP B-series 8/24c SAN Switches.
We have cases open with Microsoft and EMC but to date they have not solve the problem. Dose anyone have any insight on why this may be happening.

The event # 833 “Event Type: Information
Event Source: MSSQLSERVER
Event Category: (2)
Event ID: 833
Date: 4/16/2010
Time: 6:38:06 AM
User: N/A
Computer: ATL-SQL-3
Description:
SQL Server has encountered 35 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [M:\SQL-Data\RCC.mdf] in database [RCC] (75). The OS file handle is 0x0000000000001274. The offset of the latest long I/O is: 0x00000000012000

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 03 00 00 0a 00 00 00 A.......
0008: 0a 00 00 00 41 00 54 00 ....A.T.
0010: 4c 00 2d 00 53 00 51 00 L.-.S.Q.
0018: 4c 00 2d 00 33 00 00 00 L.-.3...
0020: 00 00 00 00 ....

Event 19019
Event Type: Error
Event Source: MSSQLSERVER
Event Category: (3)
Event ID: 19019
Date: 4/16/2010
Time: 11:15:05 AM
User: N/A
Computer: ATL-SQL-3C
Description:
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 4b 4a 00 40 01 00 00 00 KJ.@....
0008: 0a 00 00 00 41 00 54 00 ....A.T.
0010: 4c 00 2d 00 53 00 51 00 L.-.S.Q.
0018: 4c 00 2d 00 33 00 00 00 L.-.3...
0020: 00 00 00 00 ....

Event Type: Error
Event Source: MSSQLSERVER
Event Category: (3)
Event ID: 19019
Date: 4/16/2010
Time: 11:15:05 AM
User: N/A
Computer: ATL-SQL-3C
Description:
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][SQL Native Client]Communication link failure


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 4b 4a 00 40 01 00 00 00 KJ.@....
0008: 0a 00 00 00 41 00 54 00 ....A.T.
0010: 4c 00 2d 00 53 00 51 00 L.-.S.Q.
0018: 4c 00 2d 00 33 00 00 00 L.-.3...
0020: 00 00 00 00 ....

[Al Kemper]

Have you looked into the power management issue described in http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02110402

[Markusk]

Hello Larry,
have you solved the SQL server issue (error 833)? What is the solution? We have the same with HP storage and DL380G6. A feedback would be great.

Thanks for your support
MArkus

[larryb]

Hi all,

We found that the cause of this error is that the server is pushing/sending commands to the disks faster than the SAN/disks can handle them.

The solution for us was to add more disks.

There is more detail to this but I don't have time to go into this right now.

Larry

[James McGovern]

Saw your question regarding I/O requests taking longer than 15 seconds. Can anyone share what other resolutions they have taken besides the Power management bulletin?

[Markusk]

Queue Depth of the HBA set to 8 (in registry).
TempDB os SQl on local disks, not in SAN.

after this action, latency got down from 300ms peaks to 10-15ms