topic Re: SQL Server Proformance issues BL685c g6 in BladeSystem - General

SQL Server Proformance issues BL685c g6

larryb — Mon, 19 Apr 2010 11:10:30 GMT

We are having an issue with performance on our Microsoft SQL server Cluster. We are getting two events in our application event log event 833 and 19019. On occasion the SQL service will fail over. The details on the events are below. The cluster is built on two BL685 g6 servers with 4x6 AMD CPUs and 64 GB of memory connected to a EMC CX4-20 SAN through two Brocade HP B-series 8/24c SAN Switches.
We have cases open with Microsoft and EMC but to date they have not solve the problem. Dose anyone have any insight on why this may be happening.

The event # 833 “Event Type: Information
Event Source: MSSQLSERVER
Event Category: (2)
Event ID: 833
Date: 4/16/2010
Time: 6:38:06 AM
User: N/A
Computer: ATL-SQL-3
Description:
SQL Server has encountered 35 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [M:\SQL-Data\RCC.mdf] in database [RCC] (75). The OS file handle is 0x0000000000001274. The offset of the latest long I/O is: 0x00000000012000

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 03 00 00 0a 00 00 00 A.......
0008: 0a 00 00 00 41 00 54 00 ....A.T.
0010: 4c 00 2d 00 53 00 51 00 L.-.S.Q.
0018: 4c 00 2d 00 33 00 00 00 L.-.3...
0020: 00 00 00 00 ....

Event 19019
Event Type: Error
Event Source: MSSQLSERVER
Event Category: (3)
Event ID: 19019
Date: 4/16/2010
Time: 11:15:05 AM
User: N/A
Computer: ATL-SQL-3C
Description:
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 4b 4a 00 40 01 00 00 00 KJ.@....
0008: 0a 00 00 00 41 00 54 00 ....A.T.
0010: 4c 00 2d 00 53 00 51 00 L.-.S.Q.
0018: 4c 00 2d 00 33 00 00 00 L.-.3...
0020: 00 00 00 00 ....

Event Type: Error
Event Source: MSSQLSERVER
Event Category: (3)
Event ID: 19019
Date: 4/16/2010
Time: 11:15:05 AM
User: N/A
Computer: ATL-SQL-3C
Description:
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][SQL Native Client]Communication link failure

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 4b 4a 00 40 01 00 00 00 KJ.@....
0008: 0a 00 00 00 41 00 54 00 ....A.T.
0010: 4c 00 2d 00 53 00 51 00 L.-.S.Q.
0018: 4c 00 2d 00 33 00 00 00 L.-.3...
0020: 00 00 00 00 ....

Re: SQL Server Proformance issues BL685c g6

Al Kemper — Mon, 26 Apr 2010 18:20:56 GMT

Have you looked into the power management issue described in http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02110402

Re: SQL Server Proformance issues BL685c g6

Markusk — Sat, 17 Jul 2010 17:23:08 GMT

Hello Larry,
have you solved the SQL server issue (error 833)? What is the solution? We have the same with HP storage and DL380G6. A feedback would be great.

Thanks for your support
MArkus

Re: SQL Server Proformance issues BL685c g6

larryb — Mon, 19 Jul 2010 12:10:23 GMT

Hi all,

We found that the cause of this error is that the server is pushing/sending commands to the disks faster than the SAN/disks can handle them.

The solution for us was to add more disks.

There is more detail to this but I don't have time to go into this right now.

Larry

Re: SQL Server Proformance issues BL685c g6

James McGovern — Fri, 12 Nov 2010 13:51:52 GMT

Saw your question regarding I/O requests taking longer than 15 seconds. Can anyone share what other resolutions they have taken besides the Power management bulletin?

Re: SQL Server Proformance issues BL685c g6

Markusk — Sat, 13 Nov 2010 11:41:49 GMT

Queue Depth of the HBA set to 8 (in registry).
TempDB os SQl on local disks, not in SAN.

after this action, latency got down from 300ms peaks to 10-15ms