topic Re: Poor performance P4500 Campus SAN with 4 x switches in StoreVirtual Storage

Poor performance P4500 Campus SAN with 4 x switches

Paul Large — Fri, 18 Mar 2011 11:21:42 GMT

We have a P4500 campus SAN that was originally connected to 2 x 3Com GB switches with 2Gb trunk connection using VMware Round Robin and two GB connections per ESX host - 1 switch per site. To provide switch redundancy at each site, we added a second switch to each site - Site 1 now has 2 x switches with 2Gb trunk, site 2 has the same and we have a separate 2GB trunk linking the sites. when we move P4500 so that one GB connection goes into each switch the performance decreases significantly. If we move both GB connections to the same switch then performance is much better.

any ideas?

Re: Poor performance P4500 Campus SAN with 4 x switches

Cajuntank MS — Sat, 19 Mar 2011 03:41:17 GMT

Have you read the best practice for performance article dealing with jumbo frames, adaptive load balancing, and flow control? http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01750150/c01750150.pdf?jumpid=reg_R1002_USEN

Re: Poor performance P4500 Campus SAN with 4 x switches

Paul Large — Sat, 19 Mar 2011 13:27:59 GMT

Yes, I have LACP for 2GB connection (2 x 1GB) between the switches and Flow Control enabled on all switch ports. Does flow control need to be enabled anywhere else - on the P4000 for example?.

Have not enabled Jumbo frames as I have read that it rarely makes any difference.

Re: Poor performance P4500 Campus SAN with 4 x switches

Cajuntank MS — Sat, 19 Mar 2011 20:47:49 GMT

Yes. Flow control must be an end to end implementation. NICs on SAN nodes, ports on switches, and NICs on servers.
You said you have LACP enabled between your switches but did not say what you were doing on your SAN...ALB? LACP?

On the Jumbo Frames part, some switches don't play nice with Jumbo Frames and Flow Control turned on at the same time. I have HP Procurve switches which I have not had an issue with myself.

Re: Poor performance P4500 Campus SAN with 4 x switches

Paul Large — Sun, 20 Mar 2011 20:21:14 GMT

OK, I did not have Flow Control enabled on the P4000 which I have now enabled. Initial tests show a great improvement in performance :)

Re: Poor performance P4500 Campus SAN with 4 x switches

Jitun — Sun, 20 Mar 2011 20:27:17 GMT

Here is a small list that I check...
Please check on the Node, Switch and Host Server
1.) Enable Flow Control on network switches and adapters. Flow control ensures a receiver can make the sender pace its speed and is important in avoiding data loss. (Recommended)
2.) Ensure Spanning Tree Algorithm for detecting loops is turned OFF. Loop detection introduces a delay in making a port become usable for data transfer and may lead to application timeouts.
3.) Segregate SAN and LAN traffic. iSCSI SAN interfaces should be separated from other corporate network traffic (LAN).

Networking best practices
â ¢Use Non-Blocking switches and set the negotiated speed on the switches.
â ¢Disable unicast storm control on iSCSI ports. Most switches have unicast storm control disabled by default. If your switch has this enabled, you should disable this on the ports connected to iSCSI hosts and targets to avoid packet loss.
â ¢Servers should use dedicated NICs for SAN traffic. Deploying iSCSI disks on a separate network helps to minimize network congestion and latency. Additionally, iSCSI volumes are more secure whenâ ¦ Segregate SAN & LAN traffic can be separated using port based VLANs or physically separate networks.
â ¢Configure additional Paths for High Availability; use either Microsoft MPIO or MCS (multiple connections per session) with additional NICs in the server to create additional connections to the iSCSI storage array through redundant Ethernet switch fabrics.
â ¢Unbind File and Print Sharing from the iSCSI NIC â on the NICs which connect only to the iSCSI SAN, unbind File and Print Sharing.

Re: Poor performance P4500 Campus SAN with 4 x switches

Snakehead — Sat, 31 Mar 2012 14:34:50 GMT

Hi,

I got P4500G2 x 2, ESXi5 x 2, H3C switch. All running @ 10GB.

P4500G2 network raid 10, (802.3ad bonding).

ESXi5 x 2 with software iscsi, storage path (RR).

H3C switch using (IRF=Stack).

We have 2 guest windows to copy data and delete from each another. It just a simple test, however I see the disk queue length over 6 on the CMC monitor. Besides we found the error message of the disk lantency increased come out frequency.

Please help...

Re: Poor performance P4500 Campus SAN with 4 x switches

David_Tocker — Mon, 14 May 2012 09:53:48 GMT

A disk queue lenght over 6 is not probably something to worry about. We have one cluster in our setup (P4300 g2 X 2) that can hit upto about 80 on the queue lengh and the customers do not seem to notice. Real world testing is always going to better than numbers in the CMC. We have found that we can run upto about 300 RDS users with their SBS servers on a single pair of P4300s without the customers reporting any performance issues. Throughput seems to be much less important than latency, make sure you have flow control enabled on the iSCSI ports on your VM hosts and the P4000 ports on your switches. Jumbo frames appear to cause issues on the procurve 2810 switches, i dont know about the 2910 als?