iSCSI performance issue

[DAC_1]

Hi There,

Over the last couple of months we have been experiencing extreme performance issues with our SAN and iSCSI network to the point where we have to shut down all connecting systems and bring them back up 1 by 1 for the issue to go away. When the problem occurs all running VM’s on both VMware and Citrix XenDesktop become either extremely slow or non-responsive and the ESX/Xen hosts stop seeing the iSCSI storage and our Exchange servers will take the storage groups off line since it cannot see the iSCSI storage.

A quick overview of our setup is as follows.

Dell EqualLogic 4 members containing 3 x PS5000 (Model 70-0115, 2 x 70-0111) and 1 x PS6000 (70-0202) connecting to 2 x Cisco 2960S running OS 12.2. All ports on the SAN are set at 9000 MTU and all ports on the switches are set to 9000 MTU for jumbo frames.

In to these two switches the following are connected – the switches have been setup as follows http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps6021/white_paper_c11-563477.html.

VMware 4 vSphere running on HP ProLiant DL585 G5's with 4x4core AMD 2.2Ghz processors and 70Gb of memory. VMware connects to the SAN using a QLE406xc iSCSI HBA and set at 9000 MTU

XenDesktop 4 running on a HP blade system C7000 with BL560. using a mixture of NetXtreme II BCM57711E 10 Gigabit PCIe and 82571EB Quad port Mezz card to connect to the SAN, the MTU is currently left at the standard MTU size as XenDesktop 4 does not fully support jumbo frames.

Exchange 2003 cluster. Exchange connects to the SAN using a QLA4052C and the MTU is set to 9000 MTU.

The iSCSI network is currently managed by a 3rd party and as of yet they have not been able to give us solution to this on-going problem.

Please don’t hesitate to ask for more information as I would like to get to the bottom of this ASAP.

thanks

[rick jones]

I cannot speak directly to iSCSI performance, but will point-out that *all* stations in a given broadcast domain (devices able to communicate with one another at layer 2 or below) must have the same MTU. You can only change MTU on either side of a router.

If all the communication is TCP, and the small MTU is only at the "edge" of the broadcast domain then *generally* the TCP MSS exchange will paper over the misconfiguration, but relying on that isn't such a good idea.

[Antonio Milanese]

Hi,

well you have a lot of moving parts here, with several potential firmwares,drivers,software and interoperability issues:

1) which version on vSphere: 4.0 4u1/u2 4.1 4.1u1
2) which version of qla4xxx and bnx2x are installed (bnx2x driver has always been a pain in the ass!)
3) Xendesktop is running on top of which O.S.?
4) which C7000 IC modules are installed
5) C7000 firmware levels (OA,IC,HBA,NIC,ecc.)
6) Equalogic firmwares levels
7) O.S. service pack levels and HP PSP levels
8) do you have LACP links..is (m/r)STP running

My first advice is to look for specific vendor/version issues.

Secondly as Rick Jones has said if jumbo frames are not end-to-end bad things happens (tm) so try to disable jumbo frames and left only flow control (do you have flow control enabled right?).
On ESX side try to switch from hwiscsi to swiscsi and more generally disable TOE features of broadcom driver on every O.S (broadcom TOE is definitely broken!)

Another thing to investigate is if there is a common pattern when things go bad, and so check:

- vmkernel esx logs
- Equalogic array logs
- windows event and HP agents logs
- switches port stats and logs

BTW a good read for ESX+iSCSI is:

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vsphere.html

Regards,

Antonio

[Frank Gessner]

Hi DAC, I ran into comparable Problems a year ago, when the customer installed a antivirus SW on the Winbox acting as iSCSI-Server. Just a hint...
regards, Frank