StoreVirtual Storage
1748239 Members
3611 Online
108759 Solutions
New Discussion юеВ

Very Poor Perfomance (P4300 & VMware)

 
Gavin Race
Occasional Contributor

Very Poor Perfomance (P4300 & VMware)

We have a HP StorageWorks P4300 G2 with 2 nodes setup in a cluster (running SANiQ 8.5).

The SAN is connected to 2 vSphere 4.1 ESX hosts running on HP Blades. The blades have iSCSI Qlogic HBAs installed and setup to connect on the SANs VIP (using Dynamic Discovery - Send Targets) and statically to each LUN.

Path Selection to each LUN is Round Robin (VMware).

To get from the SAN, the packets pass through a Cisco 3750G switch to HP Gb2Ec switches in the Blade enclosure(via etherchannels) which are mapped to the iSCSI HBAs.

Unfortunately, any VMs running on the SAN are running incredibly slow and we are recieving errors of high utilisation about 90%. There are only about 4-5 VMs running on the hosts and they're not doing anything intensive.

Any ideas or suggestions for what might be causing it or what we can try to resolve the bottleneck somehow??

Any thoughts would be greatly appreciated.

Gav.
6 REPLIES 6
Uwe Zessin
Honored Contributor

Re: Very Poor Perfomance (P4300 & VMware)

Did you install this patch?

- 10092-00: SAN/iQ patch to address iSCSI session disconnect issue with QLogic HBA. Applies to SAN/iQ 8.5

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=12169&prodSeriesId=4118659&prodNameId=4118705&swEnvOID=54&swLang=8&mode=2&taskId=135&swItem=co-88775-1
.
Gavin Race
Occasional Contributor

Re: Very Poor Perfomance (P4300 & VMware)

Thank you for your response.

I was looking at updating to SANiQ version 9.0. Would you recommend this?

Is this patch rolled-up within this update and is 9.0 stable?

Kindest regards.
JacobS_4
Occasional Advisor

Re: Very Poor Perfomance (P4300 & VMware)

We have a very similar environment. We're using ESX 4.0 hosts running on BL460c servers in a C3000 chassis with Gbe2C blade switches. The blade switches connect to the SAN through two HP Procurve 2910al switches. We're using the ESX Software iSCSI initiator and binding it to the Broadcom NICs in the blades.

We also are experiencing poor performance. It used to be a lot worse when we were using HP 2610 switches, those just didn't have the port buffers to keep up. Moving to the 2910 switches helped quite a bit. I'd also make sure that you have flow control enabled across the board and if you can disable STP. Enabled Jumbo frames on everything didn't seem to make much of a difference.

Our Lefthand nodes have their NICs configured in an ALB team. The two NICs in are spread across both 2910 switch for redundancy (SW01, SW02). The strange thing we've noticed is that an iSCSI request will come from our ESX host over SW01 to NIC1 of the ALB team. Then the Lefthand will send back the return data over NIC2 of the team. The problem with this is that the SW02 that the second NIC is connected to hasn't learned the mac of the destination server, because the original request went through SW01. So SW02 has no choice but to flood the traffic out of all of its ports. This is really affecting throughput and at times will even cause ESX to report problems accessing a volume. It seems to me like it's a problem with LHs implementation of ALB.
Jitun
HPE Pro

Re: Very Poor Perfomance (P4300 & VMware)

Yes the Patch 10092 - ESX 4.1/QLogic HBA disconnect fix is included in 9.0

Check out complete info regarding Upgrading SANiQ 9 at

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c01886197/c01886197.pdf

VMWare specific Enhancements
VMware VAAI vStorage offloads ├в Full copy, Block Zeroing, and Hardware Assisted Locking for faster VM deployment and less load on ESX servers

Server aware sites ├в Servers and server clusters can be assigned to sites to improve load balancing

iSCSI session management improvements ├в Sessions are automatically rebalanced after systems come back online and when systems are added. Sessions are automatically disconnected when volumes are unassigned from servers

Upgrading to 9.0 is certainly Recommended.
I work for HPE
--------------------------------------------------------------
How to assign points? Click the KUDOS! star!

Accept or Kudo

Aart Kenens
Advisor

Re: Very Poor Perfomance (P4300 & VMware)

JacobS_4,

 

Did you find a solution for your problem regarding the ALB?

I have the same problem. 

 

We have a multisite config 28.8TB P4500(4nodes) connected towards 4*HP2910al, switches connected with CX4 cable and 10Gig fiber.

Like your setup each node is connected towards 2 switches.(flow control & jumbo frames enabled)

 

Now  the lefthand is still in testing stage (we have an EVA currently) and so I put one VM on a lun on the lefthand and ran IO meter on that VM.

Then I watched the traffic on my HP2910al switches and seeing flooding it too my other switches where my host is not even connected to? So its flooding switches from the other site :/

 

Is this a misconfiguration on my end or ??

 

 

 

MarcGaethofs
Occasional Visitor

Re: Very Poor Performance (P4300 & VMware)


@Aart Kenens wrote:

JacobS_4,

 

Did you find a solution for your problem regarding the ALB?

I have the same problem. 

 

We have a multisite config 28.8TB P4500(4nodes) connected towards 4*HP2910al, switches connected with CX4 cable and 10Gig fiber.

Like your setup each node is connected towards 2 switches.(flow control & jumbo frames enabled)

 

Now  the lefthand is still in testing stage (we have an EVA currently) and so I put one VM on a lun on the lefthand and ran IO meter on that VM.

Then I watched the traffic on my HP2910al switches and seeing flooding it too my other switches where my host is not even connected to? So its flooding switches from the other site :/

 

Is this a misconfiguration on my end or ??

 

 

 



There is no solution for this... In the setup the max tp is 1 Gb/s which would be about 120 MB/S...It's by design.  Only way to get more out of the box is adding 66xx switches.  Then you will get 2 Gb/s which would be 250 MB/s.

The next step is to add 10 Gb nic's into the nodes and connect them to the 10 Gb sfp ports of the 66xx switch.. Then add more nic's in the servers.  

Btw a 4 nodes setup should deliver 48 x 250 IOPS x 64 k blocks = 750 MB/S tp... and now the real world : we only could get 450 MB/s... Poor performance.. Poor usable space .... (3 nodes P4500 = 21.6 TB but on raid 5 en Nraid 10 (or 2way mirror) you will get 6.7 tb usable = 70 % lost)