topic Re: 4-node P4300 poor performance in StoreVirtual Storage

4-node P4300 poor performance

fusiongroup — Mon, 04 Jun 2012 23:06:59 GMT

Hi,

I've recently set up a 4-node P4300 (2x 7.2 Starter SAN w/1Gb NICs) and I'm just not seeing the type of performance I would expect.

I have set the cluster up as Network RAID10 and all the space assigned to a single RAID5 LUN with around 20 Hyper-V VMs on there.

Each server has the HP DSM installed with 3x 1Gb NICs connected to the iSCSI network and I've set MPIO to round robin.

If I shut down one of the VMs and copy it to the same location, I see transfer speeds of around 80MB/sec. In the performance monitor IOPS only ever hit around 700 and queue depth never goes over 14. I can see all 3 NICs in use in task manager usually hovering at around 30-60% utilization.

I also have another SAN from a different vendor connected to the same server. This SAN also has 1x RAID5 LUN (over 10 disks) but only has 2x 1GB NICs and I get exactly the same copy speed of 80MB/sec.

Surely I should be seeing faster speeds than this? Each node has 2x1GB NICs and each volume is over 16 disks, my IOPS seem low (from what I've been reading I should be able to hit anywhere up to 3500) and the queue isn't saturated.

Also if I try copying a large amount of data between the VMs I see speeds of 10-35MB/sec. Between the VMs on the other vendor's SAN I get the same 80MB/sec.

Can anyone give me any pointers on how to troubleshoot?

Thanks,

James

Re: 4-node P4300 poor performance

oikjn — Tue, 05 Jun 2012 01:23:02 GMT

80MB is typical of a fully saturated 1gb link. Sounds to me like there is a link agrogation issue. Since you say you are seeing traffic on your three server nics I don't think the issue is on the server. What are you using for bonding NICs on the SAN? If you are using something like LACP that requires switch configuration as well you might have an issue with the configuration on the switch that is preventing the ports from bonding...

so there is my vote... switch config issue :)

Re: 4-node P4300 poor performance

fusiongroup — Tue, 05 Jun 2012 15:03:34 GMT

Hi,

It would seem like there is a 1Gb bottleneck now you point it out but I'm not sure where that could lie as I thought I'd configured plenty of bandwith.

Currently the bonds on all four nodes are set to ALB so no need for 802.3ad on the switch ports.

The server cluster active node currently has 2x 1Gb NICs plugged into switch A and 1x 1Gb NIC plugged into switch B of a two switch stack. SAN nodes 1 & 3 are connected into both switches in the stack. Both switches have a 1Gb fibre link and 1Gb ethernet link in a LAG group connected to switch C. Switch C has a server cluster passive node connected to it with 3x 1GB NICs, plus all the NICs from SAN node 2 & 4 connected to it.

In theory the bandwith should be 3Gb from the active server cluster node to SAN cluster nodes 1 & 3. There should be a 4GB uplink from the stacked switch A & B to switch C.

I'll double check the LAG configuration though. If the you're right and the LAG isn't working properly then I guess there would be a 1Gb bottleneck to switch C. As the DSM wites to nodes 2 & 4 at the same time as it writes to 1 & 3 then it would make sense I would see the bottleneck.

I will be able to test by moving a couple of VMs to the server passive cluster node and then initiating a copy from the alternate vendors SAN. This will force the data to go over the possibly incorrectly configured LAG and hopefully I should see a significant performance decrease if this is the case.

I'm currently waiting for some SPF+ modules to be delivered so that I can increase the fibre link speeds to 10Gb. With some server NIC re-assignment I could also assign 6x NICs to iSCSI traffic. Hopefully this should eliminate any bottleneck issues when they go in.

Many many thanks for the pointers. I'll now go and see if I can arrange some down-time!

James.

Re: 4-node P4300 poor performance

fusiongroup — Tue, 05 Jun 2012 16:33:14 GMT

After re-reading the Link Aggregation section of my switches manual it appears that I cannot LAG the SPF fibre links, only ethernet ports. This would mean I'm not getting the 4Gb speed I thought I was.

Either the two LAG'd ethernet ports on the stacked switch A & B are creating a 2Gb link to switch C, or the SPF ports are acting as a two single paths (via Spanning Tree) to switch C and I'm getting 1Gb.

I'll make the 10Gb modules a priority and try LAGing some more ethernet ports for the time being.

Thanks for all your help :)

Re: 4-node P4300 poor performance

fusiongroup — Thu, 07 Jun 2012 23:01:13 GMT

The bond wasn't configured correctly. I was attempting to bond two fibre ports and it just didn't like it.

I created a new bond purely for ethernet ports and the speed has increased 6 fold.

Thanks again.

Re: 4-node P4300 poor performance

dagatle — Mon, 11 Jun 2012 11:57:05 GMT

If you have only 1 LUN you should consider to have more. The iSCSI IO traffic for 1 lun is only on one node if you have 2 nodes on each site then you should have at least 2 luns. Then the IO load is on 2 of your nodes instead of only one.