Blade c7000 bandwidth locked at 1Gb?

[Adam Baack]

We're just setting up our first c7000 blade system with two flex-10s. We were told that we'd be able to get 10Gb throughput between the blades, but we're barely seeing above 1Gb.

We have server 2003 R2 installed on two blades with 10Gb assigned to one NIC on each blade. Doing a transfer barely goes above 10% utilization.

Also, our uplinks on the flex-10s are 1Gb SPFs, but our traffic is inter-vlan (we've tried a dummy vlan too that is local) and shouldn't be going out the uplinks.

Any ideas what we're doing wrong?

[Diego Castelli]

Are you sure you bought the correct options for having 10Gb? Can you please report the NICs, the Flex-10s, and the enclouser P/Ns?

Are u using Virtual Connect?

[Adam Baack]

Enclosure - C7000 (P/N 507019-B21)
BladeSystem c7000 DDR2 Onboard Administrator with KVM (456204-B21) - Firmware 2.60

ProLiant BL460c G6 - ROM I24 07/25/2009

Interconnect - HP VC Flex-10 Enet Module - Firmware 2.31 (p/n 507778-B21)

Shows 8 LOMs with no mezzanine cards.

[Vincent Berger]

How are you doing the test ? If you're using Windows file sharing on Windows 2003, then don't. Use something better

[Adam Baack]

Using iperf at the moment... don't get over 1Gb/sec. Also, we're using the default drivers PSP gave us, so I don't know if we need special drivers or tweaking of the NIC settings.

[The Brit]

Are you communicating across the backplane, i.e. are you using a VC Ethernet network which was set up with no external uplink ports.

If you are not, then you need to make sure that your uplinks are 10GB and your network infrastructure is also 10GB.

You should be able to get the 10GB (well maybe 6) across the backplane, if thats where the conversation is taking place.

Dave.

[Vincent Berger]

Just to be sure, could you disconnect the uplinks and try the test again ? Just to make sure you don't have a configuration problem that makes the traffic go out of the enclosure.

[Adam Baack]

Yes, we're running with a Uplink Set with no uplink ports configured. It's all internal to the backplane.

My co-worker found that running iperf -P which creates multiples processes (connections) he can get it up to 67% of the 10Gb. Is this necessary or are we missing something? He has to set it to like 50 processes to get to that speed.

[rick jones]

I'm not terribly Windows savvy, but my understanding is it does not support MSI-X interrupt messaging. That being the case, the interrupts from the core LOM (Broadcom BCM 57711E) ports will probably go to only one CPU each. I would check to see if during your transfer(s) any one core was getting saturated.

Also, have you made any TCP window size adjustments? It is entirely possible that the defaults in Windows 2003 are insufficient to drive a 10G link terribly fast. Tput < WindowSize/RoundTripTime is normally thought of as a WAN thing, but for 10G it can be a LAN thing as well.

Did you really mean to say inter-vlan as in between vlans, or did you mean to say intra-vlan as in within the vlan.

[JKytsi]

Is it even possible to one Windows box to saturate 10Gb link ? Or get even close to that.

[Diego Castelli]

Hi, sometimes OS must be tweaked to reach the maximum bw.

have a look at Page 9 here:
http://www.nanog.org/meetings/nanog43/presentations/Dugan_Iperf_N43.pdf

and at the links provided.

Anyway, if you reached the 67% with parallel transfers, probably the matter isn't in the hardware or in the fw, but in the OS's TCP management.

did you try with UDP in iperf?

HTH.

Bye

[rick jones]

There is another thread in the forums somewhere where someone with a G6 (?) blade and Windows Server 2003 is getting >> 1 Gig/s. They are though, being held-back by the lack of MSI-X support in Server 2003

I am more familiar with netperf :) but if iperf isn't much different, by default it probably uses just the platform's default socket buffer size, which is unlikely to be sufficient for 10 G networking.

What is the CPU utilization of *each* core when you are doing the transfers?