BladeSystem - General
cancel
Showing results for 
Search instead for 
Did you mean: 

Blade c7000 bandwidth locked at 1Gb?

Adam Baack
Occasional Advisor

Blade c7000 bandwidth locked at 1Gb?

We're just setting up our first c7000 blade system with two flex-10s. We were told that we'd be able to get 10Gb throughput between the blades, but we're barely seeing above 1Gb.

We have server 2003 R2 installed on two blades with 10Gb assigned to one NIC on each blade. Doing a transfer barely goes above 10% utilization.

Also, our uplinks on the flex-10s are 1Gb SPFs, but our traffic is inter-vlan (we've tried a dummy vlan too that is local) and shouldn't be going out the uplinks.

Any ideas what we're doing wrong?
11 REPLIES
Diego Castelli
Trusted Contributor

Re: Blade c7000 bandwidth locked at 1Gb?

Are you sure you bought the correct options for having 10Gb? Can you please report the NICs, the Flex-10s, and the enclouser P/Ns?

Are u using Virtual Connect?
Diego C.

MS MCSA Server 2003

HP Accredited Integration Specialist
Adam Baack
Occasional Advisor

Re: Blade c7000 bandwidth locked at 1Gb?

Enclosure - C7000 (P/N 507019-B21)
BladeSystem c7000 DDR2 Onboard Administrator with KVM (456204-B21) - Firmware 2.60

ProLiant BL460c G6 - ROM I24 07/25/2009

Interconnect - HP VC Flex-10 Enet Module - Firmware 2.31 (p/n 507778-B21)

Shows 8 LOMs with no mezzanine cards.
Vincent Berger
Valued Contributor

Re: Blade c7000 bandwidth locked at 1Gb?

How are you doing the test ? If you're using Windows file sharing on Windows 2003, then don't. Use something better
Adam Baack
Occasional Advisor

Re: Blade c7000 bandwidth locked at 1Gb?

Using iperf at the moment... don't get over 1Gb/sec. Also, we're using the default drivers PSP gave us, so I don't know if we need special drivers or tweaking of the NIC settings.
The Brit
Honored Contributor

Re: Blade c7000 bandwidth locked at 1Gb?

Are you communicating across the backplane, i.e. are you using a VC Ethernet network which was set up with no external uplink ports.

If you are not, then you need to make sure that your uplinks are 10GB and your network infrastructure is also 10GB.

You should be able to get the 10GB (well maybe 6) across the backplane, if thats where the conversation is taking place.

Dave.
Vincent Berger
Valued Contributor

Re: Blade c7000 bandwidth locked at 1Gb?

Just to be sure, could you disconnect the uplinks and try the test again ? Just to make sure you don't have a configuration problem that makes the traffic go out of the enclosure.
Adam Baack
Occasional Advisor

Re: Blade c7000 bandwidth locked at 1Gb?

Yes, we're running with a Uplink Set with no uplink ports configured. It's all internal to the backplane.

My co-worker found that running iperf -P which creates multiples processes (connections) he can get it up to 67% of the 10Gb. Is this necessary or are we missing something? He has to set it to like 50 processes to get to that speed.
rick jones
Honored Contributor

Re: Blade c7000 bandwidth locked at 1Gb?

I'm not terribly Windows savvy, but my understanding is it does not support MSI-X interrupt messaging. That being the case, the interrupts from the core LOM (Broadcom BCM 57711E) ports will probably go to only one CPU each. I would check to see if during your transfer(s) any one core was getting saturated.

Also, have you made any TCP window size adjustments? It is entirely possible that the defaults in Windows 2003 are insufficient to drive a 10G link terribly fast. Tput < WindowSize/RoundTripTime is normally thought of as a WAN thing, but for 10G it can be a LAN thing as well.

Did you really mean to say inter-vlan as in between vlans, or did you mean to say intra-vlan as in within the vlan.
there is no rest for the wicked yet the virtuous have no pillows
JKytsi
Honored Contributor

Re: Blade c7000 bandwidth locked at 1Gb?

Is it even possible to one Windows box to saturate 10Gb link ? Or get even close to that.
Remember to give Kudos to answers! (click the KUDOS star)

You can find me from Twitter @JKytsi
Diego Castelli
Trusted Contributor

Re: Blade c7000 bandwidth locked at 1Gb?

Hi, sometimes OS must be tweaked to reach the maximum bw.

have a look at Page 9 here:
http://www.nanog.org/meetings/nanog43/presentations/Dugan_Iperf_N43.pdf

and at the links provided.

Anyway, if you reached the 67% with parallel transfers, probably the matter isn't in the hardware or in the fw, but in the OS's TCP management.

did you try with UDP in iperf?

HTH.

Bye
Diego C.

MS MCSA Server 2003

HP Accredited Integration Specialist
rick jones
Honored Contributor

Re: Blade c7000 bandwidth locked at 1Gb?

There is another thread in the forums somewhere where someone with a G6 (?) blade and Windows Server 2003 is getting >> 1 Gig/s. They are though, being held-back by the lack of MSI-X support in Server 2003

I am more familiar with netperf :) but if iperf isn't much different, by default it probably uses just the platform's default socket buffer size, which is unlikely to be sufficient for 10 G networking.

What is the CPU utilization of *each* core when you are doing the transfers?
there is no rest for the wicked yet the virtuous have no pillows