Link Aggregation in ESX 4 and Flex-10

[ChristianWickham]

BL 490 G6
Flex-10 (Firmware 2.12)
Cisco 3750 core switches (4 in a stack)
ESX 4.0 with Virtual Center 4, vNetwork Distributed Switch.
BL 490 blades have 2x LOM which have been set to 4x 5Gbps links.
Both Flex-10 modules have 3 uplinks to the Cisco 3750 switches, each at 1Gbps, all in one vNet. These are assigned to LOM1-a and LOM2-a (we have other links assigned to LOM1-b and LOM2-b).

We are trying to get multi-Gigabit connectivity to Virtual Machines in ESX4, using the above infrastructure. In Virtual Connect, we can see that 3 links are active (and 3 are standby) - so we would expect at least 3Gbps. These links have been configured on the swich with

interface Port-channel17
description HP Blades
switchport trunk encapsulation dot1q
switchport mode trunk
interface GigabitEthernet1/0/49
description HP Blades
switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 17 mode active

In the ESX console, we can see that the network cards are operating at 5000 Full Duplex, but none of the virtual machines are getting more than 1Gbps.

Has anyone successfully been able to get more than 1 Gbps to an ESX virtual machine by aggregating multiple uplinks at 1Gbps each? Can anyone advise me if there are configuration options within the Flex-10 Virtual Connect that I can set? VMWare support forums directed me to gain HP hardware support.

[JonathanT]

Have you compared your configuration to the Virtual Connect Cookbook? There are 18 different configuration scenarios in the latest version of it.

[ChristianWickham]

Yes, I have consulted the cookbook "HP Virtual Connect Ethernet Networking Scenario Cookbook_ Single Domain Scenarios.pdf" about how to set this up, and I have followed the scenarios 2, 6, 8, 10 but cannot follow 11 as we are using ESX and not Windows.

I posted this message because the Cookbook has not answered or helped with my specific issue, even after following the scenarios recommended configuration. Has anyone reading this been able to get more than 1Gbps with Link Aggregation (LACP) through the Flex-10 with ESX? If no-one else has been able to do it, then my configuration that followed the cookbook must be correct!

[HEM_2]

Correct me if I'm wrong but don't the Virtual Machines only have a 1Gb NIC?

The VC uplink bandwidth would be separate from the port speed on the Virtual Switches connecting to the Virtual Machines.

Can you add multiple NICs to a Virtual Machine and aggregate those NICs?

[Vincent Berger]

How are you measuring the bandwidth you're getting in the VM ? If you're relying on the link speed reported by the guest OS, that just reflects the virtual hardware presented by ESX and not the actual speed you could reach. If you're making a test with a single TCP connection, this will use only 1 of the uplinks as packets belonging to a TCP stream are not broken up in multiple members of a LACP group.

[ChristianWickham]

Maybe I am wrong, but I was under the impression that the network cards in ESX are not limited at all to any bandwidth, and that if you increase the bandwidth to the host, that the VMs would get this increase? The Virtual Network Card models I have tried are "Flexible" or "VMXNET3" and not "E1000".

If it is the case that the Flexible network card is either 100Mbs, 1Gbps or 10Gbps, then this answers my question. Otherwise, has anyone seen a network bandwidth that differs from any of these values in a virtual machine?

[Vincent Berger]

You are correct but increasing the bandwidth to the host by adding more links to an LACP group may or may not increase the bandwidth to the VM depending on:
- how you measure the bandwidth in the VM as I explained in my previous post
- other potentially limiting factors such as CPU resources available to the VM and the vSwitch

[Adrian Clint]

I'm installing Flex-10 & ESX4 next week and was considering the points raised in this.
But I am thinking that the question is about how the Vswitch presents bandwidth to the VM which is an ESX issue. This is unrelated to the bandwidth presented by the Flex-10 NIC to the Vswitch.
I would be looking at how the Virtual machine virtual NICs are managed by the Vswitch rather than the Virtual connect config.

[ChristianWickham]

I think I have resolved this.
We upgraded all our VMWare Tools to the latest version, then upgraded VM Hardware to version 7, then changed to VMXNet3 network cards.
Now Windows sees the network cards as 10Gbps (even though each NIC is a maximum of 5Gbps, but they are teamed at the vDS level), and the BGinfo tool(from Sysinternals / Microsoft) showed that the actual link speed was 4Gbps.

Really, it does not matter to me the actual bandwidth that we get, just that we are not limited to the 1Gbps that we previously saw.