NIC Teaming

[Tim Turay]

Hi All,

Running Windows Server 2012 R2 on DL360p G8.

I am trying to build a two node hyper-v cluster running on 2 x 10 gig HP 5900 switch, all drivers and frmware are up to date.  The switches are not stacked.

During the initial testing, when I copy a large file between the servers, I got 2 GB/s.  Now when I team two of the NICs (it is showing as 20 GB/s) and I only get 1 GB/s???  Why does my performance drop when I team the NICs?  The NICs are 530SFP+.

The teaming mode is switch independent, load balancing is dynamic and standby is none.  Even when I select the standby mode as active\passive I onlyl get 1 GB/s

TIA

[waaronb]

Can you clarify if you're seeing 1 GB/s or 1 Gb/s (byte or bit)?

1 giga*byte* per second is great for a 10 giga*bit* per second network connection.

[Tim Turay]

Hi, At this stage I am very confused with the big B vs little b.

When I do the copy, the microsoft window pops up showing me GB/s transfer. When I use the server task manager and look at the ethernet performance, it is using Gbps. In the task manager it shows the NIC running at 9.9 Gbps un-teamed and at around 7ish Gbps (for both NICs) teamed. I do know that when it is teamed, the copy takes twice as long.

[waaronb]

10 gigabit per second is equivalent to 1.25 gigabyte per second.

In the real world you would never see 1.25 GB/s though since IP has overhead... you're more likely to see 1 GB/s or 8 Gb/s.

Just pay attention to whether something is showing an upper or lower case B since that's a standard way of showing bits or Bytes.

As for your specific setup, you are hooking your server to diverse switches and using the switch independent teaming mode. When you do that, you will not be able to combine the 2 ports into a single big 20 Gb port.

Instead, what happens is that Windows will basically "round robin" each connection going over one or the other of the NIC ports. So you might be able to get a total speed of 20 Gb/s but not over any *single* network connection.

Your file copy is a single connection so it would only ever use one of the network ports. Your file copy speed is also dependent on other things like the speed of the other system and if your drives can keep up with the network flow.

If you need an actual 20 Gb/s connection then you'll have to use the LACP or static modes and then on the switch side of things you'll need to aggregate those ports into a team. Since your diverse switches aren't stacked, you can't do that.

You could stack your switches or forgo the switch diversity and have both ports go into the same switch as long as the switch supports whatever teaming mode you choose.

This link may help explain the differences:
http://technet.microsoft.com/en-us/library/hh831648.aspx

For right now, I *think* what you're seeing is that Microsoft shows your team as 20 Gb/s (lower case "b" for bits). When you copy a file, it's only able to use one port for the connection and you're getting a nice 1 GB/s (upper case "B" for bytes) which is what I would expect to see.

If you were able to aggregate both ports by going into the same switch or into a stack, then you would see Windows reporting up to 2 GB/s (giga*byte* per second) which would be normal.

And if you think bits and bytes are confusing, I won't even go into the confusion caused by whether they're talking about base 2 or base 10. Like, whether 1 MB is 1,048,576 (1024^2) or 1,000,000 (1000^2) That's a common source of confusion when manufacturers advertise hard drive capacity so there's actually another standard way of representing those numbers like MiB, GiB, etc. just to avoid confusion.

[Tim Turay]

Howdy,

and thanks for your help waaronb,

So here is the latest.

I have configured the switch to use LACP on the same switch, teamed the network cards, everything looks goods (OS showing 20 gbps, switch showing the same). When I do some data transfer and looking at the task manager, I only see data going over the one network card. The second teamed card stays silent.

I did notice that when I break team and transfer some data, both network cards are in use and running in 7-9 gbps range.

Now strangely enough, the little data transfer window takes too long in calculating that I never get to see the transfer speed anymore. I can only determine network speed from the task manager.

As of right now I see zero reason to team network cards (redundancy is mute, can’t remember the last time a switch, NIC, data port has failed) and I would rather have double the throughput.

Maybe a call to HP support is in order.

[waaronb]

With the NIC teaming set to LACP, are you able to look on the switch and see if it's auto-detecting the configuration?

You can also try manually configuring the ports on the switch to create an etherchannel. It sounds like the team is set to use LACP but the switch wasn't seeing it, so it defaulted to an active/passive setup instead of an etherchannel.

[Tim Turay]

Hi Waaronb,

Here is what I am seeing:

[10GigSwitchB]display link-aggregation verbose Bridge-Aggregation

Aggregate Interface: Bridge-Aggregation13

Aggregation Mode: Dynamic

Loadsharing Type: Shar

System ID: 0x8000, 4431-9284-727c

Local:

Port             Status Priority Oper-Key Flag

--------------------------------------------------------------------------------

XGE1/0/17       S       32768   1         {ACDEF}

XGE1/0/19       S       32768   1         {ACDEF}

Remote:

Actor            Partner Priority Oper-Key SystemID               Flag

--------------------------------------------------------------------------------

XGE1/0/17       2       0       0         0x0   , a0d3-c1f7-1d54 {ABCDEF}

XGE1/0/19       1       0        0         0x0   , a0d3-c1f7-1d54 {ABCDEF}

Aggregate Interface: Bridge-Aggregation24

Aggregation Mode: Dynamic

Loadsharing Type: Shar

System ID: 0x8000, 4431-9284-727c

Local:

Port             Status Priority Oper-Key Flag

--------------------------------------------------------------------------------

XGE1/0/18       S       32768   2         {ACDEF}

XGE1/0/20       S       32768   2         {ACDEF}

Remote:

Actor           Partner Priority Oper-Key SystemID              Flag

--------------------------------------------------------------------------------

XGE1/0/18       1       0       0         0x0   , 2c44-fd97-9d58 {ABCDEF}

XGE1/0/20       2       0       0         0x0   , 2c44-fd97-9d58 {ABCDEF}

[10GigSwitchB]display interface bridge-aggregation

Bridge-Aggregation13

Current state: UP

IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 4431-9284-72b5

Description: Bridge-Aggregation13 Interface

Bandwidth: 20000000kbps

20Gbps-speed mode, full-duplex mode

Link speed type is autonegotiation, link duplex type is autonegotiation

Bridge-Aggregation24

Current state: UP

IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 4431-9284-72b6

Description: Bridge-Aggregation24 Interface

Bandwidth: 20000000kbps

20Gbps-speed mode, full-duplex mode

Link speed type is autonegotiation, link duplex type is autonegotiation

How does this look to you?  I have never done this before so to me this looks right.

[waaronb]

It seems like it might be okay. I'm only familiar with Cisco switches so I'm not totally sure about the output you shared, but it seems like it's detecting the LACP link and showing a 20 Gbps link.

When you're copying files to test the speed, are you copying them from another system that has dual 10Gbps connections? Because it won't matter how fast one server's links are, it's only as fast as the other side of the connection.

[Tim Turay · Accepted solution]

Found the answer in a recent blog post.

Using file copy to measure storage performance – Why it’s not a good idea and what you should do instead

http://blogs.technet.com/b/josebda/archive/2014/08/18/using-file-copy-to-measure-storage-performance-why-it-s-not-a-good-idea-and-what-you-should-do-instead.aspx

Thanks.