Linux Bonding Modes with Virtual Connect

[Michael Pye]

Investigating some packet drops and conenctivity issues when using RHEL 5.7 and NIC teaming (linux bonding) using mode 0 round robin. One nic to a SUSa (first vc switch) and one nic to SUSb (second vc switch). Since upgrading to virtual connect 3.30 we've noticed more frequent packet drops and time outs.

Just noticed this article:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=120&prodSeriesId=3552695&prodTypeId=329290&objectID=c02957870

"HP Virtual Connect supports bonding modes 1, 5, or 6. VC does not support modes 0 (round robin) or 7 (switch assisted load balancing)."

But we've been using round robin mode 0 before 3.30 for many months successfully, as we built our environment around HP recomendations and this document:

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01386629/c01386629.pdf

which reports that it is valid for 3.30 states:

"For Linux, VC supports any NIC bonding type that does not require 802.3ad (static or dynamic using LACP) on the server NIC ports. "

So which is it ? I suspect its now only the reduced bonding modes that are supported and mode 0 no longer is. There is no mention of this in any document other than the above web page and I've done a lot of reading around vc 3.30. I will switch to mode 1/5/6 and see if we get better results. 

[Hongjun Ma]

the 2nd doc should be right, it's also updated recently for 3.30.  I can't tell the date for the first doc but the statement doesn't seem to be correct and I'll try to request it to be deleted.

VC3.30 shouldn't change anything for Linux Bonding support. You may want to open an HP support case to do some troubleshooting.

what's the server blade/NIC model and NIC firmware/driver version?

[Michael Pye]

Thanks for the response. The date of the 1st document is 8 Aug 2011 according to this:
http://h20000.www2.hp.com/bizsupport/TechSupport/SupportTaskIndex.jsp?lang=en&cc=us&taskId=120&prodTypeId=329290&prodSeriesId=3794423

The blade is a BL465g7. We are using RHEL5.7 with the built in be2net driver v4.0.100r.

# modinfo be2net
filename:       /lib/modules/2.6.18-274.el5/kernel/drivers/net/benet/be2net.ko
description:    ServerEngines BladeEngine 10Gbps NIC Driver 4.0.100r
version:        4.0.100r

# ethtool -i eth0
driver: be2net
version: 4.0.100r
firmware-version: 2.102.517.7

I think nic is referred to also as NC551i or emulex oneconnect. There are firmware updates
I see at:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=3709945&prodSeriesId=4132949&swItem=co-101629-1&prodNameId=4132827&swEnvOID=54&swLang=8&taskId=135&mode=4&idx=1

Thanks for any help.

[Hongjun Ma]

yes, if it's BL465G7, the LOM is NC551i NIC.

I'm confirming if mode=0(round robin) is indeed supported or not and will let you know. (hopefully by the end of tomorrow). VC3.30 should change the support model. If it's not supported, then it should have started with early versions.

[Hongjun Ma]

sorry, i mean VC3.30 should not change the support model

[Michael Pye]

Many thanks for investigating this. It's possible we had the issue before but put it down to something else and now it just happens more perhaps.

[Hongjun Ma · Accepted solution]

Hi Michael,

It turns out balance-rr mode on Linux side requires "bundling" on switch side. So this is not something VC will support. Remember 2nd doc you pointed out also states that any NIC teams requiring VC to do cross-module static or LACP bundling is not supported.

Please see:

http://www.kernel.org/doc/Documentation/networking/bonding.txt

Specifically:

The balance-rr, balance-xor and broadcast modes generally

require that the switch have the appropriate ports grouped together.

The nomenclature for such a group differs between switches, it may be

called an "etherchannel" (as in the Cisco example, above), a "trunk

group" or some other similar variation.

The biggest issue with balance-rr I see from research is that packet can be deliverred out of sequence and you may have performance issue at the end.  Please see this post

http://h30499.www3.hp.com/t5/Networking/bond-balance-alb-or-balance-rr-which-is-best/td-p/4437148

The person posted the answer is actually a Linux performance expert and the author of Netperf Linux performance testing tool.

http://www.netperf.org/netperf/

So in short, both docs are valid and mode=0(balance-rr) is not something VC can support.

[Michael Pye]

Many thanks for the research and response.It would be good if you could get HP to clarify that in the documents listed above.

I think we will set mode 1 or 5 and see how we get on with those.

Cheers

Michael

[Hongjun Ma]

ok. i'll forward your feedback for doc improvement.