Configuring 10gig ixgbe network interface

[Charles H Emery]

I am configuring a pair of 10gig network interfaces for a new Netbackup Master Server,
and am looking for recommendations on how to configure it.
Q: Will we have problems if we do not use Multi-Queueing?
Q: How do I set up redundancy with the 2 links, AutoPortAggregation (APA)?
FYI: I have APA, but have had some trouble configuring it for the 10Gig Interfaces.
FYI: Have 4x1Gb APA Load Balanced working fine for 4x1Gb interfaces.
Server specs:
RX7460, with 4 CPU, 16Gig Memory, 2xFC to SAN.
2x10Gb NetIf: HP AB287-60001 PCI-X 10GBase-SR
4x1Gb NetIf, 4Gb APA Load Balanced: HP A7012-60601 PCI/PCI-X 1000Base-T Dual-port Adapter

[Steven E. Protter · Accepted solution]

Shalom,

I'd use APA, but you may need to wait for another release. These 10gig network cards, I've never heard of them. They must be pretty new.

Realistically, you should do reasonably well with a single 10Gb network connection. It's pretty fast, fast enough to run a lot of simultaneous backup streams.

SEP

[Charles H Emery]

Wow, Steve had nothin for me!
Steve, I'm giving you 8 points anyway to thank you for all the helpful responses you have given in many threads.

By the way, here is a link to an HP doc, that I found when googling for info about the hp_ixgbeconf config file [HTML]: <>
[PDF]: <>

[Charles H Emery]

Wow, Steve had nothin for me!
Steve, I'm giving you 8 points anyway to thank you for all the helpful responses you have given in many threads.

By the way, here is a link to an HP doc, that I found when googling for info about the hp_ixgbeconf config file [HTML]: http://209.85.165.104/search?q=cache:Q2aNZsSt16sJ:docs.hp.com/en/J6379-90003/J6379-90003.pdf+hpixgbeconf&hl=en&ct=clnk&cd=1&gl=us
[PDF]: http://docs.hp.com/en/J6379-90003/J6379-90003.pdf

[rick jones]

I would hope that by now, the current APA code groks 10G NICs, but indeed at one point it did not. Probably best to ring-up the response center on that one.

WRT multiqueue support that will be one of those infamous "it depends" sorts of things. A few related questions:

*) What frequency of CPUs in the rx7640 (I cannot recall if we support both the 1.6 and the 1.4's there or not)

*) how many cell boards

*) how many concurrent backup streams would be flowing through the interface(s)?

*) what is the speed of the origin of each backup stream - also 10G or will it be a bunch of 1Gs?

*) is each concurrent backup stream a separate process/thread and is there only one concurrent backup stream per process/thread?

*) can you use jumbo frames?

*) will the rx7640 be doing anything else at the same time?

You can always try some experiments with the likes of netperf - www.netperf.org and try with and without the multiqueueing and such. There are some instructions in the manual:

http://www.netperf.org/svn/netperf2/tags/netperf-2.4.3/doc/netperf.html

for the latest "released" version use .../netperf2/trunk/doc/netperf.html for the top of trunk version, which will describe how to do some reasonably accurate multiple stream tests. Feel free to contact me via netperf.org's feedback links if you need help there, or better still join netperf-talk@netperf.org

[Charles H Emery]

Sorry - meant to respond to this last week.
I will post a detailed response with the answers to Rick's queries.

FYI: I opened a call with the HP response center, and will post my results.
There word is that APA using Lan-Monitor mode should work fine. I will be working with them to try it again on Thursday morning after the 4th-of-July holiday.

[Charles H Emery]

Okay...
sorry for the long delay in posting the configuration we ended up using.
(it has been a super busy couple of months)

We are using APA with Lan_Monitor for failover, (APA for bandwidth is not currently supported for 10Gig interfaces).

Regarding the Multi-Queueing for the 10 Gig interfaces...
Multi-Queueing is not a seperate software solution, like APA. Multi-Queueing is a built in feature of the drivers for the ixgbe 10Gig cards. The newest drivers auto-magically optimize the number of queues in use for the 10Gig interface.

Assuming you are not already using jumbo-frames, this is supposed to give about a 30-50% performance boost, and allow for throughput approaching 10GB. Without it and without jumbo frames, we were getting 3 to 5 Gig throughput.

Attached is a synopsis from the folks who helped me in the HP response center.

The essential pieces of the config to take full advantage of multigueue are as follows:
=============================================
in /etc/rc.config.d/hpixgbeconf change the config to:
HP_IXGBE_DPS[*]=1
HP_IXGBE_MQ[*]=1
e.g.
HP_IXGBE_DPS[0]=1
HP_IXGBE_MQ[0]=1

Install the latest Streams and ARPA patches:

PHNE_34788 s700_800 11.23 Cumulative STREAMS Patch
This patch contains the NOSYNC level synchronization feature
of the Streams component delivered with TOUR (Transport
Optional Upgrade Release).

PHNE_35766 s700_800 11.23 cumulative ARPA Transport patch
This patch contains the features of TOUR (Transport Optional
Upgrade Release).
=============================================

[rick jones]

I'm glad things are working-out for you - a couple nits...

10 GB would be 10 GigaBytes :) I think you meant to say 10 Gb for Gigabits.

Also, while the AB287A is a "10 Gigabit Ethernet" interface it is also a PCI-X 1.0 133 MHz card, and a 133 MHz PCI-X card cannot get more than 7.X Mbit/s of throughput before it is limited by PCI-X.

To achieve 10 Gbit/s one must have either a PCI-X 2.0 266 MHz card+slot or a PCIe x8 card+slot. The latest HP Integrity servers - those based on the zx2 or the sx2000 chipset offer either of those sorts of slots depending on the ordering choices one makes. At some point before too long I expect there will be a supported PCI-X 2.0 266 MHz or PCIe x8 10 Gig card for them.