1000BaseT realistic throughput?

[A. Daniel King_1]

Hi, folks.

How do I get more out of my Gb NICs?

I've got two 1Gb ethernet cards (HP A4929A PCI 1000) on two HP-UX 11.0 servers. These NICs/servers are connected directly with a crossover cable. There is no additional networking hardware (hubs, switches, routers) between the boxes.

I'm wondering what the reasonable expected throughput would be. The throughput: 158969-184071 Kbits/sec (from ttcp; the output is attached).

Any thoughts or suggestions appreciated.

[Berlene Herren]

There are white papers on the gigabit located at http://docs.hp.com/hpux/netcom/index.html#1000Base-T%20and%201000Base-SX%20(Gigabit%20Ethernet)

including PCI Gigabit Ethernet Performance

http://docs.hp.com/hpux/onlinedocs/netcom/gbe_perf_final.pdf

Perhaps that will help

Berlene

[A. Daniel King_1]

From the specific doc mentioned, I see that, for a back-to-back config, I should be seeing some ~600Mbps, rather than the ~200Mbs I am seeing. Any idea where to start troubleshooting this?

BTW: I turned on jumbo frames (i.e., MTU=9000), and the ttcp throughput actually dropped (according to ttcp)!

I found netperf, too. The throughput according to netperf is about in the neighborhood of 45MBytes/s. Attached.

Is this normal?

[A. Daniel King_1]

Almost attached.

[Bill Hassell]

Make sure you are up to date on patches. Gigabit LANs will push the envelope on system peformance. If the servers are busy doing other things (a couple of du / or find / processes will cripple the machine. Both machines must be lightly loaded to keep a Gigabit card running at full speed.

If the two servers are in the 500Mhz range or more, then patches should clear things up. If one of the servers is a D-class then the D-class cannot run fast enough to keep up with an A, L or N-class system.

[rick jones · Accepted solution]

the ttcp test used an entirely too small send size (1024 bytes) and that probably ran smack into the Nagle algorithm when you enabled jumboframes. Even without the Nagle question the 1024 byte send size is somewhat small for achieving good throughput.

The 64K window/socket/send buffers in netperf are good, but I think that peak throughput for single stream is with socket buffers larger still - say 128KB.

Also check the CPU util on each of the CPUs. One of them may be pegged if you have lower frequency CPUs.

With JF enabled, and a large enough socket buffer, you should be able to get link-rate on a netperf TCP_STREAM test on anything with 440's or higher on the CPU frequency (iirc)

With a 1500 byte MTU the picture becomes a bit more complicated. You might want to try multiple concurrent streams of traffic.

[Peter Cvar]

Hi Daniel,

Check your ndd ip_send_source_quench parameter:
# ndd -get /dev/ip ip_send_source_quench

... if 1 change it to 0 ...

# ndd -set /dev/ip ip_send_source_quench 0

This should speed-up the transfer. Don't forget to enter new value into ndd configuration file (/etc/rc.config.d/nddconf).

Peter

[A. Daniel King_1]

Thanks for the solid replies!

I've got to mull over some on the facts here, and I'll be assigning points soon.

Some other background:

Bill,

Both machines are 16-processor V-Class machines - with 200MHz processors. Would the comments about CPU and architecture still hold true for these machines? One box (serverA) is quite busy. Aggregate load - per uptime or top - is commonly above 1. The other box (serverB) is not so busy.

Rick,

Thanks for pointing out the Nagle algorithm. I'm not a network guru, so I suspect a little more reading in this area is in order.

I'll try the differing packet size. Can you point me to a good source of information about *how* the transmission packet size is determined? Are there tuning parameters at the OS level?

Peter,

The current parameter is set to 1 for ip_send_source_quench. I'd like to understand exactly what this does before changing the value, though.

Thanks again, everyone!

[rick jones]

OK, first things first - I doubt will get link-rate on a V Class system. First, a single stream will not run any faster than a single CPU can go, and I do not think a single 200 MHz CPU can do link-rate GbE with 1500 byte MTUs. I'm not even certain of JumboFrame.

45-50 MByte/s may indeed be the most you will see. If you want to get more out of the NICs, you may have to upgrade the V to something more contemporary. I'm sure we could find a sales-rep more than happy to sell you Superdomes or rp8400s to replace them :)

Second, the PCI busses in a 200 mHz V Class are only 32bit,33MHz (aka PCI-1X), and my understanding is that the liklihood of achieving link-rate on such a bus is quite small. Especially if there is anything else going-on on the bus at the time (V Class systems have shared-buss PCI slots).

Even if you upgraded the V to faster CPU's, you still have at best a shared PCI-2X bus on the higher-end V Class systems, where one has single-slot busses on other systems.

MSS selection by TCP is a function of the link-local MTU, whether or not the destination is local/remote, what the remote states for the MSS, and to an extent the socket buffer size. If the destination is on the local net, and has requested an MSS >= that the link-local MTU supports, we will use link-local MTU-40 bytes as the MSS (ie TCP over ethernet uses a 1460 byte MSS). If the remote is remote, and we are using PathMTU discovery, we still use 1460. If PathMTU discovery is disabled and the remote is remote, we use 536 bytes.