slow file transfer speed with gigabit ethernet A6825

[Luigi_11]

hi guys
i'm setting up an oracle rac environment.
with the 2 test server (rx5660) i used a
cross cable for the heartbeat (later will
be used by service guard) over a gigabit
channel.
the 2 lan had negotiated the 1000 full duplex
speed but when i transfer file with scp,
rcp, ftp i only get 13 MB/s (megabyte)
of transfer speed.
i can get 11/12 MB/s with 100fd ethernet and
i guess i should have 100 MB/s with gigabit
ethernet.
i tried even to change the default mtu with
no success.

any suggestion?
thank in advance

[rick jones]

rx5660? anyhow, you should try some netperf TCP_STREAM tests to take filesystem and disc out of the equation. second, while I don't _know_ it owuld be a problem, gigabit does _not_ need a crossover cable for back to back. the phy's (?) figure it out on their own.

if netperf TCP_STREAM is slow, start checking statistics at netstat -p tcp and lanadmin.

ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_netstat.txt

PS - scp will definitely (more or less) be slower than ftp or rcp because it will have considerable CPU overhead for encryption/decryption

PPS - heartbeat doesn't really need bandwidth so, IIRC a gbe would be overkill. or are you running more than just heartbeat across the link?

[Slawomir Gora]

Hi,

what is your storage disk performance ?
Maby 13Mb/s it is disk limit for your system.

[Geoff Wild]

Check the switch port the interfaces are connected to - make sure the network guys set them to 1GB FDX - and not AUTO-NEGOTIATE - I have found HP GB interfaces don't perform when the switch is set that way.

Rgds...Geoff

[rick jones]

Geoff - with which switches have you had that problem? I've never had autoneg problems with GbE and always felt that GiGe _had_ to be set to auto for anything to work.

[Jay Kidambi]

Some suggestions/answers:

As Rick suggested, netperf should give a better networking performance baseline.

At Gigabit speeds you may use a crossover *or* a straight-through cable because 1000Base-T PHYs are capable of automatic crossover detection. At 10/100 speeds, you'd need a crossover for back-to-back connections and straight cables for sever to switch connections.

Auto-Negotiation is mandatory for 1000Base-T. This is required by 802.3ab for PHY master-slave resolution. This is the reason why HP-UX Gigabit ethernet products only have an "auto_on" setting for 1000FD operation. However, some switch/server products provide a fixed/manual 1000FD operation. This doesn't really mean they disable Auto-Negotiation altogether though they sometimes mistakenly claim so. Such products just disable the ability to negotiate lower speeds (10/100 Mbps). HP-UX "auto_on" setting on GigE products allows you to negotiate 10/100/1000 Mbps speeds. Therefore you may safely use this "auto_on" setting on HP-UX with 1000FD setting on the switch. On some switches "auto" negotiates only 10/100 while 1000FD (fixed/manual) must be used for 1000Base-T operation, so be careful when choosing which mode to use. Yes, this is a bit confusing, but read it twice and you'll understand it.

[harry d brown jr]

I did an internal company white paper on this using Rick Jone's netperf and other tools, and the bottom line is that your MTU is probably set to 1500, and it needs to be bumped to 9000 (Jumbo Frame) to get HIGH data transfer results.

If you are using this in production for just a heartbeat network, then jumbo frames are not needed!

live free or die
harry d brown jr

[Steven E. Protter]

I have found that the only way for HP GB cards to work correctly is in auto-negotiate mode.

Your transfer rate isn't that bad. We get 9 MB/s on a Cisco core switch with our GB cards from HP.

SEP

[rick jones]

Internal only? Bummer - I always like to see how folks are using netperf.

BTW with later driver and transport bits at least some of the GbE NICs have TSO (large send) enabled. TSO is something like a poor-man's JumboFrame, giving CPU util assistance to the sender. Alas, does naught for the reciever, but one does not need JF support anywhere in the network.

Also, with contemporary NICs and systems, it _should_ be possible to hit "link rate" with 1500 byte MTUs. Now, with the old 4926 and 4029, well....

[Luigi_11]

thanks everyone for the reply :)

the server is a 5670 :P

i'm getting crazy on the cross cable
for those reason,

a) with a cross cable i don't have cisco
switch with bad settings

b) over the gigabit (but with the cisco
switch) i will have the private lan for
the RAC comminication that need a lot
o bandwith to let the oracle db to scale
(in production we have 4 oracle rac nodes)

the same file that i use in the transfer
test is read from the disk at 30 MB/s
and with fs buffer i should archive high
peak even if not a good overall performance.

while scp use more cpu i get 12/MB with
10/100 cards on pa-risc at 850Mhz

my systems are dual itanium 1,5Ghz and the
cpu utilization reach only a 10% during the
encryption.
...
i will make more rigorous tests asap

Luigi

[jgammelg]

Im working with netperf at the moment, but have only figured out how to change the send buffer.. tcp_xmit_hiwater_def.
I would have expected tcp_recv_hiwater_def to be the receive buffer, but no matter what I set it to using ndd, it defaults to 32K.
Any ideas why ?

Luigi, There is a document on Metalink where Oracle suggest you set the Window Size to 256000 to fix the Ethernet "limitations" on gigabit Ethernet.

Does anybody have advice about the receive buffer.. Im running hp-ux 11.11 on source and 11.22 on destination server.

[rick jones]

ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_ndd.txt

However, since netperf allows you to alter the socket buffer sizes, and since that alters the window sizes, initially at least there is no need to go tweaking ndd settings.

netperf -t TCP_STREAM -- -s -S

to set the local (-s) and remote (-S) socket buffer sizes.