FTP transfer speed

[Vladimir Fabecic]

Hello
Maybe that is stupid question, but must ask.
When doing ftp transfer between two nodes (in local LAN), I can get near 500 Mbps.
But when doing ftp between two locations, transfer rate can not be more than 72 Mbps.
There is gigabit ethernet connection between these two location.
I have tried with following parameters: TCPIP$FTP_WNDSIZ, WINDOW_SCALE and TCP QUOTA settings.
What else can be done?
Did I miss something?

[labadie_1]

May be
$ set rms/network
and
$ set rms/extend=65535

?

[Steven Schweda]

Same versions of all the FTP client and
server programs? There have been some
speed-related fixes in TCPIP's FTP stuff in
the past year.

[Vladimir Fabecic]

Thanks for reply.
Labadie, will try with rms parameters.
Steven, it is HP TCPIP V5.5 - ECO 1 and OS version is 8.2 AXP on all VMS nodes.

[Wim Van den Wyngaert]

May be one of the routers on the way to the destination is in 100 Mbs ?
Had the problem in decnet with a 10 Mbs.

Wim

[Vladimir Fabecic]

Hello Wim
Network people say that everithing is OK with network switches and there are no packets lost.
And they also said that latency is 4-5 ms (miliseconds).

[Hein van den Heuvel]

speed of light limitation? how far are those sites apart?

the system can not just pump data and never look back.
There has to be an 'ack' at some point.
At that point you the delay becomes more important than the bandwidth.

Hein,

[Robert Gezelter]

Vladimir (and Gentlemen),

Changing parameters can be helpful, or it can have unintended side effects.

The first step is understanding the precise nature of the problem. I suggest using a LAN monitor to see if the problem is propagation delay. If this is indeed the problem, the problem can be tracked down.

In some cases, it is the inherent delay, in other cases it is the setup of the network. There can be surprises here. With ready availability of WireShark (formerly Ethereal) the tools are inexpensively available.

- Bob Gezelter, http://www.rlgsc.com

[Wim Van den Wyngaert]

Just to be sure : do a traceroute (mc tcpip$traceroute node) to see what route is followed. Then ask the network team if all are capable of Gigabit ethernet. Also ask if this is the correct route.

If the ethernet trace is too difficult : try
$ tcptrace/pack=1000/prot=tcp/port=21/fu/out=x.lis node
and post x.lis

Wim

[Vladimir Fabecic]

Did a traceroute before. Network guy said it is OK.
Hein, those sites are 400 km apart.
Of course there has to be an 'ack' at some point. The point is: how to optimize packet/acknowledge ratio. And maybe there is nothing to do on VMS to help. But maybe some parameter change can help.
Wim, will try to do tcptrace tomorow.
Thanks everybody for reply.

[Wim Van den Wyngaert]

You didn't specify your tcp version but on 7.3 with 5.3 I found that for every few thousands (or was it hundreds ?) of packets of data there is an ACK. Depends on the window size agreed between both parties (check in tcptrace : it's there).

Wim

[Steven Schweda]

> You didn't specify your tcp version [...]

Not soon enough, perhaps, but:

Mar 27, 2007 07:30:31 GMT
> it is HP TCPIP V5.5 - ECO 1 and OS version
> is 8.2 AXP on all VMS nodes.

[Hoff · Accepted solution]

Just who is this mysterious Network Guy?

He seems to work everywhere.

When asking a question of Network Guy, you need to be very careful what question you ask, as you can get the answer to a question you didn't intend to have asked. Many incautious questions asked of Network Guy can seemingly be interpreted as "is the network down?", unfortunately.

Try asking Network Guy this question: "What is the wide area link between site X and site Y? SONET?" (And to avoid breaking the spell, SONET is usually pronounced as "sonnet".) Then, when and if prompted for more: "Which SONET level; which OC?"

It looks like you don't have gigabit between the sites, which isn't a huge surprise. Low-latency bandwidth is expensive, and the cost of low-latency bandwidth rises very quickly. (An example of a high-latency high-bandwidth link: the "FedEx network packet"; a FedEx truck chock-full of LTO or Ultrium cartridges, or of terabyte disks.)

It looks like you might have an OC-3 SONET connection between the sites.

To get gigabit-grade speed between sites, you'd likely need OC-18 or OC-24 -- and you might well have to buy OC-48 if the local carriers doesn't offer those other lower-bandwidth packages. These SONET connections are correspondingly rather more expensive than OC-3 (or FedEx), if the local carriers even offer these speeds between Site X and Site Y.

Yes, I can speak some Network Guy. :-)

Stephen Hoffman
HoffmanLabs

[Wim Van den Wyngaert]

Just did some testing. On a deault config an ack is sent for about every 5 packets.
I did ucx set prot tcp/delay on the destination node(was nodelay for Sybase !) and the ack was sent every 20 packets. But this only on a slow alphastation with 10 Mbs and a 20K blocks file.

Redid the test with a 220Kblocks file and then the ratio was 1 ack for every 35 packets.

Wim

[Wim Van den Wyngaert]

But looking in detail I find that at the end of the ftp sessions an ack was given every 500 packets !!! Most acks were given in the beginning of the session.

Wim (glad my test proved my memory was right)

[Hein van den Heuvel]

So you used SET PROT TCP/WINDOW_SCALE.
What is the WNDSIZ setting you tried?

Speed of Light = 300KM/ms
400km, round-trip = 3ms... close enough to the 4 - 5 ms delay stated.

If the protocol was entirely synchroneous this would allow for 250 exchanges / second.
So the 72Mb/sec =~ 9MB /sec = 360KB / exchange = 240 packets @1500 / exchange (not counting overheads).

Google for :"window size" latency tcp ftp gives lots of interesting reads of course:

http://www.asperasoft.com/technology/comparisons/gigabit.html

http://www.supercomp.org/sc2002/paperpdfs/pap.pap327.pdf


Cheers,
Hein.

[Wim Van den Wyngaert]

BTW : my window scale was 61K.

Wim

[Wim Van den Wyngaert]

But it decreases to 10K during the operation.
So, it's just enough.

Wim

[Jan van den Ende]

Re Hein:

>>>
Speed of Light = 300KM/ms
400km, round-trip = 3ms... close enough to the 4 - 5 ms delay stated.
<<<

Yeah, common error.
Most glasses have optical density ~ 1.5
That makes the speed of light in such glass ~200KM/ms.... and now you are close to theoretical.

Only way of speeding up is less round-trips, ie., more data per ACK.
(theoretically, the lightspeed in air (close to that in vacuum) can be used, but only by electromagnetic (light, radiom microwave...) along a direct line of sight. That brings a whole lot of other challenges)

hth

Proost.

Have one on me.

jpe

[Vladimir Fabecic]

Answer from network guy to Hoff:
"The answer to Hoff: WAN link is 2 times gigabit ethernet over DWDM.
Because of the distance (400km), average round trip time is 7 ms (this
is measured by network performance tools)."

He also asks:
"What is the maximum TCP window size supported?"

[Wim Van den Wyngaert]

If window scalling enabled : 1,073,741,823
Otherwise 64 K.

BTW : in the tcptrace you can watch in the RCV packets (the acks) how the window size is at each moment.
If it goes down in the direction 10000 you should increase it.

Wim

[Wim Van den Wyngaert]

In the tcptrace, you find 2 the words "window scale" (N). In my case it's 0 but if it's higher, you must multiply the window size you see with 2**N.

DOn't know yet how it gets higher than 1. Not by enabling window scaling.

Wim

[Vladimir Fabecic]

Window scalling is enabled.
TCPIP> sho prot tcp/par
TCP
Delay ACK: enabled
Window scale: enabled
Drop count: 8
Probe timer: 7200

Receive Send

Push: disabled disabled
Quota: 61440 61440

TCPIP>

Wim, do you mean that TCPIP$FTP_WNDSIZ can be set to 512k, for example?

[Wim Van den Wyngaert]

Found it but it's not documented (I think).

You define TCPIP$FTP_WNDSIZ as a big value (e.g. 1000000) and the scale is calculated by ftp (on the receiving side). But you need to restart ftp server (thus exit ftp).

Wim

[Wim Van den Wyngaert]

Tested it with a 350 MB file with 100 Mbit network (switch only).

I don't get above 3.8 MB per second (destination nl:). I increased quotas to 1M too but no improvement.

I checked the tcptrace and found that the scale was 3 (thus ok) but that the receiving side did an ACK for every 2 packets. Even when delayed ack was on.

But I'm on 5.3 eco 2. May be it was not yet fully working in that version.

Wim

Wim