Bandwidth Utilisation

[Phil Croome_2]

I may have a problem and would be grateful of any help.
We have an office connected to us using a slow WAN link (1Mb). There have been reports of problems downloading and uploading files from their client to our UNIX server here in the UK.
The NT guys have increased the TCP window size which has much improved the speed of transferring files from UNIX to the client but we are still experiancing problems sending files from the client to UNIX.
The setup is: 100Mb ethernet here and a 100Mb ethernet at the remote office both connected to routers using a 1Mb line. There is Round Trip Delay of 270ms.
The Question is: Is there a setting in UNIX which will spead the process of recieving data in this scenario e.g. TcpWindowSize?
thanks,
Phil

[G. Vrijhoeven]

Hi,

check out the ndd section of:
http://docs.hp.com/cgi-bin/fsearch/framedisplay?top=/hpux/onlinedocs/5187-4505/5187-4505_top.html&con=/hpux/onlinedocs/5187-4505/00/01/119-con.html&toc=/hpux/onlinedocs/5187-4505/00/01/119-toc.html&searchterms=ndd&queryid=20031217-051738

Hope this will help,

Gideon

[doug mielke]

that delay doesn't seem bad at all. Is that time for the very first connection?

Maybe it's a lag in establishing the connection, which could happen many times during the file transfer.

Having both system names in host files might help, if you're using DNS, with /etc/nsswitch.conf set to read files first. Also check that traceroute is consistant, and not using some other route when things get busy.

[Ron Kinner]

TCP/IP adjusts the window size based on its experience. If packets are lost your window drops to 1 and then doubles each time you don't lose a packet up to the maximum which is I suppose what your NT guys are tweaking. I expect your 1Mb line is dropping packets. Either because the line is noisy or because there is too much traffic trying to cross and packets are being dropped from the queue. This will cause a huge delay while the window ramps back up. On a cisco router:

show interface

and look for errors and drops. Clear Counters and let it run for a while and then check it again. If you see errors call the telco. If you see drops then look into more bandwidth, custom queueing, or access lists.

There is an enhancement for the TCP/IP protocol which understands that drops are not congestion but are errors that is designed to be used over satellite links. IETF's RFC 1323
TCP Extended Windows and RFC 2018 TCP Selective Acknowledgement but I am not sure if they are available yet.

Another possibility since your problem seems to be in one direction only is the ever popular duplex mismatch. One side of a CAT5 is set manually to Full and the other side is set to Auto. The side which is Auto will set itself to Half (Auto has no choice in the matter. That is what the RFC requires.) and will have trouble talking and experience a lot of collisions. The side set to Full will see a lot of short packets with CRC errors. Either set both sides to Full or both to Auto.

lanadmin
lan
display

(second page shows the errors and collisions)

If you are using 10.3 or better you can do

ndd -h |grep tcp
and
ndd -h |grep ip

to get a list of parameters that you can play with but I doubt that it will help.

Ron

[Todd Whitcher]

Regarding the RFC 2018, the default is to use Selective Ack if the remote side initiates it for 11.0 and 11.i.

tcp_sack_enable
Hidden for 11.0
Supported for 11i

This option enables the Selective ACKowledgement for TCP.

TCP may experience poor performance when multiple packets are lost from one window of data.
With the limited information available from cumulative acknowledgments, a TCP sender can only learn about a single lost packet per round trip time.
An aggressive sender could choose to retransmit packets early, but such retransmitted segments may have already been successfully received.

Multiple packet losses from a window of data can have a catastrophic effect on TCP throughput.
TCP uses a cumulative acknowledgment scheme in which received segments that are not at the left edge of the receive window are not acknowledged.
This forces the sender to either wait a roundtrip time to find out about each lost packet, or to unnecessarily retransmit segments which have been correctly received.
With the cumulative acknowledgment scheme, multiple dropped segments generally cause TCP to lose its ACK-based clock, reducing overall throughput.

Selective Acknowledgment (SACK) is a strategy which corrects this behavior in the face of multiple dropped segments. With selective acknowledgments, the data receiver can inform the sender about all segments that have arrived successfully, so the sender need retransmit only the segments that have actually been lost.



The offer to use sack can only be made in during the connection establishment (SYN).
It will be discarded in later packets.
If sender and receiver have agreed to use sack the receiver could send the sack TCP options in any TCP packet.

All these information are described in detail in RFC 2018. (Which is at the moment proposed standard)

Settings, Default is 2 for 11.i

0 Never use SACK
1 Always initiate SACK
2 Allow but don't initiate

ATTENTION: Official release will be in 11.i


Usable commands:
Check the current value:

ndd -get /dev/tcp tcp_sack_enable
Set the always initiate sack:

ndd -set /dev/tcp tcp_sack_enable 1


nddconf entry example:
TRANSPORT_NAME[0]=tcp
NDD_NAME[0]=tcp_sack_enable
NDD_VALUE[0]=1


You can view the help options with:

ndd -h tcp_sack_enable

[Bill Hassell]

And just a general note about WAN's with long roundtrip time (250ms). File transfer protocols that require synchronous acknowledgement will have terrible performance. Some examples are rcp, SAMBA/CIFS and NFS. ftp is much better in that acknowledgement will occur asynchronously and several packets can be sent/received before acks come back. ftp is the best protocol for efficient transmission. Your 1Mb (I am assuming that it is a T1 line) can handle perhaps 40-80 Mbytes/sec with no other traffic. If a faster WAN connection is not an option, then compressing files before transmission and using ftp is about all you can do.

[Bill Hassell]

Whoops, wrong abbreviation: 40-80Kbytes/sec for a T1 line. The lower number would be fairly normal for a WAN using ftp on a lightly loaded link.

[rick jones · Accepted solution]

So, if we take the magic formula of Tput <= Window/RTT and massage it a bit, we have that you want Window >= Tput*RTT

So, Window > 1Mbit/s * 0.270S or 270000 bits which is 33750 bytes or ~32.69 KB.

The default window size in HP-UX 11* is 32768 bytes.

By default, FTP and rcp on HP-UX 11 will use a socket buffer and hence window size of 56KB.

So, in theory, you should already have the HP-UX system advertising a sufficiently large window to achieve link-rate with that round trip delay. It might still be worth trying to up tcp_recv_hiwater_def if your file transfer protocol is something other than FTP/rcp.

That perforamnce is OK from UX to NT but not OK from NT to UX would suggest that the file transfer protocol is not a request-reply one - otherwise the round trip times would make things equally bad for both sides.

That suggests packet loss as a possibility. If you take output from "netstat -p tcp" from before an upload and then again from after an upload, and run the two through "beforeafter" (ftp://ftp.cup.hp.com/dist/networking/tools/) you can then look for indications that there were packet losses (ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_netstat.txt)

The suggestoins about SACK are interesting. Given that the default for tcp_sack_enable is indeed 2, and that means accept, but do not initiate SACK, the question comes up - which side is initiating the (data) connection for the file uploads? I can never rememeber which it is with FTP. If it is the client, then the TCP connection is already using SACK. You can go ahead and set tcp_sack_enable to 1, it likely won't hurt anything.

ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_ndd.txt

One other thing to consider is tweaking tcp_defered_ack_max. From the description, it seems the slow case is when HP-UX 11 is receiving the data - that means that HP-UX 11 is generating ACKs. HP-UX 11 has a rather clever ACK avoidance heuristic in it, which in general should deal with packet losses and subsequent regrowth of the congestion window OK, but there can be the odd case where it may not. Thus, one can set tcp_deferred_ack_max to say 2, and that might allow the congestion window to grow more quickly after a packet loss. I'm still not sure I like the decision that congestion windows should grow based on the number of ACKs rather than the qantity of data ACKed by those ACKS :(

Of course, if there are packet losses, and the WAN link is somewhat private, you might want to consider trying to find the source of the packet losses and getting it corrected. All the SACK and deferred ACK tweaking is really just treating a symptom.

[Phil Croome_2]

Thanks Rick.

That is just the information I need. I think the problem lies elsewhere rather than UNIX. My window sizes are fine.

Thanks for your help.

Phil