performance: tcp window size not effective

[Brem Belguebli]

Hi,

I have a ftp transfer (4 GB of gzipped data) between 2 HP-UX boxes (IA 64 11iv2 and PA 11i) distant by 270 ms over a DS3 WAN.

Transfers do not go over 2Mb/s even after raising tcp window (recv and xmit) size to 157286 on both sides . (ndd -set /dev/tcp tcp_xmit_hiwater_def and tcp_recv_hiwater_def

When tcpdumping the connection I see a 39321 window size . Both machines are connected to the LAN with Gb fiber cards.

Ftpd is HP-Ux 11iv2 wu-ftp 2.6.1 (PHNE 34036).

Any idea ?

[Brem Belguebli]

Hi,

Still limited transfer rate to max 2Mb/s even if I have found out the "window scale" parameter to go over the 16 bits limit of the windows size TCP option.

[skt_skt]

what r the values set for tcp_xmit_hiwater_max and tcp_recv_hiwater_max

[Brem Belguebli]

the values for tcp_xmit_hiwater_max and tcp_recv_hiwater_max are left to default to
1073725440.

[rick jones]

Tput <= Weff/RTT

Where Weff is the effective window size, which will be the lesser of:

1) the remote's advertised window (min of tcp_recv_hiwater_def and SO_RCVBUF)
2) the quantity of data sent at any one time by the sending application (the min of app's send() calls before waiting for remote response, the SO_SNDBUF size and tcp_xmit_hiwater_def)
3) the sending TCP's calculated congestion window.

So, one thing to check is whether there is any TCP retransmissions, which will keep the calculated congestion window (aka cwnd) low.

However, since you mention FTP, that brings us to something in the areas 1 and 2 - FTP under HP-UX will (IIRC) make an explicit setsockopt() call to set the SO_SNDBUF and SO_RCVBUF to values - the default IIRC is 56KB, and that can be altered by the -B option to the ftp client and ftpd daemon. To be effective (per one and two above) it has to be set at both ends.

This explicit setsockopt() goes back a very long way to when HP-UX window defaults were less than 32KB and we needed more for good FTP performance. Since then I've also been asking (back in the mists of time, I've not been repeating it) for a change in default from 56KB to either something much larger (256 KB) or to "0" which would mean "accept the system default."

I suspect that neither of those have yet happened in the, sadly and frankly, years since the requests were made. If this is the case I would encourage you to excercise your support contract and file a _defect_ against FTP because the current default is a severe performance degregadation over where performance should be over WANs.

If you want to experiment with values to use to verify your math, I would suggest netperf TCP_STREAM tests - you can control the socket buffers explicitly there per the manual which you can find at:

http://www.netperf.org/svn/netperf2/tags/netperf-2.4.3/doc/netperf.html

for the current release of netperf2.

[Brem Belguebli]

Hi,

Thanks for the information.
I've tried with the -B 256 (expressed in 1024 bytes units) on both client and server (thru inetd), but the result remains the same.

[rick jones]

With the -B option in place on both sides, what specifically do you see in the SYNchronize segments on the wire? Is there a window scaling option present in both the SYN and the SYN|ACK?

How about if you run a netperf TCP_STREAM or TCP_MAERTS test between the two systems? http://www.netperf.org/

netperf -H -t TCP_STREAM -- -s 256K -S 256K -m 256K

or something along those lines.

[Brem Belguebli]

When setting the ftpd (on the 11iv2) -B 256 (ftp stream tcp6 nowait root /usr/lbin/ftpd ftpd -l -B 256) and connecting from the 11 machine ftp -B 256 11iv2-host, I have a Window scale option (* 4) in the SYN packet but 0 inr the SYN ACK one though both machine have a tcp_xmit_hiwater_def and tcp_recv_hiwater_def set to 157286.

I couldn't netperf as the thing didn't want to compile on the 11.00 machine.

[rick jones]

Hmm, your initialpost said 11iv2 and 11i which woudl be v1 which is 11.11 :) Still, there should be a half-way decent chance of netperf compiling on 11.00 although perhaps some issues with getaddrinfo() - the "feedback" links on www.netperf.org will reach me or you can join and then post to netperf-talk on netperf.org.

If both the SYN and the SYN|ACK had a Wscale of 4 - what was the value in the window field again? One takes that value and "left shifts" it wscale times to get the "real" window size - in this case it would be a left-shift of four, or multiply by 16.

One other thing you can do that will only require netperf running on the 11iv2 system and not the 11.0 system is make sure that the 11.0 system has both chargen and discard services enabled (netstat -a | grep -e discard -e chargen) You can then from the 11iv2 system run netperf commands along the lines of:

netperf -H <11.0sys> -t TCP_STREAM -N -- -p,discard

to send data to the discard service on the remote or:

netperf -H <11.0sys> -t TCP_MAERTS -N -- -p,chargen

to have the remote send data to netperf. Only the "local" test-specific options will be effective which means we have to hope that the ndd settings on the 11.0 system actually "take."

For painful completeness, one can also use the echo service for a TCP_RR test:

netperf -H <11.0sys> -t TCP_RR -N -- -p,echo

Here are some examples:

raj@tardy:~/netperf2_trunk$ src/netperf -N -H tardy.cup.hp.com -- -p,discard
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to tardy.cup.hp.com (15.244.56.217) port 9 AF_INET : no control
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

0 16384 16384 10.02 49.48

raj@tardy:~/netperf2_trunk$ src/netperf -N -H tardy.cup.hp.com -t TCP_MAERTS -- -p,chargen
TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to tardy.cup.hp.com (15.244.56.217) port 19 AF_INET : no control
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

87380 0 -1 10.01 78.24

raj@tardy:~/netperf2_trunk$ src/netperf -N -H tardy.cup.hp.com -t TCP_RR -- -p,echo
TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to tardy.cup.hp.com (15.244.56.217) port 7 AF_INET : no control
Local /Remote
Socket Size Request Resp. Elapsed Trans.
Send Recv Size Size Time Rate
bytes Bytes bytes bytes secs. per sec

16384 87380 1 1 10.01 710.94
0 0

[Brem Belguebli]

Sorry for the typo, it is really a 11.00 (not i version). :-(

Only the Syn had a wscale of * 4 (65536 Window), the synack had 0 with a "default??" 32768 window.

I'll try to netperf chargen or discard. I guess it'll generate some load between my 2 machines (haven't read the doc yet).

[rick jones]

So, that means that if the transfer is to the system which responded with wscale0 32768 that is all the window will be, which will limit things a bit vor a 270ms WAN link.

On the 11.0 system see if there are any ndd parms relating to window scaling and triple-check that the tcp_xmit_hiwater_def and tcp_recv_hiwater_def settings "took" that the wscale was 0 and the window field 32768 suggests it did not take - perhaps a reboot has reset things if the settings weren't put into /etc/rc.config.d/nddconf

[Brem Belguebli]

triple checked the settings, Syn is sent from the 11.00 host (Wscale *4) so the settings are well applied.

The server (11iv2 machine) (wscale 0) looks well configured according to the ndd -get output. Unfortunately this machine cannot be rebooted, it runs some service guard packages that cannot be switched to another node as is.

[rick jones]

thankfully, ndd settings don't require reboot. changes to things in inetd.conf only require an inetd -c command to "take"

[Brem Belguebli]

yes,

But if somehow ndd was reading from some structure the right values ignored due to some sort of bug by the TCP layer (what looks like being happening in my case)

[rick jones]

probably best to make sure the inetd -c happens after the ndd commands and not before. still, when/if you can get netperf going that should be a good test

[Brem Belguebli]

It looks my ndd commands are not taken dynamically (I still get a wscale 0 )event though the new values are seen when ndd -get.

After a reboot (changes put in nddconf), things change as I get the right wscale value (4). tried it on a test 11iv2 machine and on a 11i machine.

The -B 256 option doesn't seem to have the desired effect as my transfer rates get worse when set.

Thanks

[rick jones]

ndd settings should be taking dynamically, but it may require a "new" socket to get the new values. so, the listen endpoint for ftpd, which is maintained by inetd, may not be updated unless you kill and restart the inetd. i would have thought that an inetd -c might be enough, but inetd may be too smart for our own good, and seeing that ftpd is still there, not re-create it.

the tcp connection exists by the time ftpd is started, so the ability of the ftpd's -B option to increase the window is limited if the inetd didn't create the listen endpoint with a "large enough" socket buffer setting, and i'm not sure if inetd has options for that or not, which would leave it at the mercy of the ndd settings, which of course as mentioned before won't be applied until the listen endpoints are re-created.

might be worthwhile seeing if the ftpd can be run as a standalone daemon.

the same issues will hold for chargen, discard and echo services...

[Brem Belguebli]

I ve run a few tests after your last post, and I couldn't make the system take the changes without killing and relaunching inetd, inetd -c just re reads the inetd.conf without regard to any buffer change (no residual ftpd was running).
By the way inetd accepts ftpd -B XXX and everytime it was set my transfers were slower.

Netperfing chargen and discard didn't work (I got error messages saying it didn't recieve no response).

I think i'm gonna give up untill they at the remote site to change this oldy PARISC 11.00 machine

Thanks for the help

Brem

[rick jones]

Odd - which version of netperf were you using, and you did remember the -N option right? Are chargen and discard actually enabled on the 11.0 system and not perhaps blocked by firewall rules?

I'd still like to see the compilation or configuration errors from trying to build netperf2 on 11.00 if you have the cycles. Sadly I have no 11.0 systems at my disposal these days. Coblers children and all that I suppose... rick.jones2@hp.com

[Brem Belguebli]

It's a 2.4.2 version, I saw after there's a .3 one.

-N option didn't work with the 2.4.2.

I could reach 8Mb/s by using rcp instead of ftp which is much better (window size 315 k).

Did some captures during ftp transfers, I noticed that on the ftp (tcp/21) communication the Window scale options are set on both side, but they're not on the ftp-data socket (tcp/20) :what makes me think this is the reason that makes FTP slower.

I'll definitely switch to rcp

Thanks for your help.

[rick jones]

Might also consider looking for some FTP patches, but if rcp is getting you where you want to be, enjoy.