ftp transmit and receive times

[steven Burgess_2]

Hi all

I have an issue with ftp transmit and receive times from a particular server

HPUX11

btlan3 card set at 100 full auto off.

Network team have confirmed that the adjacent switch port has the same speed setting as above

what /stand/vmunix | grep btlan3

100BT : btlan3.c Core 100BT Cleanup PHNE_25907 B.11.00.08 $Revision: 1.78 $ $Date: 2001/11/02 15:30:00 $

Have a look at the below

server server method kb/s secs
38 41 put 128.22 147.22
38 27 put 278.28 67.81
38 44 put 200 94.12

38 41 get 7777.48 2.43
38 27 get 1123.85 16.79
38 44 get 7322.87 2.58

41 38 put 839.52 22.48
27 38 put 213.64 88.33
44 38 put 885 21.3

41 38 get 398.08 47.4
27 38 get 1131.69 16.67
44 38 get 173 108.99

On 38 if I put a file onto other servers the transmit time is appalling, yet if I get a file from the servers the receive time is as expected

It looks to me like a dodgy cable/speed mismatch. Has anyone else seen this type of thing before

Thanks in advance

Steve

[Stefan Farrelly]

Yes, weve seen this before. First thing to do is ensure the files you are timing for the ftp transfer are sufficiently large - small files give variable results. eg. use prealloc to create a 50Mb file and try that.

We found problems with ports set to auto and cards set the same and found the solution was to set to half duplex on the HP and on the switch port - not auto or full.

Lastly once I remember we simply repatched to a different switch port and the variable ftp times went away.

[steven Burgess_2]

Hi Stefan

I am going to swap the ports on the switch and replace the cable.

The file I have been transmitting is 25mb

Thanks

Steve

[Dietmar Konermann]

Hi, Steven!

Maybe it's a good idea to rule out other possible factors... simply ftp from /dev/zero ro /dev/null, in this case 1GB:

# mknod /dev/zero c 3 3
# ftp
...
ftp> put "|dd if=/dev/zero bs=8192 count=131072" /dev/null

Best regards...
Dietmar.

[Volker Borowski]

Hi Steve,

7000KB/sec is the best what comes close to 100MBIT.
1xxxKB/sec seems to be somewhat 10MBit related.

Everything below is pure rubbish and has to contain drops and retries and duplex mismatch....

May be ftp is a bad choice for detecting this stuff. It gives you an all average and you see nothing in detail. How does a ping with a packet size of 1000 Bytes look ?

Set interval to 5 secs an count to 100.
Unplug (if possible) your server before each retry, so that the uplink has no more memory about the previous connection. Now check if for this sequence you have diffrent or constant round trip times. This will show you either concurrent traffic or drops or delays due to internal switch reconfiguration.

Then Run all 3 streams concurrently from one server.
Diffrences ?

Hard to debug !
I'd start hardcoding the switch configuration and drop my trust in the connection/uplink LEDs.

Good hunting
Volker

[rick jones]

a couple things to look at:

*) lanadmin stats for the interface - look for FCS errors, late collisions, etc.

*) netstat -p tcp stats - look for similar things - out of order packets, retransmissions, etc

*) tcpdump of transfers to make sure that the window sizes are large enough in each case. default socket buffer size in UX FTP is 56KB - if you are up-to-date on patches,you can up that with the -B option to just about any size. all other factors asside, TCP trasnfers no faster than Windowsize/RoundTripTime, and the "windowsize" will be the minimum of the receiver's advertised window (what is in the TCP header) and the sender's SO_SNDBUF setting

[steven Burgess_2]

Hi Guys

Have been that busy I forgot about this post.

I'm afraid I will have to raise requests before I go unplugging cables from the server etc

Ricky

#netstat -p tcp | grep -i retra

580695 data packets (689001016 bytes) retransmitted
153666 retransmit timeouts

#netstat -p tcp | grep -i order
512612 out of order packets(705489288 bytes)

from lanadmin for that interface I get

FCS Errors = 9543

FCS errors are referred to as

FCS Errors A count of frames received on a particular interface that
are not an integral number of octets in length and do not pass the FCS check

Can you expand on this and the other findings

Thanks in advance

Steve

[Ron Kinner · Accepted solution]

Have your network guys go back and recheck. They are probably set to auto thinking that that will handle whatever you throw at them. FCS errors almost always are caused by a duplex mismatch. You see them on the full duplex side because the half duplex end will stop transmitting every time the full side sends something when the half is also sending. It's the abrupt stopping that causes the full side to receive FCS errors.

If you can't get them to cooperate then just set your end to half. It will work a lot better than it currently does and also prove that the network guy was wrong and you had a duplex mismatch.

I suppose it could be a bad NIC or a bad cable but the duplex mismatch is the usual reason.

Ron

[rick jones]

"Ricky"?!? I've not been Ricky for oh, 25 years or so... :)

I second Ron's comments about FCS indicating duplex mismatch. Make sure that both ends of the cable are set to the same value and all should be well.

If the interface description from the lanadmin statistics says that the card was set to auto, it implies a failure of autoneg. One that may include a problem on the HP side because if autoneg "fails" the card is supossed to (by the spec) go into half-duplex. However, my guess is that the NIC is set FULL, and the switch is set either half or auto. Hardcoding means that side will _not_ autoneg - that is a common misconception - that you can hardcode one side and the other side will detect via autoneg what that is.

Now, as for why this might not affect a file received by the UX system as much as a file sent by the UX system... One possible contributor is the TCP/IP stack in HP-UX 11 having an ACK avoidance heuristic. As the stack notices that it has received data successfully, it backs-off on how often it tries to send ACKs. It does this because ACKs are basicaly pure overhead - they consume just as many CPU cycles (to a first order) as a data segment.

[Ron Kinner]

Rick,

I don't think HP gets the credit for the fact that data leaving goes faster than data coming when the HP is set to full duplex and the other end is at half.

Instead you are sending 1500 byte packets out and getting little tiny ack packets back. The 1500 byte packets go through without any problem because even tho the other end thinks it has only half duplex we do have a separate pair of wires and there is no real collision. The acqs are the only things which are subject to thinking they have a collision and they are so small compared to the 1500 byte packets that it is not too hard to slip one through or retransmit in the times between 1500 byte packets especially when you consider that the half end waits until the other end shuts up before sending them.

In the other direction an incoming 1500 byte packet is exposed to the possibility of a "collision" for a much longer time and if after about 15 tries it can't get through it drops the packet. A lost packet triggers the congestion avoidance routine which shrinks the number of packets that can be sent without an acq down to one and every attempt to build up the number (grow the window) fails because it loses another packet to collisions. This slows things down considerably. The effect would not be so apparent if you had a simple hub and both hosts were connected to the hub. Having a switch makes it worse since a switch usually waits until it either gets the whole packet or enough of it to check that it has a valid header before sending it on which generates a small delay. Going over a WAN with its delay of 10 to 2000 milliseconds(I have circuits which go over a satellite and then over a VSAT which have a consistent 2 second ping time) + the extra time required to transmit all 1500 bytes over a small pipe really makes the difference show up. In the old days Novell's IPX was designed to send a packet and wait for an acq before sending another. This worked OK on the hubs of the day but really slowed things down when you tried to route it over a WAN.

Ron

[steven Burgess_2]

Hi

Thanks for the repsonses guys

I will let you know how I get on

Steve