Duplicate TCP acks

[Dhilip Krishnan]

Pls take a look at the tcp dump file attached.
I have noticed dumplcate acks from both the client(Win 2K - 10.1.1.13) as well as the server(HPUX 11.0 - ccux09.c2.cv.hp.com)

Things we tried include:
Turning off SACK, the seldom mentioned, undocumented TCP extension.
Changing buffer sizes.
eliminating net gear (routers, CSS, etc.)

Why is the server acking over and over?

[Ron Kinner]

The server is just telling you that a packet got dropped. Each time it receives a packet that is not the one it is looking for it will ack with the number from the packet it is expecting.
Depending on the local implementation of TCP/IP it may hold on to what it considers to be out of order packets until it receives the missing one (or the timer expires) or it may just drop it right away.

What does your network look like? Check your WAN links for dropped or damaged packets. If there is more than one route to the other end make sure the routers are not trying to do per packet load sharing on unequal speed links.

Check
lanadmin
lan
display
for errors.

check
netstat -s

Believe there was a patch to correct improper handling of out of order packets so make sure you have the latest network patches for your OS.

Ron

[Dhilip Krishnan]

Ron,
Thanx for your response.
We seem to have a workaround for the problem mentioned above.

I changed

ndd -get /dev/tcp tcp_deferred_ack_interval which returns 50

to

ndd -set /dev/tcp tcp_deferred_ack_interval 1

and booom the server sucked the file from the client as fast as pissible.

This setting seems to have solved the problem. Will conduct more tcpdump sessions to see how this parameter made the difference.

[Ron Kinner]

The delayed ack messes up the calculation of the round trip delay so that retransmissions interval is increased more than really necessary. By reducing it I expect you are speeding up the retransmissions of lost acks and masking your problem.

This is what RFC 1122 says about delayed acks:
(ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt)


4.2.3.2 When to Send an ACK Segment

A host that is receiving a stream of TCP data segments can increase efficiency in both the internet and the hosts by sending fewer than one ACK (acknowledgment) segment per data segment received; this is known as a "delayed ACK" [TCP:5].

A TCP SHOULD implement a delayed ACK, but an ACK should not be excessively delayed; in particular, the delay MUST be
less than 0.5 seconds, and in a stream of full-sized segments there SHOULD be an ACK for at least every second
segment.

DISCUSSION:

A delayed ACK gives the application an opportunity to
update the window and perhaps to send an immediate response. In particular, in the case of character-mode remote login, a delayed ACK can reduce the number of segments sent by the server by a factor of 3 (ACK, window update, and echo character all combined in one segment).

In addition, on some large multi-user hosts, a delayed
ACK can substantially reduce protocol processing overhead by reducing the total number of packets to be processed [TCP:5]. However, excessive delays on ACK's can disturb the round-trip timing and packet "clocking" algorithms [TCP:7].

Ron

[Dhilip Krishnan]

Output from netstat -s on the system.

tcp:
654339 packets sent
365738 data packets (931324376 bytes)
61 data packets (3936 bytes) retransmitted
296348 ack-only packets (70079 delayed)
0 URG only packets
0 window probe packets
7826 window update packets
126586 control packets
756438 packets received
233790 acks (for 931340985 bytes)
7443 duplicate acks
0 acks for unsent data
528802 packets (1229001953 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
0 packets with some dup, data (0 bytes duped)
0 out of order packets (0 bytes)
0 packets (0 bytes) of data after window
39 window probes
502329 window update packets
114 packets received after close
2 segments discarded for bad checksum
0 bad TCP segments dropped due to state change
5224 connection requests
6805 connection accepts
12029 connections established (including accepts)
12543 connections closed (including 558 drops)
12 embryonic connections dropped
216403 segments updated rtt (of 216403 attempts)
37 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
6170 keepalive timeouts
6034 keepalive probes sent
2 connections dropped by keepalive
0 connect requests dropped due to full queue
97040 connect requests dropped due to no listener
udp:
0 incomplete headers
0 bad checksums
0 socket overflows
ip:
649345 total packets received
0 bad IP headers
5846 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets forwarded
0 packets not forwardable
icmp:
15033 calls to generate an ICMP error message
0 ICMP messages dropped
Output histogram:
echo reply: 15031
destination unreachable: 2
source quench: 0
routing redirect: 0
echo: 0
time exceeded: 0
parameter problem: 0
time stamp: 0
time stamp reply: 0
address mask request: 0
address mask reply: 0
0 bad ICMP messages
Input histogram:
echo reply: 23
destination unreachable: 4
source quench: 0
routing redirect: 0
echo: 15031
time exceeded: 0
parameter problem: 0
time stamp request: 0
time stamp reply: 0
address mask request: 0
address mask reply: 0
15031 responses sent
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with incorrect fields(s)
0 membership reports received
0 membership reports received with incorrect field(s)
0 membership reports received for groups to which this host belongs
0 membership reports sent

I am trying to speed up an use case in our application where the client using a browser and uploads large sized PDF files to the Webserver.

Since there is no data being send back from the server to the client while uploading the file, I am assuming that the ack for the data recieved from the client is just sitting on the server since tcp piggy backing of acks, the ack gets back to the client only when the timer goes off...

While conducting tests, we eliminated all net gear, the client and the server are in the same subnet.

Thanx Ron

[Ron Kinner]

What does the other end's netstat -s say?

Ron