HP-UX 11.0 TCP Problem

[Mark Cotherman]

I have the following tcpdump for a TCP session:

The abc host is the HP-UX host. The xyz host is a Solaris host.


18:31:13.585376 abc.51519 > xyz.7001: S 840842314:840842314(0) win 32768
18:31:13.585599 xyz.7001 > abc.51519: S 3949685170:3949685170(0) ack 840842315 win 24820 (DF
)
18:31:13.587426 abc.51519 > xyz.7001: P 1:240(239) ack 1 win 32768
18:31:13.587503 abc.51519 > xyz.7001: P 240:1700(1460) ack 1 win 32768
18:31:13.595863 abc.51519 > xyz.7001: P 1700:2328(628) ack 1 win 32768
18:31:13.595881 xyz.7001 > abc.51519: . ack 240 win 24581 (DF)
18:31:13.595895 xyz.7001 > abc.51519: . ack 1700 win 24820 (DF)
18:31:13.611238 xyz.7001 > abc.51519: P 1:307(306) ack 2328 win 24820 (DF)
18:31:13.611572 xyz.7001 > abc.51519: . 307:1767(1460) ack 2328 win 24820 (DF)
18:31:13.611606 abc.51519 > xyz.7001: . ack 1767 win 32768
18:31:13.613850 xyz.7001 > abc.51519: F 3176:3176(0) ack 2328 win 24820 (DF)
18:31:17.871423 xyz.7001 > abc.51519: FP 1767:3176(1409) ack 2328 win 24820 (DF)
18:31:17.871481 abc.51519 > xyz.7001: . ack 3177 win 32768
18:31:17.872428 abc.51519 > xyz.7001: F 2328:2328(0) ack 3177 win 0
18:31:17.872450 xyz.7001 > abc.51519: . ack 2329 win 24820 (DF)



After the HP box sends an ACK with the RCV.NXT set to 1767, It appears that the HP box receives a FIN from the solaris box with a sequence number of 3176. I looked through the TCP spec and it seems that this should be handled by the TCP stack on HP. It's strange that the Solaris box is sending a sequence number of 3176.

After the first FIN is received by the HP host, a second FIN is received about 4 seconds later with the sequence number set to 1767:3176. The HP host immediately responds.

Is this a bug?? I am by no means a TCP expert.

Thanks

[Ron Kinner]

I doubt it's a bug. What you are seeing is an out of order packet. It happens sometimes that during transport one packet will find a quicker way than the preceding packet and arrive before the first one. Every self respecting TCP/IP stack has the capability to hold on to an out of order packets for a while to see if the missing packet shows up.

4 seconds seems like a long time to be lost in space so most likely a packet got lost and was resent. That would probably account for the second FIN. The missing packet was probably just a PUSH followed by a packet with just the empty FIN (note that the sequence number is really a count of the payload bytes) and when no reply was received a combined FINPUSH was sent to replace both the missing packet and the FIN packet which was received but not ACK'd.

Ron

[rick jones]

I'd say that Ron has it pretty correct - it does look like there was a lost packet that had to be retrasmitted. The stats on the Solaris system should confirm that - by showing data segments retransmitted and some retransmission timeouts. The stats on the HP-UX side should show some out-of-order segments being received (netstat stats here).

In the TCP header, the sequence number is the sequence number at which the data begins (TCP basically assigns a sequence number to every byte it sends)

Tcpdump, by showing the starting sequence number and ending sequence number and length start:end(length) is simply doing some math for your. It is as you can tell doing other math by subtracting the ISN (Initial Sequence Number) so us mere mortals can deal with data streams that appear to start with sequence number 1.

BTW, where was this trace taken? HP_UX end, Solaris end, or somewhere in the middle? Also, next time, using "HP" and "Sol" as the hostnames instead of abc and xyz will make things a triffle easier to follow :)

[Mark Cotherman]

Thanks for the responses.

The trace was done on the HP machine. This problem consistently happens on every request in the same place. Does that seem like a network problem?? It seems that a network problem would be more random in where the packets are dropped.

[rick jones]

if the problem is consistent, is the data also always the same? if so, then there is a data dependent problem somewhere in the network. The packets aren't even getting to the promiscuous mode trace point, so start with lanadmin -g mibstats on the HP-UX box and work your way back across the network to the solaris system.

If the trace is on the HP, I'm a _triffle_ surprised that there was not an immediate ACK 1767 after the out-of-order bare FIN arrived. However, I do seem to recall some stuff changing in how out-of-order bare FIN's were dealt with, so being up on the latest HP-UX transport patches might be good. I seem to remember similar things for Solaris, be sure to be up to date there too.

Even ignoring that (not that the single immediate ACK could have triggered a quicker retransmit...) there is still the issue of the lost segments themselves.

[Steven E. Protter]

Just for grins.

Go out and get every TCP patch on the UX side, install it. Make sure December 2002 Gold Quality pack is installed first.

Then go out and do the Solaris equivalent steps.

If there is a bug on either side, these steps should squash it.

Somebody isn't playing by the rules.

SEP

[Ron Kinner]

Rick,

There is apparently a bit of ambiguity in RFC 793 about whether to ack an out of order packet or not. See the discussion at:

http://www-mice.cs.ucl.ac.uk/multimedia/misc/tcp_ip/8804.mm.www/0066.html

Mark,

What is the normal ping response time for this link?

Do a standard ping and then a larger one of about 1460 bytes.

Are there any known bottlenecks? (Slow serial links perhaps?) What sort of queuing is used on these links? Are you dropping packets at the queues? I wondering if something is queuing based on packet size and letting little packets through while big packets have to sit in a long overfilled queue. That would explain how your empty FIN gets through while the fat 1409 gets lost. You might get the same effect on a bad link with high collisions or noise. A smaller packet has a much better chance of getting through than a larger. Wouldn't explain why it is always the next to the last packet that gets lost tho (if that is what you are saying).

So it could still be some sort of bug in the Solaris's TCP/IP stack. Perhaps something causes him to lose the next to the last packet of a conversation.

Ron

[Mark Cotherman]

From what I undestand, there is nothing between the HP and Sun boxes. I am asking for more info from the guys onsite.

Here is a TCP dump from the Sun box perspective.

1 0.00000 HPUX -> SOLARIS TCP D=7001 S=59410 Syn Seq=3769744967 Len=0 Win=32768 Options=
2 0.00006 SOLARIS -> HPUX TCP D=59410 S=7001 Syn Ack=3769744968 Seq=1024955216 Len=0 Win=24820 Options=
3 0.00324 HPUX -> SOLARIS TCP D=7001 S=59410 Ack=1024955217 Seq=3769744968 Len=239 Win=32768
4 0.00004 SOLARIS -> HPUX TCP D=59410 S=7001 Ack=3769745207 Seq=1024955217 Len=0 Win=24581
5 0.00019 HPUX -> SOLARIS TCP D=7001 S=59410 Ack=1024955217 Seq=3769745207 Len=1460 Win=32768
6 0.00004 HPUX -> SOLARIS TCP D=7001 S=59410 Ack=1024955217 Seq=3769746667 Len=795 Win=32768
7 0.00022 SOLARIS -> HPUX TCP D=59410 S=7001 Ack=3769746667 Seq=1024955217 Len=0 Win=24820
8 0.02617 SOLARIS -> HPUX TCP D=59410 S=7001 Ack=3769747462 Seq=1024955217 Len=306 Win=24820
9 0.00010 SOLARIS -> HPUX TCP D=59410 S=7001 Ack=3769747462 Seq=1024955523 Len=1460 Win=24820
10 0.00004 SOLARIS -> HPUX TCP D=59410 S=7001 Ack=3769747462 Seq=1024956983 Len=1412 Win=24820
11 0.00039 HPUX -> SOLARIS TCP D=7001 S=59410 Ack=1024956983 Seq=3769747462 Len=0 Win=32768
12 0.03148 SOLARIS -> HPUX TCP D=59410 S=7001 Fin Ack=3769747462 Seq=1024958395 Len=0 Win=24820
13 4.22073 SOLARIS -> HPUX TCP D=59410 S=7001 Fin Ack=3769747462 Seq=1024956983 Len=1412 Win=24820
14 0.00047 HPUX -> SOLARIS TCP D=7001 S=59410 Ack=1024958396 Seq=3769747462 Len=0 Win=32768
15 0.00237 HPUX -> SOLARIS TCP D=7001 S=59410 Fin Ack=1024958396 Seq=3769747462 Len=0 Win=0
16 0.00004 SOLARIS -> HPUX TCP D=59410 S=7001 Ack=3769747463 Seq=1024958396 Len=0 Win=24820



So, the delay is on the Sun box also in lines 12 & 13. I guess the question is should the HP box be sending an ACK to the first FIN, or is the Solaris box screwing things up by sending an out of order FIN, expecting an ACK to the FIN and not transmitting the remaining payload without receiving the ACK??

[Ron Kinner]

This time it appears that only a FIN packet was lost and after a 4 second timeout it was resent. This appears normal. The question is what happened to the missing packet. Since you say they are on the same LAN then I would start looking at the Ethernet connection for errors. What driver are you using for the NIC on the HPUX side? It wouldn't be a btlan4 by any chance? I was trying to find a btlan4 driver the other day for another poster and I came across a patch which fixed a problem where the btlan4 driver would randomly drop packets.

PHNE_22216:
1. 100BT is randomly missing packets.
2. System panics due to instruction page fault.
3. ER: Display capabilities of the switch after auto
negotiation completion in nettl log (informative).
4. ER: Provide driver revision in btlan4_ift_t_2 structure
for Q4 support.
5. ER:for Q4 (dump reading tool) : change the name of the
internal btlan structure.
6. ER: Startup scripts and conf files should show the cards
that they support.
7. netfmt gets SIGSEGV signal when HP_APA LAN_MONITOR is
active.
8. Cable disconnect for cards with AUI sends 3 events to APA
as a result of which APA fails.
9. 100BT card receive engine hangs resulting in discard of
all incoming packets.


Ron

[Mark Cotherman]

Ron,

You said that a FIN packet was lost. I don't see that. I see the same FIN's on both ends (HP and Sun) with about the same delay in between the first and second FIN. Am I missing what you're talking about?

Thanks

[rick jones]

i too thought I saw evidence of a data segment being lost in the solaris side trace - the seqno of the FIN was more than one off the last data segment in the trace.

the HP side should never send an ACK for the "other" side of a hole in the sequence space - when the solaris system sends that FIN that jumps in the sequence space, the HP system must not ack that sequence number until it has receved all the prior sequence numbers (modulo selective ACK, which isn't being used here...)

that we also see the packet missing in the solaris trace implies a couple possibilities:

1) the soalrs box droipped a segment somewhere between tcp and the tracing point (presumeably in the driver). I'd check TCP and IP and link stats (netstat -k for link stats, netstat -s for TCP and IP.

2) the "missing" data was queue waiting for the congestion window to open, the application called close(), and that triggered a FIN to be sent without also making sure that the queued, unsent data was "flushed" out the network.

Some ways to explore that might include setting the intial congestion window to a larger value - i know that UX canbe configured that way, look for a solaris ndd setting similar to tcp_cwnd_initial and increase it. That could conceivably "workaround" the bug with the FIN being sent before the queued data, you would still want to find an apropriate solaris patch

[Ron Kinner]

Mark,

You are right. I was just looking at the two FINs and didn't notice that the second FIN had data in it. I got used to the HP's tcpdump output with the nice P when it was sending data. Here I need to look at the Len. So you have narrowed it down to the Solaris screwing up since it appears to be generating an extra unneeded FIN before it finishes sending its data. It is time to hunt for a Solaris patch. You need to find out what version of Solaris you are running and what driver you are using and visit their site and see what you can find in patches for Ethernet, ARPA, network, TCP/IP or the driver.

For what it is worth TCP/IP is supposed to close after the data is all sent by
FIN
ack
fin
ACK
where the caps indicate one talker and the small letters indicate the other talker. The last data packet can start the closing sequence by including a FIN but you don't send a FIN until you have sent all of your data.

Ron

[Ron Kinner]

I just noticed that both of your traces are using the same 7001 port number. Is that AFS or? What version? Do you see the same odd behaviour with a different protocol like telnet or ftp? I'm starting to wonder if the problem isn't caused by the application's tendency to hangup so quickly after the last data packet has been sent. True the TCP/IP stack and driver should be able to handle it so I still think there is a patch you are missing but there might be an upgrade or config change to the app which might also get rid of the problem.

Ron

[rick jones]

There are a few heuristics involved in when the P (Push) bit is set by the HP-UX TCP code. Generally, the Push bit is set on the last segment of a user's send().