TCP slow from 5.1b machine to XP over long link

[Denver York]

When trying to send a large file from a 5.1b machine to an XP machine over a long(ish) link (15 miles), I see slow transfer speeds and strange patterns in network traffic. This problem does not exist when both machines are local or when going from the XP machine to the 5.1b machine.

While watching network traffic on the XP machine, I observe bursts of traffic followed by silence. The bursts last between 1 and 15 seconds. The length of the silence will be between 29 and 15 seconds ... making the time from start of the first burst to the start of the second burst always 30 seconds.

Here are my parameters from the 5.1b machine:

root@ekpcras1# sysconfig -q inet
inet:
icmp_redirecttimeout = 0
icmp_rejectcodemask = 0
inifaddr_hsize = 32
ip_max_frag_index = 64
ipdefttl = 64
ipdirected_broadcast = 0
ipforwarding = 0
ipfragttl = 60
ipgateway = 0
ipport_userreserved = 65535
ipport_userreserved_min = 1024
ipqmaxlen = 65535
ipqs = 16
ipsendredirects = 1
ipsrcroute = 1
pmtu_decrease_intvl = 1200
pmtu_enabled = 1
pmtu_increase_intvl = 240
pmtu_rt_check_intvl = 20
subnetsarelocal = 1
tcbhashnum = 1
tcbhashsize = 1024
tcbquicklisten = 1
tcp_compat_42 = 1
tcp_cwnd_segments = 8
tcp_dont_winscale = 0
tcp_keepalive_default = 1
tcp_keepcnt = 8
tcp_keepidle = 30
tcp_keepinit = 150
tcp_keepintvl = 30
tcp_msl = 60
tcp_mssdflt = 536
tcpnodelack = 0
tcp_recvspace = 262144
tcp_rexmit_interval_min = 2
tcp_rexmtmax = 128
tcprexmtthresh = 3
tcp_rttdflt = 3
tcp_sendspace = 65536
tcp_ttl = 128
tcptwreorder = 0
tcp_urgent_42 = 1
udpcksum = 1
udp_recvspace = 262144
udp_sendspace = 262144
udp_ttl = 128

Thanks for any help,
D.

[Ivan Ferreira]

Slow connections in one direction only usually is a problem with the network adapter/switch autonegotiation.

You could try manually setting the network adapter speed with lan_config. You can also disable the autonegotiation at the switch.

Check the output of netstat -ni, if you have errors and collitions, this is the most likely cause.

[Denver York]

Thanks for the input.

I've manually set the speed and duplex already.

The problems actually exist either direction across the link, but depend upon which OS is used as the sender (Unix-XP = bad, Unix-Unix=bad, XP-Unix=good, XP-XP=good).

Ideas?

D.

[Ivan Ferreira]

You could try incresing:

tcbhashsize

You can double the values. The maximum size is 16384.

Also, you could disable PMTU.

pmtu_enabled

Do you have errors in the netstat -ni output?

[Michael Schulte zur Sur]

Hi,

what kind of nic and network equipment do you have?

greetings,

Michael

[Al Licause · Accepted solution]

Try running tcpdump on the Tru64 side and/or Ethereal on the PC side. In particular look for the window sizes.

What you might find is that one or the other, usually the PC, is shutting down the window size indicating that it's buffers are full or it can't take any additional input. In particular, look for the window size to shrink to zero.

If this is not the case, try to identify which side has sent the last packet and see who is waiting on who.

Also look for dropped packets, often indicated as duplicates or duplicate acks and/or the same packet sent multiple times.

For this situation, packet tracing is often the best method for troubleshooting.

[Denver York]

Thanks guys!

Ivan, I've increased tcbhashsize with no help. I've also tried disabling pmtu with no help either. Thanks though.

Michael, I don't know the details of the NICs or other network equipment involved, but I'm pretty sure it's not hardware related because of:

1. It's only a problem for Unix-MS and not the other way around.
2. It's not a problem when the machines are both local.

Could I be wrong? Absolutely, but I'm looking at the software side of things first.

Al, I suppose I'm gonna have to pull out some of these tools. Never used them before as this is not my primary function (Generation Control Engineer for an electric power utility), but I gotta do what I gotta do.

I'll post with what I find.

Denver

[Michael Schulte zur Sur]

Can you post

netstat -Iee0 -s
?
ee0 is the name of your nic.
ifconfig -a shows all.

Michael

[Hein van den Heuvel]

- For Gigabit you need autonegotian on right?

- The software knob I woudl try first is: nodelack=1

- I've seen this behaviour mostly with switch/cable/nic incompatibilities but software can help to avoid/trigger the problem.

fwiw,
Hein.

[Denver York]

Guys, I forgot to mention that I see no errors or collisions in netstat -ni.

Michael:

nr0 Ethernet counters at Wed Feb 1 09:49:06 2006

17854564 seconds since last zeroed
3692816193673 bytes received
2987168843752 bytes sent
4624875346 data blocks received
4214565833 data blocks sent
2128188743 multicast bytes received
31032427 multicast blocks received
2985954 multicast bytes sent
46731 multicast blocks sent
0 blocks sent, initially deferred
0 blocks sent, single collision
0 blocks sent, multiple collisions
0 send failures
0 receive failures
# netstat -Itu0 -s

tu0 Ethernet counters at Wed Feb 1 09:49:16 2006

17854574 seconds since last zeroed
3692816695497 bytes received
2987168884141 bytes sent
4624876004 data blocks received
4214566284 data blocks sent
2128189511 multicast bytes received
31032439 multicast blocks received
2985954 multicast bytes sent
46731 multicast blocks sent
0 blocks sent, initially deferred
0 blocks sent, single collision
0 blocks sent, multiple collisions
0 send failures
0 receive failures
# netstat -Itu1 -s

tu1 Ethernet counters at Wed Feb 1 09:49:19 2006

17854577 seconds since last zeroed
2202036771 bytes received
76882857 bytes sent
32134714 data blocks received
1149301 data blocks sent
2128259185 multicast bytes received
31033556 multicast blocks received
3103998 multicast bytes sent
48124 multicast blocks sent
0 blocks sent, initially deferred
0 blocks sent, single collision
0 blocks sent, multiple collisions
0 send failures
0 receive failures


Al, I've tried to run tcpdump but get:

root@ekpcras1# tcpdump 'host 192.168.3.205 and (port ftp or ftp-data)'
tcpdump: pfopen: nr0: No such file or directory
tcpdump: your system may not be properly configured; see "man packetfilter"
root@ekpcras1# sysconfig -q net
net:
arpkillc = 1200
arpkilli = 180
arprefresh = 120
arpunicast = 1
ether_cl_scheduler = 0
ifqmaxlen = 65535
lo_devs = 1
lo_def_ip_mtu = 4096
netisrfair = 1
netisrpriority = 33
netisrthreads = 1
netisrwakeupthreshold = 1
npacketfilter = 256
nslip = 1
route_rate_limit = 0
screen_cachedepth = 8
screen_cachewidth = 8
screen_maxpend = 32

Does ether_cl_scheduler=0 imply that my kernel isn't configured to use this tool?

I'll work on installing ethereal on the PC.

Hein, I've got everything on 100 MB. I'll try the nodelack though.

Thanks guys,
Denver

[Denver York]

Al,

Using ethereal, I see that when the traffic hangs, the PC has received a re-transmission then acked it. After that ... nothing from the unix machine until 30 seconds have elapsed from the prior start-up of network traffic.

Thanks,
Denver

[Denver York]

Oh, and the window size is 64240 most of the time..

Denver

[Michael Schulte zur Sur]

I remember some issue with spanning tree on the switch port. Have you disabled it?

Michael

[Ivan Ferreira]

Have you set the speed and duplex settings to the adapters before forming the nr device?

For example:

ifconfig nr1 remove ee4
ifconfig ee4 down
/usr/sbin/lan_config -i ee4 -a 0 -s 100 -x 1
ifconfig nr1 add ee4
ifconfig nr1 switch

ifconfig nr1 remove ee0
ifconfig ee0 down
/usr/sbin/lan_config -i ee0 -a 0 -s 100 -x 1
ifconfig nr1 add ee0
ifconfig nr1 switch

[Denver York]

Ivan,

I'm a little lost on what you're doing there. A brief explanation would be mucho helpful.

Thanks,
Denver

[Denver York]

cd /dev
MAKEDEV pfilt

Got it!

Now, what am I looking for. This looks like it only shows incoming packets (tcpdump -vv host 192.168.3.34). How can I see the ftp packets leaving?

Denver

[Denver York]

Michael,

spanningtree should be set up properly for our configuration. You're saying to disable it? What if I need it?

Denver

[Michael Schulte zur Sur]

Hi,

this 30 second wait has been discussed in this thread.
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=377726&admit=-682735245+1138875068814+28353475

greetings,

Michael

[Al Licause]

Denver.....in order to make certain you can run tcpdump, you need to check that the line
"options PACKETFILTER" is somewhere in your kernel configuration file (/sys/conf/{HOSTNAME}
and that the running kernel was built from this configuration file.

Then you must have the pfilt* psuedo devices in /dev/pf....it appears you've already discovered and corrected this.

Finally you need to make certain that both permiscuous and copyall modes are enabled on the interface that you will be tracing. pfconfig will show this.
The command pfconfig +c +p {interface_name} will enable them.

Finally run tcpdump: tcpdump -i {interface_name} -w {file_name} -n
You can further filter this by adding port and/or hostnames either at record time or during playback. See man tcpdump for further details on filters. The above command will write the output to {file_name}. To play it back simply run tcpdump -r {file_name}. If you give no file name, the output will go to the screen.

You mentioned a 30 second delay after the PC ack's a duplicate packet or retransmission
from Ethereal. You can playback the output of tcpdump with Ethereal which is a better way to look at it....but it sounds like you need to look more closely at how often duplicates are sent and how often the tru64 system tries to send the same packet.

It sounds as if the PC is not catching all of the data and is then going back and asking for previous data that might or might not still be in the transmission buffers. If it is not, it will need to reset the current data pointer and retrieve the old data that has already been transmitted. This is classic 30 second delay behaviour when one system overwhelms the other with large fast data transmissions or data is lost in transit.

As you said, it only happens from Tru64 to the PC and only over a routed network. So I would look closely to see if the packets are being transmitted from the unix side and never received at the PC. This will require running both Ethereal and tcpdump at either end at the same time during transmission.

As to the speed/duplex mode issues.....in a NetRAIN set, the individual devices should never be set but should be allowed to take on the characteristics of the NR psuedo device. By default, the gigabit devices will autonegotiate with the neighboring switch to determine the highest speed a duplex mode possible. In a clean network, this usually means 1000Mbps fdx.

The only time you'd want to use lan_config to specifically set these parameters is if autonegotiation fails due to an incompatibility between the nic and the switch. It should also be noted that some GigE drivers ignore any attempt to use anything other than fdx when running at 1000Mbps. If you are running at fdx, you won't see any collisions. Collisions will only occur at hdx operation.

Use the following command on Tru64unix (assuming V5.something) to view the current settings:
hwmgr get attr -cat network -a name -a media_speed -a full_duplex.

If the results are 100Mbps hdx, this usually means a failure during autonegotiation.

If you've made it this far, and by now I'm sick of listening to me....and still have questions, please continue to post them.

Al

[Denver York]

Thanks a lot guys. Al for the education and Michael for the link to something that looks similar to my problem. Yes, I did search before I posted, but didn't know what I was looking for :)

I've got ethereal running on the PC and also on a separate PC monitoring the switch port where the Unix machine is plugged into. What I see is this:

Lots of ftp data from Unix -> MS (lots of lost packets, but I don't think this is my primary problem ... yet).

Then, the PC gets a packet, sends an ack and ... silence.

30 seconds after the start of transmission (not from the time MS machine got the last packet, but from the start of the last blast of data), he gets another FTP data packet as if nothing had happened.

Looking on the Unix machine, I see that he's sending packet and all is well. Then, he just quits. However, network traffic to/from this Unix machine to/from other machines on the network does not stop. Right before he quits sending ftp packets to the MS machine, he sends a TCP packet to another machine with the PUSH and ACK flags set. Then, the other machine and he have a conversation with him always setting the PUSH and ACK flags. Does this mean anything? This this conversation taking all the bandwidth because of the PUSH flag? Then the ftp process is just hanging awaiting the end of this conversation? Anyway, ...

30 seconds after the start of the previous transmission, he starts sending more ftp data packets as if nothing happened. Right before he starts sending more ftp packets, he sends a keep-alive to a DIFFERENT MS machine on the network (I saw this more than once). That different machine ACKS the keep-alive and then he re-starts sending ftp packets.

Looking all around these start/stop sequences, I can find nothing that would indicate he's received a control from a switch or anything that he should stop sending. Plus, he continues sending data on to other machines, so I'm hard-pressed to suspect spanning-tree or other issues at the moment.

Ideas? Things I should be looking for?

Again, thanks a whole bunch. Without your help, I wouldn't be this far.

Denver

[Hein van den Heuvel]

I think you've gone about as far as you can go on this. All the information is gathered, all the homework seemingly done.
I'll poke a friendly HP engineer who 'dabled' in the Tru64 network stack to see if he has any good comments to offer here.

You may also want to start a support call if this topic does not provide a resolution.

Cheers,

Hein.

[Hein van den Heuvel]

Comments by John D.
-------------------------------------
This problem sounds like a common one where the sender is ovewhelming
the receiver, it usually happens when the receiver is a Windows box and
the sender is using larger TCP windows. Here, UNIX has a 64K send window,
you can try reducing the send buffer (tcp_sendspace) to 32K or even
smaller like 16K and see if it improves..? Of course, this will adversely
impact your local network performance, so it may not be a solution, but
will tell you if you are headed in the right direction.

The packets are either getting dropped by the PC because it's out of
buffers or they are dropped by an intermediate router. So if you send
fewer packets or send them slower like a PC does, the situation should
improve. That is probably why the PC can send to the UNIX box without
any problems, it doesn't push the link and trigger any drops.

The other thing you can try is to decrease the packet retransmit
threshold, tcprexmtthresh is set to 3, you can safely bring it down
to 2 (but no lower than that!). That will help if the receiver is
not acking the lost data enough to trigger 3 duplicate ACKs in a row.
-----------------------------------------

hth,
Hein.

[Denver York]

Okay! I've got reasonable throughput now, but ...

I did have to drop tcp_sendsize ALL THE WAY DOWN TO 8192 to get it!!! Local traffic does seem to suffer, but not drastically. Hmmm ... leave it there and move on? How do I find what equipment is causing the problem?

Thanks a bazillion!!!
Denver

[Michael Schulte zur Sur]

How is the machine patched?

Michael

[Denver York]

Michael,

KITNAME> OSF540

Not 100% familiar with patching, so, does this provide you with the answer to your question?

Thanks,
Denver