FTP Problem

[Phillip Renner]

Hi all, we're having a ftp problem between a few of our boxes and I'm hoping someone can offer a clue. We have 150 HP servers and are only having this problem on 6 boxes. When we do a FTP tranfer from box1 to box2 its takes forever. However, transfers from box2 to box1 work fine and transfers from other boxes also work ok. We try to ftp a 1M file and it keeps stopping for long periods during the transfer process. If we wait long enough the transfer will complete but this is causing performance problems. The network people have been helping and they say that they see a multitude of ack's going from box2 to box1 but no additional packets are being sent. All 3 boxes that we are having problems with are on the 10.3.194 leg of our network. The 3 boxes having problems transfering are in either the 10.3.200 or 10.3.160 legs of the network.
The HP wizards have been looking at this and they say it looks like a network issue but the network people haven't found anything. All the nic cards are running 100FD auto off and the network people say all the switches are configures for 100FD auto off. I checked the mtu packet sizes and they're all set at 1500. There are also a few NFS mounts between the boxes but we aren't working with them for this process. Anyone have any ideas???

[Pete Randall]

Phillip,

It sure sounds like a duplex mismatch to me. I know they claim everything is set to 100FD, but can you check this - or have them show you the results of a "lanadmin -x" command? You would also need demonstrable proof that the switch port is set to 100FD noauto, as well.


Pete

[Massimo Bianchi]

Hi,
which O.S. and release are that server ?

At first, i think of a name resolution problem, is there something different between those server and the others ?

- /etc/resolc.conf
- /etc/nsswitch.conf
- /etc/hosts
- .nslookuprc in each home directory

Massimo

[Geoff Wild]

First thought was the Full duplex check - but you say that it is ok.

How about, are all the boxes running the same version of ftp?

# what /usr/bin/ftp

Are all boxes patched the same?

What about routing? are you using static routes?

What does traceroute show between box1 to 2 and vice versa?

Rgds...Geoff

[Phillip Renner]

To answer the questions....

Lanadmin -x shows the lans running at 100FD. I think it would take an act of congress to get the network people to give me verifyable proof that the switch is correctly setup. Although in their defense they did move it to another port on the switch.

There is nothing different in the resolv.conf, nsswitch and hosts files from teh other boxes we have.

It appears that the ftp versions are the same. I should also mention that we have the same problem when using rcp.

The boxes are all patched at the same level and we patch them every 6 months with the patches HP recommends. They are running either 11.0 or 11i, one set of boxes we're having the problem with is 11.0 to 11.0 and the other 2 are 11.0 to 11i

And lastly, there is nothing unusual on the traceroute.

Phil

[Massimo Bianchi]

Are they connected to a hub or to switches ?

I'm thinking that on the same vlan or hub there may be lot of traffic, causing the hang.

from lanadmin, is there any error ?

Massimo

[RAC_1]

Ensure you auto_off on both sides. On few switches if it has been set to auto_nego on and same on server side it causes problems.

Ensure you have it off on both ends.

Also in netstat -ptcp, do you see large out of orders packets and retransmitt packets?

[Shannon Petry]

When you have these types of problems, 99.99% of the time it's network related and not OS related

As a few peoplt mentioned, make sure that you change the duplex to the appropriate speed in /etc/rc.config.d/hp100baseconf, and reboot.

This forces the duplex and speed.

Next, are these machines on the same subnet with the same subnet mask? Do they have the same default route?

Now for the other .01% that is OS related.

Are there name resolution issues? I.E. make sure you can both forward and reverse look up.

If this is a NIS environment, and the systems are not on the same subnet you will have problems with broadcast requests for resolution.

Regards,
Shannon

[someone_4]

Hi

Can you test the cable? If you do not have a cable tester. Maybe replace it ..


Richard

[Phillip Renner]

The HP wizards, working with our network people have found that there are packets leaving box1 that aren't arriving at box2. More info to follow as I get it..

[rick jones]

If you want to "check-up" on your network people, you can take a look at the lanadmin statistics for the NIC(s) in question:

$ lanadmin -g mibstats

if your NIC is in FD, then a duplex mismatch _may_ be indicated if you see FCS errors in the statistics. If your NIC is in HD, then a duplex mismatch _may_ be indicated if you see late collisions.

If packets are going into the weeds, then netstat -p tcp should show retransmissions on the side doing the sending, and out-of-order receives on the receiving side. The text at ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_netstat.txt may be of some help. Looking at netstat output can be assisted by "beforeafter" from ftp://ftp.cup.hp.com/dist/networking/tools/

That there are lots of ACKs from box2 to box1 when box2 is the reciever and the performance is bad would be consistent with lost traffic and out-of-order segments being received. Whenever a TCP receives a segment out of order, it is required to send an immediate ACK telling the sender which segment the receiver was expecting to receive.

WHile it may only treat the symptoms, you might also try enabling SACK, Selective ACKnowledgement. That can sometimes make things more efficient in the case of segment loss. However, the "proper" thing to do is as I think you are already trying - find the source of the lost packets. If you want to enable SACK, you would use ndd to set tcp_sack_enable to a value of 1 rather than the default of 2. ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_ndd.h