bizare problem with FTP and SCP file transfers

[John O'Neill_6]

Hi all,

I have two HP UX machines, RX2660's running HP UX 11.23, patch assessment run in October and machines fully patched up.

Recently however a bizare problem has begun to appear and i'm trying to work out if it's a firewall issue of a UNIX issue.

The two machines are having problems talkign to each ohter, here's what I am able to notice.
Now, each machine is in a different office in different subnets, with a firewall maanged VPN connecting both offices. It's meant to be a trusted VPN that allows all traffic from one site to the other.

SSH works fine between both machines
SCP works fine, as long as the file is fairly small
TCP works fine, as long as the file is fairly small

Any attempts to send larger files (say more than 5k) of either ASCII or BINARY type just failes, all you see is that 'Connection was dropped'.

Doesn't matter which user does it, root.. another user, nature of the file.. whatever.

For some reason, FTP and SCP is limited to small files no more than 5k in size.

Yet there are no problems with SSH at all, you can log into box boxes from SSH clients and each box can log into the other box fine.

There are no errors turning up in syslog.log, it just seems that for anything other than trivial file transfers the connection (in either direction) between the two machines just gets dropped.

It's got me stumped, as an experiment I fired up an old RP2470 machine at our primary site and tried to transfer files to the backup RX2660 at the other site. SAME problem, only trivial files could be sent.

So, do i have a firewall issue or an issue on my HP UX machines (which are configured the same, run same patch level and so forth)?

I've tried searching but have absolutely no idea where to look.

Please help.

Regards,

John

[John O'Neill_6]

When i do an FTP from primary machine to backup machine the ftp allows me to connect, specify a user name and password, but when i send a file of any decent size the connection just hangs, pressing CTRL-C gets this following error message.

421 Service not available, remote server has closed connection

Any ideas?

[John O'Neill_6]

OK,

Problem seems definately related to file size. The limit seems to be about 8Kb (8 kilobytes).

I am unable to send a file with more than 8k in it to the other HP UX system and visa versa. Yet I can send/receive any number of small files without a problem.

Anyone have any ideas? I've checked
'glance -t'
but no limits are anywhere near being reached and I have 20% physical memory available.

netstat isn't showing anything either...

-John

[George_Dodds]

Had similar problems at a couple of remote sites a few weeks ago and it turned out to be a firewall issue at the remote sites.

Will find out what fw changes they had to do to sort it.

[George_Dodds]

The problem was that the firewalls at the remote sites were fragmenting packets.

We couldnt even cat large files using putty, so dont know if it's the same problem as what you are having.
But hopefully it gives you something to look at.

[John O'Neill_6]

Thanks for the lead :) I'll go and start researching this line of investigation.

We're using two watchguard firewalls to manage a VPN linking the two sites to each other.

-John

[George_Dodds]

The fix that was used was to modify the maximum segment size (MSS) on the router (not sure if it was local or remote side)

Sorry i cant provide any more detail.

[John O'Neill_6]

The really strange thing is.. that I can use a windows based FTP/SFTP client to transfer large files across the link to and from the HP UX machines with no problems at all.

The problem seems to involve:
A) Problem with HPUX to HPUX
B) Across the VPN (local HPUX to HPUX is ok)
C) Files greater then 5Kb in size
D) Windows FTP/SFTP package not affected

Why is Widnows based network unnaffected whilst HP UX networking affected?

-John

[Suraj K Sankari]

Hi John,
Did you check your NIC log is there network packets are drops?
Same problem I also faced once, what I did is....
I have set the NIC speed and my switch port speed both are same means no auto-selection set it 10/100MBPS full/half duplex both the side same.
I did the same thing other side of my server then my problem got solved.

Please try I think this could also help you out.

Suraj

[John O'Neill_6]

Hi,

I'm not sure what the NIC log is, i've tried looking. How can I tell about dropped packets?

-John

[George_Dodds]

You could check with lanadmin

lanadmin - lan - ppa - enter ppa number of nic - display.

[John O'Neill_6]

Thanks for the tips, but it now looks like i've go firewall guys saying it's HP UX and HP UX saying it's the firewall.

I might need to raise a HP support Issue and get our firewall guys to look into this as well, i'm not that comfortable with changing low level settings...

I guess i'm just once again dissapointed in that my windows machines have no issues yet HP UX (Once Again) is causing me grief for some weird esoteric reason.

I might see if I can use a Windows machine to transfer the log files I need to move from one machine to the other...

I'll post back when they can tell me what's wrong.

-John

[rick jones]

Take packet traces - on both HP-UX systems, sending and receiving. Compare things and see if the firewall is messing with the traffic in some way. If you can make certain that the two systems have a reasonably close concept of time of day, it may help in the analysis to show if say the firewall is deciding on its own accord to drop the connections.

You might also take application-layer protocol out of the equation by doing some quick netperf TCP_STREAM tests between the two systems to see if those work. (www.netperf.org)

[John O'Neill_6]

Hi,

I've discovered a utility called 'nettl' which seems to be what we need to trace what's going on.

I have activated on the target machine at the other site (over the VPN) and re-created the error by trying to send a file over.

Here's what I did:

A) start 'netttl' as root on target machine
B) start 'netttl' as root on source machine
C) try to transfer file from source to target
D) stop 'nettl' on both machines

I have now got log file:
/var/adm/nettl.LOG000

But I can not examine the contents of this file, will try and locate a utility to do this.

Anything else I need to try?

-John

[rick jones]

I probably would have downloaded and used tcpdump from the HP Internet Express bits (or compiled it from sources on www.tcpdump.org :). However, either etherial or wireshark know how to read an HP-UX nettl trace. At least I think one of them does. Otherwise, you would use the "netfmt" command to "format" the nettl trace.

[John O'Neill_6]

ok, have used the command

# netfmt -Nlf /var/adm/nettl.LOG000

to read the file and all i get is this...

2004> 1000Base-T in path 0/1/2/0
Detected a faulty or disconnected cable.

---------------------Gigabit Ethernet LAN/9000 Networking------------------@#%
Timestamp : Mon Oct 01 EST 2007 10:03:21.940591
Process ID : [ICS] Subsystem : IGELAN
User ID ( UID ) : -1 Log Class : ERROR
Device ID : 0 Path ID : 0
Connection ID : 0 Log Instance : 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Only error seems to relate to a time when system was powered up but no cables connectd to it.

Anything else I can check?

-John

[Bill Hassell]

> nettl traces

The built-in nettl formatter is pretty lame. However, the leading trace and formatting tool is called Wireshark (was Ethereal in a past life) and Wireshark will decode over 400 different trace file formats. Get a copy from www.wireshark.org

[John O'Neill_6]

Hi,

I grabbed wireshark but it can't read 'nettl.LOG000' files :(

-John

[rick jones]

Grab tcpdump from either HP Internet Express or, if you have a "real" compiler installed on one of the systems, the sources for it and libpcap from www.tcpdump.org. If you go the latter route and need compilation help, the "netperf-feedback" email will reach me.

If you want to trace just ftp data connections, to/from the other system, then on one system you would say something like:

tcpdump -i -w /tmp/sideone port ftp-data and host

and then on the second system:

tcpdump -i -w /tmp/sidetwo port ftp-data and host

then the traces can be post-processed with something like

tcpdump -r /tmp/sideone > /tmp/sideone.cooked

etc etc

Best if both systems are already running xntpd and synced to a decent time source. Second best, if not running xntpd, is to run ntpdate on both - HOWEVER, that may cause a non-trivial step in time on either system, so if you are running a DB or something, beware...

[John O'Neill_6]

oooh.. wonder if this could be an issue...I'm seeing some mention of both systems having reasonable time synching.

Source sytem time: Tue Mar 24 12:25:41 EDT 2009

Target syste date: Tue Mar 24 12:38:43 EDT 2009

Target system's clock is slightly ahead of the source system, this may have been due to recent changes in daylight savings and so forth.

I would imagine that this could cause issues with things like network packets and so forth??

-John

[rick jones]

No - TCP and IP packets don't tend to carry wall-clock timestamps in them. The issue would be in terms of ease of correlating the two packet traces to more easily see if it was the firwarell initiating some packets, pretending to be one side or the other of the connection.

[Dennis Handly]

>Source sytem time: Tue Mar 24 12:25:41 EDT 2009
>Target system date: Tue Mar 24 12:38:43 EDT 2009
>this may have been due to recent changes in daylight savings and so forth.

No, DST changes only affect how times are displayed and not the system clock. So this wouldn't cause the 13 minute difference.

[John O'Neill_6]

Hi everyone, still working on this issue...

I've got multiple HP UX machines at both sites and they can't FTP anything more than 5Kb files across the VPN. They can transfer files between themselves just fine if that other machine is on the same side of the VPN.

Oddly also, TELNET and SSH sessions work just fine, but I can't mount NFS filesystems across VPN either.

Ping testing reveals 0% packet loss, despite hours of pinging. I can do hundreds of really small FTP and SCP commands with 100% success.

Could it be the firewall at either site that's doing something with the packets?

-John

[SSCHAER]

john...

strange issue indeed.


is there any chance you could perform one of these ftp manually ?

run it once as active :

ftp> open
ftp> active
ftp> put ...

and once as passive :

ftp> open
ftp> passive
ftp> put ...

try to transfer a larger than 8k file.
active/passive controles how ftp uses ports. a firewall could limit the range/number of ports a connection can use

[John O'Neill_6]

Hi,

Done, done all that.. no dice, any file larger than 5Kb fails with '421' error (as previously posted) sent back.

Even my 'old' UNIX machines which I recently powered up at both sites have exactly the same problem, before our firewall / ISP upgrade this problem didn't occur on these machines at all.

So i'm leaning towards 'something' going on with my network but so far nobody seems to be able to give me any kind of definite answer.

My UNIX machines can send / recieve files as big as you like IF they are on the same side of the VPN, but if they try to send recieve failes OVER the VPN, forget it.

I'm affraid that my depth of network analysis may not be up to further investigation here, with people here we've been able to identify what could be the issue, but not sure how a firewall can cause this problem for HP UX but not for MS Windows, who's FTP/SFTP sessions remain blissfully unnafected....

Very strange indeed.

I've got our firewall vendor/expert comming out tommorrow and he thinks it might be something to do with packet fragmentation over the VPN.

I'll let you know what happens. But i'm out of ideas.

-John