ftp file size limit?

[Tim Medford]

I am trying to ftp a large file (approx 2.5 gb) from one HPUX 11.0 server to another. The transfer gets within 30mb of finishing and keeps dying.

If I try a 'put' command from the source server, it eventually dies with a message about "sendfile: broken pipe - 452 Error writing file: Invalid argument."

If I try a 'get' command from the target server it eventually dies with a message saying "File too large
421 Service not available, remote server has closed connection".

Does anyone know what's going on here? I'm not aware of any inherit limits on file sizes in ftp?

Thanks in advance!
Tim

[Steven Sim Kok Leong]

Hi,

What is your system's ulimit setting? You should increase the ulimit if it is too small.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Darrell Allen · Accepted solution]

Hi Tim,

Since you get to within 30MB of a 2.5GB file I can assume the filesystem on the destination server has largefiles enabled, right?

Are you by chance filling the remote filesystem?

Darrell

[Bill Hassell]

You can safely assume that no file over 2Gb will transfer successfully until you have verified that the destination can accept this file. Anything larger than 2Gb requires special handling by the filesystem and is enabled by either newfs or fsadm plus an appropriate entry in /etc/fstab.

To see if largefiles are enabled on the destination (assuming another HP-UX box), use the command:

fsadm /dev/vgXX/rlvolY

where XX = the volume group and rlvolY is the raw logical volume.

If largefiles are enabled, then check for ulimit values and also use bdf to see if there is enough space.

[S.K. Chan]

FTP support large file systems, so ftping a 2.5GB file is no problem. It seems the remote system is probably not configured for large file system support.

[Sanjay_6]

Hi tim,

Take a look at the threads below,

http://us-support2.external.hp.com/cki/bin/doc.pl/sid=b8f8ab39095fc75108/screen=ckiDisplayDocument?docId=200000015247956

If you are coping files larger than 2GB make sure you have largefiles option enabled for the filesystem.

Hope this helps.

Regds

[Sanjay_6]

Hi Tim,

Here is a thread on how to enable largefiles support for the filesystem.

http://us-support2.external.hp.com/cki/bin/doc.pl/sid=31909677025c2f665f/screen=ckiDisplayDocument?docId=200000047669207

Hope this helps.

Regds

[Deepak Extross]

Have you tried doing a ftp 'reget'? You'll have to run this from the destination machine, and it should continue where the ftp left off - that is, only the last 30Megs would be ftp'd across to complete the file. Not too sure if it'll work in this case, but its worth a shot.

Also, consider splitting the large file into smaller packets say 1Gig or 0.5Gig (see man split) before ftp'ing. At the destination, you can simply 'cat' the files together to recreate your file.
I like doing this because, if the connection breaks midway, I only have to ftp the remaining file fragments across, not restart from scratch.

[Roger Baptiste]

Tim,

FTP has no filesize limit for its transfers.
You would need to make sure the destination filesystem supports LARGEFILES (which is files above 2gb). On the destination system , on the filesystem where the file is being copied do:
fsadm /dev/vg*/lv* |grep largefiles

If largefiles is not set, you can set it by
fsadm -o largefiles /dev/vgXX/lvxx

Regarding ftp, i like to use the hash command to track its progess, especially for large files.

HTH
raj

[Jason Rondeau]

I have also noticed that FTP does not work very well with large files. I think my problem is a little different however. I transfer files to and from HP3000 systems running MPE to HP-UX systems. The HP3000 systems have large files suport and and the HP-UX drives have large file suport enabled.
The problem that I see is not that FTP fails, but the fact that it doesn't transfer all the bytes and doesn't fail. I check the byte size on all files greater than 2GB before and after transfer.

Anybody have any thoughts as to why FTP doesn't transfer all the bytes and then doesn't fail?