scp issue

[Patrick Ware_1]

Hello all,

I have a user that is trying to scp a file that is over 4GB, and it is failing. Here is what happens below:

file 0% 0 0.0KB/s --:-- ETA

lost connection

They say that files under 4GB are no problem. is there a 4GB limit for scp?

[Robert-Jan Goossens]

Hi Patrick,

they say files under 4GB are no problem, did they check files above 2GB? Are largefiles enabled on the filesystem?

Robert-Jan

[Ralph Grothe]

Not sure about implied file size limitations of scp.
Have you tried dd, split or cat yet?

What about something like this?

$ ssh remlogin@remhost cat \> /mount/with/plenty/space/hugefile < /local/path/to/hugefile.src

You could similarly try with dd or pre split and cat the file on remote host,
or even copy separate splitted chunks via scp to remote host.
If your hosts have sufficient free cpu you could also try the -C option to compress prior to transferring over the wire.

[Patrick Ware_1]

Largefiles should be enabled. I built the filesystems on the server, and made sure to envoke the "-o largefiles" option for every filesystem built.

[Ralph Grothe]

Sorry, the cat might fail either.
But I think dd on sending and receiving end should work.
e.g.

$ dd if=/local/copy/of/file | ssh rlogin@rhost dd of=/remote/copy/of/file

You might want to increase the block size?

[Patrick Ware_1]

The receiving filesystem is 400GB, with only 1% used. The sending side has 529GB, with 524 remaining.

[Steven Schweda]

Knowing nothing, including what these two
systems are, their OS versions, their SCP/SSH
versions, or how they're connected, I'd
probably try something radical, like using
scp to copy files of various sizes from one
system to itself, to see if a size-related
problem is observed.

Then, if that works, repeat the procedure on
the other system.

Also of potential interest: How long did it
take for the (undisclosed) scp command to
fail?

scp has a "-v" (or similar) option, doesn't
it? ("scp -h".) Does it add any useful
info? (It couldn't subtract much, because
you haven't supplied much to begin with.)

[Patrick Ware_1]

Both are rx8640, IA64 running HP-UX version 11.23

and

OpenSSH_4.3p2-hpn, OpenSSL 0.9.7i 14 Oct 2005
HP-UX Secure Shell-A.04.30.007, HP-UX Secure Shell version

[Steven Schweda]

> Both are [...]

Well, I suppose that that's some info.

I don't see a problem fetching a large file
to an older system:

dy # uname -a
HP-UX dy B.11.11 U 9000/785 2012616114 unlimited-user license

dy # ssh -V
OpenSSH_4.3p2-hpn, OpenSSL 0.9.7i 14 Oct 2005
HP-UX Secure Shell-A.04.30.006, HP-UX Secure Shell version

from a newer one:

dyi # uname -a
HP-UX dyi B.11.31 U ia64 4235313755 unlimited-user license

dyi # ssh -V
OpenSSH_4.4p1-hpn12v11, OpenSSL 0.9.7l 28 Sep 2006
HP-UX Secure Shell-A.04.40.005, HP-UX Secure Shell version


dy # scp dyi:/usr/test/VMS732X8.IMG .
VMS732X8.IMG 100% 4959MB 6.7MB/s 12:18

dy # ls -l
total 10156240
-rw-r--r-- 1 root sys 5199986688 Mar 12 14:40 VMS732X8.IMG


Now, the other way it's a little different:

dyi # scp dy:/usr/test2/VMS732X8.IMG .
VMS732X8.IMG 100% 863MB 7.9MB/s 01:49
protocol error: expected control record

dyi # ls -l
total 1767616
-rw-r--r-- 1 root sys 905019392 Mar 12 03:12 VMS732X8.IMG

Note that 905019392 looks suspiciously like
the actual file size less 4GB, which makes it
look as if the older server is
misinterpreting/misreporting the file size.

With a file in the 2GB-4GB range, it's all
similar when it works:

dy # scp dyi:/usr/test/VMS732X4.IMG .
VMS732X4.IMG 100% 2480MB 5.8MB/s 07:05

dy # ls -l
total 5110512
-rw-r--r-- 1 root sys 2599993344 Mar 12 15:31 VMS732X4.IMG

But the failure is different here, where a
32-bit signed size value would go negative:

dyi # scp dy:/usr/test/VMS732X4.IMG .
protocol error: size not delimited


> [...] is there a 4GB limit for scp?

Apparently for some values of "scp", yes.


So, how did _your_ tests go?