scp -C does not seem to compress

[Gilbert Standen_1]

The man page says that scp -C will compress while scp copy is in progress("in-flight-compress"). But when I scp using the -C switch, the file lands at the destination uncompressed. So what does this -C switch do ? And is there a way to scp and gzip all together as one process so that the file ends at destination gzipped ?

[Patrick Wallek · Accepted solution]

I think what they are saying with this is that SSH will essentially compress the data on the source machine before the copy, and then uncompress on the destination after the copy.

Basically, this is allowing you to copy less data, but it does not actually create a compressed file at the destination.

Have a look at the compression sections of the SSH man page for more info.

If you want to compress, and have the file wind up being gzip'ed at the destination, you could try something like this:

gzip < afile > - | scp - host:/dest/dir/afile.gz

[A. Clay Stephenson]

The -C option is intended for sending large data volumes over limited bandwidth connections. The data are compressed before transmission and expanded upon receipt "on the fly" so that it merely APPEARS that -C is not implemented. In fact, it may not be implemented because both ends have to agree and not all scp's support this feature.

Note that gzip will read stdin and output to stdout so it is possible to setup a pipeline to send these files.

[Gilbert Standen_1]

Ok, good, now related to Mr. Stephenson's point, how would one determine if the 2 systems are in fact making use of the "-C" option ? I did a ps -ef | grep scp and the process information only showed the "scp" it did not indicate a "-C" option. Does that mean it is not using "-C" ? This is a case where the source files "cannot" be zipped or compressed in any way prior to sending (db files, live) but it might be desirable to compress on the end.

[Patrick Wallek]

If you look at my example, you will see that it will NOT modify the source file in any way.

gzip < afile > - | scp - host:/dir/file.gz

What the above does is:

Takes the file afile and redirects it as input into gzip, then redirects the output (the gzipped file) to stdout, which is then piped to scp as the source file and copied to the destination.

The '-' characters are valid operators. Give it a try on a test file and I think you'll see that it works pretty well.

[Steven E. Protter]

Shalom,

swlist -l | gerp -i secure

What version of Secure Shell are you using.

There have been some serious problems with recent releases ported by HP.

Its better in this case to compress the file as Patrick recommends.

Its also good to make sure you have a recent version of Secure Shell installed.

http://software.hp.com

Current version is broken but a new version is promised by month end.

SEP

[spex]

Hi Gilbert,

If you have concerns about file integrity, you can compare the cksum of the original file to that of the copy:

# [[ "$(cksum afile)x" = "$(gunzip -c file.gz | cksum)x" ]] && echo "good" || echo "data corruption"

PCS

[Gilbert Standen_1]

Mr. Walleck's cmd does work great, and it's incorporated! My small revision to it is as follows:

ssh fssctprod 'gzip /home/oracle/refresh/test.dbf > - | scp - fsscttest:/home/oracle/refresh_mmc/test.dbf.gz >/dev/null'

because the script executing this cmd sits on fsscttest.

In reply to Mr. Protter, I don't have access to "swlist" so I will have to refer that to our UNIX admins.

Spex -- the checksum is a very useful incorporation to certify the copies.

If I could take one thing with me into the next world it would be my valid login to this forum. I am so grateful for your knowledgeable help.

Gil