Data transfer to remote host over pipe(|) fails

[biswal]

Hi,

   In a ksh script , following commands are executed ( The intension is to copy ZYX directory to remote host).

 ---------------

cd /tmp/XYZ

tar cpf  -  ZYX 2> /dev/null | compress -q | remsh <remote host> "cd /tmp/XYZ; uncompress -q | tar xpf - 2> /dev/null"

----------------

Note :

   - ZYX is a directory under /tmp/XYZ on the local machine and that contains 2 data files of total size 1312 Kb.

   - Operating System is 11.31 HPUX Itanium.

After the execution,

            - sometimes the size of the 2 files  differs on local host and remote host

            - sometimes the size of the 2 files are same but the cksum are different.

Could you please let me know how to solve the issue to have the complete data transfer over the pipe(|) without any data loss.

Thanks,

Biswal

[Turgay Cavdar]

Hi,
Are you sure that source files aren't changing?

[Steven E. Protter]

Shalom,

I've seen this wkith remsh and other tools in this library.

If you try the same thing with ssh, the transfer will be slower, but may be more reliable. You do need to make sure the source files are not changing.

SEP

[Dennis Handly]

Does it fail without the compress/uncompress?

What compress(1) has the -q option?

>sometimes the size of the 2 files  differs on local host and remote host

If these are text files, what does diff(1) show?  If not, what does "cmp -l" show?

[Hakki Aydin Ucar]

I d like to add something ,but first summarize things should be sorted out:

1. Make sure source file not changing as Turgay stated ,especially  in case of dynamic source fie data during transfer to destination.
2. Need clue; what kind of error you as Dennis asked before; issue the diff command output if it is text file, otherwise use cmp
3. Remsh; you probably need full-path instead of relative-path due to its nature from restricted shell area.

[biswal]

-q option is for "quite" mode. i.e. no message will be printed on console

I have tried to rerun the test several times, I see only difference in cksum, but not in size. But earlier I had seen the difference in size.

I use the transferred file as an input to some other tool and that fails. If I manually copy the file by using rcp/scp then tool which takes the transferred file as input does not fail and the cksum of the files are same on both local and remote host.

The files under the directory are data files. I executed "cmp -l" on both the files ( local and remote) , I can only see 0 in the third column which means files are identical. But the cksum are differnt.

Please note that complete path of remsh has been used.

[Hakki Aydin Ucar]

>But the cksum are differnt.

how do you use chksum ?

both environment in HP-UX OR they are different OS ? they are both files OR source is on DVD ??

issue the command and output you saw difference here,

there is no detailed explanation to solve issue..

[Laurent Menase]

Also is it possible that some of your files are sparse files?

( du -k file , and ls -l file  doesn't give the same size)

I often do the same with gzip, but never saw any type of error.

What are the size of those files?

if you do the same locally from /tmp/XYZ to /tmp/XYZ2 do you have the same problem?