Fun with tar "directory checksum error"

[Jeff_Traigle]

I have the following line in a script implemented to transfer data between systems.

remsh ${OLDIP} "tar cf - ${XFRITEM} | compress" | uncompress | tar xf -

It's in a loop so ${XFRITEM} is defined with an existing file each time and ${OLDIP} is defined prior to the loop. The .rhosts file was created through a previous rexec invocation in the same script so I know all of the communications and OLDIP variables were setup correctly by the person running it.

We've been using the script for a few months now with systems on the same LAN with no difficulty. This past weekend, it was tried for the third or fourth time transferring files over a WAN. The first runs had no difficulties even when one ran for 24 hours due to a misconfigured router. However, this one died four times in a row with a "directory checksum error". They were able to complete the transfer manually by taring, compressing, and ftping as their old process had them doing.

My first thought of possible causes was corrupt data as it was transferred over the WAN, but I thought TCP was designed not to drop or corrupt packets. It died after varying lengths of time from 2.5 to 6 hours), but I don't know if it reached the same spot in the tranfer or not. (The data was less than 1GB total.)

I'm writing some functionality into the script so it can resume where a transfer left off so failures like this don't waste so much time, but I wondered if anyone had any ideas what might be likely causes in this circumstance?

[Steven E. Protter · Accepted solution]

I have run into several situations where remesh/rcp type results on my older systems had similar anomalies.

1) Tranferring a 12.6 GB filesystem and ending up with 12.4 GB, and binaries not functioning.
2) Transfers of of smaller files sometimes being incomplete.

The networking environment was perfect cat-5 cisco switch, clean as fallen snow.

I got rid of these results by shifting that kind of work over to ssh/scp.

I'm hinting that this protocol is old and vulnerable to these kinds of problems. Of course if it really was so bad nobody would use it any more.

I would check the networking environment from end to end. The NIC, the cable, the switch, the configuration of the switch port, everything. Make sure its consistent and isn't causing you any problems.

SEP

[Jeff_Traigle]

Hmmm... great... not really any good way to do that given that these transfers occurring over a WAN are from a system in a customer's store anywhere in the country to a system in our data center.

I would rather have done this with tools other than remsh and tar, but a requirement was that it had to work no matter what the old system was running... HP-UX 10.01, 10.20, or 11.00. (This one happened to be a 10.01 system.) That limits my utility availability for the task quite a bit. I did all of the piping to avoid coding around available disk space problems, which were troublesome in their manual processes.

Not being able to guarantee network quality or being able to use more reliable tools (unless I'm not thinking of some neat trick with a standard utility that was available and compatible across these revisions of the OS), I guess having the resume capability is going to be the only work around for them should problems like this happen again... though your experiences of incomplete files and nonfunctioning executables is a little worrisome.