scp corrupting the files

[KathyK]

Hi experts,

One of our programers was using scp to copy a file from Malaysia to our us office, the file came over showing the same size but was corrupted. here is the details:

both systems are running

HPUX 11.23
rp7420


the file is 2MB, some where in the middle of the file that's recieved around 2000 lines get dropped and replaced with null character.

If anyone knows of any problem with SCP please reply.

Always greatfull


Kathy

[Dennis Handly]

I've had corruption over NFS before. I was able to find the problem and related it to a block size issue.

You can use "xd -tx4 -tc file" to produce a dump. Then diff this with a dump from the original file.

If the data then resyncs, you may have to use an editor to remove the xd offsets from the beginning of each record.

[Armin Kunaschik · Accepted solution]

I never had such a corruption... if scp returned with return code 0.
You did check the return code after copying?

You did not say anything about the SSH version on source and destination.
A first try is to update to a current version on both sides if availabe.

If this does not help, you should consider generating md5 checksums (get the md5sum depot from http://www.software.hp.com/) and check them after transmission.

My 2 cents,
Armin

[KathyK]

Thank you both for good information, i'm going to test the checksum and compare that.

OpenSSL -> A.00.09.07e.0 Secure Network Communications Proto


Regards,
Kathy

[rick jones]

Unless there is no actual encryption happening with the scp, that such a large corruption came through the scp suggests that the sending scp was given bogus data. GIGO - Garbage In Garbage Out.

Still, if it isn't already you might consider disabling CKO (ChecKsum Offload) on the NICs on either side.

And if those 2000 lines were perhaps in some way "subversive" as far fetched as it sounds you might consider looking for a "man in the middle" situation.

[KathyK]

Rick,

thanks for the response. could you please explain how to disable the CKO on the nic card?

Thanks

[rick jones]

If we are talking about GbE cards rather than 10GbE cards, IIRC it is already disabled in 11.23 and 11.31. However, if I am remembering correctly, it would be a lanadmin kind of thing. I also think that if CKO is enabled, it will appear as one of the flags in the output of ifconfig.

Also, unless you can reliably reproduce the corruption, all of this is going to be blind shots in the dark.

[KathyK]

That's the problem, this happends in random. and I have not been able to re-produce the problem yet. but i'll keep working on it.

thanks
Kathy

[Dennis Handly]

>this happens in random. and I have not been able to re-produce the problem yet.

Were you able to look at a hex dump and see if the corruption was on nice block boundaries?

[KathyK]

Dennis,
I looked at it briefly but need to go back to it. I'm installing the md5 package on my servers now.
thanks

[ROCK_10]

Kathy,

Please check your disk space and let me know.

[Armin Kunaschik]

BTW: The OpenSSL package has nothing to do with the ssh package (because ssh is staticaly linked and does not need OpenSSL installed).
To get the ssh version do a "swlist T1471AA"
This will look like:
T1471AA A.05.00.025 HP-UX Secure Shell

Make sure you have a current version installed. Most current is 5.1.

My 2 cents,
Armin

[KathyK]

Rock/Armin,

Thanks for the response. to answer your questiosn:

both systems are hpux 11.23

stecn40:

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 606208 384968 219544 64% /
/dev/vg00/lvol1 298928 55400 213632 21% /stand
/dev/vg00/lvol8 5242880 1654624 3560432 32% /var
/dev/vg00/lvol7 8388608 4463232 3894776 53% /usr
/dev/vg01/lvusers 35872768 3361186 30479655 10% /users
/dev/vg07/lvu6 71671808 1336662 65939200 2% /u6
/dev/vg06/lvu5 71671808 1815439 65490351 3% /u5
/dev/vg05/lvu4 71671808 1732352 65568284 3% /u4
/dev/vg04/lvu3 71671808 1634687 65659805 2% /u3
/dev/vg03/lvu2 71671808 1634686 65659808 2% /u2
/dev/vg02/lvu1 71671808 4017935 63425511 6% /u1
/dev/vg00/lvol4 2097152 998848 1089760 48% /tmp
/dev/vg00/lvol6 4898816 2976872 1906944 61% /opt
/dev/vg00/lvol5 1048576 9152 1031352 1% /home

# T1471AA A.04.30.007 HP-UX Secure Shell
T1471AA.Secure_Shell A.04.30.007 HP-UX Secure Shell



And on simpk400:

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 524288 324368 198400 62% /
/dev/vg00/lvol1 511672 77904 382600 17% /stand
/dev/vg00/lvol8 5242880 1952008 3288808 37% /var
/dev/vg00/lvol7 8388608 7656712 726208 91% /usr
/dev/vg01/lvusers 35557376 21976184 13482688 62% /users
/dev/vg07/lvu6 35557376 18240992 17181248 51% /u6
/dev/vg06/lvu5 35557376 19208736 16220976 54% /u5
/dev/vg05/lvu4 35557376 24431592 11039128 69% /u4
/dev/vg04/lvu3 35557376 18687328 16746520 53% /u3
/dev/vg03/lvu2 35557376 7203456 28132472 20% /u2
/dev/vg02/lvu1 35557376 13414528 21969928 38% /u1
/dev/vg00/lvol4 2097152 633640 1452176 30% /tmp
/dev/vg00/lvol6 4096000 3657824 434800 89% /opt
/dev/vg00/lvol5 1048576 116744 924760 11% /home

# T1471AA A.04.30.007 HP-UX Secure Shell
T1471AA.Secure_Shell A.04.30.007 HP-UX Secure Shell


I have been watching the files being rcped and scped, and so far so good.

[KathyK]

One more question for you gurus, does scp or rcp have internal error checking?

[rick jones]

If you mean data checking, then I'm 99.9999% confident that rcp does not do such a thing. I'm 99% confident that scp relies on the SSL/TSL layer (I may have that wrong) and also does not have data checking itself.

[KathyK]

Thank you Rick, that's what I was asking.

[Armin Kunaschik]

I agree with Rick, I think there is no data crc check within scp and rcp. Both protocols probably rely on the usual normal TCP checksums.
Openssh 4.30 is quite old (2006). I did not hear about any issues with that version, but one never knows.

There is one way to (implicitly) create and check checksums while copying files: The tar-pipe.
Don't use scp to copy the files but tar and ssh:
# cd /sourcedir; tar -cf - . |ssh targethost "cd /targetdir; tar -xf -"

The piped tar archive contains checksums for every file which are validated on the target side. So you'll notice any high level error that occurs "on the wire". You'll don't notice any error if the source host, for some unknown reason, delivers corrupt data to the tar command. But in this case a checksum tool is useless too.

Is it possible that you tried to copy a file that was used by an application at the same time? This will produce unpredictable results and should be avoided!

My 2 cents,
Armin

[KathyK]

Armin,

thank you for you 2 cents! I will let the programmers know about the tar command. and will find out if the following was the case.

>Is it possible that you tried to copy a >file that was used by an application at the >same time? This will produce unpredictable >results and should be avoided!


Happy holidays you all.

Regards,
Kathy