SCP weird behavior

[rmueller58]

I have a process that unloads a file from our Informix database, then does a scp from the HP/Informix server to a middle man/Linux box.

When I view the files on the HP box the formatting looks perfect, I run the SCP command I lose a field of data in a straight copy.. I thought maybe it was DOS2UNIX issue but it doesn't appear to be.. It is in consistent. Perhaps an SQL issue? But the flat file appears to be in perfect order..

Any ideas of things to look at would be appreciated..

The file is generating correctly on HP,
ReferenceCode, FirstName, LastName, Gender, Group, HomePhone, EmailAddress, Institution
4040,JOHN F,JONES,M,ADMINS & SUPTS,555-555-1212,jjones@domain.tld,352,
4370,LEAH C,SMITH,F,ADMINS & SUPTS,555-555-1234,LSMITH@DOMAIN.TLD,57,

After transfer to Local Server or workstation.

ReferenceCode, FirstName, LastName, Gender, Group, HomePhone, EmailAddress, Institution
,352,JOHN F,JONES,M,ADMINS & SUPTS,555-555-1212,jjones@domain.tld
4370,LEAH C,SMITH,F,ADMINS & SUPTS,555-555-1234,LSMITH@DOMAIN.TLD,57,

[James R. Ferguson]

Hi:

The Forum will mangle whitespace; please post an _attachment_ (file) with the text to compare.

Regards!

...JRF...

[rmueller58]

Jim, See attached..

[rmueller58]

I tried to tar the files on the hp and transfer I get the same results. the odd part of this the file sizes are identical.

I may have to have our DBA look at the SQL unload files.

[OldSchool]

so, in the first line, and only the first line ( which is odd in itself) :

the first data field is deleted, and is replaced by the contents of the "next to last field" (,352)????

where is the "bad" example taken from Linux, Windows or ???

what steps were taken, or was this just a straight-forward scp of the dumped data?

[rmueller58]

Old School

On SYSTEM A (HP) I am doing an UNLOAD from an Informix table..

On SYSTEM B I initiate an SCP in BATCH mode of the unloaded FILES..

the 1st section is correct, when I initiate a simple
scp -B username@systemA:/unloadfiles.csv .

the file on SYSTEMB has lost the "4040" for record of JOHN JONES.. this goes on through out the file.. Some have it and others don't..

I've been scratching my head all afternoon, the files appear letter perfect before the SCP..

I've seen this kind of stuff with FTP and not toggling ASCII/BIN.. but not with SSH/SCP/SFTP..

[rmueller58]

BAD example from a Linux Box.

[rmueller58]

Old School and James,

I am attaching command sequence.

Step #1 produces a file.. File copies Correctly from HP to Linux

Step #2 produces 3 files, appear correct on HP, copies to Linux loses data


Step #3 is the actual Linux SCP command..

[Matti_Kurkela · Accepted solution]

What command/tool are you using to view the files (in each environment)?

Perhaps there are some invisible control characters within the data. In that case, it might be that the HP-UX viewer ignores the control characters, while the viewer on the other system obeys them.

In this case, a copy/paste from the terminal window won't allow us to see what's wrong, because at that point the damage is already done.

Use "cat -v filename" to display the file with visible symbols in place of invisible characters. Do that on both HP-UX server and the other box. Compare the results.

My first guess would be that there might be something at the end of the "jjones@domain.tld" field that causes the terminal cursor to return to the beginning of the line. Then the field separators and the Institution number overwrite the beginning of the first line.

If you still suspect the file gets corrupted in transfer, run "cksum filename" both before and after the transfer. If the checksums are the same, there is high probability that the content of the file has not changed.

MK

[rmueller58]

matt I am just using either more, cat.

I think I found an issue.. it is putting a ^M after the email and before the "Institution" Not sure why.

Thanks for pointing out more might not show me the character issue.

[rmueller58]

I am going to run the Output files through dos2ux before initiating the SCP, that seemed to do the trick.. I don't know why it worked for my 1st single script.. That is what had me stumped.

[rmueller58]

Thanks, Matt for pointing out the ^M not showing in a CAT or MORE, only in VI or editor.