File truncated in ftp

[John Ho]

Hi
Recently, I was assigned to take over a production system running on HP-UX 10.20 but with a serious ftp issue. I am new to HP-UX but have familiarity with Unix environment.

The problem I encountered is related to file truncation using ftp. We have a scheduled (Cron) script that executes a "mget *" from several remote servers (Novell, NT and Sun) daily. Sometimes, some of the files received were truncated (ie. received 130424 bytes instead of the actual 160812 bytes.) This problem is intermittent and not consistent. As I am new to this platform, I have trouble knowing where to begin to look for clues.

Would appreciate any help available. If you need addition info, I'd be most happy to furnish. Thanks in advance.

[Vincenzo Restuccia]

I have the same problem,verify your network traffic and the rules of firewall.

[John Poff]

Is it possible that the source files are held open by a process on the remote system? Are you seeing the problem with many files or just certain ones?

[John Ho]

Initially I suspected it could be a network congestion issue but ruled out as the remote server that I was having problem with, is connected to the same LAN switch as my server (ie. 100Mbs).

All the remote server does not have firewall so ruled out as well.

I was having problem with a certain files but of not particular type. Maybe out of 20 files, 2 to 3 would be truncated. Basically, all the are reports in ascii format.

[Vincenzo Restuccia]

In ftp script you have 'hash' ?

[John Ho]

I'm not sure. Do not have the script with me now. Will check when daybreak.

By the way, how would the hash affects the "file truncation" ? Can you kindly enlighten ?

[Vincenzo Restuccia]

I think a bad character in name file for hash(#),but you can see the script.
Try set -x in line 1 of script and ./script >/tmp/ctrl.log 2>&1,check ctrl.log for all errors.

Good luck

[A. Clay Stephenson]

Hi John,

Since you indicated that this is on a LAN and not a WAN, I am willing to assume this is not a network issue. I also thought that since you were transferring from so many platforms that it could simply be an acsii/binary transfer issue. I assume you have ruled that out. My best explanation for what is happening is that your cron jobs are sometimes running before some of the files have finished being written. I assume you can do interactive ftp transfers without incident.

I think you may need to modify your scripts to first do a 'dir' command and examine the timestamps and only get those files which have not been modified in perhaps the last 5 minutes.

Hope this helps, Clay

[John Poff]

The "hash" command in your ftp script will print a "#" symbol for each 1024 bytes transferred.

Also, you could put in a "dir" command in your script to get a directory listing of the files before you transfer them. This would give you some idea of the sizes of the files before you transfer them. It won't solve your truncation problem but it may give you some more information about what is happening.

[Sachin Patel]

Hi John
I am totally agree with Mr Clay. I had same problem couple month ago and solution was to use dir command and ftp file one by one in loop.

Sachin

[John Ho]

Hi Folks
Thanks for your suggestion. We'll try it out and hope it works!

Hi Sachin
It good to know that the Mr. Clay suggest works. Prior to this, were you using "mget *" command and hence experience file truncation ?

[Ovidiu D. Raita]

One reason for the truncated files could be a timeout somewhere along the line.

One point in time I had a script written in EXPECT for ftp transfers. The files that took longer than the timeout were truncated ( during the transfer ftp doesn't display anything )so I had to use hash and get arround the timeout. I bet this is the main resons has was implemented on ftp.


Regards,
Ovidiu

[John Ho]

Hi Ovidiu
I doubt that my problem is due to timeout as the servers are on LAN.

I thought that the hash command will print out the # for every 1KB trasferred. So how this help to get arround the timeout problem ? Can you enlighten.


Regards,
Ovidiu

[Ovidiu D. Raita]

Hash command can help you in call the conditions below apply:
- you're using a script for ftp
- the script times out if there's no input for a predefined period of time
- the file transfer takes longer than the time out period.

If hash is ON: after every 1K a # would be displayed, the script would have something to read on the input and would not time out (unless is unable to tranfer 1K/timeout period).

Regards,
Ovidiu