files transfer

[Admin.SIF]

I have to automate the transfer of files between two HP 9000 servers. It is necessary to verify continuously if files were put (deposited) on the first server and to move them on the second server. How to proceed: ftp, rcp, ...? Your advices will be appreciated. Thank you in advance.
Nora

[James R. Ferguson]

Hi:

If you are moving Unix-to-Unix I think I'd script the process with 'rcp'. If you really want to verify what you copied is what you think you copied, include remote shell commands to 'chsum' the file on both servers and compare the results.

Regards!

...JRF...

[Sridhar Bhaskarla]

Nora,

You can either do through automated ftp or rcp. If you are particular about security, you can install ssh and use ssh too. You can accomplish your task using either of them. If you use ftp, you can check the files on the remote system using ls or dir command compare them against what you have on the local and copy only the difference. You need to do a little bit of scripting that does the following.

If you use ftp:

1) In the first ftp session,you will get a list of files through ftp's ls command. Then you compare them against the local files and generate a diff of files.
2) in the next ftp session, you will use the above list of files to 'get' from the remote system.

It's easier with /remsh/rcp/ssh/scp also

1) Check the files using remsh/ssh command. And determine the difference.
2) Use rcp/scp to copy the files.

My recommendation would be to use ssh/scp to get the files. If security is not a concern, you can easily do it through rcp/remsh. It's better to use r'commands than ftp as you will need to hardcode the passwords somewhere.

There are other softwares like Connect Direct etc., to accomplish similary kind of tasks.

-Sri

[Santosh Nair_1]

It would be best to use rcp or scp from one server to get the file from the other. scp would probably be a better choice as it has better security. But in either case, you could do something like this.

#make sure no one is using the file.
RC=$(fuser -u $FILENAME 2>/dev/null |wc -l)

if [[ $RC = 0 ]]
then
ssp $FILENAME destsvr:
fi

The fuser command assures that no one is using the file.

Hope this helps.

-Santosh

[Alan Riggs]

You can certainly write scripts to automate this with ssh or ftp, but if you can establish a trust (.rhost) relationship between a user on each system with read priveleges for the files, then use rdist.

Rdist will do exactly what you want.

[Uday_S_Ankolekar]

Hi,

As every one said, I would rather go with rcp or scp ( If security is a concern)
you can use chksum to see the files copied properly or not
for rcp you have to deal with .rhosts file

GoodLuck
-USA..

[A. Clay Stephenson · Accepted solution]

HI:

When you say continously that could get to be a real resource hog; if you back that off to every few minutes then it becomes more manageable. I have a perl script (fileage.pl) I have used for a long time and it becomes the heart of my proposed solution. It returns a 0 if a file has NOT been modified with a certain period of time and non-zero otherwise.

Here is my cut at this but I don't guarantee the syntax is correct as I am doing this 'on the fly':

DIR=/xxxx/yyy
REM_HOST=bugs
INTERVAL=600

cd ${DIR}
ls | while read X
do
FNAME=${DIR}/${X}
fileage.pl -m -s ${INTERVAL} ${FNAME}
STAT=$?
if [ ${STAT} -ne 0 ]
then
rcp -p ${FNAME} ${REMHOST}:${FNAME}
fi
done

You could add more to this to check to see if you need to delete a file or if a file has not been modified but is not found on the remote server then do the transfer. In the above example, this should be set up as a cronjob fired off every 10 minutes (600 seconds).

I would use rcp as it is much easier to script.

Clay

[Darrell Allen]

rcp or scp is easier to automate than ftp. The primary advantage of scp is encryption of the file during transfer. The userid and password (if used) are also encrypted when initiating the connection. If you use scp to encrypt the files during transmission then it will be slower than rcp.

You might want to look into rdist or if you want to send files both ways, rsync. Both are available at HP software porting and archive centers. One is http://hpux.cs.utah.edu/

Darrell

[Irine Gibilisco]

Nora,
If your question is in 'how to detect when the files first arrive on the 1st server' (instead of how to copy them to the second server), then it is important to know how the files arrive on the 1st server, and what OS is sending them. For that, I would recommend the following:
1)Set up a dumb printer on your 1st UNIX server.
2)Then, set up a remote printer on wherever your files originate from, to do remote print to that new dumb printer on your 1st UNIX server. That will get your files over safely to your 1st UNIX box, and your dumb printer driver will know the minute they arrive.
3)Edit your 1st UNIX box's newly set up dumb driver to skip past fist 5 arguments (shift;shift;shift;shift;shift), and you'll have the name of the temporary spool file containing your newly arrived text.
4)Now you may copy or rcp or ftp it to your second server. There is no need to poll for your file's arrival - the lp spooler will do it for you.
Notes: This will only work with flat ASCII text files. Your dumb print driver will be found in /usr/spool/lp/interface/your_printer_name. If your arriving file is of large size, inside your print driver you may need to 'sleep' until you are sure that the whole file arrives, before you can copy it on. Also, with this remote printer setup, you can receive files from any other OS types capable of "lpr/lpd" (like IBM, NT, etc.)
Good luck.
Irine Gibilisco

[Admin.SIF]

Thank you very much for your answers. I am going "to compile" all this and I shall return to you for the result ... and for points which are deserved well ! I worked on another solution, using NFS and export the directory to avoid the transfer of files. Thank you Clay for your scripts, thanks to all naturally, I find that formidable to dedicate from his(her) time to the others.
Best regards,
Nora