How do you transfer data

[David_246]

Hi,

I am interested in a solution on how to transport data (automized) from server A to B.

There are several known possebility's to transport data using; FTP, scp, Sftp, rcp, NFS, ETL, etc.

Now I am interested in your solutions about data transport between different platforms. Preffered solutions are solutions that can be automized. Any product names inclusive produkt internet-links will be most apreciated. Please read data, as text-files/database-information, etc.

Your help will be apreciated and rewarded :)

Regs David

[Leif Halvarsson_2]

Hi,

If both servers is Unix, NFS is the common solution. If Windows/Unix Samba (CIFS) is an alternative.
Without knowing your problem more in detail it is hard to give a better answere.

[David_246]

Hi Leif,

Thanks for your quick response.
I am more looking for an overall solution over multiple platforms, like OS/390 and VMS and Windows and ...

So how do your transport your data from server A to B in cases platforms types differ a lot. But you always have a Unix server as server A or server B.

What other kind of products do you know, like NFS, CIFS, scp, sftp, ftp, ETL, rcp, etc.

Thanks for your feedback !

Regs David

[Stanimir]

1."Samba" is another solution.

2.You can use self-created client-server application, using sockets, to
transfer data or files between 2 systems
either with equal or not OS.
I'm sending you 2 perl-scripts, as
templates for this solution:

1. "server"-script,worked as daemon,
received data.
2. "client"-script for sendind data.


Regards,Stan

[Stanimir · Accepted solution]

Hi again!
In addition to mentioned above:

Here is the attached winzip-archive
with these 2 scripts.

[Mark Grant]

NFS is the norm for doing this and NFS is supported on all platforms that I know off. On the PC side, you need to purchase an NFS client but unix boxes should already have them. The only problem is that one machine is the server and is therefore a single point of failur for all of them and becuase NFS is connectionless, if the server goes away the clients often hang if an attempt is made to read the NFS share.

Another possibility is something like "rsync" which keeps directories structures in sync across different machines and platforms. This of course depends on the kind of data transfer you are doing.

As mentioned above, samba, is yet another alternative but is a bit fussy if all you want to do is transfer some data.

[doug mielke]

We make heavy use of NFS and CIFS, but for short or occasional transfers, when we can tolorate the open security nature, we use rcp most often.

It seems the simplest to set up, and has been more management free than FTP.

[Leif Halvarsson_2]

Hi,

A more genaral, OS independant, transfer method. FTP may be the best alternative, if I am not wrong it exist for Unix, OS/390,VMS and Windows (and several other OS) and is possible to use with automatic transfers. I know we use FTP for atomatic transfers between VMS and UNIX but the Syatem manager for this system was out of the office today, if you are interested I can give you details on Monday.

[Steven E. Protter]

When possible, I use Secure Copy from Secure Shell. When public keys are exchanged, I can do most data transfers on an automated basic, even with cron.

scp filename hostname://target directory

Attaching a doc on public keys.

There are occaisions where we revert to ftp. I am slowly but successfully pushing a standard that involves dropping telnet and ftp and any clear text password apps.

SEP

[Geoff Wild]

With different platforms and OS's, I would use ftp.

To automate it, setup .netrc file with username/password

#!/bin/sh
TIMESTAMP=`/usr/bin/date`
HOSTADDRESS=
FILENAME=`basename $1`
LOGDIR=/tmp/
LOGFILE=$LOGDIR$FILENAME.ftplog
XferMode=ascii
LOCALDIR=/somewhere/
echo "LOGFILE FOR JOB $FILENAME AT $TIMESTAMP" > $LOGFILE


#if [ `ping $HOSTADDRESS -n 5 | grep -c '100% packet loss'` -ne 0 ]
#then
# echo "$HOSTADDRESS does not answer to ping" >> $LOGFILE
# exit
#fi

ftp -v $HOSTADDRESS <> $LOGFILE
$XferMode
lcd $LOCALDIR
put $FILENAME
dir $FILENAME
pass
bye
EndFTP

STATUS=`grep complete $LOGFILE`
if [[ $STATUS != "" ]]; then
ERROR=0
l_rc=0
else
ERROR=1
fi
exit $ERROR


Rgds...Geoff

[A. Clay Stephenson]

Given your wide variety of platforms and OS's, I would tend towards FTP as the lowest common denominator. The ONLY way I now script FTP transfers is in Perl these days using the Net::FTP module which is now distributed with most Perl releases but can always be downloaded in any case from www.perl.org/CPAN.

In any event, Perl makes it trivially easy to check transfer status and take corrective action such as a re-transmit. It also allows you to use the .netrc file so that passwords do not have to be embedded in your scripts. Finally, when coming from some of the OS's what speak EBCDIC, the transfer could (with a small bit of extra coding) do the EBCDIC to ASCII translation.

Note: You only have to have Perl on one end.

Attached is an example which FTP's to a server, cd's to /tmp, forces ASCII mode, and then for every filename listed on the command line does a get and checks the status. It automatically attempts a retransfer up to 5 times on failure AND returns a 0 exit status to the calling process on success. The other nice thing about Net::FTP is that you can easily check when a file was last modified so that you don't request a file that might still be being updated.

Use it like this:
ftpget.pl file1 file2 file3
STAT=${?}
if [[ ${STAT} -eq 0 ]]
then
echo "All them files is here"
else
echo "There's trouble in River City; Status ${STAT}"
fi

[David_246]

Thanks all for your reply!

As we can see now the most common and maybe most trusted way for transporting data is still FTP.
Windows already has developed ODBC, etc to interchange data and datasources.

This might be an urgent call for Unix developers to create a programm that can interchange data amongst multiple platforms as well and has it's own authentication.

We can not afford to stand still, as you can see it will mean going backwards. Keep in mind I am not a Windows liker, but we should have a more progressive way of dealing with this matter I guess.
Do know how much more value it can give to make Unix the chain of interchanging data over multiple platforms.

Regs David

[Duncan Edmonstone]

David,

Sounds like we have some confusion here...

Transmitting data in bulk in a batch fashion:

FTP, NFS, rcp, rdist, SAMBA etc.

This ISN'T what ODBC is... ODBC is used for connecting data sources together programatically (usually database->database, or client->database).

If you *really* want ODBC for UNIX you can certainly get it:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x8645d01f8678d4118fef0090279cd0f9,00.html

And of course you can also get ODBC interfaces for perl.



Just to add an additional product to the list of batch file movers above, if you want an absloute rock-solid cross-platform file transmission tool, then look at Sterling Commerce's Connect:Direct software:

http://www.stercomm.com/solutions/products/ebi/connect/direct.html

It's certainly not cheap, but it has all the functions you could ever want, and is extensively used by banks and other financial institutions.

[Donny Jekels]

David,

throw all those tools listed above away and take a serious look at rsync.

http://hpux.connect.org.uk/hppd/hpux/Networking/Admin/rsync-2.5.6/

we use it to automate data transfer between production and dr serevrs ot terabytes of data.

have fun, you love it.....

and you can easily automate your jobs.

its extremely fast and reliable.

peace
Donny