script for copying data from production to DR server

[Amit Manna_5]

hi
I need to copy some data from production to DR server.

This is the criteria. At this moment there are around 90000 files in production server starting with MP* and around 30000 files are copied to DR manually.

We need to sync both first and then we need to put a script in place which will check if the filesystems are in sync in terms of MP* files. if not it will copy the file which has been generated recently in production.

The files should not be overwritten.


thanks and regards
Amit Manna

[Bill Hassell · Accepted solution]

The simplest method is to use rsync. Be sure to test your configuration with a very small number of files to make sure you are storing the files in the correct directories. You'll need to decide if you just want to push the files to DR or if you want the directories truly synchronized by removing files at DR that have been removed at your production site.

rsync must be downloaded from the HP software site, specifically from the Internet Express collection of programs:

http://h20392.www2.hp.com/portal/swdepot/searchProducts.do

One additional consideration is the amount of data that will be transmitted each day. Make sure that the link is fast enough to keep up with the new and changed data. This is easy to overlook with a fast network in production.

[likid0]

As Bill says, rsync is the best way to go.

I attach you a file that gives tips for rsync and other copy commands.

[Amit Manna_5]

hi Bill
thanks for your reply. But rsync is now allowed in our environment.

Is there any other way to do it



regards
Amit

[OldSchool]

Why is "rsync not permitted"?

it is one of the most widely used sync tools out there.

if you can't use it, then rdiff is probably out as well.

two remaining choices would be purchase commercial software or build your own package. the later is not neccessarily a trivial task...

and as noted previously, its not going to necessarily be the number of files, but the size of them.

google "remote file syncronization" will give you some leads.

[OldSchool]

oh, yeah..."rdist" may still be part of the std hpux installation.

it will do it, but irrc, it transmits complete files when they change. rsync transmits the differences, which should make rsync more efficient bandwidth wise.

[Bill Hassell]

Since your environment sounds incredibly restricted, you can use scp but it is quite slow. If these files are very large (50 MB or larger), then ftp is the most efficient (native HP-UX) file transfer method. You will have to script a *LOT* of the (rsync and scp) features such as subdirectories, ownership and permissions.

If your company spent the money on a DR site, then purchase NCftp which is ideal for sending files efficiently. It supports subdirectory recursion and will even create a temporary name during the transfer. When the desired filename has been completely transferred, the name will appear in the directory. This solves problems with partial transfers being used too soon.

However, don't be concerned about scp speed until you know what your slowest link speed to DR will be. A slow link will be the biggest problem for a DR site.

[Kapil Jha]

Bill is correct if your comapny can spent so much money to have a DR site they should buy something to get it synced as well.
Are you sure you have to do it manually....which storage u r using??just check if there is any utility already there u r not aware of.....
If its not there ftp can help you out.....
BR,
Kapil+

[Amit Manna_5]

HI Bill
thanks for your input. Do we need to install ssh as well for this

and how can we send some selective files?/


regards
Amit Manna

[Autocross.US]

Amit,

I use an rsync over ssh (rsync -e ssh) solution to keep several DR servers in sync with their production counterparts on a government network. It was quickly setup after downloading the above rsync depot and adding a simple script to cron to run it nightly.

I'd check with someone on your policy as rsync is different than the other 'r' commands because of this. You can also use ssh key-based authentication to automate the login info too.