script for copying data from production to DR server

[Amit Manna_5]

hi
I need to copy some data from production to DR server.

This is the criteria. At this moment there are around 90000 files in production server starting with MP* and around 30000 files are copied to DR manually.

We need to sync both first and then we need to put a script in place which will check if the filesystems are in sync in terms of MP* files. if not it will copy the file which has been generated recently in production.

The files should not be overwritten.


thanks and regards
Amit Manna

[Bill Hassell · Accepted solution]

The simplest method is to use rsync. Be sure to test your configuration with a very small number of files to make sure you are storing the files in the correct directories. You'll need to decide if you just want to push the files to DR or if you want the directories truly synchronized by removing files at DR that have been removed at your production site.

rsync must be downloaded from the HP software site, specifically from the Internet Express collection of programs:

http://h20392.www2.hp.com/portal/swdepot/searchProducts.do

One additional consideration is the amount of data that will be transmitted each day. Make sure that the link is fast enough to keep up with the new and changed data. This is easy to overlook with a fast network in production.

[likid0]

As Bill says, rsync is the best way to go.

I attach you a file that gives tips for rsync and other copy commands.

[Amit Manna_5]

hi Bill
thanks for your reply. But rsync is now allowed in our environment.

Is there any other way to do it



regards
Amit

[OldSchool]

Why is "rsync not permitted"?

it is one of the most widely used sync tools out there.

if you can't use it, then rdiff is probably out as well.

two remaining choices would be purchase commercial software or build your own package. the later is not neccessarily a trivial task...

and as noted previously, its not going to necessarily be the number of files, but the size of them.

google "remote file syncronization" will give you some leads.

[OldSchool]

oh, yeah..."rdist" may still be part of the std hpux installation.

it will do it, but irrc, it transmits complete files when they change. rsync transmits the differences, which should make rsync more efficient bandwidth wise.

[Bill Hassell]

Since your environment sounds incredibly restricted, you can use scp but it is quite slow. If these files are very large (50 MB or larger), then ftp is the most efficient (native HP-UX) file transfer method. You will have to script a *LOT* of the (rsync and scp) features such as subdirectories, ownership and permissions.

If your company spent the money on a DR site, then purchase NCftp which is ideal for sending files efficiently. It supports subdirectory recursion and will even create a temporary name during the transfer. When the desired filename has been completely transferred, the name will appear in the directory. This solves problems with partial transfers being used too soon.

However, don't be concerned about scp speed until you know what your slowest link speed to DR will be. A slow link will be the biggest problem for a DR site.

[Kapil Jha]

Bill is correct if your comapny can spent so much money to have a DR site they should buy something to get it synced as well.
Are you sure you have to do it manually....which storage u r using??just check if there is any utility already there u r not aware of.....
If its not there ftp can help you out.....
BR,
Kapil+

[Amit Manna_5]

HI Bill
thanks for your input. Do we need to install ssh as well for this

and how can we send some selective files?/


regards
Amit Manna

[Autocross.US]

Amit,

I use an rsync over ssh (rsync -e ssh) solution to keep several DR servers in sync with their production counterparts on a government network. It was quickly setup after downloading the above rsync depot and adding a simple script to cron to run it nightly.

I'd check with someone on your policy as rsync is different than the other 'r' commands because of this. You can also use ssh key-based authentication to automate the login info too.

[Bill Hassell]

As Autocross mentioned, rsync has no connection to the very unsecure command rcp, rlogin, rexec commands. rsync is available and supported by HP and is ideally suited for replication to a DR site. There's a lot to consider in trying keep files in sync: timestamp, owneership, subdirectories, symbolic links, deleting remote files when the local files are deleted, compressing to speed up transmission, and all the above are included standard in rsync.

For security, you can use ssh as the transport and now the data will be encrypted over the link.

[Yogeeraj_1]

hi Amit,

>Do we need to install ssh as well for this
>and how can we send some selective files?/

You need: HP-UX Secure Shell

It could be also that SSH is already available on your system.

To check:
SERVER1:>which ssh
/usr/bin/ssh
SERVER1:>

To do the selective copy, you can used wildcards or do a find-copy.

if you need any further assistance, please let us know.

kind regards
yogeeraj

[Amit Manna_5]

HI
We installed ssh and rsync and after that our Security team says we cannot use ssh and rsync. Any other alternative solution for this?

[Duncan Edmonstone]

If your security team don't allow ssh - how do you connect to these servers??? if ssh isn't suitably secure then I think you turn around and say it can't be done.

HTH

Duncan

[Fredrik.eriksson]

I agree with Duncan, if ssh isn't secure enough then there's some real flaw in your security department.

But, you should ask them in which manor they'd prefer the files to be moved between servers.

There are several solutions, some more secure then others, to this.

1. You can use rsync (which can be made to use TLS/SSL or just ssh-tunnel it.)
2. You can use sftp/scp (which uses ssh)
3. You can use ftp (unencrypted)
4. You can use NFS or similiar (see cifs) (can be encrypted, but gets horribly slow in my experience)
5. You can use a USB device, but I doubt you wanna run down and move it all the time :P

There probably is alot more ways to do it, but these are the ones I can tell you off the top of my head.

Basically if you need to sync the source and destination then rsync is your best option. I seem to remember that there is some other opensource software that is like rsync but has full support for SSL/TLS and support for SSH transfer... but I can't seem to remember it's name.

Best regards
Fredrik Eriksson

[Amit Manna_5]

hi
The servers are in KEON. and KEON does not support ssh. I am not sure how to go forward

[David G. Douthitt]

"The servers are in KEON and KEON does not support ssh."

I don't think most people here will know what KEON is. I looked for the acronym, but couldn't find it.

If you use rsync to transfer very large files - you may want to turn off the differential aspect of rsync and just transfer it completely.

rsync is efficient, NOT fast (and then you throw in encryption as well, slowing it down further). So when not using slow links, it may be faster to transfer the whole thing. Just add the -W option.

I agree with others - if ssh can't be used, how can you connect to the server without transmitting your password in the clear across the network for all the world to see and copy and use?

If you're not using encryption, you might as well not use passwords at all.

[David G. Douthitt]

Another thing: using the -b option to rsync will cause the file that would have been "overwritten" to be saved off as a backup file with a tilde (~) attached to it.

[OldSchool]

well...here's a little info on "RSA KEON" security. It appears to be related to Certificate Authority files and related stuff.

So I guess the questions that need answered:
1) how are you allowed to connect to these servers?
2) what recommendations does you "security team" have, since they keep shooting things down?
3) What recommendation does the var for the security software have (if any)?

[OldSchool]

http://www.opsec.com/solutions/partners/rsasecurity.html

forgot the dratted link

[Amit Manna_5]

hi
Can we use the rsync option without ssh.


Regards
Amit Manna

[Bill Hassell]

> Can we use the rsync option without ssh.

Yes. The default is to copy the data without encryption. rsync must be installed on both systems (source and destination).

[Duncan Edmonstone]

I'm still not sure I understand *why* ssh isn't allowed...

But as Bill says, you can just use rsync between the systems without the encryption of ssh - in this case rsync just uses remsh (which of course isn't a very secure protocol at all and requires a completely open "trust" between the systems and sends data in clear)

You could of xourse use rsync with remsh, and then create an IPSEC tunnel between tghe 2 systems. Probably more complex to manage and maintain than ssh, but would at least prevent the transmission of unencrypted data.

Manuals for using IPSEC are here:

http://docs.hp.com/en/J4256-90015/index.html

HTH

Duncan