topic Re: script for copying data from production to DR server in Operating System - HP-UX

script for copying data from production to DR server

Amit Manna_5 — Fri, 28 Nov 2008 11:50:24 GMT

hi
I need to copy some data from production to DR server.

This is the criteria. At this moment there are around 90000 files in production server starting with MP* and around 30000 files are copied to DR manually.

We need to sync both first and then we need to put a script in place which will check if the filesystems are in sync in terms of MP* files. if not it will copy the file which has been generated recently in production.

The files should not be overwritten.

thanks and regards
Amit Manna

Re: script for copying data from production to DR server

Bill Hassell — Fri, 28 Nov 2008 12:31:18 GMT

The simplest method is to use rsync. Be sure to test your configuration with a very small number of files to make sure you are storing the files in the correct directories. You'll need to decide if you just want to push the files to DR or if you want the directories truly synchronized by removing files at DR that have been removed at your production site.

rsync must be downloaded from the HP software site, specifically from the Internet Express collection of programs:

http://h20392.www2.hp.com/portal/swdepot/searchProducts.do

One additional consideration is the amount of data that will be transmitted each day. Make sure that the link is fast enough to keep up with the new and changed data. This is easy to overlook with a fast network in production.

Re: script for copying data from production to DR server

likid0 — Fri, 28 Nov 2008 14:20:21 GMT

As Bill says, rsync is the best way to go.

I attach you a file that gives tips for rsync and other copy commands.

Re: script for copying data from production to DR server

Amit Manna_5 — Fri, 28 Nov 2008 14:34:53 GMT

hi Bill
thanks for your reply. But rsync is now allowed in our environment.

Is there any other way to do it

regards
Amit

Re: script for copying data from production to DR server

OldSchool — Fri, 28 Nov 2008 16:02:38 GMT

Why is "rsync not permitted"?

it is one of the most widely used sync tools out there.

if you can't use it, then rdiff is probably out as well.

two remaining choices would be purchase commercial software or build your own package. the later is not neccessarily a trivial task...

and as noted previously, its not going to necessarily be the number of files, but the size of them.

google "remote file syncronization" will give you some leads.

Re: script for copying data from production to DR server

OldSchool — Fri, 28 Nov 2008 16:09:04 GMT

oh, yeah..."rdist" may still be part of the std hpux installation.

it will do it, but irrc, it transmits complete files when they change. rsync transmits the differences, which should make rsync more efficient bandwidth wise.

Re: script for copying data from production to DR server

Bill Hassell — Sat, 29 Nov 2008 00:12:17 GMT

Since your environment sounds incredibly restricted, you can use scp but it is quite slow. If these files are very large (50 MB or larger), then ftp is the most efficient (native HP-UX) file transfer method. You will have to script a *LOT* of the (rsync and scp) features such as subdirectories, ownership and permissions.

If your company spent the money on a DR site, then purchase NCftp which is ideal for sending files efficiently. It supports subdirectory recursion and will even create a temporary name during the transfer. When the desired filename has been completely transferred, the name will appear in the directory. This solves problems with partial transfers being used too soon.

However, don't be concerned about scp speed until you know what your slowest link speed to DR will be. A slow link will be the biggest problem for a DR site.

Re: script for copying data from production to DR server

Kapil Jha — Sat, 29 Nov 2008 01:10:43 GMT

Bill is correct if your comapny can spent so much money to have a DR site they should buy something to get it synced as well.
Are you sure you have to do it manually....which storage u r using??just check if there is any utility already there u r not aware of.....
If its not there ftp can help you out.....
BR,
Kapil+

Re: script for copying data from production to DR server

Amit Manna_5 — Tue, 20 Jan 2009 15:18:39 GMT

HI Bill
thanks for your input. Do we need to install ssh as well for this

and how can we send some selective files?/

regards
Amit Manna

Re: script for copying data from production to DR server

Autocross.US — Tue, 20 Jan 2009 15:57:37 GMT

Amit,

I use an rsync over ssh (rsync -e ssh) solution to keep several DR servers in sync with their production counterparts on a government network. It was quickly setup after downloading the above rsync depot and adding a simple script to cron to run it nightly.

I'd check with someone on your policy as rsync is different than the other 'r' commands because of this. You can also use ssh key-based authentication to automate the login info too.

Re: script for copying data from production to DR server

Bill Hassell — Wed, 21 Jan 2009 00:40:44 GMT

As Autocross mentioned, rsync has no connection to the very unsecure command rcp, rlogin, rexec commands. rsync is available and supported by HP and is ideally suited for replication to a DR site. There's a lot to consider in trying keep files in sync: timestamp, owneership, subdirectories, symbolic links, deleting remote files when the local files are deleted, compressing to speed up transmission, and all the above are included standard in rsync.

For security, you can use ssh as the transport and now the data will be encrypted over the link.

Re: script for copying data from production to DR server

Yogeeraj_1 — Wed, 21 Jan 2009 06:43:19 GMT

hi Amit,

>Do we need to install ssh as well for this
>and how can we send some selective files?/

You need: HP-UX Secure Shell

It could be also that SSH is already available on your system.

To check:
SERVER1:>which ssh
/usr/bin/ssh
SERVER1:>

To do the selective copy, you can used wildcards or do a find-copy.

if you need any further assistance, please let us know.

kind regards
yogeeraj

Re: script for copying data from production to DR server

Amit Manna_5 — Thu, 22 Jan 2009 14:54:01 GMT

HI
We installed ssh and rsync and after that our Security team says we cannot use ssh and rsync. Any other alternative solution for this?

Re: script for copying data from production to DR server

Duncan Edmonstone — Thu, 22 Jan 2009 15:06:13 GMT

If your security team don't allow ssh - how do you connect to these servers??? if ssh isn't suitably secure then I think you turn around and say it can't be done.

HTH

Duncan

Re: script for copying data from production to DR server

Fredrik.eriksson — Thu, 22 Jan 2009 15:35:50 GMT

I agree with Duncan, if ssh isn't secure enough then there's some real flaw in your security department.

But, you should ask them in which manor they'd prefer the files to be moved between servers.

There are several solutions, some more secure then others, to this.

1. You can use rsync (which can be made to use TLS/SSL or just ssh-tunnel it.)
2. You can use sftp/scp (which uses ssh)
3. You can use ftp (unencrypted)
4. You can use NFS or similiar (see cifs) (can be encrypted, but gets horribly slow in my experience)
5. You can use a USB device, but I doubt you wanna run down and move it all the time :P

There probably is alot more ways to do it, but these are the ones I can tell you off the top of my head.

Basically if you need to sync the source and destination then rsync is your best option. I seem to remember that there is some other opensource software that is like rsync but has full support for SSL/TLS and support for SSH transfer... but I can't seem to remember it's name.

Best regards
Fredrik Eriksson

Re: script for copying data from production to DR server

Amit Manna_5 — Thu, 22 Jan 2009 15:44:45 GMT

hi
The servers are in KEON. and KEON does not support ssh. I am not sure how to go forward

Re: script for copying data from production to DR server

David G. Douthitt — Fri, 23 Jan 2009 16:57:15 GMT

"The servers are in KEON and KEON does not support ssh."

I don't think most people here will know what KEON is. I looked for the acronym, but couldn't find it.

If you use rsync to transfer very large files - you may want to turn off the differential aspect of rsync and just transfer it completely.

rsync is efficient, NOT fast (and then you throw in encryption as well, slowing it down further). So when not using slow links, it may be faster to transfer the whole thing. Just add the -W option.

I agree with others - if ssh can't be used, how can you connect to the server without transmitting your password in the clear across the network for all the world to see and copy and use?

If you're not using encryption, you might as well not use passwords at all.

Re: script for copying data from production to DR server

David G. Douthitt — Fri, 23 Jan 2009 16:58:57 GMT

Another thing: using the -b option to rsync will cause the file that would have been "overwritten" to be saved off as a backup file with a tilde (~) attached to it.

Re: script for copying data from production to DR server

OldSchool — Fri, 23 Jan 2009 17:07:07 GMT

well...here's a little info on "RSA KEON" security. It appears to be related to Certificate Authority files and related stuff.

So I guess the questions that need answered:
1) how are you allowed to connect to these servers?
2) what recommendations does you "security team" have, since they keep shooting things down?
3) What recommendation does the var for the security software have (if any)?

Re: script for copying data from production to DR server

OldSchool — Fri, 23 Jan 2009 17:07:47 GMT

http://www.opsec.com/solutions/partners/rsasecurity.html

forgot the dratted link