Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

Keep files in sync between Live Cluster and D/R Cluster

 
SOLVED
Go to solution
Scott Langworthy
Occasional Advisor

Keep files in sync between Live Cluster and D/R Cluster

Is there a way to keep an entire disk, or certain directories "in sync" between my Live Cluster and my D/R Cluster?

My current scheme of copying files which have been modified/since=yesterday and ftp'ing them over to the D/R Cluster, isn't necessarily the best way, and I am wondering what other options I can use that might make my life easier.

My two clusters are in different locations and are not tied together, which is why I am ftp'ing them currently.
12 REPLIES 12
Hoff
Honored Contributor

Re: Keep files in sync between Live Cluster and D/R Cluster

Clustering across two or three sites is the standard approach here; that works up to 500 miles with standard support, and can also potentially operate over rather longer spans if you involve HP with the configuration.

Some of the StorageWorks software and StorageWorks controllers can optionally allow remote data replication.

RTR and explicit synchronization within the application can provide this.

The Unix way is with rsync but (AFAIK) that's not been fully ported over to OpenVMS, or with the use of a grid. Some folks use NFS here.

Depending on what you are synchronizing, a distributed version control system such as mercurial (Hg), SVN or git or such, depending on what sorts of data you're working with.

Modifications within the application to add transactions can be useful here, even if not using RTR or such as part of the design. Appropriate application coding can help both BACKUP to be more consistent, and can help improve and ease disaster tolerance; to make the applications themselves more reliable, and easier to manage and maintain.

Some databases will remote-replicate for you, if you ask them nicely.

Poke around with Google and such. This topic arises every couple of months. (Search for OpenVMS and rsync and cluster or such, and you should find some options.)
Chris Barratt
Frequent Advisor
Solution

Re: Keep files in sync between Live Cluster and D/R Cluster

We are currently doing pretty much the same as you. While we would love to use some of the distributed cluster features and/or SAN replication features, available budget has curtailed this (for now).

As we are an Rdb site, we have made use of Hot Standby and have all our databases replicating over to the DR box. This covers 90% of our data.

For application files (eg. .EXE's, .COMs) we have a procedure which moves them into production. We modified this procedure to also copy to the DR box.

That leaves our RMS based data. At the moment this consists mostly of rarely changed sequential files, relative files used for data queues and a couple of indexed files. For this we have a nightly backup of each directory to a saveset on disk (as well as tape), which is later restored across to the DR directory. Obviously, this is not ideal, but for the moment we can live with any data loss that might occur in a DR situation. We are also working to remove the indexed files, and possibly look at moving the relative file functionality into databases.

We also synchronise authorisation and other system files etc on a nightly basis, but I assume you have already covered this.

Add a couple of regular "check the difference" procedures (ie. comparing directories, queues etc) and that is our DR on the cheap solution.

Don't know if it helps at all, but depending on how well your system is setup and what you are using, it is amazing what you can achieve with a few command files, Decnet and Hot Standby.

Cheers,
chris

P.S. When we first started this a few years back, I looked for rsynch for VMS but could not find a version. Ultimately, we realised what needed to be synched to DR and concentrated on that.
Scott Langworthy
Occasional Advisor

Re: Keep files in sync between Live Cluster and D/R Cluster

Thanks for the info.

First to HOFF, I wanted to thank you for your in depth info you provided. However, Chris hit this more on the head. I am not going to Cluster my two Clusters together. I don't want them dependant on each other either, I needed them to stand on their own, should the other cluster fail or be destroyed, like in a disaster situation.

Secondly, we also wanted a cheaper solution, as Chris stated. Sort of a way around some of the things you spoke of in your response.

To Chris:

Essentially I believe I am doing much of the same things you are doing. Your details are similar to mine.

We're using BMQ for replication of Database info, and we're doing a Backup to a files-11 Saveset of our Com's, and Exe's etc...and then FTP'ing the Saveset to the remote cluster and then Restoring the saveset into like directories over there.

Like you, I also wondered if there was something like an RSYNC you could use in that situation for all the files, but there really isn't.

I've also set up two types of COM files to 1) Replicate the Empty Queue Structure, and 2) a COM file to run every 15 minutes which can create a new com file on the fly, to recreate all of my jobs that were on queue as well. This, with minor tweaking, has already worked well for us in a transition type move we needed to do.

This got us around the moving of the queue manager files, which is cumbersome and difficult at best.

The one place I wondered about was the Sysuaf.Dat and the Rightslist data files.

Can these simply be copied over to the remote cluster, if all things are the same over there?

Again, thanks for the help and info.

The Brit
Honored Contributor

Re: Keep files in sync between Live Cluster and D/R Cluster

Scott,
What kind of storage subsystems are you using at the two sites? What is the separation, and what kind of connectivity do you have.

Dave
Jan van den Ende
Honored Contributor

Re: Keep files in sync between Live Cluster and D/R Cluster

Scott,

oa you wrote (on the one-cluster idea by Hoff)
>>>
I don't want them dependant on each other either, I needed t hem to stand on their own, should the other cluster fail or be destroyed, like in a disaster situation.
<<<

I can tesfify from direct experience, that the reverse is true!

(when I was still involved there, my contract has run out) we HAD one cluster, spread out over 2 sites.
Each site has capacity to run all vital appss (some administrative apps were deemed "non-vital during emergencies").
Full HostBasedVolumeShadowing across the intersite link.

At normal operation load balancing spreads the users pretty much evenly. Users connect to Application Service names; those services are normally offered on all nodes, but can (individually) be restricted for rolling app updates.

One site is where we were, the remote site is essentially a "dark" site.

A quorum node at a third site was regularly proposed, but hit by "political" veto, because it could not also be implemented for Tru64, nor Windoze.

We did slightly unbalance the votes between the "light" and the "dark" site, such that upon failure of the link the lite site would hang and the dark site continues.
This implies that upon lite site failure, the dark site just goes on, and upon failure of the dark site, the cluster will "hang" until dark site fail is confirmed and (human) IPC (later superceeded by Availability Manager) action restores Quorum.

This config has been continuously running for over 11 years, and survived outages of either site as well as intersite link.

That did NOT go unnnoticed, but essentially the vital apps REMAINED available.

To steal a text from Keith Parris, with VMS clustering D/R can mean NOT Disaster Recovery, but Disaster Resiliance.

So, you should NOT rule out a single multisite cluster!

hth

Proost.

Have one on me.

jpe
Don't rust yours pelled jacker to fine doll missed aches.
Scott Langworthy
Occasional Advisor

Re: Keep files in sync between Live Cluster and D/R Cluster

Dave,

The two Clusters I have set up, the Live one and the D/R Remote one are like so on Each Side...

1 - ES45 using 10 Logical Raid+1 Disks using the 6 internal, and 14 on an external Storageworks 4414R shelf. Commmunication is by a 4 channel Smart Array 5304A Raid Controller.

1 - DS20E using the same setup as described above for the ES45.

The two Clusters are currently about two miles apart, and we have an ATM Link between them. We are currently looking into moving the D/R site location much further away, and I have no details on that distance, or link. I have heard it will be perhaps 50 to 250 miles apart, and no idea if we will set up ATM or not.

I hope that is what you were asking. I'm not great on some of the terminolgy sometimes.

Scott
Scott Langworthy
Occasional Advisor

Re: Keep files in sync between Live Cluster and D/R Cluster

Jan,

Again, thanks for providing more info. I do agree that some of what you said makes sense. And we looked at Volume Shadowing as well, but it seemed costly for us right now to think about that.

I agree that more needs to be researched into that area of Clustering. However, we really DO WANT to keep these two clusters separate, (BUT EQUAL, as much as we can).

We don't want users using the other cluster unless we need to, etc...

But back to the point of the initial question I asked, which is really, what alternatives are there to those that you and Hoff have detailed. In that respect, Chris has gone the route I was looking for answers to.

Again thanks.

Scott

Hoff
Honored Contributor

Re: Keep files in sync between Live Cluster and D/R Cluster

[[[I am not going to Cluster my two Clusters together. I don't want them dependant on each other either, I needed them to stand on their own, should the other cluster fail or be destroyed, like in a disaster situation.]]]

You'd be wrong with the assumptions there.

The cluster will continue its operations.

A correctly-configured DT cluster can and will continue its operations when one lobe is destroyed. Depending on how you've set it up and how you want to operate it, the cluster can continue entirely transparently.

This particular capability has been proven. Two of the cases that saw wide reporting were the Commerzbank DT cluster in the World Trade Center and the fire that destroyed the Credit Lyonnaise headquarters.

Details of and press releases related to the latter case are posted around the net, and here's the HP media from the former case:

http://h71000.www7.hp.com/openvms/brochures/commerzbank/

There is/was a DT video available from HP that showed a demonstration of the fail-over occurring live; one lobe of the cluster was destroyed with a camera rolling.
Ian Miller.
Honored Contributor

Re: Keep files in sync between Live Cluster and D/R Cluster

Another consideration for having separate clusters is that one cluster can be production and one test and therefore they are subject to different audit regimes.
____________________
Purely Personal Opinion
Hoff
Honored Contributor

Re: Keep files in sync between Live Cluster and D/R Cluster

If you want to roll your own cluster (which is where this whole discussion is headed) with this cluster on a budget approach, then you certainly can head that direction.

Sure, clustering and HBVS is extremely expensive.

At the other extreme of cost, couriers and off-site (encrypted?) backups can provide a (longer-latency) recovery process, too.

In the middle ground, that's rsync and nfs and distributed version control and such. And rolling your own.

Rolling your own clustering and your own DT means implementing and maintaining your own and testing your own scheme, and that gets expensive. (I remember when we all commonly wrote our own network stacks. We seldom do that anymore.) And implementing your own will be involved, as the error sequences are inherently difficult to test. Your site will be the first site through the error and failover sequences involved here, too.

I've implemented solutions here using both ways -- roll your own, and clustering -- based on cluster requests. The former is either a point solution for a specific failover case, or it's inherently far more work. As a solution provider, the latter provides better features and better economies for the customer.

But then, how much is an outage going to cost you?

Regardless of the solution you pick here, I'd recommend testing.
Scott Langworthy
Occasional Advisor

Re: Keep files in sync between Live Cluster and D/R Cluster

To all:

Thanks for all the info. You've given me much to think about.

I will take this information and go do more research.

For now, I will close this thread, as we can go on all day about the different scenarios.

I appreciate all the feedback on the subject.

As always, Kudos to your expertise and advice.

Scott.
Scott Langworthy
Occasional Advisor

Re: Keep files in sync between Live Cluster and D/R Cluster

See previous Email for comments.