Keep files in sync between Live Cluster and D/R Cluster

[Scott Langworthy]

Is there a way to keep an entire disk, or certain directories "in sync" between my Live Cluster and my D/R Cluster?

My current scheme of copying files which have been modified/since=yesterday and ftp'ing them over to the D/R Cluster, isn't necessarily the best way, and I am wondering what other options I can use that might make my life easier.

My two clusters are in different locations and are not tied together, which is why I am ftp'ing them currently.

[Hoff]

Clustering across two or three sites is the standard approach here; that works up to 500 miles with standard support, and can also potentially operate over rather longer spans if you involve HP with the configuration.

Some of the StorageWorks software and StorageWorks controllers can optionally allow remote data replication.

RTR and explicit synchronization within the application can provide this.

The Unix way is with rsync but (AFAIK) that's not been fully ported over to OpenVMS, or with the use of a grid. Some folks use NFS here.

Depending on what you are synchronizing, a distributed version control system such as mercurial (Hg), SVN or git or such, depending on what sorts of data you're working with.

Modifications within the application to add transactions can be useful here, even if not using RTR or such as part of the design. Appropriate application coding can help both BACKUP to be more consistent, and can help improve and ease disaster tolerance; to make the applications themselves more reliable, and easier to manage and maintain.

Some databases will remote-replicate for you, if you ask them nicely.

Poke around with Google and such. This topic arises every couple of months. (Search for OpenVMS and rsync and cluster or such, and you should find some options.)

[Chris Barratt · Accepted solution]

We are currently doing pretty much the same as you. While we would love to use some of the distributed cluster features and/or SAN replication features, available budget has curtailed this (for now).

As we are an Rdb site, we have made use of Hot Standby and have all our databases replicating over to the DR box. This covers 90% of our data.

For application files (eg. .EXE's, .COMs) we have a procedure which moves them into production. We modified this procedure to also copy to the DR box.

That leaves our RMS based data. At the moment this consists mostly of rarely changed sequential files, relative files used for data queues and a couple of indexed files. For this we have a nightly backup of each directory to a saveset on disk (as well as tape), which is later restored across to the DR directory. Obviously, this is not ideal, but for the moment we can live with any data loss that might occur in a DR situation. We are also working to remove the indexed files, and possibly look at moving the relative file functionality into databases.

We also synchronise authorisation and other system files etc on a nightly basis, but I assume you have already covered this.

Add a couple of regular "check the difference" procedures (ie. comparing directories, queues etc) and that is our DR on the cheap solution.

Don't know if it helps at all, but depending on how well your system is setup and what you are using, it is amazing what you can achieve with a few command files, Decnet and Hot Standby.

Cheers,
chris

P.S. When we first started this a few years back, I looked for rsynch for VMS but could not find a version. Ultimately, we realised what needed to be synched to DR and concentrated on that.

[Scott Langworthy]

Thanks for the info.

First to HOFF, I wanted to thank you for your in depth info you provided. However, Chris hit this more on the head. I am not going to Cluster my two Clusters together. I don't want them dependant on each other either, I needed them to stand on their own, should the other cluster fail or be destroyed, like in a disaster situation.

Secondly, we also wanted a cheaper solution, as Chris stated. Sort of a way around some of the things you spoke of in your response.

To Chris:

Essentially I believe I am doing much of the same things you are doing. Your details are similar to mine.

We're using BMQ for replication of Database info, and we're doing a Backup to a files-11 Saveset of our Com's, and Exe's etc...and then FTP'ing the Saveset to the remote cluster and then Restoring the saveset into like directories over there.

Like you, I also wondered if there was something like an RSYNC you could use in that situation for all the files, but there really isn't.

I've also set up two types of COM files to 1) Replicate the Empty Queue Structure, and 2) a COM file to run every 15 minutes which can create a new com file on the fly, to recreate all of my jobs that were on queue as well. This, with minor tweaking, has already worked well for us in a transition type move we needed to do.

This got us around the moving of the queue manager files, which is cumbersome and difficult at best.

The one place I wondered about was the Sysuaf.Dat and the Rightslist data files.

Can these simply be copied over to the remote cluster, if all things are the same over there?

Again, thanks for the help and info.

[The Brit]

Scott,
What kind of storage subsystems are you using at the two sites? What is the separation, and what kind of connectivity do you have.

Dave

[Jan van den Ende]

Scott,

oa you wrote (on the one-cluster idea by Hoff)
>>>
I don't want them dependant on each other either, I needed t hem to stand on their own, should the other cluster fail or be destroyed, like in a disaster situation.
<<<

I can tesfify from direct experience, that the reverse is true!

(when I was still involved there, my contract has run out) we HAD one cluster, spread out over 2 sites.
Each site has capacity to run all vital appss (some administrative apps were deemed "non-vital during emergencies").
Full HostBasedVolumeShadowing across the intersite link.

At normal operation load balancing spreads the users pretty much evenly. Users connect to Application Service names; those services are normally offered on all nodes, but can (individually) be restricted for rolling app updates.

One site is where we were, the remote site is essentially a "dark" site.

A quorum node at a third site was regularly proposed, but hit by "political" veto, because it could not also be implemented for Tru64, nor Windoze.

We did slightly unbalance the votes between the "light" and the "dark" site, such that upon failure of the link the lite site would hang and the dark site continues.
This implies that upon lite site failure, the dark site just goes on, and upon failure of the dark site, the cluster will "hang" until dark site fail is confirmed and (human) IPC (later superceeded by Availability Manager) action restores Quorum.

This config has been continuously running for over 11 years, and survived outages of either site as well as intersite link.

That did NOT go unnnoticed, but essentially the vital apps REMAINED available.

To steal a text from Keith Parris, with VMS clustering D/R can mean NOT Disaster Recovery, but Disaster Resiliance.

So, you should NOT rule out a single multisite cluster!

hth

Proost.

Have one on me.

jpe

[Scott Langworthy]

Dave,

The two Clusters I have set up, the Live one and the D/R Remote one are like so on Each Side...

1 - ES45 using 10 Logical Raid+1 Disks using the 6 internal, and 14 on an external Storageworks 4414R shelf. Commmunication is by a 4 channel Smart Array 5304A Raid Controller.

1 - DS20E using the same setup as described above for the ES45.

The two Clusters are currently about two miles apart, and we have an ATM Link between them. We are currently looking into moving the D/R site location much further away, and I have no details on that distance, or link. I have heard it will be perhaps 50 to 250 miles apart, and no idea if we will set up ATM or not.

I hope that is what you were asking. I'm not great on some of the terminolgy sometimes.

Scott

[Scott Langworthy]

Jan,

Again, thanks for providing more info. I do agree that some of what you said makes sense. And we looked at Volume Shadowing as well, but it seemed costly for us right now to think about that.

I agree that more needs to be researched into that area of Clustering. However, we really DO WANT to keep these two clusters separate, (BUT EQUAL, as much as we can).

We don't want users using the other cluster unless we need to, etc...

But back to the point of the initial question I asked, which is really, what alternatives are there to those that you and Hoff have detailed. In that respect, Chris has gone the route I was looking for answers to.

Again thanks.

Scott

[Hoff]

[[[I am not going to Cluster my two Clusters together. I don't want them dependant on each other either, I needed them to stand on their own, should the other cluster fail or be destroyed, like in a disaster situation.]]]

You'd be wrong with the assumptions there.

The cluster will continue its operations.

A correctly-configured DT cluster can and will continue its operations when one lobe is destroyed. Depending on how you've set it up and how you want to operate it, the cluster can continue entirely transparently.

This particular capability has been proven. Two of the cases that saw wide reporting were the Commerzbank DT cluster in the World Trade Center and the fire that destroyed the Credit Lyonnaise headquarters.

Details of and press releases related to the latter case are posted around the net, and here's the HP media from the former case:

http://h71000.www7.hp.com/openvms/brochures/commerzbank/

There is/was a DT video available from HP that showed a demonstration of the fail-over occurring live; one lobe of the cluster was destroyed with a camera rolling.

[Ian Miller.]

Another consideration for having separate clusters is that one cluster can be production and one test and therefore they are subject to different audit regimes.