keeping mirrored NAS in sync

[doug mielke]

I have 2 NetAPPs NAS filer arrays, connected to N-class servers, located 10 miles apart, connected by a T-1 line. (My DR site)

I use NetApps' Snap Mirror product to keep them in sync, which only transmitts the block level changes, and it works great. ( 1 day of Oracle activity transmitts in 3 hours, entire database takes 6 days.

My problem is that occasionally, through DBA error, comm failure, etc. I'm forced to re mirror the 2 databases, ( the 6 day process)

Is anyone else doing this? If so, can I use tapes to load remote site( remote site has no tape, but I'll buy one if it works) Is there another way?

I'm afraid the re-mirror will try to re write it all anyway, since it's at the 'blockj' level of the filer.

NetApps support can't tell me if this works or not, but I need a way to re sync, other than loading the array on a truck so I can remirror at ethernet speed, or waiting 6 days lor the s l o w t-1 connection.

Any ideas?

[Tim D Fulford]

rsync... maybe.... I've only got this to work using equivalences

rsync will synchronise only the differences, so as long as the data is stored as cooked data rsync may help!

http://hpux.connect.org.uk/hppd/hpux/Networking/Admin/rsync-2.5.6/

Alternatively use an E-ONE link or even better, proper high bandwidth WAN link!

Regards

Tim

[doug mielke]

More details:
the mirrored volumes at the DR site are intended to be a 'hot standby' of all of our production instances.

In theory, in a catostrophic failure, I can break the mirror, point the DR site server, or any other server on the planet, to the DR NAS box.
(at the not-so-blazing T-1 speed)

It appears rsych would require some sophistcation of quiece db, and archive log application. It would make my head hurt.

This whole project was Champagne DR on Beer budget from the start, but it looks like more bandwidth is the only viable solution...

C'mon folks, help a fellow Beer Buddy out with some creative magic {:^o

[Tim D Fulford]

If you are doing a Beer DR plan.... you probably need two or more plans to fiddle with as if one fails there is always another in the wings/pub....

o You are already doing one (T-1 replication).
o IF your databse supports snapshots (most recent ones do), simply
a) checkpoint database,
b) snap-shot-it (NetApp supports this)
c) uncheck your DB.
a-c normally takes less than 1 second (depending on data size, number of x-actions & freezing time etc). Then use T-1 & rsync/rcp whatever to sync sites.
o Lastly tape scenario. I'd use nightly level 0 database backups & ship off the logical logs 3 times a day (or what ever you operators can manage). You can do a full restore from level 0 & do log roll forward.

Regards

Tim

[doug mielke]

Fine ideas all,

We have our 4 levels of backup / redundancy in place, including tape, hot standby databases, local and remote, and nightly NAS hot backup.

And, I mirror my hot standbys as the offsite DR solution.

My major pain is in getting the 2 hot sites / database in sync in the first place. It's broken now, so I'm down to 3 backup plans (plus my liberal pool of hot spares in my drive arrays.

It's worth pondering more over suds, how to but I bet there's a truck in my Monday plans....

[Tim D Fulford · Accepted solution]

I'm just putting this reply in to raise the post to the top again in the hope someone with other bright ideas will respond....

Sounds like you have a good set of plans, I hope the company realise you are providing a service that would cost mucho $$$/???????????? if done via a third party! I'm signing off now & going to grab a beer some pizza, & consume infront of TV or video game & bed (the exciting life I live!!!)

Regards

Tim

[doug mielke]

Results:
It took 8 days on the T-1 line, shared with 20 or so users at our warehouse / DR site to remirror the 140 gig volume.

After the mirror, a resync. required 19 hours to process the 8 days of changes. Then, a whopping 6 hours to process the 19 hours worth of changes queued during the resync. Slow, but it looks like it worked.

So, after 10 days, I have a valid mirror that required no heavy lifting.

Signed,
Still looking for Alternatives.