Mirrordisk/UX to remote disks 2,000 miles away

[John Hall]

I have an HP-UX 9000 (rp3440) server running HP-UX 11.11 with an Oracle database on a local EMC 3000 SAN. We use LVM not Veritas Volume Mgr. I want to use Mirrordisk to keep an "exact" copy of the database on a remote DS2405 disk array that is 2,000 miles away. I have spoken with an HP representative who said I need a "protocol adapter" to attach the DS2405 to our WAN (he said Brocade makes a good protocol adapter and we kind of prefer Brocade anyway).

So, unless there is a FAR BETTER solution, I guess my question is, "As the system writes to the local EMC SAN will Mirrordisk have any latency issues writing to the remote DS2405?" By this I mean does Mirrordisk need to wait for the write to the remote disks to complete before doing the next local write? If yes, then the write latency would slow down the local server and this is unacceptable. There are probably third party products that cost $$$ but we are trying to use our existing software - like Mirrordisk/UX.

[Steven E. Protter]

Shalom,

You need a ton of bandwitch to do this.

You could also do it without mirror/ux with a dd command with an output file say on nfs at the other end of the connection.

The problem is that in either case the reliability of and usefullness of the image is doubtful.

Any communication issue with mirror/ux and either the main system slows to a crawl or the mirror copy goes stale without notification to anyone.

With dd the image would simply become corrupt and is difficult to use anyway.

If its a database you wish to replicate, the bucks required of Oracle Dataguard or whatever they call it today is worth it. If you care about the backup, you should use the right tool.

SEP

[DCE]

John,

Co-incidently, I have been researching the scenario you are talking about.

What I have found is that you need to configure your system to do asychronous writes versus synchronous writes. This allows for far better performance, but a at a slight risk. You could lose the data in the buffer in the event of a disaster.

I am not sure if LVM has an asynchrous write option. If you are mirroring SAN's I would think you would mirror at the hardware level and let the SAN take of everything.........

[Bill Hassell]

If you thought you would save money by using Mirror/UX, the fiber optics network you need to support the required bandwidth will cost THOUSANDS of dollars per month to lease. Disks are extremely fast when compared to the very slow rate of a network connection. Here's a useful chart:

(all numbers in useful megabytes per second)

Broadband or T1 = .05 to .8
LAN/10 = .5 to 1
LAN/100 = 5 to 8
LAN/1000 = 50 to 90
disks = 20 to 60

These are full duplex speeds taking into account that you can't get much more than 50%-70% usage on a network and also counting the overhead bits for each packet.

So ask your network provider what it will cost for a 1000 Mbit network connection. If you opt for anything slower, EVERYTHING on your computer will be drastically slowed down. Mirror/UX writes each new record to the primary and secondary disks. It then waits for both to complete, checking both disks for proper status. Only when both disks have completed successfully, then the next disk record can be written.

The task you describe is a common request but the data rate requirements will be so expensive, all the other required hardware will be unimportant as far as cost.

[OldSchool]

If your goal is to have a back-up site for disaster recovery, check this out:

http://www.networkworld.com/newsletters/stor/2003/0811stor2.html

Also, put "data replication disaster recovery" in your favorite search engine.

[TwoProc]

EMC itself offers packages that do this better than you can cobble together yourself. However, it would require an EMC storage server on both sides of the 2000 mile span. This method is typically how it's done, not usually by an "LVM mirror across the continent" approach.

[John Hall]

Steven - Database sychronization is paramount. In the event of a local system (SAN) failure a switchover to the remote backup site would require VERY little to no loss of data. So anything that leads to unreliable "replication" is not an option.

DCE - DS2405's don't do hardware mirroring. Besides we don't want to mirror the data inside the remote disk array itself - we want to mirror the data in the local SAN to the remote DS2405.

Bill - Your answer was perhaps the most useful in that it makes me begin to realize that this remote mirrordisk solution may no be doable. We absolutely cannot afford to have the local server slowdown due to the server waiting for mirrordisk to complete it's writing to the remote mirror.


Therefore, maybe Plan B is required. How do I "mirror" local data to a remote disk array 2,000 miles away.

[TwoProc]

As referred to in my posting:

Usally this is done at the storage system level. Like an HP XP series storage server with the software options that do the very thing you're asking. Also, the HP EVA 8000 servers do this as well, I know a company that does this and pushes over 500 miles easily using the 8000 platform.

[A. Clay Stephenson]

Your only realistic option is EMC (or at least like arrays) on both ends. EMC's approach is called SRDF. If you go to www.emc.com and search on SRDF, you should find your answer. HP's equivalent to SRDF is called ESCON.

[Steven E. Protter]

Shalom again John,

I'd say a pair of EVA-4000 or EVA-6000 boxes an a ton of bandwidth should do the job.

Mirror/ux is clearly not the tool for the job.

If its just the database, Oracle dataguard will operate with block by block change (called delta) replication, which would significantly lower the band width requirement.

As with anything there are a dozen different ways to get the job done. I think a product like Oracle dataguard or whatever they call it now should do the job with less expense.

Without an adequate supply of money, this goal will not be accomplished. It is quite possible to limit the investment, but not to the $1000 or so per cpu that mirror/ux costs.

SEP

[Michael Steele_2]

Your going to have to do this at the disk array level. HP's XP calls it Continuous Access (CA) and ESCON-WAN, which is a converter that may or may not be provided by HP.

Redundancy of your WAN nodes are critical, else, you have a single point of failure.

You also might want to look into ServiceGuard's Continental Clusters, which discusses this kind of reliance on Continuous Access in XP's.

You are also opening up a can of worms when it comes to what HP will and will not support. For example, HP will not support some of Oracle's Standby solutions. HP will insist on testing anything you create, and will foot you with the bill for any of this testing. So your going to have to look into what your HP support options will before starting any projects.

[Alzhy]

I don't think MirrorDisk/UX will not be able to do what you want.

Your decision and possible solutions will depend on the following:

BANDWIDTH
Do you have dark-fibre between your 2 sites or will it be all IP? There are already a number of products out there that are so called SAN-extenders that work over dark-fibre OR IP.

HARDWARE ARRAY/SAN ARCHITECTURE
Do you want to be beholden ("vendor-locked") to your storage infrastructure vendor?

HOST VOLUME MANAGER
Will you have plans on staying with LVM or spending a few dineros to upgrade the base VxVM that you already have?

In your case, since you are using a simple Fibre JBOD array (DS2405) - the only solutions that may work for you are all Veritas Solutions - namely VxVM's Mirroring or FlashSnap Technology or Veritas Volume Replicator. All of these products are already part of HP-UX 11i and merely requires activation (license) and configuration. These solutions should work regardless of acceptable SAN bandwidth and most importantly regardless of Storage Array used. If in the end your environment grows and you upgrade to a storage array from ABC Corp on your primary site and another array from XYZ Corp. on the failover side - all your processes should still work.


Hope this helps and good luck in your decision.

[John Hall]

Ok, just one more whack at this and then it's time for ice cream! (er, I mean points)

Oops, we actually have an EVA-5000 locally (I think I misstated that we had an EMC SAN locally) and I'm not sure we have enough $$$ for another EVA at the other end. Doesn't the shipping of data to the remote site require a lot of bandwidth along the way? I get the impression from some of these responses that an EVA at each end is all that's required. Also, we are moving away from Dataguard that's why we were hoping this cheaper Mirrordisk solution would work.

I think I'll take a quick peek at ESCON.

[TwoProc]

John,

I "think" that it's possible to mirror on the remote system to a smaller box. Now, whether or not the 5000 is already "the smaller box", I don't know, but I think you can mirror from say an 8000 to a 5000. So, I think you should call HP and ask if you could mirror from a 5000 to a 3000 over to another location, and what the costs are.

As for whether or not it will take a lot of bandwidth, I think that depends on how much write activity you have on the 5000 currently. It only has to be big enough to mirror those writes, as the local reads are "free" to the mirror machine. Most often, databases have overwhelming read activity, and the percentage of activity that is actually writes is pretty small.

[A. Clay Stephenson]

Well, it's not often that words like cheaper, less expensive, ... collide with online data replication in the same sentence. You need to either be ready to pony up the cash or rethink the problem. Solutions like Continuous Access (HP) or SRDF (EMC) can reduce the bandwidth requirements by doing asynchronous copies at the expense of latency. Obviously, if your primary database is undergoing heavy, almost continuous updates then even if buffered bandwidth eventually comes into play.

If you must keep your Data Centers ~3200km apart then one option that is often overlooked is a nitely tape sent via FedEx. That would get you to within 1 day. Another option to consider is locating your data centers much closer together so that private fibre links are possible. Online replication at a distance and cheap are just hard to do.

[Bill Hassell]

As Clay and John have said...long distance replication will be VERY expensive. If you don't have the $$$ for an EVA at the remote end, you're locked out of any real time replication design as the disk array will be one of the smallest costs for the system. Mirroring places enormous requirements on latency. When Mirror/UX first came out, HP strongly recommended identical disks and identical interfaces. Writing data on mirrors ALWAYS runs at the slowest speed between the disks.

The reason that the (high priced) HP and EMC solutions for remote data replication are often better choices is that the data is replicated asynchronously with specialized hardware that runs across all zones, leaving the local machines virtually unaffected. So Mirror/UX is cheap but your disks must be in the same building.

And clay replies with an old data axiom from 20+ years ago. The fastest, lowest cost data transfer speed is a box of tapes in a FedEx truck (and it still hasn't changed more than 20 years later). Since a small box of Ultrium 960 tapes can be shipped overnight coast-to-coast for less than $50, that translates to several terabytes in less than 24 hours. Assuming a box of tapes with 5 terabytes total, that means 200 Gb/hour or about 50 Mbytes/sec. You can buy a bunch of EVAs for the cost you'll spend each year on a 500 Mbit/sec network connection. Yes, latency will be about 24 hours...so the choice is time, or money.

[A. Clay Stephenson]

And related to time and money is this borrowed shamelessly from Cartalk (and I'm sure that Click and Clack borrowed it at least equally shamelessly from someone else):
-----------------------------------------
Theorem: The less you know the more money you make.

Proof:
We know that:
I) Time is Money
II) Knowledge is Power
III) Power = Work / Time (by definition)

By simple substitution:
Knowledge = Work / Money
Knowledge * Money = Work
Money = Work / Knowledge

It follows that as knowledge goes to 0, money goes to infinity.
-----------------------------------------

[John Hall]

Clay you are too funny - and I do know who Click and Clack are - I waste another good hour listening to them almost every weekend.

Bill, Clay, Steven your reputations are very highly regarded and I really appreciate the candid honesty with which everyone has contributed to helping me realize that Mirrordisk probably isn't going to be a solution to our remote location.

I will do the points now but need to throw one more question out to everyone. Here's another plan we are considering. We intend to use our local EVA-5000 as the primary Oracle 9i database repository. We are planning on connecting the EVA to a Brocade and also connecting a DS-2405 to the Brocade then use Mirrordisk to mirror this database from the EVA to this local DS-2405. Will the slowness of the DS-2405 (compared to the EVA) make this a show stopper? Will the added layer of the Brocade be the straw that breaks the back of good response time for my users? (But first consider the mirror between the EVA and DS-2405.)

[A. Clay Stephenson]

If I understand this latest configuarion, both your EVA and DS2405 are local; in that case, Mirror/UX makes sense but you are concerned that there will be a performance impact. Mirror/UX is always going to be limited by the slowest device. If this is mainly a read-intensive operation (as database applications tend to be) then your DS2405 through the Brocade should be fine. The performance hit will occur during periods of high write activity. As to whether of not this is acceptable to your users, well, that is what testing is for.
I would do the initial mirroring at off-peak times (because the lvsync's are going to take a long time) and then start getting some performance metrics during normal operations.

This would be a good test as to whether locating the remote data center relatively nearby (ie within Fibre range) makes sense but even in that case, I would really look at a solution that used Disk Array to Disk Array behind-the-scenes copying (SRDF or CA).

[Bill Hassell]

When you mirror the EVA to the DS, all your writes will complete at DS speeds. Mirroring (which is writing) will run at the slowest disk's speed.

[Josiah Henline]

I believe you are looking at the wrong thing.

A continental cluster with MC/SG and Oracle syncing software is what you are looking for. Keep in mind that these are high dollar projects.

If you do not have the needed bandwidth, you can set rman(oracle backup software) to work and keep copies of you backups, up to the hour, for a restore elsewhere.