SecurePath and HP-UX11iV2

[Bob Purdy]

I am running a pair of RP4640 servers with ServiceGuard A.11.16 and SecurePath 3.0F. This customer also has pruchased a pair of EVA 5000 SANs, one used as a primary and the other is replicated as a backup for disaster recovery. We have installed BusinessCopy/ContinouousAccess agents on the 4640's as well. The SecurePath software appears to be working as it should as we have only one path to each lun on the EVA 5000. The problem I have is that the customer says that when the EVA is failed over to the backup, the combination of SecurePath and BusinessCopy/ContinuousAccess should cause the 4640 to use the backup SAN in place of the primary. We ahve tried everthing we can think of to make it work but I am convinced that it will not work the way they want it to and I am suggesting that we will need to use the SecurePath to control the multiple paths to the EVA luns and use the MirrorUX to gain the redundancy between the two SANs. Can anyone suggest a solution, please?

[Duncan Edmonstone]

What excatly are you trying to achieve here?

SecurePath will provide path redundancy to a LUN on 1 EVA - if the same LUN is CA'd to another EVA, secureptah will *not* recognise that as the same LUN - it will still provide path redundancy to that replicated LUN, but will treat it as a seperate LUN in its own right.

What you are describing *could* be implemented manually, or even automatically if you purchased Metrocluster/EVA, or as you state you could mirror between the EVAs using MirrorDisk/UX, but you wouldn't require Continuous Access for that.

I think you need to get striaght *exactly* how you want this configuration to behave, and then we can see how best that should be implemented.

HTH

Duncan

[Bob Purdy]

I understand what you mean. This customer bought the servers from The company I work for and they bought the EVAs from a consulting company. This consutling company told them that by using SecurePath and BusinessCopy/ContinuousAccess it would be possible to have a seamless failover from the primary to the backup. I do not believe this is the case and I just needed someone who knows the operation of the SecurePath and BusinessCopy/ContinuousAccess to confirm that this will or will not work.

[Alex Georgiev · Accepted solution]

Bob, it will not work the way that your customer is expecting it to work. Sounds like your customer is expecting the program to continue running on the same host...

To my knowledge, the one way to setup automated failover with Continuous Access is to have each host "point" to a separate EVA (see the attached image). If EVA1 fails, that should, in theory, cause node A to fail, too. ServiceGuard on node B should detect that, and start up the package by activating the volume groups and mounting the logical volumes that reside on EVA2. You can probably have the package startup script stop the CA replication (if it hasn't stopped already), to prevent any damage to the data that may result from a partial failure of either EVA1 or node A.

The all sounds good on paper, and probably works well in a setup where each host+EVA pair is at a different site. There are, however, some problems with this CA replication setup. Here are the ones that I am aware of:

1. The failover may be automated, but certainly isn't seamless.

2. In my experience, when EVA1 fails, that will not necessarily cause the host (node A in the picture) to fail. What really happens is that every processes doing I/O with EVA1 will simply freeze up (at least in my personal experience). Node A will respond to pings, accept new telnet/ssh sessions, and continue to run every other process just fine, as long as that process does not depend on EVA1. In fact, the 'spmgr display' command will show all paths as failed... Yet, somehow those I/Os to EVA1 just refuse to time out. So you will need to have a ServiceGuard expert help you figure out how to detect this kind of failure, if you choose to go with the host+EVA "paired" setup.

3. If you cluster is using a lock disk, AND that disk resides on one of the two EVAs, AND that EVA fails, then your cluster will not fail over! In a 2 node cluster the lock disk (if you are using one) needs to be accessible for a fail-over to happen. You might be able to do synchronous replication of the lock disk with CA... I don't know. Again, a ServiceGuard expert can help you figure out how to avoid that situation.

4. Finally, Continuous Access takes up disk space. You need to leave a good amount of space unallocated, for the CA logs to work as they are intended to work. In my company we decided that at nearly $3,000 per Fibre Channel disk, space is too valuable to give to CA.

So, with that said... I am currently doing LVM mirroring between two EVAs. That gives me automated AND seamless failover, and a less complicated setup. I would recommend it to anyone over CA as cheaper and simpler setup. Nevertheless, your mileage may vary!

I only have two more things for you:

FOR YOUR OWN AND YOUR CUSTOMER'S SANITY SAKE, make sure the EVAs are running the latest version of controller firmware (aka "VCS code")! Especially if a lot of snaps and deletions will be happening. The latest version is 3.028.

The other thing to keep in mind is that IF the LVM mirror breaks (say one EVA goes offline), AND the cluster switches over during that time, then you may not be able to sync up the mirrors once the EVA comes back online. You may have to break the mirror and reestablish it... and then vgexport/vgimport the volume group on the other ServiceGuard node. Do your homework on LVM, I guess!

Good luck! Hope that helps!

P.S. What do you mean when you say that "We have installed BusinessCopy/ContinouousAccess agents on the 4640's". BusinessCopy and CA run on the EVA itself, not on the host. Do you mean the RSM management interface is installed on hosts?

[Bob Purdy]

I really appreciate your response. I am stuck in the middle of this one. The consultant that sold the customer the EVAs is really a windows oriented house and they seem to be clueless when it comes to HP-UX and I don't know the SecurePath product.

The customer has been told they can have the primary EVA fail and the system will automatically and seamlessly start using the backup EVA with no interaction and no effect on the cluster. The consultants have assured me that this is the way it works in the Windows world and they assumed it would work the same way with HP-UX. So much for consultants.

I might be able to use the paired EVA/4640 approach but I think that would require adding another lock disk not part of either EVA. I also have no idea how the replication between the EVAs is done. If there is any delay in replicating the data there is certainly a risk that in a failover some data might not be accurately replicated. We already have mechanisms in the cluster to detect when critical process fail and to fail the packages over but the cluster must unmount all file systems to do a failover and if I/O is not possible this cannot be accomplished.

I think the best solution is as you suggest to use the MirrorDisk route. I will probably leave the Secure Path running as this provides load balancing between the HBAs and takes care of the alternate paths at the same time. I need to find out how to remove the ContinuousAcces software. Since it is installed using a Java app rather than swinstall I don't know if there is an uninstall program for it.

We have just upgraded the firmware on the EVAs this past week so I think we have that covered and as far as the failover sync is concerned, we have the option to re-sync at vg activation or not.

And yes, I mean we have installed the RSM Agents on the 4640s. This is the software I will need to remove.

Againn, thanks very much for your help. It is much appreciated.

[Ted Buis]

Minor point, but you either have an rp4440 or an rx4640. I think you should look at Metrocluster with Continuous Access EVA T2403BA and possibly Continentalclusters T2346BA for greater distances. The Disaster Tolerant Solutions Manual B7660BA might be a good after first reviewing:
http://h71028.www7.hp.com/enterprise/cache/4171-0-0-0-121.html

[Duncan Edmonstone]

Bob,

You should read this in the storage forums:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=965765

I think the issue here is around semantics (what exactly does 'seamless' mean?)

Anyway, have a read.

HTH

Duncan

[Bob Purdy]

I thank ou gentlemen for your help on this one. I will probably take the quicker way out and use the mirror software since we have a deadline to take this thing live next week and the mirror solution is already installed on the systems. BTW they are indeed rx4640's. I just looked at the tags to verify. Thanks again for all the help.

[Ted Buis]

Are there two SG clusters or just two servers and two EVA's? How far apart are the two EVAs? There is greater latency with greater distance. The MirrorDisk/UX approach can work fine in a campus cluster in a single SG cluster. If you stretch it further, the latency on writes can negatively impact performance. Using Asynchronous CA can help eliminate the impacts of latency but some extra glue is needed to avoid the "seams". If you have two clusters or are at over 100KM (or maybe 200KM, then you need Continentalclusters and the prerequisites for it.

[Bob Purdy]

This is a single 2 node cluster with the 2 server and 2 EVAs less than 50 feet apart right now. Eventually one server and one EVA will be moved to another data center a block away from one another. This is a configuration we have used before with success so I have no concerns about it working. I am glad I had the chance to learn some new info from this one.

[Ted Buis]

MirrorDisk should do nicely in that environment and be less expensive. Normally, MirrorDisk will help performance with reads (up to 30%) and hurt write performance (~10%) assuming that you have the paths on separate HBAs. You may also be able to increase the queue depth on those devices beyond the default of 8 per LUN. I think the EVA's depth is 2048, so if you multiply 8 times the number of LUNs, you will see what you are leaving unused, and can adjust accordingly.

[Bob Purdy]

Actually we already have the mirror software installed on the servers so it will not cost anything extra to use it on the EVAs as well.

[Ali Merchant]

1) Does HP-UX Mirroring use synchronous or asynchrous replication?

2) Will the host take a performance hit if the data is to be written to the production and DR disks in terms of reads and writes simultaenously? When I think of HP-UX Mirroring the first word that comes to mind is "disk thrashing" I could be wrong?

3)How reliable is the data replication with HP-UX Mirroring ? What If there is an issue with the ISL between 2 sites? Is HP-UX Mirroring a smart replication technology built in the system or its dumb to pass the data and realize that the link is down and cause confusion.Does it hold the data in the queue if there is traffic congestion. Does it hold the data in the logs so once the link comes back up it can then pump information to the DR Site.

4)What if the primary EVA had 300 GB Fiber Channel Disks and DR EVA had 250 GB FATA Disks. How does HP-UX handle the mirroring of its disk pair in relation to peformance when compared with continuous access which is controller based replication

5) Are there any advantages to installing RSM Agent 1.1 on HP-UX if business copy functionality like snapclone and snapshot is rarely used but CA is a big piece for replication. Does the agent facilitate in dr tests or during failback and failover procedures? what purpose does the agents really serve in relation to CA

6) What happens if there is pending transaction that is in the process of being tranfered to the DR site and the ISL links breaks. And at the same time one of the mirror sets on the DR has hardware problems How does HP-UX Mirroring handle that when the link is established but its partner in the volume group is not up. And what happens after the hardware issues is resolved on the DR Site in terms of replication

[Ted Buis]

1) MirrorDisk is synchronous since it is host based.

2) If there are two independent paths (HBAs), then MirrorDisk will have about a 10% penalty for the overhead of dual writes or more if there is substantial distance involved. However, on reads, MirrorDisk will have about a 30% advantage since it will read from the least busy path. The distance that Bob is doing of 50 ft now is not a problem and can easily be extended to 300m later or more with switches with long wave FC.

3) MirrorDisk/UX is very reliable and stable with more than a decade of use. MirrorDisk will re-sync the mirrored lvols once the path is restored.

4) Different size disks don't matter to MirrorDisk/UX, since it is mirroring at the logical volume level, not the physical disk or LUN. MirrorDisk has nothing to do, and no knowledge of Business Copy or CA which are array based mirroring and replication products. Those tools have their own advantages, and should be considered on their merits for the particular application.

5) I am not familiar with RSM agent 1.1, but ServiceGuard can function very nicely with MirrorDisk/UX. However, if CA is going to be used on the EVA then MetroCluster extensions for the EVA are needed. If two separate clusters are being considered and/or distances >100km then Continental Clusters should be considered.

In the US, HP has a High Availability team that can help with more complex environments, and I suggest that you involve them if you are here. There may be similar teams for non-US sales.