Cluster in Campus Disaster Recovery

[PSS SYS ADMIN]

Hi everyone,
I have to create a new hpux 11i cluster with DataProtector using 2 L2000 and disks on EMC2 DMX in srdf with another DMX.
The host1 and the DMX1 are in the building1, the host2 and the DMX2 are in the building2.
Assuming the host1 and the DMX1 are down, there's a solution that could auto switch the package from host1 to host2 and use automatically the disks on the DMX2?

Regards...
PSS

[Matti_Kurkela]

Sounds like a textbook case for MetroCluster if you ask me.

MetroCluster is essentially ServiceGuard with the extra functionality to "talk" to the storage system, to make it aware of which storage has the more current information.

If you choose to implement a MetroCluster, I'd recommend that you build in a way to test the disk failover. Perhaps you could build the cluster with two packages: the real production package and one small "failover test" package, which would have no other function but would have all the disk-related parameters (except disk size - the test package could be very small) configured identically to the production.

Each time the HP-UX systems are patched, storage firmware is upgraded, or any other significant changes are done, you can use the test package to ensure that the disk failover mechanism works. You can also use it to familiarize the sysadmins to the MetroCluster functions.

If your systems are anything like ours, the chances to test the actual failover in production might be strictly limited - you will need an alternative way to test the disk failover, which is the critical event in MetroCluster.

To get the maximum utility out of MetroCluster, you need to be 100% confident it is set up correctly. If this is the only MetroCluster you have, it is very important it is easy to test.

[PSS SYS ADMIN]

In the DataProtector support matrix only MC/Service Guard is supported.

[Chan 007]

Hi PSS,

My question is are having the same subnet for the 2 Clusters? If not then are will end up in to clusters where only manual recovery is on. Like if one building is off then a SA will start the package on the Building 2.

Also, if you have a single cluster without metro cluster you may end up with Split Brain, both will go down due to quorum and try to build cluster. (Assuming 2 Servers in Site 1 and two in Site 2)

So unless you provide full information with how many system are in picture, networking information it is not possible for members to provide more information.

Chan

[PSS SYS ADMIN]

The host1 and host2 are on the same subnet but in 2 different builings (1 & 2).
The hosts will form a cluster with DataProtector using, in standard configuration, the disks on the Emc2 DMX1 that is in the building1.
This DMX1 is in srdf with the DMX2 that is in the building2.
If the host1 crashes the Dataprotector package cannot switch to the host2 using the disks on the DMX2 but using the disks on the DMX1.
I have to manually recreate the package to use the disks on the DMX2.
There's an "escamotage" to solve this limitation so the monitoring team could switch package and disks without rebuilnding the package?

[Sยภเl Kย๓คг]

Hi,

How far these buildings?, if they are so close and you have sufficient bandwidth, go for 2 node cluster with mirrored storage.

Regards,
Sunil

[PSS SYS ADMIN]

Hi SuniL,
the DMX1 and the DMX2 are connected in SAN with a 2Gb fc.

[Chan 007]

Hi PSS,

1. Copy you vg maps to the second server.
2. Unpresent the DMX1 to HOST2
3. Present DMX2 to HOST2
4. Ensure SRDF copies are doing fine.
5. Create VG
6. VGimport the VGs and vgchange it
7. Stop the SRDF for time being
8. Mount them.

The step 6 & 8 needs to be done in the DR

Chan

[Sยภเl Kย๓คг]

Hi,

I am aware of that. There are many solutions avaiable for you.

Why I suggested this solution, is to have very minimum user interaction, once after the site setup is done. If you go for an SRDF or such solution, you need more manuall interaction once the site is up ( During DR Also).

You need to present DMX1 and DMX2 disks to both Hosts. AT host1, create VGs, LVOLs and then mirror the storages. Get the VG Map files with VG-IDs. Deactivate the VG and import the VGs in host2 with this map file, with the mirror. Configure cluster as normal.

This would help you to retain a mirror copy of data and also a 2 node cluster.

Regards,
Sunil

[Steven E. Protter]

Shalom,

The MetroCluster or Continental Cluster is a good way to work with this situation.

Another is a high speed line between the systems and oracle parellel processing and regular rsync between the machines/clusters.

Regards,

Shmuel

[melvyn burnard]

Metrocluster is a Toolkit that you add on to be used withing a Serviceguard cluster, to allow packages to switch between nodes on sites that are geographically dispersed.
The data replication is not done using mirroring, but hardware replication such as EMC SRDF, which you have.
There should be no reason why DP would not be useable or supported in this environment.
Take a read of the Designing Disaster Tolerant Clusters at:
http://docs.hp.com/en/ha

[PSS SYS ADMIN]

for SuniL
your solution is very good but I have dubts about the cluster lock. on which disk will be configured?
And the performance on disk access?

for melvyn & others
the MetroCluster is in ServiceGuard or must be purchased apart?

[Chan 007]

Hi PSS,
For metro cluster to work you must have one more node (Minimum) - that is an arbitrator.
Even that will be part of the same network.
Ideally this can be placed in a 3rd Building to provide more BC.

See compatibility and requirements
http://docs.hp.com/en/B8325-90041/ch01s03.html

Chan

[melvyn burnard]

Metrocluster is an additional purchaseale piece of software.
Also, if you read the manual at:
http://docs.hp.com/en/B7660-90017/B7660-90017.pdf
you will get more information.
You do NOT need a third node for arbitration, you can use a Quorum Server, but it does need to be in a third location.

[PSS SYS ADMIN]

And what If I create 2 package in Serviceguard.
Dataprotector1 on storage DMX1 and Dataprotector2 on storage DMX2?
The DMX1 is in srdf with DMX2.
The package Dataprotector1 is the standard one.
At the DMX crash, presenting the DMX2 disks, the package2 could be started.
My doubt it's if the operation to make the disks on DMX2 visible to the hosts could be done by a script.

[Chan 007 · Accepted solution]

Hi PSS,

I would like a SA to do the job. As you are in a BC/DR which is business critical, how can you depend on operations to do this task.

Some times the disks may need a fsck, sometimes you might need to do a vgchange manually, fewer time fs will not mount.

So even though it is possible to have a script for this solution and this will work, I recommend this should be a Unix SA job. This will also help SA to plan for restoring back when the Primary site is ready.

Chan

[melvyn burnard]

well in Metrocluster, you would just fail the package over from node 1 (site1) to node2 (site2) the Metrocluster does all of the work for you

[PSS SYS ADMIN]

SuniL,
I need to clear my doubts about the cluster lock disk on the solution you suggested.
I'll have to create 2 cluster lock disks for every storage?
And how is the performance on disk access with software mirroring?

[Chan 007]

Hi PSS,

On Performance on mirroring- Use Hardware RAID/Mirroring. That will be better. In fact DMX array will provide that.

On CLuster locks please see the link below
Melvyn's answer will give you some boost.

http://forums1.itrc.hp.com/service/forums/bizsupport/questionanswer.do?threadId=944696

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=657875

Chan