OpenVMS cluster & CA EVA - system disks in a DR group - a chicken and egg problem

[Steven Fletcher_1]

Hi all,

We have a two node OpenVMS Alpha 8.2 cluster, and two EVA5000's with Continuous Access EVA running VCS 3.026. Both OpenVMS nodes have dual HBA connections to the SAN.

Both nodes boot from seperate system disks on our primary EVA, with a CA EVA Data Replication group configured to replicate all of the cluster's system & data disks onto our secondary EVA.

However, CA EVA only actively presents paths to the disks on the current source EVA. Paths to the destination EVA are not presented until the destination EVA becomes the source EVA. i.e. in our case the OpenVMS hosts will only ever see 4 active paths to their system disks on the source EVA.

HP's documentation suggests that MC SYSMAN IO AUTOCONFIGURE/LOG should be run on the OpenVMS node after a DR group failover to detect the new paths to the disks - however we have a chicken & egg situation - it seems to be impossible to run that command (or even to login to the node) after a DR group failover, because as far as OpenVMS is concerned, the node's known paths to its system disk have dissapeared and all disks (including the system disk) have gone into a mount verify state.

We have Replication Solutions Manager installed on the nodes and similarly, it fails to respond to any remote RSM commands (such as DiscoverDiskDevicesForDRGroup) once the system disk has gone into a mount verify state.

So, the only solution available to us is to shutdown both nodes, failover the DR groups, and boot up again...

Is there no other solution available to us? One option might be to manually define paths to a DGA device, but if this is possible then I don't know how to find out about it!

[EdgarZamora]

This is a tough problem and the possible solutions aren't pretty. One possible solution might be to volume shadow your system disk to a local SCSI disk, too. In the event of a disaster with your primary EVA once the system disk on the EVA times out your system should theoretically continue with the local system disk. Then you should be able to find your new paths. Something like that.

By the way, do you have remote alphas at the secondary EVA site? Might be a good idea for DR purposes.

[Jan van den Ende]

Craig,

this is one aspect of the show, that CA and VMS-Style clustering are mutually contradictionary.

Just configure your remote disks as remote SAN devices, and forget about CA.

You did not specify the intersite distance, but up to 10, maybe 15 KM, just ignore it.
Then up to some 500 KM round trip, use shadowing's SITE mechanism.

And beyond that, you REALLY do need REAL, in-depth, specialized advise. In view of the other costs involved in such setup, it is relatively peanuts.

Should that be the case, come back for some names.

Proost.

Have one on me.

jpe

[Steven Fletcher_1]

Edgar,

Yes, we've got a split site setup - OpenVMS node 1 and EVA1 are on the primary site, node 2 and EVA2 are on the secondary site. They are separated by a distance of around 8km. Both nodes normally reference the disks on EVA1.

I may adapt your idea by removing both of the current system disks from the EVA DR group and using volume shadowing by writing to disks on both EVA's, if that's possible. (having a local system disk is another piece of hardware to look after!)

Jan,

Sorry, but I don't understand why you say CA & clustering are contradictory, could you explain further?

Also you've totally thrown me by suggesting we configure "remote disks as remote SAN devices" - am I missing something, because as far as I'm aware, the EVA disks are already remote from the system, and are "SAN devices"?

[Jan van den Ende]

Craig,

>>>
Sorry, but I don't understand why you say CA & clustering are contradictory, could you explain further?
<<<

In short (and leaving out a lot of detail) because CA is a -replication- tool, with an active (master) member, and an INactive (slave) member; whereas with Shadowing ALL members are (well, nearly) equivalent and ACTIVE.
So, with CA, if you loose the active member, a failover is needed (and MUCH worse, after repair (either) failback (or reconfiguration to reverse master/slave).
With Shadowing, if ANY member survives, the shadowset survives.

>>>
Also you've totally thrown me by suggesting we configure "remote disks as remote SAN devices" - am I missing something, because as far as I'm aware, the EVA disks are already remote from the system, and are "SAN devices"?
<<<

Ambiguity is mine. I should have written "disks at remote site".

>>>
Both nodes normally reference the disks on EVA1.
<<<

So, nodes at both locations can reach the EVA at the local and at the remote site? Very good! Just shadow each disk to have a member at each SAN.

-- Special note if you plan to do some manipulation of the shadowsets for backup activities: do _NOT_ reduce a shadowset to just one member. Much better to have a 3rd member leave and join as needed. The current disk prices do not warrant taking any risks!

hth

Success!

Proost.

Have one on me.

jpe

[Steven Fletcher_1]

Jan,

Now you describe it, I think I know why shadowing everything is not a good idea for us. The company application design means that some of the disks must be mounted (and written) by both nodes at the same time. Perhaps I should have mentioned this earlier!

As I understand it, with shadowing, if the inter-site link were to fail even for a short period, both nodes could continue to write to the surviving shadow set member independently of each other, and we would be left with a situation where the database would be left in an inconsistent state at both sites - it would be impossible to merge the data after the link was restored.

We may be able to get away with shadowing for just the system disks in order to keep our nodes alive during a planned failover, but I think it would be impossible to drop CA entirely (because we mount disks on more than one node, then we do need to retain some manual control over which data is 'active' during an unplanned failover)

[Jan van den Ende]

Craig,

>>>
As I understand it, with shadowing, if the inter-site link were to fail even for a short period, both nodes could continue to write to the surviving shadow set member independently of each other, and we would be left with a situation where the database would be left in an inconsistent state at both sites - it would be impossible to merge the data after the link was restored.
<<<

You are doing both Clustering and Shadowing a gross injustice here!!!

The whole concept of the Quorum scheme is to prevent JUST THAT at ANY cost. (very simplified:) If connectivity is lost, EVERYTHING freezes (except trying to re-establish connectivity) for a configurable amount of time, and if things are not normalised in time, one or more node(s) are kicked out, probably losing some sessions, but GUARANTEEING data integrity. (it will be up to the database to handle transaction integrity, ie, rollback if needed).

The phenomenon you seem to fear, is what can happen to (undeservedly termed) "clustering" on various other OSses.

If it can give you some peace of mind:
Our cluster of 4 nodes (two nodes each at approx 8 KM separate) on two SANs with connectivity to both sites shares ALL disks, each with shadow members on both SANs.
We (have to) run 5 DIFFERENT databases.
Of those, the nature of the PROGRESS database is such that the engine MUST run on one node, and every userprocess has to communicate with that process.
But Rdb, DBMS, and RMS are fully operational on all nodes simultaniously.

hth

Proost.

Have one on me.

jpe

[The Brit · Accepted solution]

Craig,
I think you need to research the HBVS solution further. Clearly the CA solution is not working for you.
Create each volume as a two-unit set where one unit is from each of your EVA5K's and mount all of the disks on all of the systems at both sites (note the 8km separation can be ignored).

Add a quorum disk(/node) at one of your sites, (or preferably at a completely different site) to act as the tie-breaker in the event that you lose your intersite link, and make sure that you have the "votes" properly assigned. (Note also that the Quorum Disk must be visible to all systems at both sites (and cannot be a shadowed volume).)

Now, if you lose SAN connectivity, then some of your units will drop out of their respective volumes (after SHADOW_MBR_TMO seconds). The volumes involved will go to mount verify and (if connectivity is not restored within MVTIMEOUT) the system will eject the unit which is causing the problem (if it didn't leave after SHADOW_MBR_TMO).

Assuming that your Cluster Interconnect is still intact, then all systems will continue to function using the unit supplied by the surviving EVA5K. If the IC is the problem, then the quorum calculation will keep your primary site functioning while the remote site will enter a "quorum hang". Users at the remote site could simply reconnect to the primary site in order to continue working.

When connectivity is restored, then the remote system will become active again, however the units being supplied by the remote EVA5K will have to re-synch.

Dave.

[Steven Fletcher_1]

Thanks to all for their input especially regarding volume shadowing. We'll certainly look into it - the more I read about it the more I understand why it would work :-)

[Steven Fletcher_1]

Will look into HBVS for further information