HF20C - 6.0.0.400-991061-opt - sync repl - handover not working

[Fred Blum]

We have two sites, Site A with Array A, Storage Pool A, Datastore A and Site B with Array B Storage Pool B, Datastore B.

Array A and Storage Pool A are synchronously replicated to Array B and vice versa for Storage Pool B. 

We experienced a system crash at Site B so the upstream Storage pool B became unresponsive. Management interface communication on Array B appeared green but the interfaces to the disks were offline and Array B, Controller A was down.  Hardware status was not showing. As we could not get it back up we tried a handover of the upstream Storage Pool B to the downstream Array A. As the array B appeared down we did a forced handover also when the remote Array B is down.

Problem is this handover never succeeded and is still trying to execute. 

We had to create a clone of the last succesful remote snapshot of the downstream DatastoreB to a new DatastoreC to be able to mount that on VMware and get our VM's back on line. 

Problem is we can't clean up Array A or reconfigure it as the command queue is waiting for the handover operation to finish frist. That won't happen anymore after 2 days. 

Support isn't of much help as I was quoted support renewal in 2023 direct and again later by our reseller for 3x the price of the direct quote  We, reseller and customer, tried adressing this issue with HPE but never received an explanation.  I confirmed the direct quotation but HPE never made it effective which we have been raising with HPE since 2023..

Are there ways to clean up the Array B and change it to a stable standalone config with both the DatastoreA and cloned DatastoreC?

TIA,

Fred

[buzzsubash]

This issue needs extensive log verfication and real time troubleshooting over zoom/teams. But with the array out of support, it is going to be tough. The reason is, without looking at the logs from array, it is extremely difficult to share a workaround straight. 

I can list couple of items to be checked.

- Handover gets stuck, if there is excessive delay in replication. Can you check from array UI on monitor > Replication tab to see the volume/colelctions? If there is a real progress in replication, I would recommend to wait, if not may need reseeding.
- If you look at the Nimble CLI guide, we have an option to abort the handover, and once we do it, the volume collection should become online. (Please note, if handover is stuck due to any other issue, it may fail as well.)

Consider the above as a generic troubleshooting steps and implement at your own risk. And ensure proper backup of the data is in place before attempting any action. 

I can see the case you have created and is pending with OPS team, will keep it on my watch list.

[Fred Blum]

Hi Subash,

As I wrote the replication partner array was down and as I iniated the Handover with the option checked to do a handover even in case of the replication partner being down, the handover should not wait for replication or communication with the replication partner array but accept it as being down and force the handover. That did not happen.

Are you saying that Handover of a synchronously replicated volumes will not work as a Disaster Recovery strategy? That it will only work when both systems are online? hat was never communicated to me in the past.

Fred

How I miss the LEFTHAND days, inexpensive, trustworthy, delivered reliable active/active, autonomous failover 

[buzzsubash]

Fred,

That is correct, to use handover, the replication partners must be alive and sync. This is mentioned in the admin guide as well.

If a volume collection is out of sync (in this case), all you had to do was unconfigure synchronous replication for the volume collection, to access the downstream volume which is on Array A, Storage Pool A. 

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you are satisfied with the answers then kindly click the "Accept As Solution" button for the most helpful response so that it is beneficial to all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".