OmniStack down - how to act?

[elange]

Hello,

we still have a cluster of Legacy Hosts (Cisco) to be replaced next year. With the support finally ending in September we are now out of it.

So of course one of the accelerator cards had to die on us (supercap message) And also on the weekend,.

While i will try to buy some best effort support hours/days on monday from HPE of course now its the weekend and i am not sure how to continue for now. I do have external storage available and i could at least vmotion a good deal  of VMs from that Simplivity cluster storage to the other external storage..

However i am not sure whats going to happen once we get the other node up and running again. Will it cause issues if i storage migrate these VMs (currently still running as another node took over the storage IP from the OmniStack that went down) once the other node comes up? I would expect not because the node that went down i would expect would get the  info of the in between changes (vmotions) from the still running nodes.

The goal will be if i get support to remove the faulty node from the cluster (its a 5 node cluste and we can do that as basicly the cluster includes one Host thats basicly running only one VM anyway. We should have enough space to remove that one node.also available for the cluster recareat secondary datasets over the 4 remaining nodes.

Since the Federation will refuse to remove the node during its faulty state. i dont dare to force the removal now.

So should i just move as many VMs as possible to the other storage or will i creater a bigger issue?

Any help would really be appreciated.

Best regards,

ELA

[gustenar]

Hello @elange 

To me it's safer to try to have a healthy federation before attempting storage Vmotions. If you plan on removing the host that had the TIA card failure, and if you are getting support assistance, make sure the replicas are recreated in other nodes and all VMs have storage HA before moving them. 

[elange · Accepted solution]

Quick update:

The issue has been resolved with the Support. We removed the Host successfully.

While we thought we should have enough Space in the remaining Federation for rebuilding the backups and secondary datasets that was not the case. due to the current balancing. So we ended up with Hosts having way over 30 percent while the largest VM was rebuild on the one with the lowest ressources.  Hence i had an interesting night deleting stuff when reaching the 10% limit and being unable to call the support as out of contract you can buy support but its definately business hours only.

So i found some stuff to delete in that night and it worked out ok.

Also we found that certificates were expired inside of Simplivity on all Hosts in all clusters so we had them replaced too.

Still a bit of a nightmare as it took us 5 days to get support for the legacy nodes.

Waiting for the delivery of the new systems and hoping for no further cards dying before we get them.