How to recover from a total cluster loss

[kurotan]

I would like to confirm about recovery.

I would like to know how to recover if the cluster environment cannot be restored due to a malfunction of the manager or hypervisor host.

I thought that I could recover it as follows.

・Rebuild the hypervisor host and manager
・Reconfigure the previously set backup and image buckets
・Run "Reindex all searchable data" in the utility in the manager settings

I thought that by doing the above, the virtual images and backups would be restored to a state where they could be accessed from the manager, but nothing was restored.

If I had a backup, I thought I would be able to create a VM again from it, but in this state it is not possible.

Is there any documentation that describes how to restore the environment?

[PeterTzvetanov]

Hi kurotan,

it always depends on what you have as environment and what is down. 

For example, you the VME is "toasted", but you have a backup of the DB to an external NFS or CIFS server, you could power off that VME appliance, install a new one and once you managed to login into the new one, you could follow the restore procedure and the rest will sync. If we are talking about server is "toasted" and had to be reinstalled, it can be just removed and readded again after the restore if the VME and its DB.

1. Install a fresh new VME with the same name and FQDN.
2. Don't create any group/clouds and only login over ssh and stop the morpheus-ui with morpheus-ctl stop morpheus-ui and upload the SQL backup to the DB.
3. Run a "cat /etc/morphes/morpheus-secrets.json"
4.Lets restore now the DB via "/opt/morpheus/embedded/mysql/bin/mysql -u morpheus -h 127.0.0.1 morpheus -p < /tmp/morpheus_backup.sql"
5. Once that is finished I would now start the morphes-ui via morpheus-ctl start morpheus-ui and monitor via "morpheus-ctl tail morpheus-ui" the startup.
6. Once that finished and we see the morpheus ASCII logo, we login into the GUI and monitor what is happening and how it is syncing with the hosts and we should be recovered. 

With regards

Peter Tzvetanov

Support Engineer

Peter.Tzvetanov@hpe.com

Technical Support Phone Numbers

HPE Operations

You can manage your cases on HPE Support Center

[kurotan]

Thanks for your reply.

"You could follow the restore procedure and the rest will sync."
Can you please tell me where this document is?

In the event of a catastrophic failure of a hypervisor host, I understand that I can remove the host from the cluster and add another host.

I'm assuming a catastrophic failure of the VME Manager.

In this case, I understand that I would need to delete the VME Manager, redeploy it on the existing hypervisor host,
and then recover the database, etc.

Since MySQL was running on the VME Manager, I would need to restore that data, but I lacked the information and targets needed to connect via CLI.

By default, an item called "Morpheus Appliance" was included in the backup.
https://support.hpe.com/hpesc/public/docDisplay?docId=sd00006691en_us&docLocale=en_US&page=GUID-1C1ADB18-710D-40A4-ADCA-660DC7988190.html

It says that it's backing up databases, etc.

Will it be possible to recognize the "Mourpheus Appliance" backup by re-registering the target storage with the same settings after rebuilding the VME Manager?

Unlike the VM, the backup I got was only available as a download, and there was no option for restore.

I tried downloading it, but it appears to be a MySQLDump file.
As I expected, recovery will be performed using mysqldump, but I still don't have enough information to operate it.

[PeterTzvetanov]

Hi kurotan,

regarding the backup & restore of the appliance, you need enable the backup of the DB of the backup job "Morpheus Appliance". This will save the SQL DB in a zip file that normally recommended if having an external NFS share or CIFS share or you download it yourself from time to time via the download button of the particular backup. The restore itself is the following in case of having the SQL available:

1. If the VME is totally toasted, you jsut shutdown via virsh.
2. Now we can install the new VME on another ubuntu host in the cluster(we will cleanup later the original).
3. Once we finished up the install itself, but haven't yet started to create any group and/or clouds, we login over ssh to the VME and also upload the SQL zip file to it via WinSCP.
4. Now we stop the morpheus-ui via morpheus-ctl stop morpheus-ui.
5. Run a "cat /etc/morpheus/morpheus-secrets.json" so that you have the mysql morpheus user password and also we can extract/unzip the SQL backup and rename it if you want to morpheus_backup.sql.
6. We run now "/opt/morpheus/embedded/mysql/bin/mysql -u morpheus -h 127.0.0.1 morpheus -p < /tmp/morpheus_backup.sql"
7. Once that is finished, we can run an "morpheus-ctl start morpheus-ui" and once it is finished with the service initialization, we can look into the GUI how it is syncing.

With regards

Peter Tzvetanov

Support Engineer

Peter.Tzvetanov@hpe.com

Technical Support Phone Numbers

HPE Operations

You can manage your cases on HPE Support Center

[kurotan]

Thank you for your replay.

I tried it.
I tried to rebuild the VME Manager using the hpe-vm command, but the network settings were not configured correctly, and I was unable to rebuild the VME Manager.

For example,
- The netplan/50-cloud-init.cfg file does not have a fixed IP address. (DHCP settings were configured.)
- The netplan/50-cloud-init.cfg file has a fixed IP address, but eth0 is not up. (I confirmed this by connecting to the VME Manager using the virsh console command.)

[PeterTzvetanov]

Hello kurotan,

send a mail to hpe-sw-trial-vmessentials@hpe.com and we will have look.

With regards

[kurotan]

For the VM manager virtual machine whose "50-cloud-init.cfg" file had DHCP settings,

I was able to manually edit the "50-cloud-init.cfg" settings and get the network up and running.

The WebUI seemed to be running, so I stopped the UI and then attempted to restore the database.
It took some time, but the database was successfully restored.

The cluster status seems a little strange, so I will investigate further,

but I wonder if there is a problem with the network settings of the manager deployed from the hpe-vm command menu being incorrect?

[zayanhani]

following this… I always assumed having a proper backup would be enough to spin everything back up, but looks like it’s more complicated than that. do the backups need to be linked to some specific metadata or config that gets lost during a total rebuild? curious if there is an order of operations or if something else needs to be restored first before the manager recognizes the backups.