How to shut down and start a VME cluster

[kurotan]

Is there any documentation on how to shut down and start a VME cluster?

[dccolpitts]

I've been shutting my testing infrastructure every night when I call it quits for the day.   My approach is to shutdown all the VMs except the Manager in the Manager WebUI, and then take note of which host the Manager is on.  Then I Putty the host and "sudo virsh list --all"   (that is a double dash incase the text editor here screws with it) to confirm the Manager is still on that host, and then I run "sudo virsh shutdown ManagerName".   It seems to take between 1 and 2 minutes to shutdown, which I confirm with another "sudo virsh list --all".  Once all the VMs are off, I "sudo shutdown --poweroff" on each host.  Once those are shutdown and powered off, I then shutdown my external storage in the normal manner (in my case, it's a MSA2062, so I Putty a controller and "shutdown both"), wait 10 minutes and then I turn off the breaker that supplies power to my storage, hosts, and switches.  When I come in the next morning, I just flip the master breaker back on and wait 10 minutes (my hosts are configured to wait 2 minutes before automatically powering on so that the SAN has a head start on them).   Once everything is back on, I Putty my host, do another "sudo virsh list --all" and find out which one is hosting the manager and then "sudo virsh start ManagerName".  To be honest, I haven't looked yet to see how (or if) VMs can automatically be started when the host starts.  That was actually on today's list of things to check.

dcc

[kurotan]

Thank you for your reply.

I'm currently shutting down in the same way.

I have a concern.

When I look at the VME manager processes, it seems that MySQL, Nginx, RabbitMQ, Opensearch server, etc. are running.

Is there a correct shutdown procedure to maintain data integrity?

[Arnout_Verbeken]

Kurotan,

"virsh shutdown" does a gracefull shutdown of the VM.
"virsh destroy" does a hard stop of the VM.
So you should be good with "virsh shutdown"

You can also just shutdown the VME Manager from within VME itself (since you should have converted it from the start to "Managed").  You can do this with:
Stop/Start/Restart Server  -->Stops, Starts or Restarts the Virtual Machine.  --> https://support.hpe.com/hpesc/public/docDisplay?docId=sd00006560en_us&page=GUID-FB7D15CC-A606-42DC-9DBF-27778D20282D.html

This option will do the following:
If there is access to the guest, a guest os shutdown/restart is attempted.  Otherwise we will fall back and call the hypervisor for a force.

I hope this helps.

[kurotan]

Thank you for your reply.

So you're saying that there's no need to stop the services individually and that it's okay to stop them with virsh shutdown?

[Arnout_Verbeken]

dccolpitts,

By default, VME will not autostart VM's once the hypervisor starts.  It would be rather silly that VME decides to poweron all VM's that you have shutted down yourself before.

Note that this is different from the  "Automatically Poweron VM's" setting in Cloud and Cluster.  This setting is actually your HA setting, meaning thet VME will check his last-known state of the VM and will take actions accordingly.  So if for VME, a VM has a last-known state of "ON" and at next check by VME, the VM is powered-off, VME will start the VM himself.
Autostart will start vms if they are powered off for a reason outside of a user choosing stop in the Morpheus ui.  Without a heartbeat datastore, this only occurs at cluster sync.  With heartbeat datastores, this happens withing seconds.

--> conclusion:  Make sure you shutdown your VMs from within VME if possible so VME knows the correct state. 

If you really want to start some/all VM's automatic during poweron of the host, I guess you could write a bash script that runs at boot.  It could containm "virsh list --all" to get a list of the VM's and a loop to "virsh start VMname".

[CalvinZito]

My colleague @Arnout_Verbeken gave a complete answer but I'm looking at the documentation and discovered something:

Looking at the HPE Morpheus VM Essentials Software documentation for v8.0.8. Page 82 talks about going into Maintence Mode. Can't find page 82? When you're looking at the HTML document, under the "Sign out" button on the top right, there's a "Create PDF" button. That PDF has page numbers. I personally prefer a PDF over HTML docs. 

I'm also looking at the HPE Morpheus Enterprise Software docmentation for v8.0.8. Again, on page 387, there's a section there about HVM clusters and Maintence Mode.  Seems like there is a lot more information in the Morpheus Enterprise documentation about clusters than in the VM Essentials documentation. I've asked our docs person why don't we have the details in the Morpheus Enterprise doc in the VM Essentials doc. 

Hopefully we will soon but I'd suggest looking at the Morpheus Enterprise doc. And note that VM Essentials features are in Morpheus Enterprise. 

[dccolpitts]

Calvin - yeah - I think I pointed that out a couple of days ago on the private TechPro Slack channel.  The documentation posted for HPE VME verses Morpheus Enterprise is rather sparse.

And I agree with you - PDF is so much better than that silly web viewer.

dcc

[dccolpitts]

Arnout - rigth now with VMware (both stand alone host and vCenter managed hosts), if there is a site power outage and systems go down because the UPSes ran out of battery, when the power comes back on, the SAN and hosts automatically boot, and VMs start without any interaction from the IT team (yeah - it might be a VC HA process, I don't know).  So the use case would be if the power goes out, and the UPS management software gracefully shuts down the VMs including the VME Manager (or maybe there is no management software and the customer just wants stuff to turn back on), it would be nice for them to restart (or atleast the VME Manager at a minimum).  If we could designate VMs and start orders like a standard ESXI host, that would be great (so that AD servers come up first, then whatever the customers wants next...).

dcc