SimpliVity OVC Shutdown times

[CadenLange]

Hi

My experience with OVC shutdown duration is that it takes quite a long time and I'll like to know if there are ways to speed this up.

With a two node cluster I can shutdown all the running VM's in about 5mins (I don't have that many) and this will leave me with just the 2 OVCs. I shutdown the first OVC using he svt-shutdown-safe option and this can take about 10-15mins. It seems a little quicker if I using the vCenter plugin to shut it down rather than SSH commands - is that expected?

However, the second OVC (which I must use SSH for as I have noe shutdown the vCenter server) takes about 20mins. So this means that shutting down the two OVCs can take over 30mins  - and this is after I've already stopped all my virtual machines. Is there a quicker way to gracefully shutdown the OVCs once I have no running VM's? I don't want to risk just powering off as this may impact the integrity of the environment when I start things back up again

many thanks

[gustenar]

Helllo @CadenLange 

Thank you for using HPE Simplivity Forum. 

The procedure you are using is the correct one to ensure a graceful shutdown of the OmniStack Controller. It takes time due to the internal operations it has to perform before shutting down the OVC. In general 15-20 minutes shutdown time is considered normal for the OVC.

Thanks.

[jlangmead]

Hi

Many thanks for the reply.

I just want to confirm that there isn't a way to reduce the shutdown time if it can be ensured that all the guest VMs are already shutdown - are there any manual commands perhaps to stop the svtfs and then to shutdown the OVC quicker?

The main reason for asking is that I need to reduce the UPS runtime for this procedure and I can shutdown all my vm's in under 10 mins but it then takes a further 40mins to shutdown the OVCs - seems a little crazy. I don't want to risk any corruption by risking the UPS run out of battery power before the OVCs and ESXi hosts have shutdown but equally struggling to justify the need for additional UPS runtime batteries just to allow the time for the OVCs to shutdown within an already shutdown environment.

thanks in advance

[Randolfini]

Hi Jlangmead,

I found that extended OVC shutdown times can be caused by OVC losing access to the arbiter and vCenter.

Are you trying to automate an auto-shutdown in case of power loss?

Are you running the arbiter and vCenter on this two node cluster?

If so, then here are my recommended steps to shutdown:

1. Migrate Arbiter and vCenter (either VCSA or Windows VM) to local storage of one of the two nodes.

2. Shut down the remaining VMs except the vCenter, Arbiter, and OVCs.

3. On the node NOT hosting the vCenter and Arbiter, gracefully shutdown the OVC

4. On the node HOSTING the vCenter and Arbiter, both of which are now on local storage, gracefully shutdown the OVC

5. Both OVCs should shut down in a reasonable time, I believe faster than 20 minutes each

6. Now you can shutdown gracefully the vCenter and Arbiter and proceed with host shutdowns if necessary

You don't want to force the shutdown faster with any commands. If it takes a long time then chances are the OVC(s) have split-brained because of lost access to vCenter and Arbiter.

[jlangmead]

Hi Randolfini

Yes - I need to automate the shutdown using the UPS functionality when it's running on battery and need to make sure the shutdown completes before the UPS battery is exhausted. The arbiter is running on a different server (not on the cluster), however vCenter is running on the cluster.

You're absolutely correct on shutdown times being reduced when vCenter is still up and the interesting thing is that both OVC's shutdown in under 3 mins in that scenario. The first OVC is fine as I can shut this down whilst vCenter is on the shared datastores and managed by the other node. The second OVC is still a problem though as it takes just as long to storage vmotion the vCenter server to the nodes local datastore as it does to shutdown the OVC without vCenter running - plus I don't really want to use the local datastore for anything other than the OVCs as HPE don't really like this - albeit even for a temporary location to help with the shutdown. Also it then means it needs to be moved off again when powering everything back on - but it was useful to do this to test the theory.

It does beg the question though as to what the last OVC is doing during the shutdown that means that it can shutdown in 3mins when vCenter is running but takes 20mins when it's not. Or perhaps it's to do with running the shutdown command as the svtcli user rather than administrator@vsphere.local??

thanks again - very interesting to know that actually shutdown times can be reduced to just a couple of minutes per OVC if vCenter can remain accessible to the final OVC.

thanks Jon

[wardspruytte]

Regarding the OVC shutdown times, my experience is that whenever the OVC's are running for quite some time (x weeks, months) without being restarted in the meantime, the shutdown takes a lot longer in controlled shutdown scenario.

I'm struggling to get APC Powerchute Network shutdown to shut all controllers down within the given timeframe. By Default Powerchute sets the shutdown time to 300sec and startup time to 480sec. Even after changing the 300sec to 480sec for shutdown time, in most cases it does not go well. 

In the recent past I've performed UPS tests using APC combined with Simplivity setups in both 2-node and 3-node clusters. In both situations, I've experienced issues whereas the shutdown is not being performed within the given time.

APC first performs a controlled shutdown of all VMs using the priority groups, where my vCenter is being shut together with the virtual Domain Controllers in the highest priority group, being last to go down. Then both OVC's are being shut by the APC software. vCenter is no longer available at that time, Arbiter is still connected. There is no option to shut vCenter down after the OVC's in my case.

In a 2-node cluster, one of the OVC's is taking way longer to shut down, if not both. 
In a 3-node cluster, one OVC was not shut down within the given time. 
Powerchute marks that ESXi host as failed to enter maintenance mode, and startup procedure is not performed correct.

adjusting the shutdown time to 20min might be a workaround, however I would like to point out that APC software is adding the startup time to the shutdown time by design, in order to get the complete shutdown time. This adds a whole lot of waiting time for the shutdown procedure and not all UPS devices are able to add this amount of extra time because of a limited runtime available.

Any other ways to have the OVC's shut down in a faster way? 

[Sunitha_Mod]

Hello @wardspruytte,

Thank you for posting! Since you have posted in an old topic and there is no response yet, I would recommend you create a new topic using the create "New Discussion" button, so the expert can check and guide you further.