VME Cluster maintenance - how to reboot one node at a time

[CricutTech]

We have a new (4) node cluster. I put (1) node into maintenance mode and from the VME told that node to stop. That did not go well. It took a long time to reboot with a lot of sigkill calls, failed unmounts and term timeouts. Once the node was back online the cluster looked good (pcs cluster status) but all of the VM's had been stopped. I had to manually restart the VME VM.

What is the proper way to do rolling server (host) maintenance in a VME cluster? Obviously the way I did it was not correct. Thanks.

PT

[DA9]

Related question, how is Ubuntu patching on the hosts done?  What is the recomended way to perform host maintenance?

[fbroussey]

Hi all,

I have the same need. How to reboot each host without going into the host console ? I don't see any restart option in the vmemanager web interface ...

Thanks

[Arnout_Verbeken]

There is indeed no restart/poweroff option in the VM Essentials Manager.  So you have to reboot via ssh or via a console (Open Console in VME or via iLO Console).

Procedure for patching/rebooting:

• Put host in maintance in VM Essentials Manager.
• Open Console
• apt update
• apt upgrade
• reboot
  While rebooting you can check the "pcs status" on another node.  The node you are rebooting should be in a "stopped" state.
  If you see a different state, you can use the following command:
  pcs node standby NODENAME
  
• Once the HVM node is back online,  let it join the cluster again with
  pcs node unstandby NODENAME
  and verify with pcs status that it has joined the cluster
• Exit maintenance from VM Essentials Manager.
• Proceed to next HVM node

In all my tests and reboots, the HVM node went cleanly in a stopped state.

[NayakSandeep]

There is an option to Stop & Start the Host (Infrastructure --> Hosts --> Select the host and go to actions. You will see option to stop/start.

But this is not the recommended method when the node is part of the cluster.

If you want to take the node out for maintenance when it is part of a cluster, you may do so. Follow this link.

https://support.hpe.com/hpesc/public/docDisplay?docId=sd00006775en_us&page=GUID-967D7E5A-D701-4580-BFCB-0EB3588D2725.html 

[DA9]

Arnout,
I agree with you that these are the steps to reboot a host, and it has worked for me too.  But, have you watched the console of the host when it is trying to shutdown?  For me, the host is having issues with unmounting file systems and stopping processes.  It is not a clean shutdown.  I have opened a case with Support on this.

Would it be possible for you to run a test and watch your host as it reboots?  Thanks.

[fbroussey]

Thans for the answer. It's a shame that HPE didn't provide a Reboot option...

[dya]

When restarting a node, is it sufficient to simply place the host into maintenance mode using VM Essentials Manager?

While placing it into maintenance mode will prevent virtual machines from running on that node, won't the Pacemaker resources remain active?

When restarting a node in a Pacemaker cluster, the typical procedure is as follows:

1. pcs node standby <node>
2. reboot
3. pcs node unstandby

Given this, wouldn't it be safer to both put the host into maintenance mode via VM Essentials Manager and also execute `pcs node standby <node>` when restarting a node?

This is an environment using iSCSI connections with GFS2 for the datastore.

[Arnout_Verbeken]

The PCS ressources remaing active.  The management layer will prevent that any VM's run on a host in maintenance.

I have never put a host in "pcs standby" when patching/rebooting, but I guess it could indeed be an extra safety measure.
There is no doc describing that it's actually needed, but you can certainly do those actions.

[CricutTech]

Have you watched the console of the host when it reboots? Sigkills and timeouts trying to unmount drives. It's not a clean shutdown.

Get Outlook for Android<>
________________________________
The information contained in this E-mail is intended only for the use of the individual(s) named above and may contain either privileged or confidential information. If the person actually receiving this E-mail or any other reader of the E-mail is not the named recipient or the employee or agent responsible to deliver it to the named recipient, any use, dissemination, distribution, or copying of the communication is strictly prohibited. If you have received this communication in error, please immediately notify us by telephone 1-877-727-4288 or E-mail and return the original message to us via interoffice or U.S. mail.