Block storage is unmounted

[kurotan]

I tried to connect to the manager this morning, but there was no response.
Upon checking, I found that block storage mounts had suddenly been disconnected from all hypervisor hosts.

When I looked with multipath -ll, the connections were still active.

However, the corosync service was stopped, and there was no pacemaker process.

Have you ever experienced a similar problem?

What could the cause be?

[PeterTzvetanov]

Hello kurotan,

can you have a look into the "dmesg -T" across the hosts. Also it is important on what type of servers(model + eventually virtualized on what on top) are you running the ubuntu as base for your VME cluster environment + storage backend information is important.

With regards

Peter Tzvetanov

Peter Tzvetanov

Support Engineer

Peter.Tzvetanov@hpe.com

Technical Support Phone Numbers

HPE Operations

You can manage your cases on HPE Support Center

[kurotan]

The environment is as follows:

Server: Cisco UCS UCSC-C240-M3S + DELL PowerEdgeR750

Storage: NetApp A150

This is not the configuration listed in the compatibility.

I wanted to proceed with the verification, so I rebuilt it, so I can't share the results of the command.

[PeterTzvetanov]

Important is that it isn't good to mix hosts(no matter if different vendors and/or same vendor as it will lead to different NIC orders, different CPU model/rev., device filenames etc.) in the VME cluster on the one side. Also highly recommended is to not have them too old because you have drivers implementation that needs to talk to the firmware of the hardware and in case of too old firmware and too new drivers, can lead to unexpected results. Also you need to check the "dmesg -T", because this might give you an indication what happend. Also crosscheck that the auto updates be disabled across all hosts as that can lead to pacemaker and corosync interactions:
root@morph-esx1:~# cat /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
APT::Periodic::Unattended-Upgrade "0";
root@morph-esx1:~#

With regards

Peter Tzvetanov

Support Engineer

Peter.Tzvetanov@hpe.com

Technical Support Phone Numbers

HPE Operations

You can manage your cases on HPE Support Center

[kurotan]

Thank you for your answer.

I understand that mixing models is not recommended, but in reality, I think models will be mixed when expanding or replacing a cluster, so I hope that restriction will be lifted.

Also, regarding CPU models, you need to be careful because the manager's "CPU MODEL" function is not working properly at the moment.

However, regarding NICs, even if the port device name is different for each hypervisor host, can't the problem be avoided by replacing it with a bond device?

(I think that even a single Ethernet port can be replaced with a bond device by setting the bond setting to active/standby.)

The hypervisor settings in my environment were as follows.

----------------------------
root@ucs-3:/etc/apt/apt.conf.d# more 20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
----------------------------
The manager settings were as follows.
----------------------------
root@vmcontrol:~# more /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::Enable "0";
----------------------------

Should I change both the hypervisor and manager to the following?
----------------------------
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
APT::Periodic::Unattended-Upgrade "0";
----------------------------

Also, I understand that disabling automatic updates means that I need to put the hypervisor or manager into maintenance mode and run apt update and apt upgrade (and reboot if necessary) periodically, is that correct?

[PeterTzvetanov]

"Also, I understand that disabling automatic updates means that I need to put the hypervisor or manager into maintenance mode and run apt update and apt upgrade (and reboot if necessary) periodically, is that correct?" - Yes, exactly..

Regarding the NICs and the bonds, I am not sure if it will be supported even that it sounds correctly, the cluster and the VM Essentials is aware of the NIC backends. About the CPU Model, I am aware of it, hopefully solved with the 8.0.8.

Peter Tzvetanov

Support Engineer

Peter.Tzvetanov@hpe.com

Technical Support Phone Numbers

HPE Operations

You can manage your cases on HPE Support Center

[kurotan]

Thank you for your replay.

So it's unclear whether the NIC will be supported.

Could you please answer this as well?

*************************

Should I change both the hypervisor and manager to the following?
----------------------------
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
APT::Periodic::Unattended-Upgrade "0";
----------------------------

[PeterTzvetanov]

Hi,

I don't think that it will  be ever supported. All KVM configuration require to have some specifics which MUST be applied across the cluster. Comparing it with VMware is a bad idea as the way how vSphere do the clustering is entirely different. Regarding the question about the disable of the auto updates is a MUST in a clustered environment, because if you want to update something, it needs maintenance window and downtime for the environment or the entity in question.

With regards

Peter Tzvetanov

Support Engineer

Peter.Tzvetanov@hpe.com

Technical Support Phone Numbers

HPE Operations

You can manage your cases on HPE Support Center

[kurotan]

I understand that the settings on the hypervisor host are required,

but if the manager is also required, it is desirable to disable automatic updates after deployment.

Is automatic update disabled even with the following settings?
----------------------------
root@vmcontrol:~# more /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::Enable "0";←This setting
----------------------------

[PeterTzvetanov · Accepted solution]

Regarding the question for the manager, the updates are disabled on the VME 8.0.6 anyway, but on the hosts it is ALWAYS recommended to disable all updates. If updates are needed, you will need downtime for the entire environment and then you can apply you settings.. This is NOT like VMware.

With regards

Peter Tzvetanov

Support Engineer

Peter.Tzvetanov@hpe.com

Technical Support Phone Numbers

HPE Operations

You can manage your cases on HPE Support Center

[kurotan]

Thanks for your reply.

It doesn't answer my question, but the above configuration information is for VME8.0.7.

In other words, I understand that this setting disables automatic updates.