HPE Morpheus VM Essentials Software
1870510 Members
3823 Online
110549 Solutions
New Discussion

Re: HVM iptables rules

 
ChrisDavis
Occasional Advisor

HVM iptables rules

Hi All,

We have two independant Morpheus Enterprise 9.1.0.1 installs each with a three node HVM cluster layout 2.0 (26.04)
Lets call them Cluster A and Cluster B. The Morpheus instance managing Cluster A, also has another cluster (layout 1.3) specifically for SDN.

I can see from the Cluster Layout blueprint (1.3 & 2.0) that it uses/installs iptables-persistent, and there are a number of rules added to allow connectivity.
image.png
It also shows that is saves the iptable rules after updating.
We recently had an issue where we were intermittently loosing connectivity (quorum) to some hosts, and suspected MTU issues, which we fixed.
Sometime after this we experienced hosts dropping from the cluster (although quorum was OK)
We could not ping from one host to the other, and https queries to the agent (:7443) on some hosts failed.
Comparing host configuration between clusters we found that the iptables were significantly different.
On the working cluster (B) the hosts had the rules from the cluster layout.
On the failing cluster (A) the hosts did not have these rules, had references to hosts in another cluster (SDN)

Cluster A example..
image.png

Cluster B example..image.png

I assume that the Cluster A hosts did initially have the cluster layout rules ??
Where have they gone?
What processes modify these rules?
Does the saved / persistent rules get reloaded at boot?
Does the running rules get saved to the saved files?

TIA
Chris




8 REPLIES 8
CalvinZito
Neighborhood Moderator

Re: HVM iptables rules

Here's my Copilot notebook response:

This doesn't look right to me. The cluster provisioning process includes firewall configuration during Host Preparation, so I would expect the required cluster communication rules to have been established when the cluster was created.
 
What concerns me more is that Cluster A now appears to have firewall rules referencing hosts from the separate SDN cluster, while Cluster B retains the expected rules. I don't see anything in the 9.1 documentation that explains that as expected behavior.
 
The documentation also doesn't give me enough to say what process subsequently changed those rules, or whether the persistent rules should be automatically regenerated. I wouldn't recommend manually copying the rules from Cluster B to Cluster A until we understand what owns them, since another process could simply change them again.
 
This looks like something that may need engineering investigation. If you have HPE Support entitlement, I'd open a case. If this is a trial environment without Support, let me know and I'll see if I can get this in front of the appropriate HPE Morpheus team.
 
In the meantime, if possible, preserve the current iptables-save output and /etc/iptables/rules.v4 from all three Cluster A hosts before making further changes.
 
@DiegoDelgado - anything you'd add?


I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo
ChrisDavis
Occasional Advisor

Re: HVM iptables rules

Hi Calvin,

I agree that copying ClusterB rules isn't ideal, however without applying the rules the hosts will not be seen as active cluster members, or maintain Quorum. Also if the hosts need a reboot, then the rules are lost, as the running iptables rules haven't been copied to the /etc/iptables/rules.v4 save location that is specified in the cluster layout blueprint scripts. (don't want to overwrite until we have a solution / engineering input)
...but it's a workaround I must use ATM, as we have limited time running the POC, and were also beta-testing another HPE product we already use with VMWare on Morpheus, and have a feedback deadline we need to meet.

Yes, these are POC/trial environments without support.

CalvinZito
Neighborhood Moderator

Re: HVM iptables rules

Thanks, that additional detail helps. Given the testing deadlines, I understand why you're restoring the rules as a workaround to keep moving.

The fact that the cluster works after restoring the rules, but loses them again after a reboot, makes me even less comfortable suggesting additional changes without understanding what's managing the firewall configuration. The documentation says firewall configuration is part of Host Preparation, but doesn't explain what should be rewriting or persisting these rules afterward. 
 
 
Since this is a trial without Support entitlement, I've tagged Diego Delgado from the VM Essentials PM team. I think this needs product input so we can understand what's happening rather than keep working around it.
 


I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo
ChrisDavis
Occasional Advisor

Re: HVM iptables rules

I might have found the reason the firewall rules changed from the cloud layout version...
The cloud the cluster is associated to has a local firewall enable setting.

I think as part of an SDN test (my memory is a little hazy) that the setting was enabled.
Not sure if this was part of the documented setup.
The cloud has two clusters. An SDN and non-SDN cluster associated to it.

The questions still remain..
What is the default behavoir of Morpheus / HVM in relation to firewall changes?
What processes modify the hosts IP tables?
If I need to change rules on the hosts, what is the process?
...and now additionally
What does the Clouds Local Firewall enablement actually do? The hosts already have local firewall enabled by default, and this switch changes the default to a non-functioning set of rules that break the cluster?

image.png

ChrisDavis
Occasional Advisor

Re: HVM iptables rules

Just found this too… So I now know that the enablement does, so it's just a case of how, when and what utilises this feature to change the rules to non-functioning rules?

 

Firewall Management

When “Local Firewall” is enabled for a Cloud (see the Advanced Options section on the Add/Edit Cloud modal), the HPE Morpheus Software Agent can manage host or VM IP Table (firewall) rules for Linux workloads.

From <https://docs.morpheusdata.com/en/9.1/getting_started/functionality/agent/features.html>

... OR maybe it's this host setting...
image.png

…or maybe they both do, and they're clashing ?

CalvinZito
Neighborhood Moderator

Re: HVM iptables rules

Can you telll me what license you are using? You've said a few things that have me wondering - I originally thought you had the VM Essentials trial but SDN isn't supported with that. 



I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo
ChrisDavis
Occasional Advisor

Re: HVM iptables rules

... using a rolling temporary Enterprise Trial license...

image.png
CalvinZito
Neighborhood Moderator

Re: HVM iptables rules

Looks like you have the Enterprise POC license. I'll reach out to the team who manages those to see what help they can give you. 



I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo