A smart move for the enterprise network: Kubernetes multi-zone deployment with Calico

[ServicesExperts]

Kubernetes isn’t designed for multi-tenancy, but here’s a way to achieve zone-based isolation of workloads within the same Kubernetes cluster.

by Hans Emmanuel, Chief Solution Architect, Cloud Native Computing Practice, HPE Pointnext Services

As we know, Kubernetes doesn’t provide multi-tenancy out of the box. There are some workarounds for achieving multi-tenancy using different tenancy models, depending on the requirement. But the truth is that Kubernetes is not designed in a multi-tenant pattern. And when it comes to networking, the container networking interface (CNI) spec is not concerned about the network segregation of workloads. So, the Kubernetes CNIs are not intended to provide L2/L3 network isolation out of the box. The CNI-backed network policies are the Kubernetes object used for network-level isolation of workloads, which in most cases leverages the firewall rules in worker nodes.

But what if it's required to deploy worker nodes across multiple network zones, due to various concerns from application owners and other stakeholders? And in some cases – for example, to be aligned with different compliance requirements – it is mandatory to have separation of physical and network workloads.

Usually, separate Kubernetes clusters (in a cluster-as-a-service model) are used when it’s pivotal to have the isolation of workloads. But sometimes running and managing multiple Kubernetes clusters causes some operational burden.

In this blog, I’ll explain an approach that HPE used for one of our customers, with Calico CNI in BGP (border gateway protocol) mode to achieve the zone-based isolation of workloads within the same Kubernetes cluster.

We used HPE ProLiant DL360 Gen10 servers as the worker nodes. The diagram below shows a high-level view of the deployment topology. Here the worker nodes are deployed across different isolated network zones. Inter-zone traffic will be crossing the core firewall. The key point in this topology is the BGP route reflectors per zone. As shown in the diagram, worker nodes in the yellow zone are peered to the corresponding route reflectors, which will make sure that the Calico-advertised routes will be contained within the zone.

The datacentre is using leaf-spine topology and virtual routing and forwarding (VRFs) used in network fabrics for the multi-tenancy at L3 level. Route reflectors are peered towards corresponding VRFs in border leaf switches. All the inter-VRF (zone) traffic will be crossing the core FW, and only permitted traffic will cross it.

In this topology, the worker nodes in a zone don’t have any idea about the workloads running in worker nodes in other zones. Even if a workload in one zone needs to talk to a workload in another zone, it would be routed towards core FW and only the allowed traffic will flow.

Conclusion: Though multi-tenancy is not an out-of-the-box solution in Kubernetes, sometimes we need to extend it to meet technical expectations and security requirements. Here we achieved this with Calico CNI, with its extensive BGP capabilities.

Technology services consulting from HPE Advisory & Professional Services can help you get the most out of your Kubernetes multi-tenancy design and implementation. We understand that once cloud-native workloads reach production maturity, it is inevitable to design and implement a higher level of network security and performance standards. The Global Cloud-Native Computing practice in HPE Advisory & Professional Services can help you build your enterprise-grade network design and configuration, drawing on our deep expertise and experience of cloud-native computing technologies.

To learn more, see our HPE Container Adoption Service solution brief.

Learn more about HPE Pointnext Services and how we help you stay ahead of what's next.

Hans Emmanuel is a Chief Solution Architect in HPE’s Cloud Native Computing Practice Area, HPE Pointnext Advisory & Professional Services. He started his career as a Linux server engineer back in 2010 and has since worked on a variety of private cloud solutions and cloud-native technologies. Hans has worked on DevOps and development projects; design and implementation of Devops/DevSecOps pipelines; and self-managed Kubernetes clusters.

Services Experts
Hewlett Packard Enterprise

twitter.com/HPE_Pointnext
linkedin.com/showcase/hpe-pointnext-services/
hpe.com/pointnext