Multi-Tenancy Patterns and Which One Is Correct for You

Namespaces

A Namespace is a a mechanism for grouping resources within a single cluster. You can add some isolation mechanisms and limits via RoleBindings that scope a role to a specific Namespace, Network policies and Quotas. They are very cheap and easy and as they are Kubernetes native, every controller already understands namespaces

Some of the core issues of using namespaces per tenant are the following:

• Cluster-scoped objects are shared. For example CustomResourceDefinition, ClusterRole or PersistentVolume, and admission webhook configs live outside any namespace. If Tenant A installs a CRD with the same group/version as one Tenant B needs at a different schema, you have a conflict with no boundary to save you. This is the single biggest structural limitation of namespace-based tenancy: There is no way to give two tenants different versions of the same CRD in one cluster.
• The API server, etcd, scheduler, and kubelet are shared. A tenant that manages to trigger API server overload affects everyone. There's no per-tenant API server.
• Lack of network segmentation. Without a CNI that allows for Network Policies, there is no network boundary between namespaces and pods can communicate across them.
• Nodes are shared across namespaces. Unless using tains and tolerations to bound teams to nodes, there not only is a noisy neighbor risk, but if an attacker escapes to the underlying node, shared across namesapces, namespace level enforcements can theoretically be bypassed.

Because of that, namespaces are well used for internal teams with the same trust boundary, where gloabl resources dont need to be different by tenant. To have some level of security, it is advised to bind roles to namespaces, use networkpolicies if possible, quotas and also an external policy engine for further enforcements. Moreover creating a new namespace is a cluster operation and unless you want to give teams the permissions to create namespaces and adjust ResourceQuotas, this becomes a bottleneck quickly.


Cluster-per-tenant

On the other hand, one can give every tenant a fully separate Kubernetes cluster, with a separate API server, separate etcd, separate nodes. This eliminates most of the issues I defined for namespace isolation, however comes with drawbacks itself.

• Duplication of components. Control plane and core components like the monitoring-stack or CNI get duplicated. This leads to additional cost and gets more complx to manage at scale.
• No resource sharing across tenants. A quiet tenant's idle capacity can't be borrowed by a busy tenant's cluster. You either overprovision every cluster or accept poor utilization.
• Fleet management becomes its own discipline. Before you start creating clusters for tenants, you should build a system to manage them centrally and keep drift to a minimum.
   

• Compute isolation is only as strong as the underlying node model. In the default shared-nodes mode, a "tenant" is still just Linux cgroups/namespaces on a shared kernel, the same compute isolation strength as plain namespaces. To get materially stronger compute isolation you move to private nodes, which reduces the resource-sharing efficiency argument.
• Per-tenant control-plane overhead adds up. Each vCluster is its own API server + datastore process that needs CPU/memory headroom, needs upgrading, and can itself become unhealthy (etcd/SQLite corruption, API server OOMs) independent of the host. Running hundreds of vClusters means monitoring hundreds of small control planes.
• Not all resources/features virtualize cleanly. Certain operators that manage cluster-wide state (service meshes, some CNIs, cert-manager cluster issuers) don't virtualize well and often need to live only at the host level, which tenants then don't have control over.
• The host cluster is still a single point of compromise. If the host cluster's control plane is compromised, or if RBAC on the host cluster is misconfigured, an attacker can potentially reach into every vCluster's synced resources, as they are all stored in the hosts etcd. For instance, when a developer inside vcluster-teamB creates a Secret with a database password, vCluster's syncer mirrors it down into the host cluster as a real object in namespace host-ns-teamB. If someone on the host cluster has a broad ClusterRole like get/list secrets bound cluster-wide, they can now run kubectl get secret db-password -n host-ns-teamB -o yaml and read team B's credentials directly from the host, even though they have zero access to vcluster-teamBitself and team B's vCluster RBAC would have denied them.

Capsule
 

How to choose

Summary