gVisor multi-tenant Kubernetes with user-space kernel isolation

Why namespaces are not a tenant boundary

In a multi-tenant Kubernetes cluster, every pod on a node shares one Linux kernel, so a single kernel exploit in one tenant's workload crosses into every other tenant on that node. Namespaces, RBAC and quotas separate tenants in the control plane. At runtime, the only wall between them is the host kernel's attack surface. With gVisor, multi-tenant Kubernetes gets a second boundary right at that point.

For example, NetworkPolicies show the problem clearly. They are enforced by translating policies into iptables rules and eBPF filters on the host. An attacker who escapes a container onto the host can simply rewrite those rules and reach any other tenant's pods.

gVisor adds a second boundary. As a result, a workload that breaks out of its process lands inside gVisor's own user-space kernel, not on the host. From there it still has to get past a set of components before it can touch the node, the host network stack or another tenant.

What gVisor is

gVisor is a user-space reimplementation of the Linux syscall ABI, written in Go. A tenant's syscalls are serviced by gVisor's code in a separate process, not by the host kernel. When gVisor does need the host, it builds those calls from its own state instead of forwarding them, so nothing the tenant passed in ever reaches the host kernel unchanged. For tenant isolation, this means a tenant never talks to the shared kernel directly. Each pod gets its own userspace kernel, and the shared one only sees a small, fixed set of requests from it.

Three components, none trusted with everything

gVisor splits its kernel into separate processes so that no single one can both run attacker-controlled code and reach the host freely. Since directfs became the default in 2023, the split between Sentry and Gofer is looser than it was. --directfs=false restores the strict version.

The stub runs the tenant's code but cannot issue a single host syscall. Every call traps into the Sentry. The Sentry is the per-pod kernel. It intercepts each syscall and reimplements it in Go, which makes most of it memory safe. Its own host syscalls pass a strict seccomp filter that by default blocks socket, exec and connect. With directfs it may open files, but only relative to descriptors the Gofer gave it, with O_NOFOLLOW and never into procfs; in hardened mode, file syscalls are blocked too. It runs in its own user, mount and PID namespaces, inside cgroups and a pivot_root. The Gofer runs outside the sandbox and sets up file access. It builds the container's mounts inside its own mount namespace and pivot_root, then hands the Sentry a descriptor for each mount point.

Following one open() through the sandbox

A tenant's file descriptor under gVisor exists nowhere on the host. Take this call:

int fd = open("/data/foo.txt", O_RDONLY);

int fd = open("/data/foo.txt", O_RDONLY);

int fd = open("/data/foo.txt", O_RDONLY);

1. The call traps before it becomes an ordinary host syscall.

2. The Sentry resolves the path against its own view of the container filesystem. It handles permissions, symlinks and mount traversal itself.

3. It calls openat on the host itself, relative to a mount descriptor the Gofer donated when the sandbox started. Its seccomp filter only allows this with O_NOFOLLOW and never into procfs, so neither .. nor a planted symlink leads out of the container filesystem.

4. The host returns fd 15 into the Sentry's own descriptor table.

5. The Sentry creates application fd 3 in its internal table and returns it.

In hardened mode (--directfs=false), step 3 becomes an RPC to the Gofer. The Gofer opens the file and passes the descriptor back, and the Sentry needs no file syscalls at all.

Under runc, fd 3 would be a real entry in the host's descriptor table, visible in /proc/<pid>/fd. Under gVisor there are two tables. The tenant's fd 3 is an index into a Go data structure inside the Sentry; the host only knows the Sentry's fd 15.

Netstack: a private network stack per tenant

gVisor runs its own network stack, netstack, inside the Sentry, so a tenant's TCP state, control messages and packet assembly never touch the host's network stack. Only finished link-layer frames leave the sandbox, written to the virtual device in the pod's network namespace that Docker or Kubernetes set up.

Source: Anjali, T. Caraza-Harter, M. M. Swift, Blending Containers and Virtual Machines: A Study of Firecracker and gVisor, VEE 2020.

This matters for the NetworkPolicy problem from the intro. A tenant who compromises their workload ends up in the Sentry, which has no way to reach the host's iptables rules or eBPF programs. The policies keep being enforced on the host side, outside the attacker's reach. There are, however, two caveats.

On the one hand, netstack can be switched off in favour of host networking for better throughput. Doing so hands the tenant's packet handling back to the host kernel and gives up most of this isolation. On the other hand, netstack does not remove the host's networking code from the path. Coverage measurements show gVisor still exercises substantial parts of the Linux networking code, so bugs there remain reachable (talk).

Netstack's isolation, however, has a measurable throughput cost. In iperf3 runs by Anjali et al., gVisor with netstack reached roughly 1 to 3 Gbps aggregate across 1 to 10 instances, while the host, Firecracker and LXC sat near 9 to 10 Gbps. Notably, switching gVisor to host networking closed most of the gap.

These numbers predate several years of netstack optimisation.

Platforms: Systrap and KVM

gVisor needs a way to catch a tenant's syscalls and page faults. It calls these mechanisms platforms.

What an attacker gets after breaking out

A tenant who fully compromises the Sentry gains little more than they already had. With directfs, they hold host descriptors for the pod's own mounts and a seccomp filter that lets them open files only below those, without following symlinks or entering procfs. In hardened mode they are limited further, to what the Sentry-to-Gofer RPC interface allows.

Consequently, everything reachable through it already belonged to the tenant. What a real escape needs is simply absent: /proc/sys, host /proc entries, /dev/kvm, the container runtime socket, other tenants' volumes. A hostPath volume is inside the Gofer's view by definition. Mount something sensitive into a tenant pod, and a compromised Sentry gets it. In a multi-tenant cluster, hostPath should be denied for tenant namespaces outright.

Running gVisor multi-tenant Kubernetes with a RuntimeClass

Kubernetes picks gVisor per pod through a RuntimeClass, so you can sandbox untrusted tenants and leave trusted system workloads on runc. gVisor's runtime binary is runsc, and containerd talks to it through a shim like any other runtime.

Register the runtime with containerd on each sandbox node:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
  runtime_type = "io.containerd.runsc.v1"

Expose it to the cluster:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

Then select it in the tenant's pods:

spec:
  runtimeClassName: gvisor

spec:
  runtimeClassName: gvisor

spec:
  runtimeClassName: gvisor

A field in a pod spec is a request, not a boundary. For real tenant separation, you should enforce it via admission policies.

The price: overhead by workload type

How much gVisor slows a workload depends on how often it crosses into the kernel. CPU-bound work pays almost nothing, pure I/O work pays the most, and most typical applications land somewhere in between.

However, many published benchmarks predate directfs, which gives the Sentry direct file descriptor access and removes most Gofer round trips. Therefore, benchmark on a current release before ruling gVisor out for I/O-heavy tenants.

Conclusion

gVisor turns the shared node kernel from a tenant's direct attack surface into something reached only through a small, fixed set of calls from a memory-safe, heavily confined process. For clusters that run code from tenants who do not trust each other, that is a real second boundary, at a cost most applications can absorb.

However, gVisor works only as part of a policy, not as a switch:

• Enforce runtimeClassName: gvisor for every tenant namespace through admission control.

• Ban hostPath, hostNetwork and privileged pods for tenants.

• Keep netstack enabled for tenants unless network throughput clearly outweighs isolation.

• Keep NetworkPolicies. gVisor keeps them out of a tenant's reach rather than replacing them.

In case you are interested in sandboxing AI agents using lightweight VMs, my colleague Jannis wrote an article on exactly that: liquidreply.net/blog/ai-agent-sandbox-amazon-eks-kata-containers

Sources

• Optimizing gVisor filesystems with Directfs, gVisor blog, 2023

• gVisor filesystem user guide

• Anjali, Tyler Caraza-Harter, Michael M. Swift (Authors Info & Claims), Blending Containers and Virtual Machines: A Study of Firecracker and gVisor, VEE '20: Proceedings of the 16th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pages 101–113, 2020, doi.org/10.1145/3381052.3381315 (talk)

Get in contact with us if you want to build tenant isolation for your workload platform: liquidreply.net/contact