Usernetes generation 3: a Kubernetes cluster built without root on the host

Two deployment modes, and generation 3 is the one that added Kubernetes-in-Kubernetes

The project states its goal in one sentence: Usernetes deploys a Kubernetes cluster without requiring root privileges on the host, so as to mitigate potential container-breakout vulnerabilities. From that, two deployment modes branch out. The default is Kubernetes-in-Docker, where the cluster runs inside Rootless Docker, Rootless Podman or Rootless nerdctl. The documentation is candid that this mode resembles Rootless kind and Rootless minikube, and the difference it claims is that Usernetes supports creating a cluster with multiple hosts. The second mode, Kubernetes-in-Kubernetes, runs Usernetes inside an existing cluster as UserNS-enabled pods with hostUsers set to false, and it has its own README in the kubernetes directory. The generation history explains why the modes exist. Generation 1, on the gen1 branch from 2018 to 2023, was written in the style of Kubernetes The Hard Way and was hard to use because it lacked kubeadm support. Generation 2, on the gen2 branch from 2023 to 2026, switched to Kubernetes-in-Docker for simplicity, which made kubeadm support possible. Generation 3 is the current line and adds the Kubernetes-in-Kubernetes mode.

The component choices are fixed: kubeadm, containerd, runc, Flannel or Calico

The component list is short and it tells you what you are actually running. Cluster configuration is handled by kubeadm, the CRI is containerd, the OCI runtime is runc, and the CNI is Flannel by default, with Calico available in VXLAN mode and only in the Kubernetes-in-Docker mode. Containerd and runc inside a rootless engine is the interesting part: a rootless container engine runs the container runtime as an unprivileged user inside a user namespace, so the kubelet is never root on the host either. The build pins the base image to a kind node image by digest, and the Dockerfile fetches CNI plugins, Helm, Flannel and the Calico operator archives, verifying each against a checked-in SHA256SUMS file before unpacking it. The pinned versions in the build arguments are CNI plugins v1.9.1, Helm v4.2.4, Flannel v0.28.9 and Calico v3.32.2. This is a deliberately reproducible image, and it also means that when a component needs a security update, you rebuild the image rather than patch a running node.

Host preparation is the real cost: cgroup v2 delegation, lingering and kernel modules

The requirements section is where a reader learns whether this fits their machine, and it is not a short list. You need a supported host operating system, with Ubuntu 22.04 recommended, Rocky Linux 9, AlmaLinux 9 and Fedora listed with an unclear minimum version. You need a rootless container engine, either Rootless Docker at v20.10 or later, which is the recommendation, Rootless Podman at v4.x or later, or Rootless nerdctl at v1.6 or later. Then the host has to be configured. systemd lingering is enabled so your user services survive logout, and cgroup v2 delegation is required so the user session can pass controllers to containers.

sudo loginctl enable-linger $(whoami)

sudo mkdir -p /etc/systemd/system/[email protected]

sudo tee /etc/systemd/system/[email protected]/delegate.conf <<EOF >/dev/null
[Service]
Delegate=cpu cpuset io memory pids
EOF

sudo systemctl daemon-reload

Kernel modules and a sysctl setting follow, and the modules file is what makes the Flannel and Calico VXLAN paths work.

sudo tee /etc/modules-load.d/usernetes.conf <<EOF >/dev/null
br_netfilter
vxlan
EOF

sudo systemctl restart systemd-modules-load.service

The sysctl value relaxes reverse path filtering to the loose mode of 2, which is needed for the routing setup inside the namespace. The project points at scripts in the init-host directory for automating these steps, so the manual sequence above is a description of what those scripts do.

make up, make kubeadm-init, make install-cni is the whole cluster lifecycle

Usage is delegated to the Makefile, with make help as the entry point, and the documented sequence for a single host is three commands.

make up
make kubeadm-init
make install-cni

The first brings up the node container, the second initializes the cluster with kubeadm, and the third installs the CNI. Then you make kubectl usable by generating a kubeconfig and pointing the environment at it.

make kubeconfig
export KUBECONFIG=$(pwd)/kubeconfig
kubectl get pods -A

The multi-host flow extends the same idea: make join-command produces a command file, you copy it to the other host and run make up kubeadm-join there, and make sync-external-ip makes the API server reachable from the other hosts. Debugging and teardown have their own targets, including make logs, make shell, make kubeadm-reset and make down-v, and the last line of the usage block taints away the control-plane taint so you can schedule workloads onto the node. One instruction in the compose file is worth repeating: use make up rather than docker compose up, because the YAML depends on variables that are set by the Makefile.

Everything is tuned through environment variables exported by the Makefile

The customization table is a list of environment variables with defaults, and the Makefile exports most of them. CONTAINER_ENGINE selects the engine and is automatically resolved to docker, podman or nerdctl. CNI selects flannel or calico and has to be set on make up, because the value is baked into the node container by the compose file to enable the CNI specific hacks in the entrypoint, and the install scripts fail if the value is inconsistent between make up and make install-cni. NODE_NAME defaults to u7s- plus the host name, and NODE_SUBNET defaults to a 10.100.%d.0/24 address computed from a hash of the host name. POD_SUBNET is 10.244.0.0/16 and SERVICE_SUBNET is 10.96.0.0/16, and the Kubernetes-in-Kubernetes mode uses different defaults so the inner cluster does not overlap the outer one. The port variables are PORT_ETCD at 2379, PORT_KUBELET at 10250, PORT_FLANNEL at 8472, PORT_CALICO at 4789, PORT_CALICO_TYPHA at 5473 and PORT_KUBE_APISERVER at 6443. There is one extra knob worth explaining, CALICO_VXLAN_IIFNAME, which defaults to eth0 and becomes lo with Podman rootlessport, because the forwarded VXLAN packets then arrive on the loopback interface rather than eth0.

The limitations are port exposure, host mounts and the privileged node container

The documented limitations are specific and worth reading before planning a deployment. Node ports cannot be exposed automatically, and the fix is to edit the compose file to expose the additional node ports by hand. Most host files are not visible with hostPath mounts, and the fix there is also to edit the compose file to mount additional files. The list of limitations continues past the point where the documentation is quoted here, but the first two entries already tell you the shape of the trade-off. The other one is visible in the compose file itself: the node service is declared privileged. That is not a bug, it is the mechanism by which a rootless container can provide the kernel level capabilities a kubelet expects, and it is worth understanding rather than glossing over, because it means the isolation story is about the outer user namespace boundary rather than about an unprivileged container. The node service also mounts the repository read only at /usernetes, the boot and modules directories read only, and tmpfs for /run and /tmp.

How this differs from rootless kind and from plain kind

The comparison the project itself draws is with Rootless kind and Rootless minikube, and the stated difference is multi-host support, so that is where the real comparison should be focused. kind and minikube are single-node tools aimed at a local development cluster you throw away, with kind in particular building a node container image and running kubeadm inside it. Usernetes adds the ability to bring up a cluster spanning several hosts, and it does that with the same component set of kubeadm, containerd and runc, plus a CNI choice between Flannel and Calico in VXLAN mode that kind does not offer in that form. The cost is that kind needs no host preparation beyond a container engine, while Usernetes needs cgroup v2 delegation, systemd lingering, kernel modules and a sysctl change on every host, and the multi-host flow requires a join command file, an scp and an ssh. A static cluster manager would give you node lifecycle, upgrades and networking as a product, which is the trade you make when the isolation of user namespaces matters more to you than the operational conveniences.

Editorial conclusion

Adopt Usernetes if you need a disposable multi-node Kubernetes cluster for development, for security research on container isolation, or for testing workloads in a user namespace, and you can give a host the cgroup v2 delegation and kernel modules it asks for. Do not adopt it as a production platform, because node ports are not exposed automatically, host files are mostly invisible to hostPath mounts, and the deployment shells out to make targets that assume you have already prepared the host by hand. Before you plan around it, confirm that your host runs cgroup v2 with delegation, check that your container engine version is at or above the documented minimum, and read the node service definition in docker-compose.yaml, because the node container runs privileged, which is the tension the project is working around rather than removing.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. rootless-containers/usernetes on GitHub