# System Upgrade Controller: node upgrades as declared Plans and privileged Jobs

> The Rancher System Upgrade Controller turns node upgrades into a Kubernetes custom resource: a Plan selects eligible nodes by label, the controller runs a privileged Job on each, and the node is labelled with the applied version once the Job succeeds.

**rancher/system-upgrade-controller** — In your Kubernetes, upgrading your nodes

- Repository: https://github.com/rancher/system-upgrade-controller
- Stars: 980 · Forks: 113
- Language: Go
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/rancher-system-upgrade-controller

## A Plan is an outstanding intent to mutate nodes

The central idea is one custom resource. The project introduces a CRD called Plan, and the README defines a Plan as an outstanding intent to mutate nodes in your cluster. Everything else in the system exists to reconcile that sentence.

The API lives under the upgrade.cattle.io group with a v1 version, and the type definitions are in pkg/apis/upgrade.cattle.io/v1/types.go, which the README points at for up to date details on defining a plan. Autogenerated API documentation for the Plan type is published at doc/plan.md, so the schema can be read before any YAML is written.

The scope is deliberately wide. The README describes the project as a general purpose, Kubernetes native upgrade controller for nodes, and states that it attempts minimal imposition of opinion. What an upgrade actually does is defined by the content of a container image and the command you give it, not by a fixed set of supported operations. That is a different design from a controller that knows how to upgrade, say, a specific operating system, and it is why the controller treats a reboot as one possible effect of a command rather than as its own feature.

## Label selectors decide eligibility, and the controller labels the node back

A plan does not name nodes. It names a selector, and the controller applies the plan to whichever nodes match. The example plan in the README uses matchExpressions to combine several conditions:

```yaml
  nodeSelector:
    matchExpressions:
      - {key: plan.upgrade.cattle.io/k3os-latest, operator: Exists}
      - {key: k3os.io/mode, operator: Exists}
      - {key: k3os.io/mode, operator: NotIn, values: ["live"]}
```

The same expression does two jobs at once. It limits the upgrade to nodes that opted in by carrying the plan's label, and it excludes nodes booted from a live CDROM. A value of disabled for that label on a node causes the controller to skip over it, which is the opt-out path.

The label name is not arbitrary. The README points out that the name portion of the label matches the plan name, and that the controller will tag the node with this very label, with the value being the applied plan.status.latestHash. In other words, the label you opt in with becomes the record of what was applied. That feedback loop is how the controller knows which nodes are already at the target version and can skip them, and it is why renaming a plan changes which nodes it selects.

Two details follow from the same design. A node that has never been labelled never matches, so a fresh node joining the cluster stays on the old version until someone applies the label. And because the label carries a hash rather than a human readable version, reading cluster state means resolving the plan status rather than reading a string off the node.

## The upgrade runs as a privileged Job with a writable /host

The controller does not talk to nodes directly. It schedules a Job, and the Job's pod carries the privileges needed to mutate a host. The README lists the design constraints explicitly: content delivery via container image, an operator overridable command, and a very privileged job, pod and container.

The privilege set is the part to read carefully. The job runs with host IPC, host network and host PID namespaces, it holds CAP_SYS_BOOT, and the host root file system is mounted at /host with read and write access. Cordoning and draining are optional, in the manner of kubectl.

A Job model has a second advantage that is easy to miss. A pod is retried and can be inspected, so a failed upgrade leaves a failed Job and its logs behind, rather than a shell history on a machine you cannot see. The cost is the opposite: these pods are as close to host root as a Kubernetes workload gets, and the README's own note about optional opt in and opt out through node labels is about limiting which machines ever get one of them.

## Deploying with kustomize and pinning a release manifest

Installation is one command. The README offers it in the time honoured tradition of a one-liner:

```bash
kubectl apply -k github.com/rancher/system-upgrade-controller
```

The comment in the README calls this Y.O.L.O., which is the project's own framing of applying a moving reference from the default branch rather than a pinned artefact. The pinned route also exists: manifests/system-upgrade-controller.yaml in the repository holds the most up-to-date manifest, and since release v0.4.0 a manifest specific to each release has been uploaded to the release artifacts page, for example the v0.4.0 asset under releases/download/v0.4.0/.

Building the controller from source is a Go project. go.mod declares the module github.com/rancher/system-upgrade-controller with go 1.26.0 and a toolchain directive for go1.26.5, and it pins the Kubernetes libraries at v0.37.0, including k8s.io/kubernetes v1.37.0 and k8s.io/api, apimachinery, client-go and kubectl. The build itself runs through docker buildx with the Dockerfile in package/, and the Makefile exposes script targets for build, ci, test, validate, package, release and the e2e runs:

```makefile
SCRIPT_TARGETS := build ci e2e-sonobuoy e2e-verify package package-controller package-e2e-tests release test validate
```

The dependency set also tells you what this controller is built to talk to: rancher/wrangler/v3 and rancher/lasso for Kubernetes client machinery, k8s.io/pod-security-admission, which matters in clusters where Pod Security Admission is enforced, and github.com/kubereboot/kured v1.13.1.

## Channels resolve a version, or you pin it yourself

The plan carries a version field and a channel field, and they are alternatives. The README states that the value for channel is assumed to be a URL that returns HTTP 302, with the last path element of the value returned in the Location header taken as an image tag, after munging the plus sign to a hyphen. Providing a value for version prevents polling and resolution of the channel.

In practice a channel is a redirect endpoint that always points at the newest release, and the controller polls it to learn which image tag to run. A version turns that off and fixes the target. For a k3s example the README shows both:

```yaml
  concurrency: 1
  channel: https://github.com/rancher/k3os/releases/latest
  version: v0.10.0
```

This is the difference between a rolling upgrade and a repeatable one. A channel driven plan means the target moves on its own, which is convenient for staying current and awkward for auditing what ran on which node. A version pinned plan is reproducible, at the cost of needing a new plan edit to move forward.

The plan also carries the service account the pod should use and the tolerations the upgrade pods need. The example appends architecture tolerations to a default of node.kubernetes.io/unschedulable with the NoSchedule effect, which is what allows a job to land on a node that has been cordoned.

## prepare, drain and concurrency define the order of events

Within a single node, the plan defines the sequence. The README states that the prepare init container, if specified, runs before cordon and drain, which runs before the upgrade container, and that prepare shares the same format as the upgrade container:

```yaml
  prepare:
    image: alpine:3.18
    command: [sh, -c]
    args: ["echo '### ENV ###'; env | sort; echo '### RUN ###'; find /run/system-upgrade | sort"]
```

Draining is optional and configurable:

```yaml
  drain:
    force: true
```

The README notes that deleteLocalData and ignoreDaemonSets default to true, and recommends disableEviction and skipWaitForDeleteTimeout to prevent upgrades from hanging on small clusters. That last point is a genuine operational trap rather than a warning about exotic setups: a cluster with too few nodes for eviction to succeed can leave an upgrade Job waiting indefinitely.

Across nodes, concurrency caps how many nodes are upgraded at once, and the example sets it to 1. Nothing in the documentation suggests the controller will coordinate with itself for you, so the concurrency value is the whole of your blast radius control.

The examples directory shows how the mechanism adapts to different targets: examples/k3s-upgrade.yaml for upgrading k3s itself, examples/ubuntu/bionic.yaml for apt-get style upgrades of arbitrary packages at pinned versions, and two kernel upgrade examples for Ubuntu 18.04 on AWS EC2 instances and on generic virtual machines.

## Idempotence is the requirement the project cannot enforce

The README warns about this in its most direct line: one should take care when defining upgrades by ensuring that such are idempotent, there be dragons.

The reason is structural. The controller will re-run the upgrade command on a node whenever the node does not carry the applied plan hash label, and the version of the command is whatever is in the container image you reference. If the command is not idempotent, a repeated run can half apply an upgrade, and because the same command may run again on a later node or on the same node after a re-label, the damage is not necessarily visible in the Job status.

This is also the point where the design pushes back on the idea of a general purpose upgrade controller. Because the controller imposes minimal opinion about what an upgrade is, it cannot check whether your command is safe to run twice. That check has to live in your command and in your review of it.

The alternative approach is worth naming. The go.mod requires github.com/kubereboot/kured v1.13.1, so this controller is built to cooperate with a dedicated reboot coordinator rather than to displace it. The difference in approach is the granularity of what is scheduled: a reboot coordinator acts on a node as a unit, while a Plan schedules an arbitrary privileged container command and lets that command decide what a node upgrade means, including whether it ends in a reboot at all.

## Conclusion

Use the System Upgrade Controller if you run Kubernetes nodes that can be upgraded by a privileged container command, and you want that upgrade expressed as a reviewed resource with per node eligibility, concurrency limits and drain behaviour rather than a script run over SSH. Do not use it for upgrades that are not idempotent, for hosts where a writable /host mount and CAP_SYS_BOOT are unacceptable, or where the release manifest for your Kubernetes version is not published. Before you adopt it, verify two things in your own cluster: that the API type in pkg/apis/upgrade.cattle.io/v1/types.go covers the fields your policy needs, and that your node labels make the plan's matchExpressions select the right machines.

## FAQ

### How do I install the Rancher System Upgrade Controller?

The README gives the one-liner kubectl apply -k github.com/rancher/system-upgrade-controller. For a pinned version, manifests/system-upgrade-controller.yaml in the repository holds the current manifest, and since release v0.4.0 each release also publishes its own manifest as a release artifact.

### What custom resource does the System Upgrade Controller introduce?

A Plan, in the upgrade.cattle.io/v1 API group. The README defines a Plan as an outstanding intent to mutate nodes in your cluster, and the type definitions live in pkg/apis/upgrade.cattle.io/v1/types.go with generated docs at doc/plan.md.

### How does a Plan choose which Kubernetes nodes to upgrade?

Through spec.nodeSelector matchExpressions, which can require nodes to carry a label whose name portion matches the plan name, and can exclude nodes by label value. After a Job succeeds, the controller labels that node with the applied plan.status.latestHash.

### What privileges does the node upgrade Job receive?

The README lists host IPC, network and PID namespaces, the CAP_SYS_BOOT capability, and the host root file system mounted read and write at /host. Cordoning and draining a node are optional, in the manner of kubectl.

### How does a Plan decide which version to install?

The channel field is assumed to be a URL returning HTTP 302, where the last path element of the Location header is taken as an image tag, with plus signs munged to hyphens. Setting version instead prevents polling and resolution of the channel, which pins the target.

## Sources

- [Issues](https://github.com/rancher/system-upgrade-controller/issues)
- [License: Apache-2.0](https://github.com/rancher/system-upgrade-controller/blob/master/LICENSE)
- [rancher/system-upgrade-controller on GitHub](https://github.com/rancher/system-upgrade-controller)
- [README](https://github.com/rancher/system-upgrade-controller/blob/master/README.md)
- [Releases](https://github.com/rancher/system-upgrade-controller/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/rancher-system-upgrade-controller