Agent Substrate: A Kubernetes layer for high-density agent workloads

What Agent Substrate actually solves

Agent Substrate addresses a specific operational problem: running thousands of stateful agent processes on a cluster without paying for each one's idle time. The README is explicit that agent-like applications tend to be idle most of the time. Instead of giving every agent its own pod, the system maps a larger set of actors onto a smaller set of ready workers. This is not a general-purpose scheduler. It is a control plane that manages the lifecycle of actor sandboxes, assigns actors to workers in real time, and routes traffic to them. The intended user is a team running large-scale agent deployments, particularly those involving AI agents, that need higher density than Kubernetes alone provides. The system is framework agnostic, so it can host agents built with ADK, LangChain, Claude Code, or CodeX, as long as they run as OCI containers.

The actor-worker model and its mechanism

The core mechanism is a mapping between actors and workers. An actor is an application, such as an agent, with a lifecycle that can be suspended and resumed. A worker is a Kubernetes pod that is ready to host an actor. The system relies on the observation that agents are mostly idle, so many actors can share a small pool of workers. When an actor needs to run, it is assigned to an available worker. When it becomes idle, it can be suspended and its state snapshotted. The README claims sub-second resume and suspend operations, and the demo shows 250 stateful actors multiplexed across 8 physical pods. The state persistence mechanism uses full-state snapshots that preserve volatile RAM and filesystem state across hibernation cycles. This is not a process fork or a container restart. It is a full snapshot that can be restored onto any worker, which is what enables the teleport-like behavior.

Sandbox technologies and Kubernetes integration

Agent Substrate supports multiple sandbox technologies, including microVMs and gVisor. The README says it manages standard OCI containers at the kernel level via gVisor, which is how it remains framework agnostic. The control plane uses Kubernetes for infrastructure provisioning and worker lifecycle management, specifically Pods and Pod autoscaling. Agent Substrate adds its own scheduling and control on top to achieve lower latency than Kubernetes alone. The design choice is to build on Kubernetes rather than replace it, which means you get consistent infrastructure management across all workload types. For reinforcement learning scenarios that span agentic, inference, and training cycles, the README claims holistic infrastructure optimizations, though it does not detail how those optimizations work. The system is low-opinion, so the workloads it manages do not have to be literal AI agents, but they must fit the actor model with suspend and resume.

Getting it running: the quickstart path

The development quickstart requires Go, kubectl, and docker on your machine. The project manages other dependencies, including kind, automatically via Go. The steps are concrete. First, run hack/create-kind-cluster.sh to create a cluster and local registry. The script supports IPv4 by default, with IP_FAMILY=dual or ipv6 as overrides. Second, run hack/install-ate-kind.sh --deploy-ate-system to install ate, PostgreSQL, and rustfs. Then run the same script with --deploy-demo-counter to install the counter demo. After that, install the kubectl plugin with go install ./cmd/kubectl-ate. The plugin is named kubectl-ate, and it is used to create an atespace and then an actor. The example commands are: kubectl ate create atespace demo, then kubectl ate create actor my-counter-1 -a demo --template=ate-demo-counter/counter. The README mentions port-forwarding the network r, but the text is truncated there, so the exact forwarding command is not visible.

A genuine limitation: early development and API instability

The most significant limitation is the project's maturity. The README states plainly that Agent Substrate is in early development, is not ready for production use, and that the APIs are almost guaranteed to change. There are no backward compatibility guarantees at this stage. This means any code you write against the current APIs may break with the next commit. The project is also very young, with the initial commit and release v0.0.0 dated the same day as the last push. The contributing guidelines note that the project may not review or merge contributions that do not align with its immediate focus on the core system and demos. So if you need a stable runtime, this is the wrong tool today. Another limitation is the reliance on snapshot-based suspend and resume. If an agent holds state outside the snapshot, such as external database connections or in-memory caches that are not captured, those will not be preserved across hibernation cycles. The README does not discuss how to handle such external state.

Alternatives: a different approach to density

The most direct alternative is to run agents as plain Kubernetes pods with autoscaling, which is what Agent Substrate builds on. The difference is that Kubernetes alone does not provide sub-second suspend and resume or the actor-to-worker mapping. With plain pods, an idle agent still occupies a full pod, and scaling down means terminating the pod and losing its state unless you implement external persistence. Another alternative is to use a serverless container platform like Knative, which scales to zero when idle, but it does not preserve in-memory state across cold starts. Knative's approach is to restart from scratch, not to snapshot and restore. Agent Substrate's approach is fundamentally different: it keeps the actor's state in a snapshot and restores it onto any worker, which allows for much faster resume than a cold start. If your agents are stateless, Knative or plain Kubernetes is simpler. If they are stateful and idle-heavy, Agent Substrate's model is worth evaluating, but only if you can tolerate its instability.

Maintenance and upgrade cost, and license implications

The project is licensed under Apache-2.0, which is permissive and allows commercial use, modification, and distribution, with the condition that you include the license and attribution. There is no copyleft requirement, so you can use it in proprietary systems. However, the README includes a note that this is not an officially supported Google product and is not eligible for the Google Open Source Software Vulnerability Rewards Program. That means you should not rely on Google for security fixes or official support. The maintenance cost is high at this stage. The project has a weekly community meeting and a Google Group, but the core system is being built out, and contributions are only selectively accepted. The upgrade cost is unpredictable because APIs are not stable. You will need to track the main branch closely and be prepared to update your integration with each release. There is no release history beyond v0.0.0, so there is no track record of migration paths or deprecation policies.

Editorial conclusion

Adopt Agent Substrate only for experimental or research deployments where you can tolerate API churn and frequent breaking changes. Do not use it in production or for any workload that requires stable interfaces or long-term support. Before adopting, verify that your Kubernetes cluster matches the supported versions (latest stable and previous minor), that you can run gVisor or microVMs in your environment, and that your agents can tolerate snapshot-based suspend and resume without losing in-memory state that is not captured. The project is very young, so expect to contribute fixes or wait for the core system to stabilize.

Official sources

1. Official README
2. Project repository
3. Release notes