kubernetes/test-infra: The CI Monorepo Behind the Kubernetes Project

Scope and Intended Audience for kubernetes/test-infra

kubernetes/test-infra was created to consolidate everything required to run CI for the Kubernetes project in a single, auditable repository. The README describes it as containing tools and configuration files for the testing and automation needs of the Kubernetes project.

The repository serves three overlapping groups. SIG members use it to define and update CI job configs for their subprojects. On-call engineers use its dashboards to monitor test health and investigate failures. A smaller group of infrastructure contributors maintains the tooling itself. External contributors who want to understand how the Kubernetes project runs its CI, or who want to add a job for a subproject, need to read this repository.

kubernetes/test-infra is a monorepo. It does not expose a single installable package. It contains tools, configuration, and code generation scripts that work together under the control of the Prow instance at prow.k8s.io. The architecture diagram the README references at docs/architecture.svg is the single best reference for how the components connect.

Prow: The CI Engine and the Self-Service PR Workflow

Prow is the CI and automation platform used by the Kubernetes project. The README notes that the Prow instance is at prow.k8s.io and that everyone can participate in a self-service PR-based workflow where changes are automatically deployed after review.

All job definitions for Prow live under config/jobs. Contributors submit YAML changes to those files via pull requests against this repository. After review and merge, changes deploy without a manual step. The README lists four paths for job management: adding or updating job configs (config/jobs/README.md), deleting job configs, testing job configs locally, and triggering jobs on PRs using bot commands at go.k8s.io/bot-commands.

Tide is the Prow component that batches pull requests and merges them when all configured status checks pass. Its dashboard is at prow.k8s.io/tide. The Deck dashboard at prow.k8s.io shows running and recently completed jobs. The PR Status view at prow.k8s.io/pr shows what a given pull request still needs before it can merge. These three surfaces are the primary windows contributors use to check CI state.

Setting Up the Repository and Running the Test Suite

kubernetes/test-infra is primarily a Go codebase with some Python tooling and TypeScript for the browser-facing dashboard components. The Go module root is declared in go.mod:

module k8s.io/test-infra

For contributors working on the tooling itself, the Makefile provides a unified test target:

make unit

This target runs two sub-targets. go-unit executes hack/make-rules/go-test/unit.sh and py-unit executes hack/make-rules/py-test/all.sh. The Makefile also exposes a clean target that removes the _output/ directory. Build configuration in package.json shows that the repository also uses Bazel as a secondary build system for the TypeScript and browser components, with tools such as rollup and terser listed as dev dependencies.

For contributors focused only on CI job changes, the workflow does not require building the tooling. The main task is editing YAML files under config/jobs and validating them locally using the process described in config/jobs/README.md.

Testgrid, Triage, and the Dashboard Layer

Testgrid at testgrid.k8s.io shows historical test results across all Prow jobs, organized into dashboards and tabs. It answers the question of when a test started failing relative to a specific merge. Configuration for Testgrid lives in the testgrid/ directory of this repository.

Triage at go.k8s.io/triage takes a different approach. Instead of showing results per job, it clusters similar test failures across all jobs. A failure cluster that spans many unrelated jobs usually points to an infrastructure problem rather than a code regression. The triage tool code lives under triage/.

gcsweb under gcsweb/ provides a browser UI for test artifacts stored in public Google Cloud Storage buckets. The gubernator/ directory holds configuration for a related tool that also displays artifacts from GCS. Together these form a layered stack: Prow produces test results, Testgrid stores them as time-series, Triage cross-references them for shared failure signatures, and gcsweb or gubernator let engineers read raw output from specific test runs.

Supporting Toolchain: label_sync, kettle, ghproxy, and kubetest2

Beyond Prow and the dashboards, the monorepo holds several operational tools.

label_sync under label_sync/ reads a labels.yaml file and creates, updates, or migrates GitHub labels across organizations and repositories. It solves the practical problem of keeping label definitions consistent across the dozens of repos in the Kubernetes GitHub organization.

kettle under kettle/ extracts test results from GCS and loads them into BigQuery. The metrics/ directory then runs BigQuery queries to generate metrics from that test history. These two components form an analytics path for long-term CI data.

ghproxy, now maintained separately at kubernetes-sigs/prow but referenced in the README, is a GitHub-aware reverse proxy cache. It reduces API token consumption by caching responses, which keeps the Prow instance within GitHub's rate limits when issuing thousands of API calls per hour.

kubetest2, maintained at kubernetes-sigs/kubetest2, is the tool CI uses to create and end-to-end test actual Kubernetes clusters. The main repository delegates cluster lifecycle management there rather than encoding it inline.

robots/commenter under robots/commenter/ is used by some jobs to post comments on GitHub issues. It is a minor utility but shows the scope of what the monorepo holds: not just test configs but the full set of bots and scripts the project operates.

Concrete Constraints for External Adopters

kubernetes/test-infra is not designed for adoption outside the Kubernetes project. Three constraints make this concrete.

First, the infrastructure assumes Google Cloud Platform. boskos manages pools of GCP projects for CI leases. kettle writes to BigQuery. GCS is the artifact store. A team running on AWS or Azure would need to replace each of these integration points.

Second, Prow itself was extracted to its own repository at kubernetes-sigs/prow. Teams wanting to run Prow for their own project should start there, not here. kubernetes/test-infra holds the job configs and automation that operate on top of the Kubernetes-project Prow instance; it does not contain Prow's server code in the form that other projects would deploy.

Third, documentation is sparse and assumes familiarity with Kubernetes project conventions. The README points to per-tool READMEs and external links, but there is no single getting-started guide for an engineer outside the project.

Maintenance Status and License

The last push to kubernetes/test-infra was on 2026-09-25, and the repository receives continuous commits from contributors across the Kubernetes organization. The repository is not archived.

The project is licensed under Apache-2.0. This license permits use, modification, and redistribution with attribution. It does not impose copyleft requirements on downstream code that incorporates this tooling. Teams incorporating any piece of this repository into their own infrastructure should confirm that Apache-2.0 is compatible with their project's licensing terms.

Editorial conclusion

Teams already working within the Kubernetes ecosystem, such as SIG members adding or modifying CI jobs, will find kubernetes/test-infra necessary to understand. Teams outside the Kubernetes project will find it too tightly coupled to Google Cloud infrastructure and the Kubernetes organization structure to adopt directly. The right starting point for running Prow elsewhere is the separately maintained kubernetes-sigs/prow repository. Before contributing job configs here, check the config/jobs/README.md instructions for local validation, since job configs that fail local checks are rejected before merge.

Frequently asked questions

Official sources

1. Issues
2. kubernetes/test-infra on GitHub
3. License: Apache-2.0
4. README