Kedro: A Python Toolbox for Data Pipelines That Behave Like Software

The problem Kedro was built to answer

The README is unusually direct about the origin story. Kedro comes from "our collective best-practice (and mistakes) trying to deliver real-world ML applications that have vast amounts of raw unvetted data." The stated targets are the shortcomings of Jupyter notebooks, one-off scripts, and glue-code, with an emphasis on maintainable code and team collaboration. That is a specific complaint, not a general one. The failure mode it describes is a notebook that produced a model nobody can rerun, or a set of scripts where the path to the input file lives in someone's shell history. Kedro's answer is to make the pipeline itself a Python package with a fixed shape. The audience is therefore narrow and identifiable: teams of more than one person, working in Python, who need a training or feature pipeline to run again next quarter on a different machine. A solo analyst exploring a CSV is not the audience, and the README does not pretend otherwise.

Four pieces: project template, Data Catalog, pipeline abstraction, deployment

The README lists four main features, and they map cleanly onto a data flow. First, a project template based on Cookiecutter Data Science, which fixes where code, configuration and tests live. Second, the Data Catalog, described as "a series of lightweight data connectors used to save and load data across many different file formats and file systems, including local and network file systems, cloud object stores, and HDFS," with data and model versioning for file-based systems. Third, the pipeline abstraction, which performs "automatic resolution of dependencies between pure Python functions." Fourth, flexible deployment, with strategies for single or distributed machines and additional support for Argo, Prefect, Kubeflow, AWS Batch and Databricks. The mechanism that ties these together is the pure function. A node is a function; the catalog declares what each input and output name points to; the framework reads those names and builds the execution order. Because the function does not open files itself, the same node can read from local disk in development and from an object store in production by changing catalog configuration rather than code. That separation is the whole design, and it is also where the constraints come from.

Getting a project running

Installation is a single command in the README: uv pip install kedro. Conda users get conda install -c conda-forge kedro. For an unreleased version, the README gives uv pip install git+https://github.com/kedro-org/kedro@main. The README does not spell out the project-creation command in the excerpt provided, so I will not invent it; the getting-started guide linked from the README covers installation and virtual environments, and the spaceflights tutorial is the documented path to a working project. What the README does commit to is the surrounding toolchain: pytest for test-driven development, Sphinx for documentation, ruff for linting, and the standard Python logging library. Those are the tools the template wires in, which means an adopting team inherits a test directory and a lint configuration on day one. The practical first step is to run the spaceflights tutorial end to end before pointing Kedro at real data, because the tutorial is where the catalog syntax and node signatures become concrete.

Where the abstraction costs you

The pure-function requirement is the sharpest edge. A node that mutates a global, writes to a database as a side effect, or depends on a notebook variable will not fit the model without being rewritten. Teams with a large body of existing imperative scripts should expect that rewrite to be the bulk of the migration, not the installation. There is a second cost: the catalog is a second place where names must stay in sync with code. Rename an output in a function and you must rename it in configuration too, and the failure appears at run time rather than at import. The README also notes versioning only "for file-based systems," so teams expecting dataset versioning against a warehouse or a feature store will not get it from the catalog. Finally, the deployment list is a list of integrations, not a scheduler. Kedro resolves dependencies inside one run; it does not decide when that run happens. Anyone who reads "deployment on Argo, Prefect, Kubeflow" as "Kedro is an orchestrator" has misread the scope.

Kedro against Airflow, Dagster and Prefect

The honest comparison is with workflow orchestrators, because that is the neighbouring category. Airflow, Dagster and Prefect are built around scheduling, retries, backfills and a control plane that watches many runs over time. Kedro is built around the shape of the code inside one run: a project template, a catalog of named datasets, and dependency resolution between pure functions. The overlap is the dependency graph, and the difference is what happens around it. A team that needs a pipeline to run at 02:00 every day, alert on failure and backfill three weeks of history is choosing an orchestrator, and Kedro's own README points at Argo, Prefect and Kubeflow as the places that run happens. A team whose pain is that the training code is unreproducible and the paths are hardcoded is choosing Kedro, and may never need a scheduler. They are not substitutes, and the README's deployment section is effectively an admission of that.

Maintenance, releases and the licence question

The release cadence visible in the metadata is roughly every six to ten weeks across 1.4.0, 1.5.0 and 1.6.0, and the project is not archived. Python support in the README badge covers 3.10 through 3.14, which is a wide window and means an upgrade path exists for teams on older interpreters. The project is hosted by the LF AI & Data Foundation, which is a governance fact rather than a quality claim. On licensing there is a real discrepancy to resolve before adoption: the README badge says Apache 2.0 and links to LICENSE.md, while the repository metadata reports NOASSERTION. NOASSERTION usually means an automated detector could not match the file to a known licence, not that no licence exists, but the difference matters to anyone with a legal review process. Read LICENSE.md directly rather than trusting either signal. I am not giving legal advice; the point is that the two sources disagree and one of them is machine-generated. Maintenance cost otherwise looks like ordinary Python dependency management: pinning Kedro, tracking the release notes for breaking changes, and keeping the catalog in step with the node signatures.

What to check before you commit a quarter to it

Three things are worth verifying against your own code before a migration. First, count how many of your existing functions are already pure: take a data structure in, return a data structure out, no file handles. That number is your migration cost, and it is measurable in an afternoon. Second, decide whether your team needs a scheduler, because if it does, Kedro is one layer of a stack and the deployment integration list is where you start reading. Third, open LICENSE.md and confirm the terms, given the metadata mismatch. The README's own framing is the best guide to fit: it exists to address "the main shortcomings of Jupyter notebooks, one-off scripts, and glue-code." If that sentence describes your repository, the template and catalog are worth the structure they impose. If your repository is one notebook and one CSV, they are not.

Editorial conclusion

Adopt Kedro if you have a Python team that keeps rewriting the same load-clean-train-evaluate glue and wants it structured once. Do not adopt it if your work is a single script or a notebook you run by hand; the template and catalog add files you will never open. Before committing, install it with uv pip install kedro, run the spaceflights tutorial from the documentation, and check whether the nodes you already have fit a pure-function signature. Verify the license file itself, since the repository metadata says NOASSERTION while the README badge says Apache 2.0.

Official sources

1. Issues
2. kedro-org/kedro on GitHub
3. Project website
4. README
5. Releases