Koheesio: data pipelines built from reusable Steps

Koheesio breaks a pipeline into Steps rather than a graph

Koheesio takes its name from the Finnish word for cohesion, and the framework's central idea is that a data pipeline should be assembled from small units that can be written, reviewed and reused one at a time. The unit of work is the Step. Each Step is described in the project's own documentation as a way to break down tasks and workflows into manageable, reusable and testable pieces, and each Step ships with built-in logging so a run leaves a trail behind it.

The framework positions itself unusually among data engineering tools, and that position is stated rather than implied. Koheesio is explicitly not a workflow orchestration tool. The documentation names Luigi, Apache Airflow and Databricks workflows as tools built to manage complex computational workflows and generate DAGs, and says outright that Koheesio does not serve the same purpose. What it does instead is concentrate on the individual tasks that would sit inside such a workflow.

That framing matters for adoption decisions. If the gap in your stack is that nobody owns dependency ordering, retries across days, or a visual view of how tasks connect, Koheesio is aimed somewhere else. The framework targets the part of the problem that lives in a Python module: writing the transformation itself so that it can be tested without a cluster and read without a runbook.

Context and Pydantic typing inside every Step

Two mechanisms are named in the documentation, and both are visible in the packaging. The first is Pydantic, which Koheesio uses for typing, data validation and settings management. The stated effect is type safety and structured configuration inside pipeline components, which means a Step is a typed object rather than a loose function with keyword arguments threaded through it.

The second is the Context component. It exists to allow flexible customization of task behaviour, so that the same underlying Step can behave differently depending on what the surrounding job needs. Context and Step together are the shape of the framework: a Step describes the unit of work, and Context carries the settings that shape how that unit behaves at run time.

The project also says it supports multiple implementations and works with various data processing libraries or frameworks, with the stated goal of handling any data processing task regardless of the underlying technology or data scale. The repository topics reflect that breadth, listing data engineering, delta-lake, pydantic, pyspark and python.

Delta Lake and PySpark appear alongside Pydantic and plain data engineering, which is consistent with a framework meant to sit under whatever execution layer a team already runs. The packaging metadata repeats the same intent, listing data transformation, ETL, data validation and data quality as the keyword space the project occupies.

The supported Python range is declared in pyproject.toml

Koheesio is packaged with Hatchling, and the build configuration is short enough to read in full. The relevant part of the project file looks like this:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

The project metadata states the version dynamically rather than hardcoding it, and it declares a supported Python range of 3.9 up to but not including 3.13. The same line carries a comment saying 3.10 is the minimum recommended version. That is a concrete constraint for anyone running on a recent interpreter, since the upper bound excludes Python 3.13 and later.

The keyword list in the project file is worth reading as a statement of scope, because it names the job types the maintainers had in mind:

keywords = [
  "data-engineering",
  "data-pipeline",
  "data-transformation",
  "software-engineering",
  "etl",
  "pyspark",
  "spark",
  "pydantic",
  "data-validation",
  "data-processing",

Alongside those keywords, the classifiers declare the project as Development Status :: 5 - Production/Stable and mark it as Operating System independent. The project file also lists thirteen named authors, mostly with Nike email addresses, which is a reasonable signal about where the work comes from.

For upgrade planning, the important thing to note is that this is a young version line. Version 0.11.0 landed on 2026-07-27, so the number itself signals that the API surface has not yet settled into a long-term guarantee.

Getting Koheesio: PyPI, a Hatch build, and a docs site

The documentation does not give a copy and paste install command, so there is nothing here to run verbatim. It does point to the package index: the README carries a PyPI version badge and a PyPI Python version badge, both linking to the project page on PyPI at https://pypi.org/project/koheesio/. That is where the distribution is published and where the installation instructions live. The project also publishes documentation under docs/ in the repository, and the README links to a separate documentation site at https://engineering.nike.com/koheesio/.

The top-level layout of the repository tells you where to look for things. The root holds .actrc, .github/, .gitignore, CONTRIBUTING.md, LICENSE.txt, README.md, docs/, makefile, mkdocs.yml, pyproject.toml, src/ and tests/.

The source tree lives under src/, tests sit alongside it, and the documentation site is built with MkDocs given the presence of mkdocs.yml. A makefile sits at the root for local developer tasks. The .github/ directory holds the CI configuration, and the README badges confirm two workflows: a test workflow and a release workflow named CD - Release Koheesio.

So the practical first step for an engineer is to install from the PyPI project page into an interpreter that satisfies the 3.9 to 3.12 range, then read the docs/ tree for the Step and Context API before writing your first task.

Airflow and Luigi own the schedule, Koheesio owns the step body

The clearest way to understand Koheesio is against the tools it declines to be. Apache Airflow, Luigi and Databricks Workflows all manage complex computational workflows and generate DAGs. Their unit of concern is the graph: which task runs after which, how a failure propagates, when a schedule fires. Their code is frequently spread across a Task class or a function plus a dependency declaration in a separate file.

Koheesio's approach is the opposite in emphasis. The unit of concern is the task body, and the stated strengths are that a Step is small, reusable and testable, that it carries its own logging, and that it can be composed into larger workflows later by those external orchestrators. The documentation even says Koheesio steps are meant to be orchestrated with other tools or frameworks such as Airflow, Luigi or Databricks Workflows.

So the difference in approach is not feature by feature but architectural. Airflow and Luigi give you the schedule and take the task as a thin callback you register with them. Koheesio gives you a typed, logged, configurable object and hands the scheduling problem to someone else. A team that already runs Airflow can adopt Koheesio under it without replacing it. A team with no scheduler at all gains nothing from Koheesio on its own.

Where Koheesio is the wrong tool

There are several honest cases against choosing it, and most come from the same source, the project's own statement of scope.

First, it is not an orchestrator. If you need dependency management between tasks, retry policy across a multi-stage job, or a schedule, none of that comes from Koheesio. The documentation itself points readers looking for that to dedicated workflow orchestration tools.

Second, it is a framework of libraries rather than a running service, which means the operational cost lands on your team. There is no control plane to inspect, no UI, and no built-in place to see the state of a long job. The observability it provides is the logging that comes with a Step, which is a good default for a single task and much thinner than a scheduler's view of a whole run.

Third, the Python range is capped. The declared upper bound of 3.12 excludes newer interpreters, so a platform team standardising on a newer runtime has a mismatch to solve before anything else.

Fourth, the package is pre-1.0. Version 0.11.1 is the current release line, and a 0.x version number signals that interfaces may still shift between minor releases.

Maintenance pace and the Apache 2.0 license

The project is not archived, and the last push was on 2026-10-17, so the repository is being worked on. Releases confirm the pace: version 0.11.1 shipped on 2026-10-16, a pre-release of the same version, koheesio-v0.11.1a0, went out a few hours earlier the same day, and 0.11.0 had appeared on 2026-07-27.

Licensing is straightforward. The project file declares Apache-2.0, the classifiers list it as an OSI approved Apache Software License, and the repository root carries a LICENSE.txt. The README badges include a link to that license file. Apache 2.0 permits modification and redistribution with notice of changes, and it includes an express patent grant from contributors. This is a short comment on license mechanics rather than advice.

The practical upgrade cost sits with the Python version constraint and with a 0.x API. Both are known and checkable before you commit engineering time, rather than surprises discovered after a pipeline is running.

Editorial conclusion

Adopt Koheesio if your team already runs its jobs under Airflow, Luigi or Databricks Workflows and wants typed, testable task bodies with logging attached. Skip it if you are still looking for a scheduler, a dependency graph generator or a Spark runtime, because the project states plainly that it is not a workflow orchestration tool. Before committing, check the Python floor declared in pyproject.toml, since the supported range stops at 3.12 and the project points to 3.10 as its recommended minimum.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Nike-Inc/koheesio on GitHub
3. Project website
4. README
5. Releases