Apache Airflow: a scheduler for Dags that keep their shape, built on a two-stage image

Airflow states its own limit: it works best with static, slowly changing Dags

The Project Focus section is unusually direct about where the design fits. Airflow works best with workflows that are mostly static and slowly changing, and when the Dag structure is similar from one run to the next it clarifies the unit of work and continuity. The projects it puts next to itself are Luigi, Oozie and Azkaban.

That sentence is the single most useful thing in the file for a reader choosing a tool, because it names the failure case as precisely as the success case. A workflow that decides its own shape while running has no stable unit of work to point at, and the scheduler's whole model depends on the dependency set being known in advance. The payoff Airflow is buying with that restriction is the one stated next: workflows defined as code become more maintainable, versionable, testable and collaborative, and the scheduler runs tasks on an array of workers while following the specified dependencies. If your graph changes shape per run, this is the tradeoff you are being asked to make, and no configuration flag removes it.

requires-python excludes 3.15 outright, and the build backend pins exact versions

The package metadata is where version constraints live, and it is strict in two different ways. The runtime floor is declared as:

requires-python = ">=3.10,!=3.15"

The exclusion is a hard one. A resolver on Python 3.15 refuses the package instead of warning, which is a better failure than an import error later, and it means a base image has to be chosen with 3.15 ruled out rather than discovered at deploy time. The build side is stricter still, because hatchling is the declared build backend and every build requirement is pinned exactly:

[build-system]
requires = [
    "hatchling==1.32.4",
    "packaging==26.3",
    "pathspec==1.1.1",
    "pluggy==1.6.0",
]
build-backend = "hatchling.build"

The consequence for anyone building in an isolated environment is that the build needs those exact versions, plus `tomli==2.4.1` on Python below 3.11 and `tomlkit==0.15.1`, which is why the build step is more fragile than the install step in a locked-down CI image.

The production Dockerfile is two images, and it calls itself alpha quality

The header of the repository's Dockerfile states the intended use and then qualifies it: this Dockerfile is intended for production use and deployment, and it is alpha quality for now because the project is in a process of testing it. That comment is the honest summary of the file, and a reader planning a container deployment should treat it as a constraint rather than a formality.

The structure explains why. The file is multi segmented and contains two images. The first, airflow-build-image, is where all Airflow dependencies can be installed and built, including the ones that need build essentials, with Airflow itself installed into a `${HOME}/.local` virtualenv that Python also treats as a `--user` folder when a venv is created with `--system-site-packages`. The second, main, is the actual production image: much smaller, no build essentials, and it works because the `${HOME}/.local` folder is copied across from the build image.

So a failure inside the container is often a missing compiler in the wrong segment rather than a broken install, and a reader debugging it needs to know which of the two images a command lands in. The tree carries `Dockerfile.ci`, `docker-tests/` and `docker-context-files/` alongside it, which is where the project's own coverage of that path lives.

Two comment markers make the README a generated file for GitHub and PyPI at once

The README opens with a licensed header, then a `START Apache Airflow, please keep comment here to allow auto update of PyPI readme.md` marker, the project description, and a matching `END` marker. Everything between them is machine managed. The package metadata points at a different file entirely for the same purpose, with `readme = { file = "generated/PYPI_README.md", content-type = "text/markdown" }`, and a `generated/` directory sits at the top level of the tree.

The consequence for a reader is that there are two artifacts and one source, and the one you see on PyPI is not the file in the repository. A contributor editing inside those markers loses the edit on the next sync, and a reader comparing the GitHub page with the PyPI page may be looking at different text without either being wrong. The rest of the file makes the same point structurally: a `doctoc` generated table of contents carries a `DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE` instruction and lists everything from Project Focus through Voting Policy to Sponsors.

For navigation that is a strength, since the section names tell you the project's own priorities. For a reader hunting for an install command, the contents list shows the shape of the answer, and the shape is four separate sections: Installing from PyPI, Installation, Official source code, and Convenience packages.

The core ships 3.3.2 while the Java SDK is still at 1.0.0-beta1

The release list mixes maturity levels, and the tag names are what tell you which is which. Apache Airflow 3.3.2 was published on 2026-09-17 and 3.3.1 on 2026-08-12, both plain core releases. The third entry is different: `java-sdk/1.0.0-beta1`, published 2026-07-13, is a beta of a separate artifact rather than a core patch, and the tree has a `go-sdk/` directory alongside it.

So the classifier `Development Status :: 5 - Production/Stable` in the package metadata describes the core, not every artifact carrying the Airflow name. A team running Airflow itself on 3.3.2 is on a stable line. A team writing a JVM client against the Java SDK is picking up a beta whose first 1.0.0 was still in progress in July. Those are different risk decisions and the release list is the place to see them apart.

The build workflows point the same way. Status badges cover the main branch and the 3.x line on `ci-amd.yml` and `ci-arm.yml`, so arm64 is a tested target rather than an afterthought, and the last push to the default branch, main, landed on 2026-09-29.

constraints/, chart/ and the SDKs share a tree with the release tooling

The top level of the repository is where the operational surface shows. Alongside `airflow-core/` sit `airflow-ctl/` and its tests, `airflow-e2e-tests/`, `clients/`, `chart/`, `constraints/`, `dev/`, `devel-common/`, `docker-stack-docs/`, `docs/`, `contributing-docs/` and `generated/`. The linting and release configuration sits with them: `.pre-commit-config.yaml`, `.codespellignorelines`, `.hadolint.yaml`, `.markdownlint.yml`, `.rat-excludes`, `.cherry_picker.toml`, `codecov.yml`, `.gitpod.yml` and `.devcontainer/`.

Two consequences follow for a reader who wants to change or pin things. First, `constraints/` is part of the source tree rather than a downstream artefact, so whatever lands there affects what every install resolves, and the `.cherry_picker.toml` that governs backports sits next to it. Second, the container story is documented in the repository as `docker-stack-docs/`, `chart/` and `INSTALLING.md`, and the `INSTALL` and `INSTALLING.md` files at the top level sit next to `BREEZE.rst`, which is the environment management path the README's table of contents does not spell out at all.

The practical takeaway is that Airflow's install story is a set of files to read rather than one command to memorise, and that is by design: the same tree that ships the scheduler also ships the way it is expected to be deployed.

apache-magpie has its own lock file, which puts agent-written patches in the review path

The README's table of contents includes a section titled Agent-assisted contribution (apache-magpie), between Community standards and Voting Policy. The tree backs that up with `.apache-magpie.lock` at the top level, a `.apache-magpie-overrides/` directory, and an `.agents/` directory, next to `AGENTS.md` and `CLAUDE.md` and a `.claude/` directory.

What that combination means is narrow and concrete. The project has a named route for contributions drafted with an agent assistance, that route is versioned in the same repository as the scheduler, and its configuration is pinned by a lock file with an overrides directory beside it. Nothing here says an agent writes the patch on its own; it says the path exists and that the project keeps its state in files a reviewer can read and diff, which is the part that matters to a committer.

The broader convention is visible too. `CONTRIBUTING.rst`, `CODE_OF_CONDUCT.rst`, `GOVERNANCE.md`, `COMMUNITY_ESCALATION.md`, `ISSUE_TRIAGE_PROCESS.rst`, `COMMITTERS.rst` and `RELEASE_NOTES.rst` are all top level files, so process is part of the tree rather than a wiki page. The cost of that arrangement is that a contributor has to read several documents before their first patch, and the benefit is that the rules are versioned with the code they govern.

Editorial conclusion

Airflow fits teams with batch pipelines whose shape is known weeks ahead, where authoring the graph in Python and versioning it buys more than the runtime flexibility costs. It is a poor fit for jobs whose fan-out is only known at run time, and a cautious one for a container deployment, because the repository labels its own production Dockerfile alpha quality. Before you commit, confirm your Python version is not 3.15, read the Installing from PyPI section for the command rather than reconstructing one, and decide whether the constraints directory in the same tree is something you are willing to track.

Frequently asked questions

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes