dag-factory and the case for YAML-defined Airflow DAGs

YAML in, Airflow DAGs out

dag-factory is a library for Apache Airflow that constructs DAGs declaratively through configuration files. The stated benefits are blunt: build DAGs without knowing Python, build DAGs without learning Airflow primitives, and avoid duplicative code. The audience follows from those three lines. A platform team that ships forty near identical pipelines does not want forty near identical Python files, and an analytics group whose members write SQL but not Airflow does not want a two week Airflow course before the first DAG exists.

The tool sits inside Airflow rather than beside it. The minimum requirements are Python 3.10.0 and Apache Airflow 2.9 or newer, and the project states compatibility with Airflow 3, including modern scheduling and updated import paths. So this is not a replacement for Airflow, and it does not run Airflow either. It is a front end that turns a file into the objects Airflow already knows how to schedule.

That framing matters for expectations. You still need an Airflow deployment, you still get Airflow's scheduler, and you inherit Airflow's operational surface. What changes is where the pipeline definition lives.

One loader function is the whole integration point

The documentation organises the library around a single entry point, the load_yml_dags function, and the surrounding pages are variations on configuration. The examples/dags folder in the repository holds more examples, and there are two documented quickstart paths, one for the Astro CLI and one for Airflow standalone.

The configuration surface is where the design shows. The documentation covers schedule configuration, dynamically mapped tasks, defining a Python object, custom operators, defaults, callbacks and the KubernetesPodOperator. That list tells you what a YAML DAG can express and, just as usefully, what it cannot. A YAML file describes tasks, their operators, their dependencies and their parameters. Anything that requires a Python object, a callback body or an unusual operator is expressed by pointing at Python code from the YAML rather than by writing it in YAML.

Defaults are the other lever. A defaults file lets many DAGs share a common set of values, which is how a large estate of YAML pipelines avoids the same duplication the project set out to remove. The practical shape of a dag-factory deployment is therefore a directory of YAML files plus a small amount of Python for the parts that resist configuration.

Mapped tasks, callbacks and Kubernetes are the load-bearing features

Three features determine whether dag-factory fits a given team, and they are worth separating.

Dynamic task mapping is the first. Airflow can fan a task out over a list at runtime, and dag-factory exposes that so a YAML file can express a mapped task without hand-writing the expansion loop. For teams that run the same task over many partitions, this is often the whole reason to move to configuration.

Callbacks are the second. Success, failure and retry callbacks are configurable, and the callback bodies are Python. This is the honest edge of the approach: dag-factory lets you point at a callback function, and the function is yours to write. A YAML first is not a YAML only.

Kubernetes support is the third, and it is the clearest example of a dependency that has to be present. The KubernetesPodOperator is documented as a first class feature, and the project ships a kubernetes extra that pulls in apache-airflow-providers-cncf-kubernetes. The same pattern holds for the http and common-sql extras, and the all extra installs the three together. If a pipeline references a provider operator whose extra is not installed, the failure arrives at DAG parse time, which is the least convenient moment to discover a missing dependency.

Installing for development with uv and a local Airflow

The package is published as dag-factory on PyPI and is built with hatchling. Its runtime dependencies are small and deliberate: apache-airflow>=2.9, pathspec, pyyaml, packaging and typer. The optional extras carry the provider libraries, with all, kubernetes, http and common-sql available separately, and a tests extra pulling in pytest, pytest-cov, pre-commit and pandas.

The repository ships a Makefile whose setup target creates a virtual environment with uv and installs the package in editable mode with the test extras:

uv pip install --python venv/bin/python -e ".[tests]"

The target prints the activation line, which is source venv/bin/activate. For running a local Airflow to test against, the docker-run target builds the wheel first and then starts or restarts an Astro development environment in the dev directory:

cd dev && astro dev start --verbosity debug

That environment serves Airflow on port 8080, which is the port the Makefile checks before deciding to start or restart, and astro dev stop shuts it down. The repo also carries a mkdocs configuration for the documentation site, an AGENTS.md, a CODEOWNERS file and a pre-commit configuration, so the project is set up for outside contributions.

The alternative is writing the DAG in Python yourself

The real alternative to dag-factory is not another YAML tool. It is the Python DAG you would have written anyway, and the difference in approach is worth stating precisely.

A Python DAG gives you the full Airflow API, normal editor tooling, type checking and the ability to use loops and helper functions without restriction. What it costs is repetition, since every pipeline spells out the same scheduling and retry boilerplate, and a knowledge requirement, since whoever edits the file has to understand the framework rather than a config format.

dag-factory inverts the balance. Configuration becomes the common case, duplication drops, and the Airflow primitives become an implementation detail that only a few people need to know. The price is an extra layer between you and the framework, plus a constraint that the pipeline must be expressible in the schema. Whenever a DAG needs real logic, you fall back to Python objects and callbacks, and at that point you are back to maintaining code.

A pragmatic hybrid is the pattern most teams land on: YAML for the pipeline skeleton, Python for the parts that compute, and the custom Python object feature as the bridge. The project supports that path rather than pretending it does not exist.

Version 1.0 brought breaking changes, and Airflow 3 moved the goalposts

Upgrades are the part of this project that needs planning. Version 1.0 is described as introducing important improvements and breaking changes to support modern Airflow usage, and the README points to a migration guide for people upgrading from earlier versions. The library is also now compatible with Airflow 3, with modern scheduling and updated import paths. Those two changes land in the same era, which means a team moving both dag-factory and Airflow at once has two migrations to reason about and one dependency floor of Airflow 2.9 to respect.

The release history shows the shape of it. The 1.1.0 release was preceded by two alpha releases, v1.1.0a1 on 2026-04-29 and v1.1.0a2 on 2026-05-05, and v1.1.0 was published on 2026-05-07. Alphas in the same package name are the detail to watch, since a resolver that does not exclude pre-releases can quietly install one of them.

Python support is broad, with classifiers for 3.10 through 3.14 and both CPython and PyPy listed. Airflow support is narrower at the bottom end, at 2.9, which is a recent-enough floor that most estates will already be above it, but a team on an older Airflow cannot use dag-factory at all without upgrading first.

Apache-2.0, a maintained repository, and a gap between main and the latest tag

The project is licensed under Apache 2.0, with the licence text in the LICENSE file and the metadata declaring the same. That is a permissive licence with an explicit patent grant, so embedding dag-factory in a proprietary platform does not oblige you to publish anything, provided you keep the notice. The repository also ships a SECURITY.md describing how to report vulnerabilities and how reports are handled, a PRIVACY_NOTICE.md pointing at Astronomer's privacy policy, and a CHANGELOG.md, so the governance around the code is documented rather than implied.

The repository is not archived and the last push was on 2026-10-19. That is well after the newest tag, v1.1.0 from 2026-05-07, so main has moved ahead of the last published release by several months. For a library that pins Airflow behaviour, that gap is the thing to plan around: install the tagged version and you get what was tested against Airflow 2.9 and 3 at the time, and install main and you get whatever the maintainers changed since, including fixes for Airflow versions released in the meantime.

Coverage is tracked through codecov and a.coveragerc, and CI runs through a GitHub Actions workflow, both of which suggest the ordinary hygiene of a project with outside contributors. None of that tells you how fast bugs are fixed, and the changelog is where a team upgrading should look first.

Editorial conclusion

Adopt dag-factory if your team ships many similar pipelines, or if the people defining them are not fluent in Airflow primitives, since a YAML file removes the boilerplate and the duplication that comes with it. Do not adopt it for pipelines whose logic is genuinely bespoke, because a YAML DAG still needs the custom Python object escape hatch, and you would be layering a configuration format over code you had to write anyway. Verify three things first: that your Airflow version is 2.9 or newer, that the extra matching your provider is installed, and that you have read the migration guide, since the jump to 1.0 carried breaking changes. The last push was on 2026-10-19 while the newest tag is v1.1.0 from 2026-05-07, so pin the tag rather than main.

Frequently asked questions

Official sources

1. astronomer/dag-factory on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases