luigi is five commands and a daemon, and the dependency graph lives in Python

Five console commands, and three of them only look

The manifest installs five scripts, and the names tell you what the project thinks a workflow tool needs:

[project.scripts]
luigi = "luigi.cmdline:luigi_run"
luigid = "luigi.cmdline:luigid"
luigi-grep = "luigi.tools.luigi_grep:main"
luigi-deps = "luigi.tools.deps:main"
luigi-deps-tree = "luigi.tools.deps_tree:main"

Two of them are the obvious pair. `luigi` runs tasks and `luigid` is the server. The other three are the interesting part: `luigi-grep` searches something, `luigi-deps` answers a dependency question, and `luigi-deps-tree` prints a tree.

That is a design position rather than a feature list. A pipeline tool whose only command runs a workflow treats the graph as something you submit. Luigi treats it as something you interrogate, from a shell, while the workflow is failing. When a schedule of thousands of tasks breaks and you need to know which upstream change caused three hundred downstream failures, being able to ask that as a command rather than clicking through a web page is the difference between minutes and an afternoon.

The tools live under `luigi/tools/` in the package, so they are maintained alongside the scheduler rather than bolted on by a user.

The graph is a Python program, which is why dates work in dependencies

The philosophy section is the clearest statement of the design. Conceptually Luigi is similar to GNU Make, where you have tasks and those tasks may have dependencies on other tasks, and there are similarities to Oozie and Azkaban. One major difference is that Luigi is not built specifically for Hadoop, and it is easy to extend with other kinds of tasks.

Then the part that matters. Everything in Luigi is in Python, and instead of XML configuration or similar external data files, the dependency graph is specified within Python. The stated benefit is that it makes it easy to build complex dependency graphs where the dependencies can involve date algebra or recursive references to other versions of the same task.

That is a real capability rather than a convenience. A task that runs over yesterday's partition needs yesterday's date, and in a data file you either write the date out or you template it. In Python you compute it. The recursive-reference case is the same idea: a task that depends on the previous version of itself needs arithmetic, not a copy and paste.

The cost is the other half. Your schedule is a program, so there is nothing to edit in a data file and no way to inspect the graph without running Python.

Atomic local writes are the stated reason a pipeline cannot be left half-done

The toolbox paragraph names the task templates, and among them is the one that is a correctness property rather than a convenience. Alongside support for Python mapreduce jobs in Hadoop, Hive and Pig, and file system abstractions for HDFS, there are local files, and the promise is that all file system operations are atomic.

The sentence that follows explains why: this is important because it means your data pipeline will not crash in a state containing partial data.

That is the guarantee, and it is worth being precise about what it covers. It is a property of the local file system layer. HDFS has its own abstraction, and its behaviour is HDFS's. So the atomicity claim is about the case where a task writes to local disk, and a crash mid-write leaves either the old content or the new content rather than a truncated file that the next task happily reads as complete.

For a scheduler whose whole job is restarting things after failure, that is the property that makes restart safe rather than merely possible. A pipeline that can leave partial output behind is a pipeline where every retry has to reason about whether the previous attempt finished, and the examples directory includes `per_task_retry_policy.py` for the retry side of the same concern.

It is explicitly not a replacement for Hive, Pig or Cascading

The Background section names its neighbours and then rules out the obvious misreading. There are other software packages that focus on lower level aspects of data processing, like Hive, Pig, or Cascading, and Luigi is not a framework to replace these. Instead it helps you stitch many tasks together, where each task can be a Hive query, a Hadoop job in Java, a Spark job in Scala or Python, a Python snippet, dumping a table from a database, or anything else.

That is the clearest statement of scope in the README, and it is a layer statement rather than a quality statement. Luigi is a scheduler. Those are execution engines. Adopting Luigi means adopting it alongside the tools you already have, and the value is entirely in what it adds: dependency resolution across heterogeneous tasks, retries, and a view of the graph.

The task templates follow from that. The repository ships helpers for the engines people actually had in 2013 and still do, Hive, Pig, Hadoop, Spark and SQLAlchemy table dumps, and the examples directory has `pyspark_wc.py`, `terasort.py`, `top_artists_spark.py` and `wordcount_hadoop.py` alongside a plain `wordcount.py`.

For a reader choosing a tool, the question is not whether Luigi beats a query engine. It is whether you need a sequencer, and if you do, this one is written in the same language as your Python tasks.

The visualiser is a daemon, so the graph lives in a running process

The web interface is described in one sentence: the Luigi server comes with a web interface too, so you can search and filter among all your tasks. The dependency graph example that follows is a screenshot from production, where each node is a task that has to be run, green meaning already completed and yellow meaning yet to run.

The architecture behind that picture is the part worth understanding. The state that makes the graph renderable is held by the server, which is the `luigid` process, and the web interface it serves is built on `tornado`, which is one of only five runtime dependencies. Workers connect to it.

That has two consequences, one good and one to plan for. The good one is the graph: a schedule of thousands of tasks with completion state per task is exactly what a central process is for, and it is why the visualiser can show green and yellow at all. The planning item is that something long-lived has to be running, supervised and restarted, in addition to your scheduler. `python-daemon` is in the dependency list for that reason, and its version is platform-split, which is the next thing worth looking at.

Python 3.15 is excluded, and python-daemon is capped on Windows only

Two dependency decisions in the manifest are specific enough to check before you install anything.

The interpreter range is `>=3.10, <3.15`, and the classifiers name 3.10 through 3.14, which matches the README's tested list. The upper bound is the interesting part. It is not an omission, it is a ceiling, and an installer on 3.15 will refuse or warn rather than install, so the day 3.15 lands this needs a release before you can use it.

The runtime dependencies are short. `python-dateutil` for the date algebra, `tenacity` for retries, `tornado` for the web interface, `typing-extensions`, and `python-daemon` for the server process. That last one is declared twice, split by platform:

python-daemon<2.2.0; sys_platform == 'win32'
python-daemon; sys_platform != 'win32'

So on Windows you are held below 2.2.0 and everywhere else the version is open. The manifest gives no reason for the split.

One more to be aware of: the `toml` extra is capped at `toml<2.0.0`, and it exists so Luigi can read TOML configuration files from users. It has nothing to do with reading the project's own manifest, which hatchling does natively.

The PyPI description is three clauses with an abbreviation in it

What a reader sees on PyPI is not what the README says, and the difference is in the manifest.

The package description field is:

Workflow mgmgt + task scheduling + dependency resolution.

Three clauses, two of them nouns, and `mgmgt` for management. The README's own first paragraph is a sentence: Luigi is a Python package, with the tested versions listed, that helps you build complex pipelines of batch jobs, and it goes on to list dependency resolution, workflow management, visualization, handling failures and command line integration.

So the summary a pip user reads is a compressed version of a description that already existed in better shape one file away. It is a small thing, and it is the kind of small thing that is the first thing a new contributor fixes.

The build explains the plumbing. It uses `hatchling` with `hatch-fancy-pypi-readme`, which is the pattern for rendering a reStructuredText README into a package long description. And the README is RST, not Markdown, which is why the source begins with `.. figure::` and `.. image::` directives for the logo and the badges. The Python packaging community has largely standardised on Markdown for READMEs, and this project did not.

catalog-info.yaml says this is a Backstage service, and the named users are from 2013

Two things at the top level, and one claim in the README.

`catalog-info.yaml` is a Backstage software catalog descriptor. That is a Spotify-internal convention appearing in a public repository, and it tells you the project is developed inside a system where every service has catalog metadata. Beside it are `RELEASE-PROCESS.rst`, which documents how a release happens, `uv.lock` for dependency locking, `tox.ini` for the test matrix, `codecov.yml` and a `.coveragerc`.

The claim is the Who uses Luigi section. It says Luigi is used internally at Spotify to run thousands of tasks every day in complex dependency graphs, mostly Hadoop jobs, powering recommendations, toplists, A/B test analysis, external reports and internal dashboards.

Then the public evidence, and it is dated. The first named users are Spotify with a 2014 presentation, Foursquare with a 2013 presentation, Mortar Data, later Datadog, with documentation and a tutorial, and Stripe with a 2014 presentation. The README also says that because Luigi is open source with no registration walls, the exact number of users is unknown, and that based on the number of unique contributors it expects hundreds of enterprises to use it.

So the first entries in that adoption list are a decade old while the project itself is current, with v3.8.1 released on 2026-05-07 and the last push to the default branch on 2026-10-14. The stated internal use at Spotify is the part with no date attached, and the contributor-count inference is offered as an expectation rather than a measurement.

Editorial conclusion

Luigi fits a team with an existing stack of batch tools, Hive, Spark, Hadoop, a database dump, who needs something to sequence them, retry them, and show a graph of thousands of tasks to whoever is on call. It does not fit a team looking for a query engine, because the README states plainly that it is not a framework to replace Hive, Pig or Cascading, and it does not fit a team on Python 3.15, which the manifest excludes. Before adopting, check three things: that your graph is expressible as Python, because the dependency specification lives in code rather than in a data file, that you can run `luigid` as a long-lived process, since the state that makes the visualiser work lives in the daemon, and the two dependency edges worth checking first, `tenacity` for the per-task retry policy and `tornado` for the web interface.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. spotify/luigi on GitHub