Digdag: a workflow automation system for data pipelines and scheduled jobs

What Digdag is

Digdag is a workload automation system, and it comes from Treasure Data.

The unit of work is a workflow, and a workflow is a file with the.dig extension. The examples directory in the repository shows what that means in practice, and the file names are a catalogue of the features. There is a basic workflow, a call, a check task, conditions, an error task, parameter export, a loop over an HTTP response, an include, a task that sends mail, parameters passed to a child workflow, tasks that run Python and Ruby, daily, hourly and weekly schedules, shell access with environment variables, and service level agreement handling with a duration variant.

So the model is closer to a build system than to a cron table. A workflow has tasks, tasks can depend on each other, tasks can be conditional, tasks can fan out into subtasks, and a schedule is a first class thing inside a file rather than a line in a crontab.

The system runs in two shapes. There is a CLI that runs a workflow locally and produces a packaged executable, and there is a server mode that exposes a REST API, a web interface, and the scheduling across many machines. The documentation points at digdag.io and docs.digdag.io for installation and the user manual, with a separate API document and a release notes directory in the source tree.

The module layout, and what each piece is for

The repository root is a Gradle build with a long list of subprojects, and the names describe a layered design.

There is a core module, a commons module, a client module, a server module and a cli module, which is the obvious split for an application with a server and a command line front end. There is a plugin utilities module and an spi module, which is the extension point, and a standards module, which is where the formats and the shared contracts live.

The storage side is where the interesting modules are. There are separate storage modules for S3 and for GCS, so object store backends are plugins rather than being hardcoded into the core. The server side has two Guice based variants, a plain one and one built on Undertow, with the shared parts in a third module, which is the standard arrangement for running the same server on two servlet stacks.

The rest is the machinery around it: a ui module, a tests module, a client module for talking to the server, a docs module, and a docker directory. There is also a separate digdag-docs repository whose gh-pages branch hosts the built website, which is published from the documentation sources in this tree.

The split into a core, a server, a client, a cli and a set of storage plugins is what lets the same workflow definition run in a local development run and in a server that schedules it across a fleet.

The .dig model, seen through the examples

The examples directory is the best documentation of the language, and the file names tell you what a workflow can contain.

The basic file is the starting point. The call example shows one task invoking another, which is the composability primitive. The conditions file is the branching mechanism, and the check task and error task files are the two halves of validation, a task that passes and a task that fails on purpose.

Then there is the parameter story, with a params file, a params_child file and an export_params file, which together cover declaring a value, passing it to a child workflow and getting a value back out. The include file is the other composition mechanism, pulling a reusable sequence into a workflow.

The subtask generation is where the language earns its keep. There are files for Python and for Ruby, each with an args variant and a generate subtasks variant, plus a JavaScript style variable file. The generate form is how a workflow turns a list of items into many parallel tasks, which is the case that a shell script handles badly and that most scheduling tools handle not at all.

The schedule files are daily, hourly and weekly, and the shell environment variable file is how a task reads a secret from the environment rather than from the workflow file. The service level agreement files, one of them with a duration, are about alerting, and the HTTP for each file is a loop driven by a remote list.

Every one of these is a file in the examples directory, which is an unusually good way to learn a workflow language, since each one is a small complete program that demonstrates exactly one idea.

Building the CLI, and the user interface

The build instructions for the command line tool are two commands, and the second one exists for a specific reason.

$ ./gradlew cli
$ ./gradlew cli -PwithoutUi  # build without integrated UI

The output is an executable in the pkg directory, for example a jar named with the version. The withoutUi flag is what you reach for if the build fails during the user interface step because of errors from the node command, which the README explains is how the UI is excluded from the package.

That tells you something about the architecture. The web interface is not a separate application that talks to the server over a network, it is built into the package. The same command produces a server with a UI and a server without one, and the difference is a build flag.

For developing the interface itself, the README describes a different arrangement, because the interface has its own Node development server. You put a set of headers in the digdag configuration file, and the header list is long, including the allowed origin, the allowed headers, the credentials flag and the allowed methods, with a max age value. Then you install the node dependencies in the ui directory and start the development server, which comes up on port 9000.

The point of the headers is that the development server on 9000 has to be allowed to call the digdag server, and doing that in the configuration file rather than in the interface code keeps the CORS policy on the server side, where it belongs.

Testing, and the two database backends

The test story is layered, and the layer boundaries are explicit in the commands.

The full check runs the unit tests and the static analysis, and the reports tell you where the output lands. The coverage report is generated under the jacoco test report path in the build output, and there is a separate findbugs report for the static analysis findings.

The acceptance tests are a separate target and need an environment variable to run, because they exercise the built product rather than the classes. The README names the variable, and notes that the tests option Gradle provides is useful for running the digdag-tests subproject locally, and that the environment variable is needed to execute them at all.

$ CI_ACCEPTANCE_TEST=true ./gradlew digdag-tests:test --info --tests acceptance.BuiltInVariablesIT

The database story is the part worth planning for. Tests use an in-memory H2 database by default, and to run them against PostgreSQL you set an environment variable whose value is the contents of a properties file.

$ export DIGDAG_TEST_POSTGRESQL="$(cat config/test_postgresql.properties)"

So the same suite runs against two engines, and switching is a one line change. That matters more than it sounds, because a scheduler that only ever runs its tests against H2 will surprise you the first time it talks to PostgreSQL, and the reverse is just as true.

Documentation, the REST API and the release process

The documentation is a Python project and it builds the public site, and the API document is generated rather than written.

The docs live in a digdag-docs source directory, are built with Sphinx, and the site is hosted on the main domain using GitHub Pages. The build is done by a deployment step in the circle configuration, which pushes the result to the gh-pages branch of the digdag-docs repository. To build the pages locally you create a Python virtual environment, install the requirements and constraints files, and run the site task through Gradle. The README notes that this does not always update everything, because Sphinx does not manage its dependencies well, and that running the clean task first fixes it.

The REST API document is generated from the code. A Gradle task dumps the swagger file, and the server can be started with an option to expose the current API, after which a swagger UI container can be pointed at it.

./gradlew swaggerYaml  # dump swagger.yaml file

The release procedure is documented for committers only, and it is worth reading because it shows how the pieces fit. It needs an account with the Sonatype repository hosting service and the credentials in the Gradle properties file, plus a PGP key configured for signing, because the artifacts go to Maven Central. Then it is a version bump task, release notes written into a release file for the version, a clean build with the check and site tasks, a release branch, a pull request, a merge, a repeat of the build, the release task itself, a tag, a GitHub release and the upload of the packaged jar.

The build is required to run under JDK 8 for this procedure, which is a fact about the project's era that you will feel in your own toolchain.

The state of the project

Two dates tell the story.

The most recent tagged releases are v0.10.5.1 from February 2024, v0.10.5 from February 2023 and v0.10.4 from January 2022. And the last push to the repository was on 2026-09-04.

So the version numbers have not moved for a couple of years while the repository has continued to receive work. For a tool like this, that pattern is common and not alarming on its own, because the artefact that matters is the packaged jar and the running server rather than a version number, and the release notes directory in the source tree is where the history of what changed lives.

What is worth planning around is the toolchain the project documents. The development requirements are JDK 8, Node.js 12.x and Python 3, and the release procedure explicitly needs JDK 8. Node.js 12 is long out of support, and the README even shows how to install that specific version with a version manager, which tells you the project has not moved off it.

There is also a development note about IntelliJ IDEA, and it is a good example of a real friction point. Digdag uses a Java annotation processor for the immutables library, and the combination of annotation processing with Gradle in the IDE sometimes produces compile errors about a symbol that cannot be found. The recommended fix is to enable the option that delegates the IDE build and run actions to Gradle, which is a small thing to do and removes a class of confusing errors.

So the honest summary is a mature, well documented system with a large and thoughtful example set, whose build environment asks for more care than a modern Java service would.

Editorial conclusion

Adopt Digdag if your scheduled work has dependencies, retries and a web interface requirement that cron entries cannot express, because a .dig file describes a task graph rather than a single command. Do not adopt it for a handful of unrelated nightly scripts, where a scheduler and a shell script are less machinery. Before you build from source, check the toolchain, because the documented development requirements are JDK 8 and Node.js 12, and both are old enough to be a problem on a current machine.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. treasure-data/digdag on GitHub