Dagger.jl and the case for letting Julia build its own task graph

Dagger.@spawn turns a function call into a schedulable task

Dagger.jl is a framework for out-of-core and parallel execution, and its public surface is small enough to describe in one line: you wrap a call in `Dagger.@spawn` and the framework decides how to run it. The README compares the macro to `@async`, which is a fair shorthand for the ergonomics but misleading about the machinery, because an async task stays inside one process while a Dagger task can end up anywhere the scheduler has a worker available.

The useful consequence is that ordinary Julia code becomes parallel code without a separate distributed API. A nested loop body can be wrapped, and the dependencies between iterations are expressed by the data flow between spawned tasks rather than by hand written channel reads and writes. The README names nested loops, large datasets and GPU acceleration as the situations where this pays off.

Nothing about the macro hides the cost of moving data. A spawned task that reads a large array needs that array to reach the worker that runs it, and the README is explicit that Dagger figures out how to "distribute the work and move data around". That phrase carries more weight than it first appears, because data placement is where most of the surprise lives.

The scheduler plans a DAG before work reaches a worker

At the centre of Dagger.jl is a scheduler that the README describes as heavily inspired by Dask. That reference matters for how you should think about the design: the unit of work is a node in a directed acyclic graph, and the scheduler evaluates that graph to decide what can start, what must wait, and where each piece belongs.

The graph can be executed over many Julia worker processes, obtained through Distributed.jl or MPI, over threads, and over GPUs. That list is the project's whole capability surface in miniature, and it is worth separating the two axes. Threads share memory inside one machine. Distributed workers do not, so anything crossing that boundary is copied. GPUs add a third placement question, since an accelerator is not a CPU core and the surrounding work still has to run somewhere.

For a reader deciding whether to adopt the project, the practical question is whether their computation decomposes into a graph with clean edges. If it does, the scheduler has something to work with. If your workload is a long chain of dependent steps with no branching, there is very little for a scheduler to optimise.

Installing Dagger and running your first spawned task

Dagger is distributed through the Julia package manager. The README gives two equivalent routes. In the Pkg REPL mode, reached by typing `]` at the Julia prompt:

pkg> add Dagger

Or, equivalently, through the Pkg API:

julia> import Pkg; Pkg.add("Dagger")

For distributed work you load Distributed first and add workers, then load Dagger. The README example adds one worker per CPU core:

using Distributed; addprocs() # Add one Julia worker per CPU core
using Dagger

The first program to write is the one the README provides. It creates one task, derives two independent tasks from it, and then waits on a final task that prints both results:

# This runs first:
a = Dagger.@spawn rand(100, 100)

# These run in parallel:
b = Dagger.@spawn sum(a)
c = Dagger.@spawn prod(a)

# Finally, this runs:
wait(Dagger.@spawn println("b: ", b, ", c: ", c))

What you should see is a single line with the two values. The structure is the lesson: `a` is produced once and consumed twice, `b` and `c` are siblings that can run at the same time, and the `wait` call blocks until the printing task has finished. If the output never appears, the task graph is where to look first.

Data placement, not core count, decides whether Dagger wins

The README sells the framework as running tasks "in parallel across all your CPU cores, GPUs, and even multiple computers". Taken literally, that is a scaling claim, and it is the claim that deserves the most scrutiny, because the bottleneck in distributed Julia is rarely arithmetic. It is the movement of arrays between processes, and the scheduler decides for you how much movement there will be.

This produces the characteristic Dagger surprise. Work that you would have written as a plain loop over one big array may become a task graph whose edges each carry a copy, and the copies can cost more than the computation. Conversely, work that chunks cleanly across workers, such as a filter over a large column set or a family of independent model fits, can scale well because each worker touches a distinct slice.

The framework's out-of-core framing is the same idea seen from the data side. If your working set exceeds comfortable memory, the graph gives you a way to express it, but the README does not document any knob for tuning how data is partitioned or cached between workers. That absence is the reason to test with your own array shapes rather than with the small random matrix in the README example.

Where Dagger is the wrong tool

Three situations stand out as poor fits. The first is code with heavy shared mutable state. Dagger expresses dependencies through the data returned by spawned tasks, so state that must be shared and mutated across tasks is awkward to express and easy to get wrong. The second is small workloads. Spawning a task and building a graph carries overhead that a plain Julia function call does not, and for a computation that finishes in milliseconds the framework spends more time scheduling than computing.

The third is code that needs a specific piece of work to run in a specific place, say a GPU that a user selected by device index. A scheduler that optimises the graph as a whole does not offer the same level of placement control as writing the device selection yourself.

Two gaps in the documentation are worth naming directly. The README does not document a fallback path when a worker dies mid graph, so recovery behaviour for long running distributed jobs is unstated. It also does not document profiling or instrumentation of the scheduler itself, which makes it harder to answer the first question any adoption debate reaches: where did the time actually go. The repository does contain a FEATURES_ROADMAP.md file that the README points to for missing features and known bugs, and that file is the place to check before you commit.

Dagger against hand written threads and Distributed

The obvious alternative is to stay with Julia's built-in mechanisms. `Threads.@spawn` gives manual parallelism inside one machine with shared memory, and `addprocs()` with `pmap` or `@distributed` gives manual parallelism across processes. Both are simpler, both put you in control of placement, and neither requires a scheduler to infer your dependency structure.

The difference is one of where the work goes. Hand written code needs you to decide, for each piece of work, which worker runs it and how the results get back. Dagger moves that decision into the scheduler, which builds the graph from the values your tasks return. For nested loops and other shapes where the dependency structure is obvious to a reader but tedious to express by hand, that trade is usually good.

It also costs you predictability. Manual parallelism exposes the concurrency structure in your source, so a reader can see the intent. With Dagger the structure lives in the values and in the scheduler, which means you depend on the framework's choices rather than your own. The README credits Dask as the design inspiration for the scheduler, and the same trade-off that shape implies applies: you accept an automated plan in exchange for less bookkeeping.

Release cadence, licence, and what the repository holds

The project is not archived, and the last push was on 2026-10-18, with releases v0.22.5 on 2026-09-17, v0.22.6 on 2026-09-30 and v0.22.7 on 2026-10-13. Those dates show the package is being worked on now, and the patch level sequence suggests the API is treated as stable within the 0.22 line.

The licence needs a separate note. The repository is published under a custom licence that GitHub cannot classify, and the licence text lives in the LICENSE.md file at the root. Anyone planning to use Dagger in a commercial or redistributable setting should read that file rather than assume a familiar permissive licence, and no reading of it is offered here.

The repository layout is worth a quick scan before adoption. There is `src/` and `lib/` for the implementation, `ext/` for extensions, `test/` and `benchmark/` for verification, `docs/` for the manual that the README links to, plus `contrib/`, `AGENTS.md` and `CITATION.cff`. Note also that the DTable abstraction was moved out to a separate DTables.jl repository, so a project that relied on tables inside Dagger will need that second package. Help is offered through the Julia Discourse parallel category and the `#dagger` channel on Julia Slack.

Editorial conclusion

Dagger.jl suits Julia code with independent or nested work that would otherwise be threaded by hand, especially when the data set is too large to fit comfortably in one process. It is the wrong choice for code with heavy shared mutable state, for small workloads where scheduling overhead dominates, and for anyone who needs predictable placement of a specific piece of work. Before adopting it, check FEATURES_ROADMAP.md in the repository for the features you depend on, and confirm that the licence in LICENSE.md suits your organisation, since the project is published under a custom licence rather than a standard one.

Frequently asked questions

Official sources

1. Issues
2. JuliaParallel/Dagger.jl on GitHub
3. README
4. Releases