# d6tflow: task graphs for research workflows instead of production pipelines

> d6tflow is a Python library that turns a data science workflow into a graph of parameterised tasks with cached inputs, outputs and metadata, and it is aimed at researchers rerunning models rather than teams scheduling production jobs. The catch is that the project has no tagged GitHub releases, so upgrades are decided by whatever pip resolves.

**d6t/d6tflow** — Python library for building highly effective data science workflows

- Repository: https://github.com/d6t/d6tflow
- Website: https://d6tflow.readthedocs.io/en/latest/
- Stars: 949 · Forks: 72
- Language: Python
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/d6t-d6tflow

## A workflow library for people who rerun the same notebook

The README frames d6tflow as a library for data scientists and data engineers who want to build complex data science workflows easily, fast and intuitively, and it states plainly that the primary audience is data scientists building better models faster. The problem it names is concrete: a research workflow chains parameterised tasks that pass multiple inputs and outputs between each other, output data lands in dataframes, files and databases, and you have to track where everything is by hand. Worse, you often want to rerun a task with different parameters without accidentally rerunning a long-running step. The stated outcome of that mess is code that is hard to audit and does not scale. d6tflow's answer is a task graph in which each step declares its inputs, its outputs and its parameters, and the library works out what actually needs to execute. The README also positions it for data engineers doing load, filter, transform and join work over pandas, dask, pyspark, sql or athena, but as a lightweight alternative rather than a production pipeline engine.

## Task classes, parameters and the decorator that wires dependencies

The mechanism is visible in the model comparison example. A task subclasses one of the library's base classes, for example d6tflow.tasks.TaskPqPandas, and the base class decides how the task's data is persisted. A class attribute of persist = ['x','y'] declares that the run method should save those two objects, and the run method calls self.save with a dictionary of them. A second task, ModelTrain, inherits from d6tflow.tasks.TaskPickle so the fitted model is pickled, and the decorator @d6tflow.requires(GetData) declares that ModelTrain consumes GetData's output. Parameters are declared as class attributes built from d6tflow.Parameter(), and the run method reads self.inputLoad to fetch the upstream data, then writes the result with self.save and records metadata with self.saveMeta. Metadata is the part worth noting, because it is what lets a workflow report results per run rather than just artifacts. The example's final line reads flow.outputLoadMeta() to pull the model scores back out.

## One task class, several parameter values, one workflow manager

Parameterised tasks are only half the design. The dependency and parameter declarations are two lines:

```python
@d6tflow.requires(GetData)  # define dependency
class ModelTrain(d6tflow.tasks.TaskPickle):
    model = d6tflow.Parameter()  # parameter for model selection
```

The example ends with d6tflow.WorkflowMulti(ModelTrain, {'model1':{'model':'ols'}, 'model2':{'model':'gbm'}}), which registers the same task class twice with different parameter sets, and then calls flow.run() to execute model training including all dependencies. The effect is that one class definition covers both a linear regression and a gradient boosting variant, and the workflow manager schedules the shared GetData task once while running ModelTrain for each parameter combination. Scores are then read back as a mapping keyed by the parameter names, which is what makes model comparison a one-liner rather than a bookkeeping exercise. The same machinery is what the README means by experiment management, running workflows that compare different models to find the best one. The cost of this design is that the parameter set becomes part of the workflow definition, so a grid of variants grows the workflow object rather than a separate loop in your code.

## Caching is the feature that decides the cost of an experiment

The README lists cache data among the things d6tflow can do for you, and the wording is save and load intermediary calculations to reduce model training time. The base task classes make this concrete: TaskPqPandas for pandas data, TaskPickle for objects, and the persist declaration on the class telling the library which keys survive between runs. When a parameter changes, the README says the library will intelligently rerun the workflow after changing parameters, code or data, and it will intelligently continue workflows after failed tasks. In practice this is the difference between a research loop that re-trains everything on each parameter change and one that reuses the data preparation step. It is also the reason the example calls flow.reset_upstream(confirm=False) with a comment marking it as demo only and a forced re-run. That line is the visible footgun of the library: a reset invalidates cached upstream work, and in a real pipeline that costs hours of recomputation, so the method deserves more care than a one-line example gives it.

## Installing d6tflow, and what the dependency list actually pulls in

The README gives a plain pip install, an update path, and a source install for people behind an enterprise firewall.

```bash
pip install d6tflow
pip install d6tflow -U
```

That is the whole installation story for most people, and the README notes that the project is Python 3 only, so you might need to call pip3 install d6tflow if you have not set python 3 as default. The tree ships setup.py with version 26.6.6 and python_requires '>=3.5', with packages d6tflow, d6tflow.targets and d6tflow.tasks. The base install_requires list is pandas, pyarrow, markdown, openpyxl and loguru, so you get parquet-backed storage, spreadsheet reading and logging without asking. The extras cover the wider data surface: a dask extra with toolz and dask[dataframe], a cloud-base extra with universal_pathlib, a gcs extra adding gcsfs, an s3 extra adding s3fs, and an export extra with jinja2. Choosing s3 or gcs pulls a filesystem library in, so the target you persist to decides your dependency surface. A requirements.txt at the root additionally lists pandas, toolz, dask[dataframe] and pyarrow, which is a slightly wider set than the base install and a useful hint about what a working environment actually needs.

## Where d6tflow deliberately stops short of Airflow and Luigi

The README is unusually direct about this. It links a write-up on how d6tflow is different from airflow and luigi, and the difference is scope rather than technique. d6tflow describes itself as focusing on managing data science research workflows instead of managing production data pipelines, and the data engineer use case is framed as a lightweight alternative that helps productionize models faster. The same distinction shows up in the feature list: d6tflow talks about visualising task dependencies and their execution status, executing tasks including dependencies, continuing after failures and sharing output data, but there is no mention of a scheduler, a worker fleet, alerting or retry policy. If your requirement is a job that must run at 3am and page someone when it fails, this library is the wrong shape of tool.

## No GitHub releases, a version string in setup.py, and a Claude Code plugin

Maintenance facts are limited and worth stating without interpretation. The repository is not archived, and the last push was on 2026-06-28. The repository has no GitHub releases, so there is no changelog to read and no upgrade path written down for users. The only version marker visible in the tree is the version string 26.6.6 in setup.py, and the name and author fields credit the DataBolt Team. Licence is MIT, with a LICENSE file at the root, which covers the library code and places no restriction on commercial use, though it says nothing about the licence of the data you process. The repository also carries a CLAUDE.md file and documents a d6tflow Claude Code plugin in a separate repository, which adds a skill that auto-activates when pipeline files named tasks.py, flow.py or run.py are edited, along with slash commands such as /d6tflow:init-project and /d6tflow:init-gitlfs for setting up Git LFS to version data outputs. The plugin gives a hint about the intended project layout, and it is worth noting that the init-gitlfs command exists because outputs are large enough to need Git LFS rather than a plain commit.

## Conclusion

d6tflow fits an analyst who reruns the same feature engineering and model training code with different parameters and wants the expensive steps skipped, and it does not fit a team that needs scheduled, monitored production jobs because the library positions itself against Airflow and Luigi rather than with them. Before committing, check that the tasks.py, flow.py and run.py files the Claude Code plugin watches match how your team actually organises a project, and pin the package yourself, since the repository publishes no GitHub releases and the setup.py version string is the only version marker in the tree.

## FAQ

### How do I install d6tflow?

Run pip install d6tflow, and pip install d6tflow -U to update. The project is Python 3 only, so you may need pip3 install d6tflow, and behind an enterprise firewall you can clone or download the repository and run pip install .

### How does d6tflow cache intermediate results between task runs?

Tasks inherit from a base class such as d6tflow.tasks.TaskPqPandas or d6tflow.tasks.TaskPickle, and a persist class attribute lists the keys to keep, for example persist = ['x','y']. Inside run, self.save writes the data and self.inputLoad reads it back, so a rerun with changed parameters reuses the cached step instead of repeating it.

### Is d6tflow a replacement for Airflow or Luigi?

No. The README states that d6tflow focuses on managing data science research workflows instead of managing production data pipelines, and for data engineers it is described as a lightweight alternative that helps productionize models faster. It has no scheduler or worker fleet of its own.

## Sources

- [d6t/d6tflow on GitHub](https://github.com/d6t/d6tflow)
- [Issues](https://github.com/d6t/d6tflow/issues)
- [License: MIT](https://github.com/d6t/d6tflow/blob/master/LICENSE)
- [Project website](https://d6tflow.readthedocs.io/en/latest/)
- [README](https://github.com/d6t/d6tflow/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/d6t-d6tflow