dLLM: A Unified Training and Evaluation Hub for Diffusion Language Models

The Problem: Diffusion Language Models Lack a Common Toolchain

Diffusion language models (DLLMs) generate text by iteratively denoising masked or corrupted tokens, unlike autoregressive models that predict one token at a time. Research in this area has produced a dozen architectures, from LLaDA and Dream to Edit Flows and block diffusion. Each comes with its own training script, evaluation quirks, and inference loop, which makes reproducing results or comparing methods painful. dLLM addresses this by providing a single library that unifies training and evaluation on top of established Hugging Face components. The intended user is a researcher or a machine learning engineer who wants to train or fine-tune a diffusion model without writing the boilerplate for distributed training, checkpointing, and evaluation. The README positions it as a transparency and reproducibility tool, though it also serves as a practical starting point for those new to the field.

How dLLM Works: Trainer, Harness, and Example Recipes

The core mechanism is to build on existing infrastructure rather than invent new ones. For training, dLLM uses the Hugging Face transformers Trainer, which means it inherits support for LoRA via PEFT, DeepSpeed, and PyTorch FSDP. For evaluation, it wraps the EleutherAI lm-evaluation-harness, abstracting away the inference details so that the same evaluation code works across different models. The library then provides minimal recipes under the examples directory. Each subdirectory, such as examples/llada or examples/dream, contains the configuration and scripts needed to pretrain, fine-tune, or evaluate a specific model family. The README also mentions support for converting any autoregressive model into a diffusion model (the a2d example) and any BERT encoder into a chatbot (the bert example). This design means that the actual data flow is dictated by the underlying transformers Trainer, with dLLM adding the diffusion-specific loss and sampling logic on top. The result is a modular approach where adding a new model is mostly a matter of writing a new example folder.

Getting Started: Setup and Installation Commands

The setup requires Python 3.10 and a CUDA-capable environment. The README gives a concrete sequence: create a conda environment named dllm, install CUDA 12.4 from the nvidia channel, and then install specific PyTorch versions using pip with the index URL. The exact command is: conda create -n dllm python=3.10 -y, conda activate dllm, conda install cuda=12.4 -c nvidia, and then pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124. The README notes that other PyTorch and CUDA versions should also work, but these are the pinned ones. After that, you would clone the repository and follow the instructions in the specific example folder, such as examples/llada for training LLaDA. There are no release tags, so you rely on the default branch's latest commit. The installation is straightforward for anyone familiar with conda and pip, but the lack of a setup.py or pyproject.toml mention in the README means you likely need to run scripts from the repository root.

What You Can Do: From Pretraining to RL Fine-Tuning

The examples directory is the heart of dLLM. For LLaDA, you can pretrain, fine-tune, and evaluate, including the Mixture-of-Experts variant LLaDA-MoE. For LLaDA2.0 and LLaDA2.1, only inference is provided, which suggests that the training recipes for these newer versions are not yet public. Dream supports pretraining, fine-tuning, and evaluation. The a2d example converts any autoregressive model, like Qwen or LLaMA, into a diffusion model using masked or block diffusion, and the bert example turns any BERT encoder into a chat model. Edit Flows is labeled as an educational reference for training with edit operations like insertion and deletion. Fast-dLLM integration accelerates inference and evaluation for LLaDA and Dream using cache and confidence-threshold decoding. Finally, the rl example supports diffu-GRPO training for reasoning tasks. This is a broad coverage, but the depth varies. The README does not list any release versions, so the maturity of each recipe is unclear until you inspect the code.

A Genuine Limitation: Educational Focus and Reproduction Caveats

The README contains a notable caveat, even though it is commented out in HTML, it says the repository is primarily for educational purposes and does not aim for 100% exact reproduction of official models. This is a critical limitation if you need to benchmark against published results or deploy a model in production. The library abstracts away details, but that abstraction may hide assumptions that differ from the original papers. For example, the exact masking schedule or sampling temperature might vary. Another limitation is that the repository is evolving quickly, with news items from late 2025 and early 2026, but no tagged releases. This means that code can change without notice, and you cannot pin to a stable version. If you are using this in a research project, you should fork the repository or record the commit hash. Also, the README does not provide a full API reference or a command-line interface; instead, you must navigate the examples and read their individual README files. This is fine for a researcher who wants to adapt code, but not for someone looking for a plug-and-play library.

Alternatives and Comparisons: Official Implementations and General Tools

A direct alternative is to use the official codebases for each model, such as the original LLaDA or Dream repositories. The difference is that those are tailored to a single architecture and often provide the exact training and inference code used in the paper, which is better for reproduction. However, they lack unified evaluation across models and may not support modern distributed training features like FSDP out of the box. Another alternative is to use the underlying transformers library directly, adding a custom diffusion loss function yourself. This gives you full control and avoids the extra abstraction layer, but it requires significant effort to handle evaluation and multi-GPU training. For evaluation only, you could use lm-evaluation-harness on its own, but you would need to write a custom adapter for each diffusion model. dLLM's value is in consolidating these tasks, but if you only care about one model, the official repository might be simpler and more faithful.

Maintenance and Licensing: What You Need to Know

The project is licensed under Apache-2.0, which is permissive and allows commercial use, modification, and distribution, provided you retain the copyright notice. The repository is actively maintained, with the last push on July 17, 2026, and no archived status. The README references several external projects, including Fast-dLLM from NVIDIA and a separate repository for diffu-GRPO, which indicates that the ecosystem is growing. However, the maintenance cost for you as a user is non-trivial: since there are no releases, you must track the default branch, and the rapid addition of new examples means that APIs might shift. The documentation is example-centric, so upgrading to a new commit might require reading the updated example READMEs. The dependency on specific PyTorch and CUDA versions (2.6.0 and 12.4) could also cause conflicts if you have a different setup, though the README says other versions should work. There is no mention of a contribution guide or issue tracker in the provided material, but the GitHub repository itself likely has those.

Editorial conclusion

Adopt dLLM if you are a researcher or engineer who wants to experiment with diffusion language models and values a single codebase for training, evaluation, and RL fine-tuning. It is well-suited for reproducing published methods like LLaDA and Dream, especially if you already use Hugging Face Trainer and lm-evaluation-harness. Do not use it if you need a production-ready inference server or if you rely on the original authors' exact implementations, as the README itself notes that exact reproduction is impossible. Before adopting, verify that the specific model version and algorithm you need are covered by the examples directory, and check the arXiv report (2602.22661) for the latest architectural details, since the repository has no tagged releases and the default branch is the only source of truth.

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. ZHZisZZ/dllm on GitHub