Open-dLLM: a 0.5B diffusion coder with the whole stack released

The infilling table has two rows for one model and one benchmark

The code infilling results are the most instructive table in the project, and they are also the easiest to misread. Open-dCoder at 0.5B parameters appears twice. Under standard evaluation it scores 32.5 pass at one on HumanEval infill and 29.6 exact match on the SantaCoder benchmark. In a second row labelled with an oracle length, the same model scores 77.4 and 56.4. Same weights, same benchmark, more than double the score, and the only difference is how the length of the span to fill was chosen. That makes the length oracle the measurement instrument, not the model. It also explains the shape of the rest of the column: the seven-billion-parameter entries from DiffuCoder and Dream-Coder score 54.8 and 55.3 pass at one, comfortably above the 0.5B model evaluated the ordinary way and far below it evaluated the oracle way. Every model in the table is being charged a fixed-length penalty, and the penalty is proportionally harsher on a small model, because a wrong guess about span length costs more when the model has less context to recover from. The footnote that explains the evaluation protocol is cut off partway through in the copy of the readme available here, so the exact fixed-length convention cannot be checked from the documentation. Treat the pair of rows as one result with an attached caveat.

At 0.5B the model trails the 7B and 8B baselines on single samples

The standard code generation tables put Open-dCoder at 0.5B against LLaDA at 8B and Dream at 7B, and the pass at one columns are not close. HumanEval reads 20.8 for the 0.5B model against 35.4 and 56.7. On the harder HumanEval+ variant it is 17.6 against 30.5 and 50.0. On MBPP it is 16.7 against 38.8 and 55.4, and on MBPP+ 23.9 against 52.6 and 71.5. So on single-sample accuracy the small model is behind by roughly a factor of two to three, which is what a fourteen-times-smaller model should be expected to be. The pass at ten columns narrow the picture without reversing it: 38.4 against 50.0 and 59.2 on HumanEval, 38.4 against 53.4 and 56.2 on MBPP. Pass at ten measures the luckiest of ten samples and pass at one measures the first, so the two columns answer different questions and the project's own summary points at the pass at ten figures when it argues the 0.5B model competes with larger diffusion models. The table also carries two smaller diffusion baselines that change the reading. Mask DFM at 1.3B scores 9.1 pass at one on HumanEval and Edit Flow at 1.3B scores 12.8, with 6.2 and 10.0 on MBPP, while their MBPP+ cells are left empty. On both of those benchmarks the 0.5B model is ahead of models three times its size, and it is behind only the 7B and 8B entries. That is the comparison worth making, and it is still a comparison inside one table produced by one group, with the harness vendored to their own copy of it. Place the finding that way rather than as a leaderboard position. The most valuable artifact here is the evaluation setup itself. The benchmarks are HumanEval, HumanEval+, MBPP, MBPP+, HumanEval-Infill and SantaCoder-FIM, run through the lm-evaluation-harness plus custom metrics, and the harness is vendored into this repository instead of being consumed at a pinned version from outside.

Representation alignment is the route that starts from a checkpoint you already have

Most of what this repository offers assumes you want to train a diffusion language model properly. One feature does not. Representation alignment adapts an existing autoregressive language model into a diffusion one, and the project states a four-times speedup for doing it that way, based on a paper it links under the title about not retraining and aligning instead, with a tutorial in the documentation directory. The practical appeal is obvious: pretraining from raw data needs the data pipeline, the cluster and the schedule, while alignment needs a model you can already load. That is why it is the feature an outsider reaches for first. Two caveats belong next to it. The speedup figure arrives without a baseline, a batch size or a hardware description, so it is a claim you carry to the tutorial rather than something the readme lets you reproduce. And alignment produces a diffusion model, not a diffusion coder: nothing in the description of the feature says the aligned result is trained for code, so a code benchmark on an aligned model would be measuring a different thing from the tables above.

Three files ask for three different transformers versions

The dependency story is where a careful reader loses time. The install block pins transformers to 4.54.1 and liger-kernel to exactly 0.5.8. The requirements file asks for transformers with the torch extra in a range from 4.46.2 up to 4.49.0, and gives liger-kernel a floor of 0.5.5. The packaging script carries its own base list with transformers pinned to 4.54.1, a CUDA extra that accepts liger-kernel from just above 0.4.1 up to 1.0, and an NPU extra for a torchvision build. Those transformers ranges do not overlap, so following the readme and following the requirements file produce different environments, and the package metadata declares its dependencies as dynamic, which means the resolved list comes from the packaging script rather than from the metadata you can read. The same block also pulls a CUDA toolkit labelled 12.3 through micromamba and then installs torch 2.6.0 from the cu121 wheel index, so the toolkit and the wheel build are not the same minor release by construction. The comment above that line calls it the newest torch, which it is not. None of this makes the project unusable. It does mean the readme's list is the one to follow, and that anyone building an environment from the requirements file alone should expect to reconcile versions by hand.

micromamba install -c nvidia/label/cuda-12.3.0 cuda-toolkit -y
pip install ninja

# install the newest torch with cu121
pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu121

pip install "flash-attn==2.7.4.post1" \
  --extra-index-url https://github.com/Dao-AILab/flash-attention/releases/download

pip install --upgrade --no-cache-dir \
  tensordict torchdata triton>=3.1.0 \
  transformers==4.54.1 accelerate datasets peft hf-transfer \
  codetiming hydra-core pandas pyarrow>=15.0.0 pylatexenc \
  wandb ninja liger-kernel==0.5.8
# optional
pip install pytest yapf py

The package, the model and the project URLs carry three different names

One project, several identities, and the packaging metadata is where they come apart. The distribution is named open-dcoder, the published model is open-dcoder at 0.5B, the repository is Open-dLLM, and the project URLs in the metadata point at a different repository entirely, one named for diffusion LLM training rather than for this release. The version is split the same way: the metadata carries a static 0.1.0, the published release is v1.0 with a DOI attached, and the packaging script contains a helper that reads a version string out of the package's own init file with a regular expression instead of taking the number from the metadata. Three numbers for one artifact, none of them cross-checked in the files a reader is likely to open first. Two smaller signals sit in the same file. The development status is declared as alpha, which is the honest label for a research release with one model at one size. And the Python requirement is 3.8 or newer while the classifier list stops at 3.11, so nothing in the package tells you whether a current interpreter is supported.

from transformers import AutoTokenizer
from veomni.models.transformers.qwen2.modeling_qwen2 import Qwen2ForCausalLM
from veomni.models.transformers.qwen2.generation_utils import MDMGenerationConfig
import torch

model_id = "fredzzp/open-dcoder-0.5B"
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load tokenizer + model
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = Qwen2ForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, trust_remote_code=True
).to(device).eval()

# Prompt
prompt = "Write a quick sort algorithm in pyt

The quickstart loads remote code and fixes bfloat16 regardless of device

The sampling example is short and every line of it matters. It pulls the tokenizer and the model from the 0.5B checkpoint on the model hub with trust_remote_code enabled, which asks the transformers library to execute code shipped in the model repository rather than treating the download as data only. That is the standard way to load a model that ships its own modelling code, and it is also the single line in this example with consequences outside the process, so it deserves a decision rather than a copy. The dtype is fixed to bfloat16 while the device is chosen as cuda when available and cpu otherwise, which means the fallback path inherits a dtype that most CPUs handle poorly. No alternative dtype and no device override appear in the example, so a machine without a usable GPU is a path the documentation does not describe. What the example does show clearly is where the masked diffusion generation lives. The model class is not imported from transformers: it is a Qwen2 causal language model class imported from this repository's own module tree, paired with a generation config named for masked diffusion that comes from the same tree. The generation path is patched into an autoregressive class inside the project, not consumed as an installed library.

python sample.py

Vendored harnesses, a five-target Makefile, and an env file in the repository

The repository layout is where the transparency claim is either earned or not. The training framework lives in its own package directory, with the configuration tree beside it, hydra as the configuration system, and a shell script for training at the top level. The evaluation side is vendored: an evaluation directory, the lm-evaluation-harness, and a separate human-eval-infilling directory all sit inside the project so a benchmark run uses the code in this repository rather than whatever a harness release happens to be that week. That is the single most useful thing here for anyone trying to reproduce a number. The Makefile has five targets and they are worth reading as a statement of what the maintainers check. Build produces a source distribution and a wheel, commit installs the pre-commit hooks and runs them over all files, quality runs the linter and the formatter in check mode, style applies the fixes, and test runs pytest over the tests directory. The lint scope is deliberately narrow: tasks, tests, the framework package and the packaging script. The vendored harnesses and the evaluation directory sit outside it, which is the right call for code you did not write and the wrong call if you plan to modify a metric. One file at the top level deserves a look before you trust the configuration story, because it is an env file named for local use sitting in a public repository rather than a documented example file. Two more top-level artifacts suggest the intent: the sampling output is kept as a text file and there is a standalone page for viewing it, so the project expects you to read generations rather than only serve them through an interface. Alongside the English readme there are Chinese and Japanese versions, which is more translation coverage than most research repositories manage, and a documentation directory that holds the alignment tutorial among other things. None of that changes what the model scores, but it does tell you what kind of project this is: one organised around being read and reproduced rather than around being called.

Editorial conclusion

Open-dLLM is worth your time if you want to train or evaluate a diffusion language model rather than merely call one, because it is the only release in its comparison table that hands over data, training code, inference, evaluation and weights together, and because the alignment route lets you start from an autoregressive checkpoint you already have. Two things to check before you trust a number or an install. The infilling results are governed by the length oracle, so quote the fixed-length figure and the oracle figure together or you are quoting a choice rather than a result. And the dependency story is inconsistent across the install block, the requirements file and the packaging script, so pin your own environment before you start, and expect the readme's versions to be the ones that actually work with the quickstart. The 0.5B model itself will not win a leaderboard; it is a component for people building the next one.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. pengzhangzhi/Open-dLLM on GitHub
3. Project website
4. README
5. Releases