Dolma: An Open Corpus and the Toolkit That Built It
Data and tools for generating and inspecting OLMo pre-training data.
At a glance
- What is it?
- AI2's Dolma is two artifacts sharing one name: a 3 trillion token pretraining corpus released under ODC-BY, and a Python toolkit under Apache-2.0 for running taggers and deduplication over document collections. The toolkit is the part you can actually adopt for your own pipeline.
- Who is it for?
- Adopt the Dolma Toolkit if you are building a pretraining corpus and want Gopher, C4 and OpenWebText taggers plus Rust-backed deduplication without writing that filtering code yourself, and if your data already lives on a filesystem or an S3-compatible store. Do not adopt it if you need a managed service, a streaming pipeline with per-record latency guarantees, or a corpus whose licence you have not reviewed.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 22 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
Two artifacts, one repository, different licences
The name Dolma covers a dataset and a toolkit, and the README is explicit that the repository holds the source code for the toolkit only. The dataset is a separate download: 3 trillion tokens drawn from web content, academic publications, code, books and encyclopedic materials, assembled as the training corpus for OLMo and published on the HuggingFace Hub at huggingface.co/datasets/allenai/dolma. The dataset carries the ODC-BY licence, and the README links a blog post explaining the move to that licence. The toolkit source in this repository is Apache-2.0. That split matters more than it first appears: if you download the corpus and ship a model trained on it, the ODC-BY attribution obligation attaches to the data, not to the Python code you used to process it. The README points readers to a data sheet at docs/assets/dolma-v0_1-20230819.pdf for the corpus itself, which is where the composition details live rather than in this repository.
Who the toolkit is aimed at, and who it is not
The toolkit exists to curate large datasets for pretraining. That phrasing sets the audience: teams assembling a corpus from raw documents, not teams fine-tuning on a small labelled set and not teams that want a data loader. The README lists five properties, and the ordering is informative. High performance comes first, described as processing billions of documents concurrently through built-in parallelism. Portability comes second: the same code runs on a single machine, a cluster, or a cloud environment. Then built-in taggers for Gopher, C4 and OpenWebText heuristics, deduplication via a Rust Bloom filter, and extensibility with custom taggers plus AWS S3-compatible storage. If your problem is 'I have a few terabytes of Common Crawl-style text and I need it filtered and deduplicated before a training run', this is the shape of tool you are looking for. If your problem is 'I need to sample batches during training', it is not.
How curation actually flows through the toolkit
The mechanism visible from the material is a pipeline of taggers applied to documents, with deduplication as a distinct stage. Taggers are the filtering layer: the README names Gopher, C4 and OpenWebText as the ready-to-use sets, each corresponding to a published heuristic filter from the cited papers, and states that custom taggers are supported. Deduplication is separate and is implemented with a Rust Bloom filter, which is the reason the README can call it fast. A Bloom filter is a probabilistic structure: it can tell you a document is definitely not a duplicate, and it can tell you a document is probably a duplicate. That is the trade-off baked into the design, and it is the one to keep in mind when you reason about what survives the stage. The stated deployment targets are a single machine, a cluster, or a cloud environment, with AWS S3-compatible locations supported, so the toolkit is designed to read and write against object storage rather than assuming everything sits on local disk. What the README does not give is a worked example of the pipeline in code; that lives in the docs directory, which the README links as the place to learn how to use the toolkit.
Getting it installed and finding the real usage path
Installation is one line, quoted directly from the README: pip install dolma. That is the whole of the installation instruction in the material available here. There is no version pin, no optional extras list, no note about whether the Rust deduplication component arrives as a prebuilt wheel or needs a toolchain on the machine, and no Python version floor stated. Those are the first things to check against the docs before you plan a deployment, because a Rust-backed component changes what your build environment needs. The README's closing pointer is the documentation directory at /docs in the repository, described as the place to learn more about how to use the Dolma Toolkit. Beyond the install command and the docs pointer, the README supplies no config keys, no CLI flags and no config file schema. Anyone writing a tutorial that claims otherwise is inventing it. Treat the docs directory as the source of truth for invocation, and treat the pip command as the only verified setup step.
The deduplication stage is where the design choices bite
Deduplication is presented as a headline feature, and the Bloom filter implementation is the reason it can be. The limitation is structural rather than a bug: a Bloom filter trades exactness for memory and speed, so the toolkit's duplicate detection is approximate by construction. For corpus curation that is usually the right trade, since near-duplicate web text is the target and a small false-positive rate is tolerable. It stops being the right trade when the downstream use requires an auditable record of exactly which documents were removed and why, or when a false positive silently drops a document you needed. The README does not state the false-positive rate, the memory cost per document, or whether the filter parameters are exposed for tuning. Those are questions to put to the documentation before you size a cluster job. The other constraint is that the pipeline is batch-oriented in framing: the README talks about processing billions of documents concurrently, not about serving records one at a time. If your requirement is a streaming filter with per-record latency guarantees, this architecture is pointed the other way.
Where Dolma sits against writing your own filters
The realistic alternative is not another named project; it is assembling the same pipeline yourself from the underlying papers. The Gopher, C4 and OpenWebText filters are published heuristics, and the README links the Gopher, C4 and OpenWebText sources directly. A team with a modest corpus could implement those thresholds in a few hundred lines of Python and skip the dependency entirely. The difference in approach is the parallelism and the deduplication. Writing the filters is easy; making them run concurrently across billions of documents and pairing them with a memory-efficient deduplication stage is the work Dolma has already done, and it is the reason to take the dependency rather than reimplement. The counter-argument is control: a hand-rolled pipeline lets you log every document that fails a threshold and tune each rule without touching a third-party library. Dolma's value proposition is that you do not maintain that code. Whether that trade is worth it depends on how much of your filtering is standard heuristic work versus domain-specific rules that would end up as custom taggers anyway.
Version cadence and what upgrading costs you
The release history shows v1.1.2 in February 2025, v1.2.0 in June 2025 and v1.2.1 in July 2025, with repository activity continuing well past the last tagged release. That is a moderate cadence: not a project that ships weekly, not one that has gone quiet. The practical upgrade cost is not in the Python API surface, which is small enough that a pip install line covers the whole documented setup. It is in reproducibility. A curated corpus is only meaningful alongside the toolkit version that produced it, because tagger thresholds and deduplication behaviour can change between minor versions, and a corpus regenerated under v1.2.1 is not guaranteed to match one produced under v1.1.2. Pin the version in whatever environment file your pipeline uses and record it next to the corpus hash. The licence side is simple on the code: Apache-2.0 permits commercial use and modification with attribution. The dataset is a separate matter under ODC-BY, and the README's own pointer to a blog post explaining that licence switch is the signal that the terms were deliberately chosen and are worth reading rather than assuming.
Editorial conclusion
Adopt the Dolma Toolkit if you are building a pretraining corpus and want Gopher, C4 and OpenWebText taggers plus Rust-backed deduplication without writing that filtering code yourself, and if your data already lives on a filesystem or an S3-compatible store. Do not adopt it if you need a managed service, a streaming pipeline with per-record latency guarantees, or a corpus whose licence you have not reviewed. Before committing, verify the taggers you need exist in the built-in set or budget for writing your own, confirm the deduplication memory footprint at your document count, and read the ODC-BY attribution terms separately from the Apache-2.0 code licence, because the two cover different artifacts.
Community notes