calm: continuous autoregressive language models that predict chunks, not tokens

The autoregressive bottleneck, attacked by predicting a chunk

The problem CALM targets is stated at the top: modern large language models generate text one token at a time, and that is a fundamental bottleneck. The proposed shift is to predict a single continuous vector representing an entire chunk of K tokens instead of one discrete token per step.

Two consequences follow immediately from that framing. The number of autoregressive steps drops by a factor of K, which the documentation presents as the efficiency argument. And a new axis appears for scaling: instead of only parameters and data, you can scale semantic bandwidth, meaning how much information is processed in a single step. K becomes a tunable quantity rather than a fixed consequence of the token vocabulary.

The implementation is a two-stage process. First, a high-fidelity autoencoder learns to compress K tokens into one vector and reconstruct them with near-perfect accuracy. Second, a continuous-domain language model performs autoregressive prediction in that vector space.

The split is what makes the idea workable. Reconstruction quality is a separate, verifiable problem: if the autoencoder cannot round-trip a chunk of text, the language model above it is predicting in a lossy space and no amount of training downstream will fix that. The language model never has to learn the identity mapping.

The project is the official implementation of the work described in arXiv 2510.27688, with an accompanying blog post, a Hugging Face collection of models, and a `models/` directory in the repository. The licence is MIT.

Continuous output means you lose likelihood, so the toolkit has to replace it

The most interesting engineering content in the repository is not the architecture but the tooling that a continuous output domain forces on you. The documentation calls it a comprehensive likelihood-free toolkit, and the four pieces are named.

The autoencoder is the first: it learns high-fidelity continuous representations of token chunks, and it is the component with the clearest correctness criterion.

Energy-based training is the second, described as a principled and likelihood-free method for generative modelling. It matters because a model that emits continuous vectors does not naturally produce the normalised probabilities that likelihood-based training and evaluation depend on. This is the core cost of the approach: you give up the well understood machinery of next-token likelihood and have to substitute something else for both training and measurement.

BrierLM is the third, a new metric for calibrated, likelihood-free evaluation of language models. That is the evaluation-side answer to the same problem. If you cannot compute a likelihood, you need a score that still tells you whether the model's probabilities are well calibrated rather than merely plausible.

Temperature sampling is the fourth, described as controlled, high-quality text generation using only a black-box sampler. A black-box sampler means no access to internal logits or gradients, which is a much weaker position than the standard sampling methods used with discrete token models.

Together these four pieces are the actual contribution of the codebase beyond the paper. Anyone evaluating CALM should treat the quality of BrierLM and the energy objective as the deciding factor, because the efficiency claim about K autoregressive steps is only worth having if the model is still good.

The training pipeline is two scripts and a lot of data

Setup is a clone, a dependency install and a data preparation step:

git clone https://github.com/shaochenze/calm.git
cd calm

pip install -r requirements.txt

bash data/get_data.sh

That last script downloads and processes the pile-uncopyrighted dataset from Hugging Face, and the documentation warns that the dataset is large and that you need at least 2.5TB of free disk space. That figure is the single most important number for anyone planning around this repository.

Training happens in two stages, matching the two-stage model. The autoencoder is trained first, on approximately 15B tokens, with `bash train/train_autoencoder.sh`. The language model is trained second on the remaining data with `bash train/train_energy.sh`, using the proposed energy loss, with the BrierLM score computed during evaluation steps to track performance. The documentation states that the model should reach a final BrierLM score of approximately 5.72 on the validation set, and that the configuration to replicate is K=4.

The scripts are plain bash wrappers around torchrun. The autoencoder script uses a latent size of 128, two encoder layers and two decoder layers with a patch size of 4, a block size of 2048, batch size 8 per device with gradient accumulation of 4, 30000 max steps and a constant learning rate schedule at 3e-4 with 1000 warmup steps. The language model script uses 16 hidden layers, hidden size 1024, intermediate size 2752, 16 attention heads and 16 key value heads, a block size of 8192, batch size 4 per device with accumulation of 8, 250000 max steps and the same learning rate and warmup pattern with 2000 warmup steps.

Both scripts are launched as `torchrun --nnodes 1 --node_rank 0 --nproc_per_node 8`, so the reference configuration assumes eight GPU processes on one node.

The dependency pins are the practical constraint

The requirements file is short and entirely pinned, and two of those pins will shape your environment planning:

torch>=1.13.0
transformers==4.43.0
datasets==3.1.0
accelerate==0.30.1
huggingface-hub==0.35.3
tokenizers==0.19.1
safetensors==0.5.3
flash-attn==2.1.1
tqdm==4.67.1
deepspeed==0.10.0

`transformers==4.43.0` is an exact pin on a specific release of the Hugging Face library, and `flash-attn==2.1.1` pins a flash attention build that has to match your CUDA and torch versions. flash-attn in particular is compiled rather than pure Python, so it is the dependency most likely to fail to install cleanly on a new machine.

`deepspeed==0.10.0` is present for distributed training support, even though the reference scripts run with a single node. And `torch>=1.13.0` is the one loose constraint, everything else is exact.

The repository also carries a `llama3_tokenizer/` directory, and the training scripts point their `--tokenizer_name` at it. That is a useful detail: the tokenizer is vendored rather than downloaded, so the token chunking behaviour is reproducible from the repository itself.

The top level is otherwise minimal: `data/` for the preparation script, `models/` for model code, `train/` for the training entry points and the `train.train_autoencoder` and `train.train_calm` modules that torchrun invokes, plus `overview.png`, the README and the LICENSE. There is no inference server, no CLI and no chat example in the visible tree, which tells you what kind of artifact this is.

Other generative heads are mentioned but not shown

The visible documentation ends mid-sentence on the topic of alternatives. It says alternative training scripts are provided for generative heads based on Diffusion and Flow Matching, and that they are available, but the section is cut off there.

That is worth flagging because it points at the shape of the research. CALM's contribution is the continuous-domain formulation, and once you are generating in a continuous space you need a way to sample from it. The energy-based training described earlier is one such head. Diffusion and flow matching are two more, which suggests the authors treat the head as a swappable component rather than the core contribution.

Practically, this means you should read the source under `train/` before assuming the energy path is the only one that works. The documentation does not describe the configuration differences between the heads, the hyperparameters that apply to each, or whether the reported BrierLM figure of approximately 5.72 is specific to the energy head. Treat that number as the reference point for the configuration actually shown rather than a property of CALM as a whole.

The autoencoder and the language model are the two halves that the paper's central claim rests on, and those are the parts with explicit scripts and documented hyperparameters in the visible documentation.

What this repository is for, and what it is not

CALM is a research artifact, and the repository is shaped like one. The evaluation target is a single configuration with K=4 on a validation set, tracked with a metric the same authors introduced, and the reproduction path runs through eight GPU processes over tens of terabytes of text.

That shape makes it valuable in a specific way and unhelpful in others.

It is valuable as a reference implementation. If you want to build on continuous latent language modelling, the autoencoder design, the energy objective, the BrierLM metric and the temperature sampler are the parts worth reading, and having them in one tree with the training configurations spelled out is more useful than the paper alone.

It is unhelpful as a product dependency. There is no inference API here, no way to load a model and generate text through a documented interface, and no checkpoint release workflow described in the visible documentation, though models are linked on Hugging Face. If your goal is to run CALM in an application, you would be writing the inference path yourself.

It is also not a small experiment. The 2.5TB disk requirement is for the dataset alone, before any checkpoint storage, and the reference run is 30000 steps of autoencoder training followed by 250000 steps of language model training. Reproducing the numbers requires multi GPU hardware and days of compute.

The interesting middle ground for many groups is training a smaller autoencoder, at a smaller K, on a smaller corpus, and using the scripts as a starting point rather than a target. That is a legitimate way to explore the idea, and the config overrides mechanism in the scripts makes it possible. What you cannot do is assume the reported quality transfers, because the quality depends on the autoencoder reconstructing accurately and the autoencoder needs scale to do that.

Editorial conclusion

Adopt calm if you are reproducing the CALM paper, building on continuous latent language modelling, or want the BrierLM metric and temperature sampler as reference implementations. Do not adopt it as a drop in chat model: the repository ships training scripts for a research pipeline, not an inference library or a chat interface, and reproducing the headline configuration needs at least 2.5TB of disk for the dataset plus eight GPU processes. Verify first which patch size K you need, since the paper's reported configuration uses K=4 and the semantic bandwidth argument changes with K, and check the pinned dependency versions in requirements.txt, including transformers==4.43.0 and flash-attn==2.1.1, against your environment before starting a long run.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. shaochenze/calm on GitHub