HRM-Text: a 1B pretraining framework that runs on 8 to 16 H100s

What HRM-Text is for, and who should care

Most people who search for this project are looking for something else. The name collides with a texting abbreviation, and the search data around it is full of questions about what "hrm" means in a message. The repository is not that. It is sapientinc/HRM-Text, a 1B text generation model built on the HRM architecture, described in the README as strengthened by task completion and latent space reasoning, and shipped with the framework used to pretrain it rather than only with weights.

The audience is narrow and specific. The README frames the pitch as pretraining a foundation model from scratch for roughly $1000, with 130-600x less compute and 150-900x less data than whatever baseline the authors are comparing against. That is a claim about the training recipe, not about the model being small. The two reference configurations are an L size at 0.6B parameters on 8 H100s for about 50 hours, and an XL size at 1B parameters on 16 H100s across two nodes for about 46 hours, with the price estimate based on $2 per H100 hour. If you are a lab, a university group, or a company with a spare node and a corpus you want a model to absorb, this is the intended use. If you want a chat model to drop into a product tomorrow, the pretraining framework is not what you need.

The hierarchical recurrent core and the data path around it

The architecture is a hierarchical recurrent one, and the README pairs that with PrefixLM sequence packing. The repository layout supports the description: models/ holds the architecture, multipack_sampler.py handles packing, dataset_new.py handles data, and pretrain.py is the single entry point for both pretraining and the SFT stage. Training itself is PyTorch FSDP2, with FlashAttention 3 kernels underneath.

The data flow is the part worth understanding before you plan anything. HRM-Text does not read a raw corpus. It trains from sampled, tokenized data produced by a separate repository, sapientinc/data_io, which cleans, tokenizes, and stratified-samples the corpus. The README gives two deployment shapes. On a single node you run the pipeline and pretraining on the same machine and sample into shared memory at /dev/shm/sampled. On multiple nodes you keep data_io and the tokenized data on shared storage, expose that directory to every pretraining node, and run stratified sampling independently on each node. The README states that sampling is fast and deterministic, so every node produces the same in-memory training data. That determinism is the load-bearing assumption: if it does not hold in your setup, the nodes will train on different data and nothing in the config will warn you.

Epochs tie the two stages together. HRM-Text uses 4 training epochs by default, and the README is explicit that if you change epochs in the training config you must change the sampling command to match. This is a manual invariant, not an enforced one.

Installing HRM-Text and running a first L-size job

There is no pip package and no single install command. The README recommends the published Docker image, which contains the full environment. You need Docker to see your GPUs, for example through NVIDIA Container Toolkit. From the repository directory:

docker run --gpus all --ipc=host --network=host -it \
  -v "$PWD":/workspace \
  sapientai/hrm-text:latest

The --ipc=host flag matters for shared memory, and the volume mount puts the repository at /workspace inside the container. If you go the source route instead, the README says to install PyTorch, CUDA, and FlashAttention 3 first, and points at docker/Dockerfile for the tested versions rather than listing them. Then:

pip install -r requirements.txt

The requirements file pins nothing, and it includes flash_attn_3, vllm, and lm-eval[hf,vllm] alongside torch and transformers. Expect the dependency resolution to be the slow part of the install.

Before a long run, the README suggests verifying NCCL across nodes and logging in to Weights & Biases, since training metrics go there:

wandb login

The first real job is the L-size reference run on one 8xH100 node:

OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 \
torchrun --nproc_per_node=8 pretrain.py arch/size@arch=L lr=2.5e-4 global_batch_size=172032

Checkpoints are saved every epoch under checkpoints/. When you want to look at results, evaluation loads the latest checkpoint epoch automatically if ckpt_epoch is not given:

python -m evaluation.main ckpt_path="checkpoints/..."

Evaluation typically needs one 80 GB GPU, and the README notes that if it runs out of memory you can lower the batch size with generation_config.batch_size=16. Benchmark data is pulled on demand through Hugging Face datasets, so the evaluation step needs network access.

Hopper-only attention and the sharded checkpoint trap

The clearest constraint in the README is hardware. It states that Hopper-class GPUs are the expected training target because the attention path depends on FlashAttention 3. That is not a soft recommendation. If your cluster is A100 or older, you are outside the supported path, and the README offers no fallback attention implementation.

The second constraint is checkpointing on multiple nodes. Each node saves only its own shard, and the README recommends mounting shared storage for that reason. The suggested layout puts HRM-Text/checkpoints and data_io under a single /shared directory, with identical paths on every node, which the README says avoids version drift between ranks and keeps FSDP2 checkpointing straightforward. Getting this wrong is quiet: training continues, and you discover the problem when you try to resume or export.

There is also a gap around failure recovery. The README documents resume_from for the SFT stage, but it does not document rollback or restart semantics for a pretraining run that dies mid-epoch, and it does not say what happens to a partially written shard. For a 46 to 50 hour job, that is the operational question you will care about most, and the documentation leaves it to you.

Fine-tuning, export, and what the two reference runs actually report

The SFT path is full-parameter only. The README says so directly, which rules out LoRA-style adapters. Input is JSONL with one object per line, and condition defaults to direct:

{"instruction": "<full prompt>", "response": "<expected output>", "condition": "direct"}

Data preparation runs through scripts/prepare_sft_data.py, and the README warns that --epochs must equal the training epochs because there is one pre-shuffle per epoch. Training then launches with the cfg_sft config and resume_from pointing at the pretrain checkpoint.

For getting weights out, conversion/ converts a checkpoint to Transformers format, and the README notes that EMA weights are used by default when EMA is present in the checkpoint. That default applies to evaluation and export both, which is convenient but worth knowing before you compare an exported model against a training-time metric.

The reference benchmark table reports two rows. L at 0.6B on 8 GPUs over 50 hours: GSM8k 77.6%, MATH 51.2%, DROP 78.6%, MMLU 56.6%, ARC-C 75.9%, HellaSwag 52.7%, Winogrande 67.6%, BoolQ 85.0%. XL at 1B on 16 GPUs over 46 hours: 84.7%, 56.5%, 82.3%, 60.7%, 81.9%, 63.4%, 72.4%, 86.2%. The README labels these as benchmark results from the reference runs. Treat them as the authors' numbers under their configuration, not as something you can expect to reproduce on a different corpus.

HRM-Text versus simply fine-tuning an open checkpoint

The honest alternative is not another pretraining framework. It is not pretraining at all. If your goal is a model that follows instructions on your domain, you can start from an existing open checkpoint and run supervised fine-tuning, which needs one GPU rather than eight or sixteen and takes hours rather than two days.

The difference in approach is what you get at the end. Fine-tuning inherits the base model's pretraining corpus, its tokenizer, and its biases, and it cannot teach the model a domain the base never saw at scale. HRM-Text gives you the whole stack, including the tokenizer, so the model's knowledge is a function of the data you fed through data_io. That is the trade: roughly $800 to $1472 in GPU time and a multi-stage pipeline, in exchange for a model whose pretraining you control. If you cannot articulate why you need that control, the fine-tuning path is cheaper and faster. The repository also ships simple_inference_engine.py, which suggests the authors expect people to run the pretrained model directly, but the README does not document that script.

Licence, maintenance and upgrade cost

The repository is Apache-2.0, which is permissive and includes an explicit patent grant. That covers the code in this repository. It does not automatically tell you the terms attached to the Hugging Face model weights at sapientinc/HRM-Text-1B, which are distributed separately, or to the data_io pipeline, which lives in its own repository. If you plan to ship something built from the exported checkpoint, check the model card rather than assuming the code licence carries over. This is not legal advice.

The repository is not archived, and the last push was on 2026-09-04. There are no retrieved releases, so there is no tagged version to pin and no changelog to read. Upgrades therefore mean tracking the main branch, and the Docker image tag is latest, which is a moving target. Combined with an unpinned requirements.txt that includes flash_attn_3 and vllm, rebuilds of a working environment are the main ongoing cost. The README does not describe a compatibility policy between the framework and the data_io pipeline, so a change on either side is something you find out about by running it.

Editorial conclusion

Adopt HRM-Text if you have Hopper-class GPUs, a reason to pretrain rather than fine-tune an existing checkpoint, and the appetite to run sapientinc/data_io first, because nothing in the training path works without sampled tokenized data. Do not adopt it if your hardware is pre-Hopper, if you only need instruction-following behaviour, or if you expect a single-command install: the README points at Docker or a source build whose tested versions live in docker/Dockerfile. Before committing GPU hours, verify three things in this order: that FlashAttention 3 builds in your environment, that your tokenizer and epochs match between the sample_tokenized.py run and the training config, and that your multi-node checkpoint directory is on shared storage, since each node saves only its own shard.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. sapientinc/HRM-Text on GitHub