MobileLLM: Training Code for Sub-billion Parameter Language Models

What MobileLLM actually ships

The README opens by describing the repository as containing the training code of MobileLLM, the model family introduced in the paper MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases, published at ICML 2024. That sentence is the scope. What you get is a pretraining harness plus an evaluation script, not a packaged model. The weights live elsewhere: an October 2024 news entry states that MobileLLM models are publicly available on HuggingFace.

The intended user is someone with a multi-GPU cluster who wants to reproduce or extend a small language model. The README's own cost table makes the audience concrete. Training on 1T tokens using 32 NVIDIA A100 80G GPUs takes roughly 3 days for the 125M model, 6 days for 350M, 8 days for 600M, 12 days for 1B, and 18 days for 1.5B. Anyone without that hardware is looking at the wrong half of this project. The HuggingFace collection is the useful half for them.

There is a second layer of confusion worth flagging. The news section has moved well past the original paper. November 2025 brought MobileLLM-R1.5, a HuggingFace collection whose 950M model is claimed to outperform DeepSeek-R1-Distill-Qwen-1.5B on all evaluated math and coding benchmarks. September 2025 brought MobileLLM-R1, which the README says matches or surpasses Qwen3-0.6B on MATH, GSM8K, MMLU, and LiveCodeBench using roughly 2T pretraining tokens against Qwen3-0.6B's 36T. January 2026 brought news that MobileLLM-R1 was accepted to ICLR 2026. None of that changes what this repository contains. The training code here is for the original MobileLLM family, and the README does not describe the R1 or R1.5 training recipes in this repository.

The four design choices behind the architecture

The README names four components that were integrated to build MobileLLM: SwiGLU activation, deep and thin architectures, embedding sharing, and grouped-query attention. The paper abstract in the README frames these as design factors considered together to obtain high-quality models below one billion parameters.

Deep and thin is the choice that most changes the shape of the work. Instead of widening a small model, you stack more layers at a smaller width. That shifts the memory profile during training: more layers means more activations to hold, and the README's data-parallel layout of 1x8 GPUs per node assumes you have enough per-device memory to fit whatever depth the config specifies. Embedding sharing removes a separate output projection matrix, which matters proportionally more at 125M than at 1.5B, since the vocabulary embedding is a fixed cost that does not shrink with model size. Grouped-query attention reduces the number of key and value heads relative to query heads, which is the standard trick for cutting KV cache size at inference. That last point is where the repository's boundary becomes visible: the architecture is designed for inference efficiency, but the code shipped here is the training side.

The README reports that MobileLLM-125M and MobileLLM-350M achieve a 2.7% and 4.3% accuracy boost over preceding 125M and 350M state-of-the-art models on zero-shot commonsense reasoning tasks, and that the design philosophy scales to 600M, 1B, and 1.5B. The result tables back this up on the named benchmarks. On the 125M table, MobileLLM-125M averages 46.3 against OPT-125M's 42.6, GPT-neo-125M's 42.9, and Pythia-160M's 42.5. The -LS variant, which appears in the 125M and 350M tables, scores 47.0 and 52.1 respectively. The README does not explain what LS denotes. That is a gap, and it is the kind of gap that matters if you intend to reproduce the -LS configuration rather than the base one.

Data layout and the two scripts you edit

Getting this running is a three-step process the README lays out plainly. Step 1 requires Python 3.9 and PyTorch 2.0 or later, then pip install -r requirement.txt. Step 2 is data preprocessing, and the constraint here is structural rather than algorithmic.

You divide a tokenized dataset, or tokenize your own, and distribute it across the total number of training nodes, where each node comprises 1x8 GPUs. The directory layout is a basepath containing numbered subdirectories 1 through #nodes, each holding one or more .jsonl files. Every line of those files is a key-value pair of tokenized data in the form {"token_ids": [1,2,3,4,...]}. The README states the training code is compatible with the preprocessing method in the LLM360/amber-data-prep repository, which is the practical route if you would rather not write a tokenizer pipeline.

Step 3 is training. The pretrain.sh script initiates training on a 1x8 node setup using torchrun. Two parameters need your attention. Set --train_data_local_path to the preprocessed data from Step 2, and set --input_model_filename to ./configs/{model_size}/, where {model_size} is one of the sizes in the configs directory. Then run bash pretrain.sh.

Multi-node scaling is where the README gives a warning worth reading twice. The learning rate in the script is tuned for 1x8 nodes with a batch size of 32. If you increase the number of nodes or the batch size, you need to increase the learning rate linearly. The README says the script can be modified to adjust --nnodes and other settings for slurm or torchx configurations, but it does not supply those modifications. You write them. Evaluation is a separate script: download the models, update the checkpoint path in eval.sh, and run bash eval.sh.

Where the repository stops

The paper title says on-device use cases. The repository does not contain on-device anything. There is no inference engine, no quantisation script, no conversion tool, no runtime binding, and no latency or memory measurement in the README. The only evaluation described is bash eval.sh for Wiki, plus the zero-shot commonsense reasoning tables, which are accuracy numbers on academic benchmarks.

This is the most important thing to understand before adopting it. If your goal is to run a small language model inside a phone application, this repository does not address that goal. The HuggingFace collection does, at least in the sense that it gives you weights. What you do with those weights on a device is outside this project's scope, and the README never pretends otherwise, which is honest but easy to miss given the paper title.

The second limitation is the learning rate rule. Linear scaling with batch size and node count is a heuristic, and the README states it without qualification or a pointer to the reasoning. If you are training at a different scale than 1x8 nodes and batch 32, you are on your own for the hyperparameters. The cost table compounds this: it lists days for 32 A100 80G GPUs at 1T tokens, and nothing else. Different hardware, different token counts, different throughput.

Third, there are no releases. The repository metadata shows no releases retrieved, and the README has no versioning scheme for the training code. You are tracking main. For a research codebase that is normal, but it means a recipe that worked in one clone may not be the recipe in the next.

How it compares to the alternatives in its own tables

The README's result tables are the most useful comparison material, because they name the models MobileLLM is measured against. On the 1B table, MobileLLM-1B averages 57.3 against Falcon-1B at 56.3, MobiLlama-1B at 55.2, TinyLlama-1.1B at 54.2, Pythia-1B at 48.7, and BLOOM-1.1B at 46.9. On the 600M table, MobileLLM-600M averages 54.3 against MobiLlama-800M at 50.7, Qwen1.5-500M at 48.8, and BLOOM-560M at 44.2.

The interesting comparison is MobiLlama, because it is the closest in intent. MobiLlama-800M is a larger model than MobileLLM-600M and scores 3.6 points lower on the average of these eight benchmarks. That is a real difference in approach, not a rounding artifact. The README attributes MobileLLM's position to the four design choices above. MobiLlama is not described in this material beyond its numbers, so the architectural contrast cannot be drawn from what is supplied here.

The Qwen1.5 comparison is the one to watch as you go up in size. At 600M, MobileLLM-600M beats Qwen1.5-500M by 5.5 points on the average. At 1.5B, Qwen1.5-1.8B averages 56.5 and the MobileLLM-1.5B row is truncated in the supplied README, so the final comparison cannot be read off. The 1.5B table also includes GPT-neo-2.7B at 52.8 and OPT-2.7B at 55, both larger models scoring below the 1.5B class leaders. The pattern across all five tables is that parameter count alone predicts little on these benchmarks, which is the paper's premise stated as data.

Maintenance, licensing, and what the repository does not tell you

The repository is not archived and the last push is dated 2026-04-30, so it is being touched. There are no tagged releases, no changelog in the supplied README, and no migration notes. Upgrading means pulling main and re-reading pretrain.sh, because the README does not document what changed between the ICML 2024 version and the version that produced the 600M, 1B, and 1.5B results. The README does say the design philosophy was extended in an updated version, so the configs and script have changed at least once without a version marker.

On licensing, the repository metadata reports NOASSERTION. That is not a licence. It means the automated classifier could not identify one from the repository contents. The README contains no licence section, no usage terms, and no statement about commercial use or redistribution. The pretrained weights are hosted on HuggingFace and may carry terms that differ from whatever governs this code, and the README does not discuss the relationship between the two. This is a question for your legal team, not for a review, but the practical point is that you cannot answer it from the README alone.

Two more gaps. The README does not state a minimum GPU memory requirement for any model size, so the feasibility of the 1.5B configuration on your hardware is something you determine from the config, not from the documentation. And the -LS variant appears in the results without an explanation of what LS stands for or how to select it in the configs directory. If you need that configuration, you are reading the code.

Editorial conclusion

Adopt MobileLLM if you are pretraining a sub-billion parameter model from scratch on at least one 1x8 A100 node and want a published architecture configuration plus the training script that produced it. Do not adopt it if you need an inference engine, quantisation pipeline, or on-device runtime, because the repository contains none of those and the README never claims otherwise. Before committing GPU time, verify two things in the repository itself: whether configs/{model_size}/ contains the size you intend to train, and whether the licence file grants the rights your use case needs, since the repository metadata reports NOASSERTION rather than a named licence.

Official sources

1. facebookresearch/MobileLLM on GitHub
2. Issues
3. README