MOSS-003: 16B parameters, four plugins, and a pinned torch 1.13.1

The plugin training data covers exactly four tools

moss-moon-003-sft-plugin was fine-tuned on about 300,000 plugin-augmented multi-turn dialogues, added on top of the 1.1 million conversations behind moss-moon-003-sft. Those dialogues cover four tools and no others: a search engine, text-to-image, a calculator and an equation solver. The plain sft checkpoint carries no plugin ability at all, so the memory-saving swap down to it also removes tool use entirely. Running the search side is separate work: MOSS WebSearchTool is a standalone project for deploying the search plugin.

Preference training is described in full, and none of its weights exist

The training path has four stages. moss-moon-003-base comes from self-supervised pretraining on roughly 700B words of Chinese, English and code, at a stated cost of about 6.67 x 10^22 floating point operations. moss-moon-003-sft adds fine-tuning on about 1.1 million multi-turn dialogues, built from around 100,000 user inputs collected during MOSS-002 internal testing plus gpt-3.5-turbo, and that stage is where instruction following and the ability to avoid harmful requests come from. The plugin build adds the 300,000 plugin dialogues on top. Stage four trains a preference model, moss-moon-003-pm, and all three of its outputs are marked as coming: the pm itself, moss-moon-003, and moss-moon-003-plugin. The repository publishes no GitHub releases, so there is no tag to watch either. The project names its own limits for the released stage, since a small parameter count and the autoregressive paradigm leave room for misleading replies containing factual errors and for harmful content carrying bias. Consequence: the artifact you can download stops before the stage that was meant to fix factuality and safety.

The 12GB figure covers one turn, not the full 2048 context

The moss-moon series has 16 billion parameters. At fp16 it runs on one A100 or A800, or across two 3090s; at int4 or int8 it runs on a single 3090. The hardware table gives three figures per precision, all at batch size 1: memory to load the model, memory to complete one round of dialogue as an estimate, and memory to reach the maximum conversation length of 2048. fp16 needs 31GB, then 42GB, then 81GB. int8 needs 16GB, then 24GB, then 46GB. int4 needs 7.8GB, then 12GB, then 26GB. The model list quotes the 12GB and 24GB numbers as the memory needed for inference, and those are the one-turn column. A 24GB 3090 runs a 4-bit conversation, not a full 2048-token one. Consequence: sizing a machine from the int4 figures in the model list understates the requirement by more than half once conversations get long.

Quantized checkpoints cannot be split across two GPUs

The deployment notes say it twice: quantized models do not support model parallelism, and only single card deployment is currently supported for a quantized model. The fp16 path, by contrast, documents two card inference across a pair of 3090s using the accelerate helpers:

>>> from accelerate import init_empty_weights, load_checkpoint_and_dispatch
>>> os.environ['CUDA_VISIBLE_DEVICES'] = "0,1"
>>> model_path = "OpenMOSS-Team/moss-moon-003-sft"

That sharding is exactly what disappears once you switch to a quantized checkpoint. A 4-bit model that fits on about 12GB of one card stays on one card. Consequence: if your largest single card cannot hold the precision and the context length you need, the supported move is a shorter window, and pairing a small card with a large one to fit int4 weights has no documented path.

Clone, a python=3.8 conda environment, and a pinned torch

Three steps put the code in place:

git clone https://github.com/OpenMOSS/MOSS.git
cd MOSS

conda create --name moss python=3.8
conda activate moss

pip install -r requirements.txt

The requirements file that install pulls is mostly unpinned, but two lines are exact: torch==1.13.1 and transformers==4.25.1. A third line brings in triton, which the README says currently supports Linux and WSL only, not Windows or Mac OS. The conda step creates the environment with python=3.8. Check which torch build your environment already carries before installing, because requirements.txt pins torch==1.13.1.

The first call on a single A100 or A800, or on CPU, uses the sft checkpoint in fp16 at about 30GB of VRAM:

>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("OpenMOSS-Team/moss-moon-003-sft", trust_remote_code=True)
>>> model = AutoModelForCausalLM.from_pretrained("OpenMOSS-Team/moss-moon-003-sft", trust_remote_code=True).half().cuda()
>>> model = model.eval()

Consequence for a Mac or Windows user: the quantized path is not supported, because triton currently supports only Linux and WSL.

trust_remote_code executes the checkpoint's own Python

Both example loaders set trust_remote_code=True on the tokenizer and on the model, so pulling moss-moon-003-sft runs custom modeling code that ships inside the Hugging Face repository rather than a library you already had installed. Read that code before it runs on a machine holding anything you care about, because neither example shows a verification step, and the two card path calls snapshot_download whenever the local model path is missing, which folds a network fetch into the same operation. Two inference stacks sit side by side at the top level: a transformers set (moss_inference.py, moss_cli_demo.py, moss_api_demo.py, plus gradio and streamlit demos) and a jittor set (models_jittor/ and moss_cli_demo_jittor.py). Nothing in requirements.txt installs jittor, so that route cannot be brought up from the documented install.

Where deployment lives: gradio here, flutter and Go elsewhere

Serving is split across four repositories that this one only points at. MOSS Vortex is named as the deployment and inference solution, MOSS WebSearchTool covers the search plugin, MOSS Frontend is a flutter interface for MOSS-003, and MOSS Backend is a Go backend for MOSS-003. What you get here is two single process browser demos, moss_web_demo_gradio.py and moss_web_demo_streamlit.py, plus moss_api_demo.py and a moss_api.pdf file. The difference in approach is worth stating plainly: the local demos are scripts you launch against a loaded checkpoint, while the flutter client and the Go server are separate codebases you assemble, version and keep compatible yourself.

finetune_moss.py ships without the recipe that drives it

The top level file list includes finetune_moss.py, a configs/ directory and SFT_data/, which holds the conversation corpus used for moss-moon-003-sft. The table of contents carries a fine-tuning section with its own software dependencies and usage method subsections, and finetune_moss.py and configs/ are at the top level; torch==1.13.1 is pinned in requirements.txt. One licensing detail deserves the same attention: three licence files sit at the top level, LICENSE, DATA_LICENSE and MODEL_LICENSE, on a project declared as Apache-2.0, and the README has badges linking to all three.

Editorial conclusion

Adopt moss-moon-003-sft or the plugin build if you need Chinese and English dialogue, can supply your own GPU, and want the fine-tuning corpus next to the weights. Do not adopt it for speech, for sharded quantized inference, or when the preference-tuned model is the reason you came, since none of those ship in this repository yet. Verify two numbers before anything else: whether your single largest card clears the int4 figure for the context length you intend to run rather than the one-turn figure, and whether torch==1.13.1 fits the CUDA build you already run. Those two checks decide whether deployment is a weekend or a quarter.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. OpenMOSS/MOSS on GitHub
4. Project website
5. README