MOSS-VL: an open-weight video model built for streams, not files

Three checkpoints for three different moments of a video project

The usual shape of a video question answering model is a batch job. You hand it a file, it encodes frames, then it writes an answer. MOSS-VL takes that sequence apart. The project describes an open-weight series of three 11B models built on a unified cross-attention architecture, and the variant named MOSS-VL-Realtime is meant to sit on a live feed. A user can speak at any point in the stream and the model answers from the frames it has received so far instead of waiting for an end of file.

The other two checkpoints split the work along different lines. MOSS-VL-Instruct targets offline use, with the README pointing at complex long-video understanding and in-depth dialogue. MOSS-VL-Base is an open pre-trained foundation offered for continued pre-training and downstream fine-tuning. So this is not three sizes of the same thing. It is one family of weights for three points in a project's life: watching something that is happening, analysing something that already happened, and building something new on top.

Cross-attention runs perception beside language generation

The architectural bet is separation of duties. The README states that MOSS-VL-Realtime uses a cross-attention design that decouples visual encoding from language reasoning, and says this reduces response latency on dynamic video streams. The stated payoff is a system that runs multimodal perception and text generation in parallel over a continuously arriving stream, so a new frame does not have to wait behind a finished generation pass. Interleaved image, video and text inputs are supported natively.

Two behaviours follow from that arrangement in the project's own description. Interruption is native: a question can arrive at any timestamp and the model replies on the currently received frames, with the README claiming open-source state of the art latency for streaming interaction. Correction is also native, since cognition updates as the feed continues and earlier outputs can be revised when a state change appears.

The third behaviour is the least common in this class of model. MOSS-VL-Realtime is described as having proactive silence: it stays quiet and keeps observing when context is insufficient or no key event has happened. That turns a video model into something closer to a participant in the scene, and it also means a gap in the output is not automatically a failure.

Absolute timestamps and XRoPE put the model inside the timeline

Timing is handled explicitly rather than approximated. Every sampled frame is anchored to a precise time marker through dedicated special tokens, and the model uses those markers to reason about pacing, duration and motion dynamics. The README also states that the approach adapts natively to variable frame rates, which matters for camera streams that drop frames or for screen captures assembled from separate segments.

Position encoding comes from XRoPE, Cross-Attention Rotary Position Embedding. It maps text tokens and video patches into a single 3D (t, h, w) coordinate space, and the README connects this to patch-level and moment-level grounding across the entire video. The practical consequence is that a question about a particular instant has a coordinate system to point at, rather than a flat sequence of patches to average over.

Whether that grounding transfers to a domain with very different visual statistics is something you would have to check on your own footage. The technical report published on arXiv covers the training curriculum, the real-time inference system, and offline and streaming evaluations, so the published numbers describe the settings the authors chose.

There is no install command, so the first session is a checkpoint download

It is worth knowing before you plan a deployment that the README gives no install command. The path the project points to is the model collections: a MOSS-VL collection on Hugging Face, a collection on ModelScope, and a hosted space for trying the demo. The repository itself is mostly assets and tooling, with top-level directories for inference, fine-tuning, quantization and docs, a third_party folder that now holds the SGLang-Omni realtime backend and the realtime demo application, a device_utils.py script, and pinned Python dependencies.

The dependency pins are strict and worth reading before anything else:

--extra-index-url https://download.pytorch.org/whl/cu128
torch==2.8.0+cu128
transformers==4.57.1
accelerate==1.12.0
flash-attn==2.8.1
torchcodec==0.7.0

That block is the requirements.txt at the top of the repository. The cu128 wheels and the extra index URL mean the install expects a CUDA 12.8 environment, and flash-attn plus torchcodec are compiled components that have to match the platform. A separate requirements-npu.txt covers other accelerators. Since no install command is documented, expect to resolve these pins yourself rather than to run one line and move on.

The offline paradigm this project argues with

The default paradigm in the field, as the README describes it, is watch first, answer after: decode a clip, sample frames, encode, then produce one reply, often with retrieval or a memory step to cover a long file. MOSS-VL-Realtime challenges the shape of that pipeline rather than the model class. Perception stays running on an open stream and text generation runs beside it, so the unit of work is a conversation that unfolds over time rather than a question that terminates with an answer.

That is a genuine difference in approach, and it carries a real cost. Batch models can be cached, retried and scheduled, and their output is stable once written. A streaming session has to hold state across minutes of video, and any downstream system built around a settled final answer has to be rewritten to consume partial output that the model may later correct. The project frames its streaming design as a departure from that default, which suggests a design position rather than a faster version of the same thing.

The practical question is whether your problem is a conversation or a query. If it is a query, the same repository also gives you a path that does not require the streaming behaviour at all.

Where MOSS-VL is the wrong tool

Not every workload wants a model that may decline to answer. Proactive silence and mid-stream correction are what you want in a live assistant and unhelpful in a batch pipeline that needs one complete answer per file. For archives of recorded video and for reproducible output, MOSS-VL-Instruct is the checkpoint to look at, and the README does not document a switch for forcing a full pass or for switching off the real-time behaviour.

The weights are also large. The parameter count is 11B across the series, and the project publishes 24 GiB quantized checkpoints with FP8 and NF4 variants for Instruct-0708 and Realtime, which puts a floor under the memory ceiling even in quantized form. The quantization tutorial covers FP8-Dynamic and NF4 recipes, KV-cache quantization, and how to quantize your own fine-tuned checkpoints, so the footprint is adjustable rather than fixed.

Hardware coupling is easy to miss. The stack pins flash-attn and torchcodec against CUDA 12.8 wheels, and the repository carries a separate requirements file for NPU deployments, so operators on other accelerator stacks should expect to do the dependency work themselves. The README does not document a CPU-only path, and the top-level tree lists no Dockerfile.

Fine-tuning paths and the state of the repository

Fine-tuning is the part of the stack that other projects support most concretely. The README records integration with ms-swift, where image and video inference runs through swift infer and LoRA or full-parameter tuning through swift sft, and integration with LlamaFactory, which offers LoRA and full fine-tuning workflows out of the box. Both are separate projects, and the integration amounts to a bridge from their standard configuration to these checkpoints. The repository's own finetune directory and the list of open-source datasets used in training are published as well.

The maintenance picture is easier to read than the model card. The repository is not archived, and the last push was on 2026-10-14. There are no GitHub releases, so version identification runs through checkpoint names such as Instruct-0708 and through dated news entries rather than tags, and a deployment needs to record which checkpoint it loaded. The licence is Apache-2.0 for the repository contents. Weight and dataset terms still need reading in the files themselves, by whoever owns that review, rather than inferred from a line in a README.

Editorial conclusion

MOSS-VL suits teams that need a model to watch a live camera feed or a long recorded session and talk about it, and the open weights plus the FP8 and NF4 variants plus the ms-swift and LlamaFactory paths remove most of the reasons to build a video stack from scratch. It does not suit a batch pipeline that needs one reproducible answer per file, and the absence of an install command, a Dockerfile, and a footprint below 24 GiB of quantized weights puts real work on whoever deploys it. Before you commit, settle the two things the documentation leaves open: whether your stream and frame rate sit inside the settings the technical report evaluates, and whether the checkpoint you plan to download matches the memory budget and accelerator stack you actually run.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. OpenMOSS/MOSS-VL on GitHub
4. Project website
5. README