# VideoChat3: a fully open 4B video model built for time, motion and live streams > A generalist video MLLM from Nanjing University with token-efficient I3D-ViT compression and evidence-aware frame resolution, released with weights, full training data and training code. **MCG-NJU/VideoChat3** — A generalist video MLLM built for fine-grained motion, long-form reasoning, temporal grounding, and online proactive response. - Repository: https://github.com/MCG-NJU/VideoChat3 - Stars: 349 · Forks: 11 - Language: Python - License: not declared - Published: 2026-09-18 · Updated: 2026-09-18 · Language: en - Canonical page: https://hysenlabs.com/projects/mcg-nju-videochat3 ## One model for four video problems Video understanding tends to fragment into specialties: one model for clip captioning, another for long-video question answering, another for finding the moment something happens, another for reacting to a livestream as it happens. VideoChat3, from the MCG group at Nanjing University, is a 4-billion-parameter generalist built to cover those four in one system: fine-grained motion perception, hour-long story reasoning, temporal grounding, and online proactive response to live streams. The openness claim is unusually strong for the category. The release sequence in the README shows weights and training data on 2026-07-17, evaluation code on 2026-07-24, training code on 2026-07-27, and an updated arXiv paper on 2026-08-25. Model weights, the complete training datasets and the training pipeline are all public, which makes this one of the few video MLLMs a research group can actually reproduce end to end rather than merely prompt. ## The architecture: compress, then look closer Two mechanisms carry the efficiency story. I3D-ViT compresses visual tokens sixteen-fold while preserving spatiotemporal evidence, attacking the token explosion that makes video models expensive. Adaptive Frame Resolution is the evidence-aware half: frame resolution increases only when closer visual inspection is needed, so an hour of video is not processed at uniform high cost, and a moment of fine motion gets the pixels it requires. Training drew on three datasets named in the README: Academic2M, LV116K and OL617K, spanning the capabilities above. The design reads as a deliberate answer to the two standard failure modes of video models, paying full price for redundant frames, and losing the evidence trail when compressing too aggressively, with the grounding capability positioned as the bridge between understanding and proof. ## Training on your own stack The training implementation lives in xtuner-videochat3, built on XTuner V1, and includes the model, the data pipeline, staged training configurations, initial checkpoint construction, and launchers for both Slurm and non-Slurm clusters. Installation follows the README: ```bash cd xtuner-videochat3 pip install -e ".[video]" ``` After paths are set in the chosen configuration, a launcher starts the run, with a single-node stage example invoking eight processes through the stage1 training script. The complete training guide covers initial checkpoint construction, every stage configuration, Slurm commands and manual multi-node setup. Staged training with published checkpoints per stage is the notable reproducibility feature: a lab can start from the same initial state the authors did rather than approximating it. ## Evaluation and first inference Evaluation runs through vlmevalkit-videochat3, built on VLMEvalKit, supporting all offline and online video benchmarks evaluated in the paper, with its own guide for environment setup, model downloads and runs. For first inference, the README points at the model card on Hugging Face for environment and weights, then lists the dependencies: ```bash pip install torch transformers accelerate qwen-vl-utils pip install decord opencv-python-headless ``` with flash-attn as an optional extra installed without build isolation. Two inference implementations ship in the demo directory, one for standard model inference and one for the online proactive-response mode, which is the capability most video models omit entirely: reacting while the stream is still running rather than answering after the file ends. ## Limits to weigh The boundaries are the honest ones of a research release. There is no licence file in the repository tree, so reuse terms are unstated despite the fully open framing; check the Hugging Face model card before any production or commercial plan. Hardware demands are those of a 4B video model with training scripts aimed at multi-GPU and Slurm clusters; the README documents the cluster path in detail and says little about laptop inference. The project is young and moving: the last push was on 2026-08-25, the same day the paper update landed, with no releases or tags. Benchmark positioning comes from the paper rather than the README, which tabulates no numbers itself. For a team adopting VideoChat3, the realistic flow is: read the paper for the benchmark claims, pull the weights for inference, and use the staged checkpoints only if you intend to reproduce or extend training. ## Against closed video models and partial releases The comparison that matters is with closed video understanding APIs and with open models that release weights but not data. Closed APIs are convenient and opaque: no training recipe, no data provenance, and a dependency on someone else's rate limits. Partially open research releases publish weights and stop, leaving reproduction to guesswork about data mixture and staging. VideoChat3's differentiator is the complete chain: weights, full training data, staged checkpoints, training code on a published framework, and evaluation code covering the paper's benchmarks. The trade is that you operate it, on GPUs, with a cluster-oriented codebase. For research groups studying video temporal reasoning, or companies with data-residency needs and GPU budget, that completeness converts the model from a demo into a foundation. ## Conclusion VideoChat3 fits research teams and product groups that need video understanding with temporal grounding or live proactive response, and have the GPUs to run and possibly retrain a 4B model on the published staged pipeline. Skip it if you need a licensed, supported API, if your use is single-image or text-only, or if you cannot cluster-train, since the README's strength is the full training chain rather than a lightweight endpoint. Verify the openness claim firsthand: download the weights, run the standard demo on one of your own clips, then check whether the online proactive mode fits your latency budget before planning anything around it. ## FAQ ### What can VideoChat3 do? It is a 4B generalist video MLLM covering fine-grained motion understanding, long-video reasoning, temporal grounding and online proactive response to live streams, combining I3D-ViT token compression with adaptive frame resolution. ### Is VideoChat3 fully open? The README's release list shows model weights, complete training data, evaluation code and training code all published, with staged training checkpoints included. The repository tree itself carries no licence file, so check the model card for reuse terms. ### How do I run VideoChat3 for the first time? Follow the Hugging Face model card for setup and weights, install torch, transformers, accelerate and qwen-vl-utils plus decord and opencv-python-headless, then use one of the two demo implementations for standard or online inference. ## Sources - [Issues](https://github.com/MCG-NJU/VideoChat3/issues) - [MCG-NJU/VideoChat3 on GitHub](https://github.com/MCG-NJU/VideoChat3) - [README](https://github.com/MCG-NJU/VideoChat3/blob/main/README.md) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/mcg-nju-videochat3