Video-ChatGPT: a video conversation model and its evaluation benchmark

What Video-ChatGPT is for, and who it is actually for

The problem the project addresses is that a general vision-language model answers questions about a single image, while a video question is usually about change over time: what happened, in what order, and what it implies. The README states that Video-ChatGPT combines an LLM with a pretrained visual encoder adapted for spatiotemporal video representation, and that the model generates conversation about videos rather than a single caption. The second half of the contribution is evaluation. The authors describe a quantitative video conversation evaluation framework, and the repository links leaderboard entries for VCGBench-Diverse, the video-based generative performance benchmark, and zero-shot question answering on MSVD-QA, MSRVTT-QA, TGIF-QA and ActivityNet. That combination of a model and a scoring harness is the reason to look at this repository rather than a generic multimodal codebase. The intended user is a researcher who needs a baseline and a way to score it. A team that wants a supported API for captioning user uploads is not the intended user, and the release notes support that reading: the March 2025 update announces Mobile-VideoGPT with higher throughput, and the June 2024 update announces VideoGPT+ with state-of-the-art results on multiple benchmarks. The maintainers are pointing people forward, not maintaining this as a service.

The architecture the README describes, and where it stops

The README gives one sentence of architecture and one diagram: an LLM plus a visual encoder adapted for spatiotemporal representation. Everything else about the data flow has to be inferred from the repository layout, which is the honest limit of what can be claimed here. The training entry point is a script named train.py, invoked with a configuration file, and the README's training section directs you to download the pretrained projector and the LLaVA and Vicuna weights before running it. That ordering tells you the pipeline: video frames are encoded, projected into the language model's token space, and the language model is what produces the answer. The projector is the trained part; the base weights are frozen or reused. The published VideoInstruct-100K dataset supplies the 100K video-instruction pairs the README describes, generated by an annotation framework the authors say is scalable. Two things are worth flagging. First, a spatiotemporal encoder that pools frames into a fixed number of tokens will lose fine temporal detail, and the README does not describe how many frames are sampled or how that number is chosen, so you cannot predict from the documentation alone how the model behaves on long or fast-cut video. Second, the evaluation harness is described as quantitative and generative, meaning it scores free-text answers, which makes the score dependent on the judging procedure as much as on the model.

Getting it running: the commands and checkpoints the README names

The README documents three paths. For a hosted trial, it points to the online demo at ival-mbzuai.com/video-chatgpt. For local use, the offline demo is documented under a section titled Running Demo (Offline), and the training path is documented under Training. Training is invoked as python train.py with a config argument, following the convention used across the LLaVA family, and the README instructs you to fetch the pretrained projector together with the LLaVA and Vicuna checkpoints before starting. The datasets and extracted features are distributed through a SharePoint folder linked from the June 2023 update, and VideoInstruct-100K is separately available on Hugging Face at MBZUAI/VideoInstruct-100K. Evaluation is documented in the Quantitative Evaluation section, with the benchmark also published as a standalone site. The practical constraint is that this is a multi-gigabyte setup: a visual encoder, a 7B-class language model, a projector, and a 100K instruction dataset. If you only want to score an existing model rather than train one, the evaluation benchmark is the smaller commitment, and the VCGBench-Diverse dataset of 4,354 human-annotated QA pairs across 18 video categories is published on Hugging Face under MBZUAI/VCGBench-Diverse.

The licence covers the code, not the whole stack

The repository is licensed CC-BY-4.0. That is a content licence, not a software licence, and it is unusual for a training and inference codebase. It permits sharing and adaptation with attribution, which is permissive enough for academic reuse, but it does not include the patent grant that Apache-2.0 carries, and it says nothing about the weights, the datasets or the third-party base models the setup instructions tell you to download. The LLaVA and Vicuna checkpoints are distributed under their own terms, and the videos you feed the model carry their own rights. If you plan to redistribute a fine-tuned model or to use outputs commercially, the licence file in this repository is not the whole answer, and this is a question for your own legal review rather than something the README resolves. The attribution requirement is easy to satisfy and worth satisfying deliberately: cite the ACL 2024 paper by Maaz, Rasheed, Khan and Khan.

Where this is the wrong tool

The clearest failure mode is expecting throughput. The repository's own March 2025 update promotes Mobile-VideoGPT on the grounds of roughly double the throughput, which is an implicit statement about the cost of running this model. A 7B-class language model plus a video encoder is not a cheap per-request pipeline, and nothing in the README suggests batching or streaming optimizations. The second limitation is maintenance. The last push is dated September 2025, but the substantive update entries end with pointers to successor projects, and there are no retrieved releases, so there is no versioned artifact to pin against. You are tracking a branch. The third case is domain fit. The benchmark covers 18 video categories and the instruction data is general-purpose; a team working on, say, surgical video or factory inspection has no evidence here that the instruction tuning transfers, and would be better served by fine-tuning on their own annotated clips using the published pipeline rather than by adopting the released weights. Finally, if your requirement is a captioning API with an SLA, a research repository with SharePoint-hosted checkpoints is the wrong shape of dependency regardless of model quality.

VideoGPT+ and Mobile-VideoGPT: the same authors, different trade-offs

The README itself names the alternatives, which is the most useful signal in the document. VideoGPT+ is announced as achieving state-of-the-art results on multiple benchmarks, and the same update bundles a semi-automatic video annotation pipeline and the VCGBench-Diverse benchmarks. Mobile-VideoGPT, announced in March 2025, is positioned on throughput rather than accuracy. The difference in approach is worth stating precisely: Video-ChatGPT is the 2023 baseline that established the instruction dataset and the evaluation harness, while the successors reuse that evaluation scaffolding and change the model. That means the benchmark outlives the model. If you are choosing where to spend effort, the evaluation framework and VCGBench-Diverse are the durable artifacts, and the model checkpoint is the part most likely to be superseded. Comparing against a general multimodal model such as LLaVA applied frame by frame is a different proposition again: that approach treats video as a bag of images and loses the temporal adaptation the README describes, which is exactly the gap this line of work exists to close.

What to verify before you build on it

Check that the SharePoint links in the June 2023 update still resolve, because that folder is where the README says models, datasets and extracted features live, and SharePoint links tied to a university tenant are the most fragile part of the setup. Confirm the LLaVA and Vicuna checkpoint paths named in the training instructions are still published, since those are external dependencies the repository does not control. Decide early whether you need the model or the benchmark, because they carry very different costs: the benchmark is a dataset and a scoring script, while the model is a training and inference stack. If you intend to publish comparisons, read the VCGBench-Diverse paper alongside the dataset, since a generative benchmark is only as meaningful as its judging criteria. And treat the absence of releases as a real constraint: pin a commit hash rather than tracking main, because there is no tag to fall back to.

Editorial conclusion

Adopt Video-ChatGPT if you are reproducing or benchmarking video conversation research and can work from a research repository: the training code, offline demo and VideoInstruct-100K are published, and the evaluation benchmark is the part with the longest shelf life. Do not adopt it as a production video Q&A service; the code is a 2023 research release, the weights are hosted on SharePoint rather than a model registry, and the repository's own update log now points to VideoGPT+ and Mobile-VideoGPT. Before you commit, verify three things: that the SharePoint links still resolve, that the LLaVA and Vicuna checkpoints named in the setup instructions are still downloadable at those paths, and that CC-BY-4.0 covers what you intend to redistribute, since that licence applies to the repository and not automatically to third-party base models or to the videos you feed it.

Official sources

1. Issues
2. License: CC-BY-4.0
3. mbzuai-oryx/Video-ChatGPT on GitHub
4. Project website
5. README