LingBot-Video: an MoE video model for embodied intelligence

What LingBot-Video is for, and who should care

Most open video models are trained to look good. LingBot-Video is trained to look plausible as physics. The README describes it as the first open-source large-scale MoE video generation model dedicated to embodied intelligence, and the training mix is the evidence: massive web video plus more than 70,000 hours of embodied data, with a reward system that scores aesthetics, physical rationality and task completion separately. That combination targets robotics and world-model research more than short-form video production.

The intended user is someone who needs a generated clip to be consistent with a physical scene, for example a manipulation task or a camera move through a room that has to obey contact and gravity. The repository also ships RBench, an evaluation set for embodied video generation, which tells you the authors expect you to measure this rather than eyeball it.

If you want a general-purpose text-to-video toy for social clips, the embodied bias is overhead. The model still does T2I, T2V and TI2V, but you are paying for a training recipe you will not use.

The two-stage prompt pipeline is the actual interface

The README is blunt about this: DiT inference is designed to consume structured JSON captions, not casual natural-language prompts. So the real entry point is not the diffusion script, it is the rewriter.

Step one takes the plain user prompt and expands it into JSON. Step two runs Auto Negative, which prunes the negative prompt for that specific caption instead of using a fixed list. Only then does the unified runner see `--prompt_json` and `--negative_prompt_json`.

The rewriter is a Qwen3.6-27B base model with a LoRA adapter, and the README warns that the two stages must keep their semantics: step one uses the base VLM without the adapter, step two uses the same base VLM with the adapter enabled. That is an awkward constraint for serving. You either run two endpoints or one server that can select the adapter per request. The bundled single-process `transformers` backend does not scale, and the README points at vLLM or SGLang for throughput rather than shipping that path itself.

This is the part I would flag hardest. A video model whose quality depends on a 27B language model rewriting your prompt is a two-model deployment, and the documentation treats the serving topology as your problem.

Installing LingBot-Video from the repository

The root `requirements.txt` pins a specific PyTorch nightly build, not a stable release. The README states that this file includes the recommended PyTorch build for inference, and the file itself points at the cu130 nightly index. Clone, create a virtual environment, then install.

git clone https://github.com/Robbyant/lingbot-video
cd lingbot-video

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip

pip install -r requirements.txt
pip install -e .

The base requirements cover direct DiT inference and the rewriter through the `transformers` backend. SGLang is optional and separate, and the README says to install it only when using SGLang Diffusion or the fused / FP8 MoE runtime.

python -m pip install --no-deps -r requirements-sglang.txt

The README recommends Python 3.10 or newer, `transformers` 5.8.1, `diffusers` 0.39.0, `peft` 0.19.1, `json_repair` 0.30 or newer, `decord` 0.6.0 or newer and `safetensors` 0.4.5 or newer. Note that `pyproject.toml` is looser than `requirements.txt` on several of these, so the requirements file is the version you should trust for a first run.

A first run: rewrite, prune, then generate

Model weights are not in the repository. You download the Dense 1.3B or MoE 30B-A3B package from Hugging Face or ModelScope, plus the rewriter base model and the LoRA adapter, and point three environment variables at them.

export MODEL_DIR="<path_to_lingbot-video-model>"
export REWRITER_BASE_MODEL="<path_to_rewriter_base_vlm>"
export REWRITER_ADAPTER="<path_to_rewriter_lora>"

Then produce the structured caption. The README gives this example for text-to-video with a five second duration, writing `prompt.json`.

python rewriter/inference.py --backend transformers --mode t2v \
  --prompt "<plain_user_prompt>" --duration 5 --output prompt.json

Auto Negative runs next and writes `negative.json`. The README notes that if you skip it, you must also remove `--negative_prompt_json` from the generation command, which is a clean way to tell that the two are coupled.

python rewriter/auto_negative.py --backend transformers --mode t2v \
  --caption prompt.json --output negative.json

Finally the unified runner. `--backend` selects `diffusers` or `sglang`, and the README says that if the SGLang package is not installed the runner falls back to direct diffusers and prints a warning.

export BACKEND=diffusers

python scripts/inference.py \
  --backend "$BACKEND" \
  --model_dir "$MODEL_DIR" \
  --run_refiner \
  --mode t2v \
  --prompt_json prompt.json \
  --negative_prompt_json negative.json \
  --output "<output_dir>/base.mp4" \
  --refiner_output "<output_dir>/refined.mp4" \
  --height 480 \
  --width 832

You should end up with a base clip and a refined clip. The README's own example is truncated after the resolution flags, so treat the remaining arguments as something to read from `scripts/inference.py` rather than guess.

Multi-GPU memory behaviour and where it bites

The README addresses multi-GPU inference with two flags. `--enable_fsdp_inference` shards the base DiT and the refiner DiT across GPUs, and `--enable_vlm_fsdp_inference` does the same for Qwen3-VL when that also needs sharding. The two are independent, which is useful because the language side and the diffusion side have different memory profiles.

The limitation is stated plainly: FSDP reduces GPU memory pressure after loading, but each rank still constructs the transformer on host memory before sharding. So a 30B-A3B MoE checkpoint has to fit in system RAM on every rank first. Adding GPUs does not fix a machine that cannot hold the model in host memory, and it does not reduce the load time before sharding begins.

The README also points at an inference performance benchmark document for the Dense and MoE speed, memory and exactness matrix. That is the right place to check before buying hardware, and it is also the document that would tell you whether the claimed roughly 3x inference speedup from the MoE architecture applies to your sequence length and resolution. The README states the ~3x figure as a highlight; it does not break it down in the main file.

Where LingBot-Video is the wrong choice

The structured-caption requirement is the biggest filter. If your pipeline produces free-form prompts and you want a single call that returns a video, LingBot-Video is not that. You have to stand up the rewriter, which means a second large model, and you have to keep the two LoRA states separate per the README's warning. Teams that treat prompt rewriting as optional will get outputs that do not match what the DiT was trained to consume.

Second, the pinned torch build is a nightly from the cu130 index. That is a real operational constraint: nightly wheels move, and the README pins an exact date-stamped build rather than a range. Anyone with a locked-down environment that only allows stable PyTorch releases will have to deviate from the recommended setup and accept the consequences.

Third, there is no mention of ComfyUI, GGUF or quantised single-file distributions in the repository or the README. If your workflow is node-based or you run models on consumer hardware through a quantised loader, this project does not document a path for you. The related searches around ComfyUI and GGUF have no corresponding answer in the README.

Alternatives and the difference in approach

The natural comparison is a dense open video model run through `diffusers` directly. LingBot-Video actually includes that path: the `diffusers` backend is described as the direct diffusers reference path, and the Dense 1.3B checkpoint exists precisely so you can run the same pipeline without the MoE routing. If you only want to evaluate whether the prompt pipeline helps, start with Dense 1.3B and the diffusers backend, then move to MoE 30B-A3B with SGLang if throughput matters.

The other axis is SGLang Diffusion versus direct diffusers. The README treats SGLang as the higher-throughput runtime and diffusers as the reference. The difference is not just speed: the fallback behaviour means a missing SGLang install silently changes your execution path to diffusers with a warning, so a benchmark that does not check which backend actually ran can compare two different runtimes without noticing.

Against a general-purpose hosted video API, the trade-off is control versus setup. LingBot-Video gives you the weights, the rewriter, the evaluation set and the licence, and asks you to assemble a two-model deployment. A hosted API asks for a key and gives you no say in the prompt pipeline.

Licence, maintenance and upgrade cost

The repository is Apache-2.0, and `pyproject.toml` declares `license = { text = "Apache-2.0" }`. That covers the code in this repository. It does not automatically cover the model weights, which are distributed separately on Hugging Face and ModelScope, and the README links a `LICENSE` file for the project. Check the licence attached to each checkpoint and to the Qwen3.6-27B rewriter base model separately before commercial use; the rewriter base is a third-party model, not a Robbyant release.

The last push to the default branch was on 2026-08-05, and the README's latest news entry is from the same date, announcing the RBench evaluation release. The repository is not archived. There are no retrieved releases, so upgrades are tracked through commits and documentation rather than versioned tags.

Upgrade cost is dominated by the pinned nightly torch build and the exact version pins in `requirements.txt`. Moving `transformers` from 5.8.1 or `diffusers` from 0.39.0 is not a routine bump when the rewriter depends on specific adapter behaviour. Budget for re-running the rewriter and Auto Negative steps after any dependency change, because a different tokenizer or chat template can shift the JSON captions that the DiT consumes.

Editorial conclusion

Adopt LingBot-Video if you already have a GPU box with enough system RAM to hold a 30B-A3B checkpoint in host memory before FSDP sharding, and if you are willing to run the rewriter and Auto Negative steps rather than typing a prompt. Skip it if you want a one-line text-to-video call, if you need a ComfyUI node, or if you cannot source the Qwen3.6-27B base VLM for the rewriter. Verify three things first: that your torch build matches the pinned 2.12.0.dev20260220+cu130 nightly, that the rewriter adapter loads on top of the base VLM without the LoRA in step one, and that your system RAM survives loading the MoE transformer before sharding.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. Robbyant/lingbot-video on GitHub