bytedance/Lance: a 3B unified model for image and video understanding, generation and editing

What Lance actually is, and the problem it targets

Most teams that need image and video capabilities end up stitching together separate checkpoints: one model to caption or answer questions about an image, another to generate one, a third to edit one, and something else again for video. Each has its own preprocessing, its own tokenizer, its own prompt format. Lance is ByteDance's attempt to collapse that stack into a single 3B-active-parameter model that handles image and video understanding, generation and editing inside one framework.

The target user is not an application developer looking for an API. It is a researcher or an ML engineer who wants to study how far a unified model can go at a small parameter count and a bounded compute budget. The README is unusually direct about this: it states that Lance is a research project rather than a polished product model, and that the released checkpoint was trained with up to 128 A100 GPUs, with training conducted up to 768x768 image generation and 480p, 12 FPS video generation. The stated goal is to share a research artifact for studying unified image and video understanding, generation and editing under a relatively small model and limited compute budget.

That framing matters more than any benchmark chart in the repository. A 3B active-parameter model that covers five task families is a different proposition from a specialist model that does one of them well. Lance is interesting precisely because it is small enough to reason about and broad enough to be a single artifact. It is also, by the authors' own admission, uneven.

How the unified framework is put together

The repository layout tells you most of what the README leaves implicit. There is a modeling/ directory for the model definition, a config/ directory for configuration, a common/ directory, a data/ directory for data handling, and a train/ directory alongside train.md and train_dataset.md for the fine-tuning path. Inference is driven by inference_lance.py with a shell wrapper, inference_lance.sh, and there is a separate lance_gradio.py for the interactive interface. The changelog entries confirm that the Gradio interface supports image and video generation, editing and understanding, and that Image-to-Video generation was added on 2026-05-29.

The design claim is multi-task synergy: one model, one set of weights, shared representations across understanding and generation. The README's title calls this 'Unified Multimodal Modeling by Multi-Task Synergy', and the highlights describe a staged multi-task training recipe rather than a single monolithic training run. That is the mechanism to pay attention to. A staged recipe means the task mix and the stage boundaries are part of the artifact, which is why the repository ships train.md and a train_dataset.md describing the data side.

What the README does not give is a layer-by-layer architecture description. There is no diagram of the tokenizer, the vision encoder, the diffusion or flow component, or how the understanding and generation paths share parameters. For that you have to go to the arXiv technical report linked from the badges. If you are evaluating Lance for a specific pipeline, budget time for the paper before you budget time for the code.

Installing Lance and running a first inference

The repository pins its dependencies in requirements.txt, so the first step is a clean environment. The list is long and tightly versioned: transformers==4.49.0, diffusers==0.29.1, accelerate==1.13.0, gradio==5.31.0, decord==0.6.0, opencv-python==4.7.0.72, numpy==1.23.5. Torch is not pinned in the file at all, which means you supply your own build matching your CUDA version. Install into a virtual environment rather than your system Python, because several of these pins (numpy 1.23.5 in particular) conflict with what modern distributions ship.

pip install -r requirements.txt

After the dependencies resolve, weights come from Hugging Face. The README links the model repository at huggingface.co/bytedance-research/Lance, and the requirements list huggingface-hub==0.29.1, so a standard download path applies. The repository does not document the exact checkpoint directory layout or the expected local path in the README text, so check the model card for the file list before assuming a folder name.

The repository ships a shell entry point for inference, inference_lance.sh, alongside inference_lance.py. The README does not reproduce the script's arguments, so read the script before running it; the flags it passes are the authoritative list. For an interactive first look, lance_gradio.py launches the Gradio interface, which the changelog says covers image and video generation, editing and understanding. Expect the first run to spend most of its time loading weights, and expect output quality to move around with resolution and prompt, which the README warns about directly.

The research-artifact caveat is the real limitation

The note at the top of the README is the most useful paragraph in the repository. It says output quality may vary across prompts, resolutions, duration, motion complexity and editing scenarios, and that the authors see further opportunities to improve the post-training recipe. That is not marketing hedging. It is a statement that the post-training stage is the known weak point, and that the released checkpoint is the current best effort rather than a finished one.

The practical consequence is that Lance is the wrong tool for anything with a quality floor you cannot move. If you are building a product where a user submits a prompt and expects a usable image or clip every time, a model whose authors describe output quality as varying by prompt and resolution will fail that contract. The same applies to editing workflows where a bad edit is worse than no edit.

There is a second constraint that is easy to miss. The training description mentions up to 128 A100 GPUs and video at 480p and 12 FPS. Those are the conditions the checkpoint was trained under. Running it outside those conditions (higher resolution, longer duration, faster motion) is not covered by the README, and the authors name duration and motion complexity as sources of variance. Treat 480p and 12 FPS as the documented operating point, not a suggestion.

Finally, there are no releases retrieved for this repository, so versioning is by commit. The last push was on 2026-07-14. If you need pinned, dated artifacts, that is a gap you have to manage yourself.

Where Lance fits against specialist models and vLLM-Omni

The obvious alternative is not a single competitor but the specialist route: a dedicated image generation model, a separate video generation model, and a vision-language model for understanding. That approach gives you a best-in-class component for each task and lets you upgrade them independently. It also triples your integration surface, your dependency graph and your evaluation work, and it means three sets of weights in memory instead of one.

Lance's bet is the opposite: accept some quality loss per task in exchange for one artifact, one config, one training recipe. The README's own framing supports reading it that way, since the stated goal is studying unified modeling under a small model and limited compute budget. If your interest is the unification question itself, Lance is a rare open artifact at this scale. If your interest is the best possible video, the specialist route wins on the axis you care about.

A second, more concrete alternative is serving through vLLM-Omni. The 2026-06-03 changelog entry states that Lance is now supported in vLLM-Omni, with a recipe at github.com/vllm-project/vllm-omni/blob/main/recipes/ByteDance/Lance.md. That is a different approach to the same model: instead of the repository's own inference_lance.py path, you serve Lance through vLLM's infrastructure. The README does not compare the two paths or state which one the maintainers recommend, so the trade-off between them is undocumented. If throughput matters to you, that recipe is worth reading before you commit to the bundled script.

Fine-tuning, licence and the cost of staying current

Fine-tuning code landed on 2026-06-17, with the guide in train.md, and the repository also carries train_dataset.md and train_dataset_zh.md for the data side. That is the upgrade path if the released checkpoint does not meet your bar: you are expected to bring your own data and your own compute. The README does not state minimum GPU requirements for fine-tuning, so the only scale reference available is the training description of up to 128 A100 GPUs. Assume fine-tuning is a cluster activity until you find evidence otherwise.

On licensing, the repository carries Apache-2.0. That is a permissive licence and it is the only licence statement in the repository. What Apache-2.0 does not settle is the status of the model weights themselves, which live on Hugging Face rather than in this repository, and the licence terms attached to a model checkpoint are a separate question from the licence on the code. Check the model card for the weight terms before you plan a deployment. This is not legal advice; if the distinction matters commercially, get it reviewed.

Maintenance cost is the other number to price in. The last push was on 2026-07-14, and the changelog shows a dense run of updates through May and June 2026: inference code and weights on 2026-05-18, the technical report on 2026-05-19, the Hugging Face Space on 2026-05-25, the expanded Gradio interface on 2026-05-26, Image-to-Video on 2026-05-29, vLLM-Omni support on 2026-06-03, and fine-tuning code on 2026-06-17. A repository moving at that pace will change under you. Pin a commit and expect to re-read train.md when you move.

Who should pick this up, and what to check first

Take Lance seriously if you are researching unified multimodal modeling at small scale, if you want a single checkpoint that spans image and video understanding, generation and editing for experiments, or if you are evaluating multi-task synergy recipes and want a reference implementation with training code attached. The presence of train.md, train_dataset.md and a modeling/ directory means you can inspect the recipe rather than only the outputs, which is the point of a research artifact.

Stay away if you need a dependable image or video generation service, if your workflow cannot tolerate output quality that varies by prompt and resolution, or if you need dated releases rather than commit-based snapshots. The authors' own note rules Lance out for those cases, and no amount of benchmark imagery changes that.

Before you invest time, verify three things in order. First, the Hugging Face model card for the weight licence and the actual file layout, since the repository does not document either. Second, the arXiv technical report for the architecture, because the README never explains how the understanding and generation paths share parameters. Third, whether vLLM-Omni or inference_lance.py is the serving path you want, since the repository documents the first only as a link and the second only as a script you have to read. If those three checks come back clean for your use case, Lance is one of the few open 3B unified models you can actually fine-tune.

Editorial conclusion

Adopt Lance if you are a researcher or ML engineer who wants a single small checkpoint that covers image understanding, image generation, image editing, video generation and video understanding, and who is comfortable reading train.md and modeling/ to see how it fits together. Do not adopt it as a production image or video service: the README explicitly calls it a research project rather than a polished product model and warns that output quality varies across prompts, resolutions, duration, motion complexity and editing scenarios. Before committing, check the Hugging Face model card for the exact weight revision and hardware requirements, and confirm whether your GPU budget matches the training scale described (up to 128 A100 GPUs, 768x768 images, 480p video at 12 FPS).

Frequently asked questions

Official sources

1. bytedance/Lance on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README