OmniLottie: Generating Lottie JSON Animations from Text, Images and Video

The gap OmniLottie targets: Lottie JSON is structured, not a picture

A raster image generator outputs a grid of pixels. A Lottie file is a JSON document describing layers, shapes, keyframes and interpolation, and a player such as lottie-web or the Lottie Android and iOS libraries reads that document. The two outputs fail in different ways. A blurry pixel is still a picture; a malformed keyframe is a file that will not open. OmniLottie's stated goal is to generate that JSON directly from multimodal instructions, which the README describes as "texts, images, and videos". The intended user is a researcher or an engineer building an animation tool who wants a model checkpoint rather than a hosted service, and who is willing to accept research-grade output quality in exchange for running the generator on their own hardware. The repository also ships MMLottie-2M, described as two million annotated Lottie animations, and MMLottieBench, a benchmark for vector animation generation. That combination (weights, dataset, benchmark) is what makes the release useful to someone comparing approaches rather than just demoing one.

Parameterized Lottie tokens: the mechanism the title refers to

The paper title is "Generating Vector Animations via Parameterized Lottie Tokens", and the README's own note explains the measurement basis: inference time is counted per OmniLottie Lottie token, while the paper reports per JSON code token "for fair comparison with baseline methods". That tells you the model does not emit raw JSON characters. It emits tokens that stand for Lottie parameters, and a decoder step turns those tokens into the JSON document. The architecture is a pretrained VLM adapted to this output space, which is why the release is framed as "the first family of end-to-end multimodal Lottie generators that leverage pre-trained Vision-Language Models". Text, image or video goes in as the conditioning signal; parameterized Lottie tokens come out; the JSON is written to the path given by --output. The practical consequence is that output length is not fixed. A simple bouncing ball and a multi-layer character animation will not cost the same, and the README's timing table is the only published guide to that spread.

Two model formats, and which one to pick

The README documents two checkpoints layouts. The Original Format uses pytorch_model.bin and is driven by inference.py and app.py; it exists for people who downloaded the weights before HuggingFace format support landed on 2026/03/20. The HuggingFace Format uses model-*.safetensors plus config.json, is driven by inference_hf.py and app_hf.py, supports from_pretrained() and automatic downloading, and is labelled "Recommended for new users". The README states both formats produce identical results, so the choice is about tooling, not quality. If you are starting fresh, the HF path is shorter: the model_path argument accepts either a local directory or the Hub identifier OmniLottie/OmniLottie, and the Gradio app reads the same value from a MODEL_PATH environment variable. The single published checkpoint is OmniLottie(4B) at 8.46 GB, updated 2026-03-02. There is no smaller variant listed, so the 4B size is the floor for self-hosting.

Getting it running: conda, CUDA 12.1 and the three inference modes

The install path is explicit. Clone the repository, then conda create -n omnilottie python=3.10 and conda activate omnilottie. The README states the environment was tested with CUDA 12.1, and the PyTorch install line is pinned: pip install torch==2.3.0+cu121 torchvision==0.18.0+cu121 --index-url https://download.pytorch.org/whl/cu121, followed by pip install -r requirements.txt. Weights come down with huggingface-cli download OmniLottie/OmniLottie --local-dir /PATH/TO/OmniLottie after pip install huggingface-hub. Text-to-Lottie is python inference_hf.py --model_path OmniLottie/OmniLottie --text "A bouncing ball" --output output.json. Image-to-Lottie adds --image image.png alongside --text "rotating animation". Video-to-Lottie swaps in --video video.mp4 and needs no text flag in the example shown. For a browser UI, MODEL_PATH=/PATH/TO/OmniLottie python app_hf.py, or point MODEL_PATH at the Hub identifier. The pinned torch and CUDA versions are the constraint most likely to bite: if your cluster is on a newer CUDA, you are choosing between matching the README exactly or accepting that the tested configuration no longer applies.

The timing table is the real budget document

The README publishes one hardware figure: 15.2 GB of GPU memory, and 8.34, 16.68, 33.38, 66.74 and 133.49 seconds for 256, 512, 1024, 2048 and 4096 tokens respectively. Read the shape of that curve rather than the first number. Cost scales linearly with token count, so a 4096-token animation is roughly sixteen times the wall clock of a 256-token one. The README also flags that this is per Lottie token, not per JSON code token, and that the paper uses the other basis. Anyone comparing OmniLottie's published speed against a baseline should check which unit each side is using before drawing a conclusion. What the table does not tell you is how many tokens a typical prompt produces, so the practical question (does a character animation fit in a latency budget) cannot be answered from the README alone. You need to generate a few outputs and count.

What is missing: training code, and any quality claim you can check

The open-source plan is a checklist, and one box is empty. Project page and report, dataset, inference code and weights, the Gradio demo and the benchmark are all marked done. Training Code is not. So you can run OmniLottie, evaluate it on MMLottieBench and fine-tune it with your own loop if you write one, but you cannot reproduce the training run from this repository. For a team whose requirement is a generator aligned to a specific illustration style, that is the deciding limitation: you are starting from a 4B checkpoint and building your own training path, not adapting a supported recipe. There is also a gap between the model and the artifact. The README describes generating Lottie JSON, and it does not describe a validation or repair step for the JSON that comes out. Whether a given output loads cleanly in a Lottie player is something you find out by trying it. That is normal for a research release and it is still the first thing to test.

Alternatives, and where the difference actually lies

The nearest hosted option is LottieFiles Motion Copilot, which generates Lottie animations from text inside a commercial web tool. The difference is not model quality, it is where the control sits. Motion Copilot gives you a browser and an account; OmniLottie gives you a checkpoint, a conda environment and a JSON file on disk. If you need to embed generation inside a build step, run it offline, or inspect the token stream, the hosted tool cannot help and the local checkpoint can. If you need a working animation today with no GPU, the hosted tool wins on effort. The other comparison worth making is to OmniSVG, the sibling project whose commented-out header is still visible at the top of this README. OmniSVG targets static vector graphics; OmniLottie targets animation, and its output is a timeline of keyframes rather than a single drawing. If your requirement is a static SVG, OmniLottie is the wrong tool regardless of how the demo looks. A third option is a general code-generating LLM prompted to emit Lottie JSON. It needs no GPU and no 8.46 GB download, and the README's framing of OmniLottie as a VLM adapted to parameterized Lottie tokens is precisely the argument for why a general model would struggle with the format's structure.

Maintenance, licence and what to verify before you depend on it

The repository is Apache-2.0 and not archived, last pushed 2026-04-06, with the paper accepted to CVPR 2026. No releases were retrieved, so there is no tagged version to pin against; the practical pin is a commit hash plus the model revision on the Hub. The checkpoint is dated 2026-03-02 and the HuggingFace format support landed 2026-03-20, which means the HF path is newer than the weights it loads. Apache-2.0 covers the code in this repository, and the README does not state separate terms for the weights or the MMLottie-2M dataset, both of which live on Hugging Face. Check the licence shown on each Hub repository before you ship anything built on them; that is a permissions question for your own counsel, not something the GitHub licence file settles. The community ComfyUI plugin is a separate project by another author and carries its own maintenance. On cost: the moving parts are the pinned torch and CUDA 12.1 pair, the 8.46 GB download, and 15.2 GB of GPU memory at inference. None of those shrink on their own, and the training code box is still unchecked, so any fine-tuning work is yours to own.

Editorial conclusion

Adopt OmniLottie if you need a research-grade, self-hosted path from a prompt or a reference image to a Lottie JSON file and you can absorb a 15.2 GB GPU footprint plus the per-token times in the README. Do not adopt it if you need a trained-on-your-brand pipeline, because the open-source plan still lists Training Code as unchecked, or if you need a guaranteed renderable animation on the first try. Before you commit, run inference_hf.py on one text prompt, one image and one video from the MMLottieBench set, load each output.json in a Lottie player, and compare the token count against the 8.34 to 133.49 seconds per 256 to 4096 tokens table.

Official sources

1. Issues
2. License: Apache-2.0
3. OpenVGLab/OmniLottie on GitHub
4. Project website
5. README