Tuna-2: a unified multimodal model that skips the vision encoder

Stripping the vision stack out of an earlier design

Tuna-2 is the official implementation of a unified multimodal model, and the way it arrives at its design is by subtraction. The README describes the project as simplifying Tuna by progressively stripping away its visual encoding components.

The first removal is the VAE, which produces Tuna-R, described as a pixel-space unified multimodal model that relies solely on a representation encoder. Tuna-2 goes further and bypasses the representation encoder entirely, using direct patch embedding layers for raw image inputs. The claim in the overview is that Tuna-2 using pixel embeddings outperforms both Tuna-R and Tuna across a diverse suite of multimodal benchmarks.

The three model variants are named in the inference options as none_encoder, siglip_pixel and vae, and those correspond to Tuna-2, Tuna-R and Tuna respectively, with none_encoder as the default. So the codebase carries all three architectures rather than only the final one, which is what makes the comparison reproducible rather than a claim in a paper.

The project is a collaboration between Meta, the University of Hong Kong and the University of Waterloo, with a project page and an arXiv entry linked from the README. The primary language is Python, the licence is Apache-2.0, the default branch is main, the last push was on 2026-07-22, and the repository has no GitHub releases.

Installing with uv, and the versions that are pinned

Installation is a clone and a setup script, and the script creates a virtual environment with all the dependencies:

git clone https://github.com/facebookresearch/tuna-2.git
cd tuna-2
bash scripts/setup_uv.sh   # creates .venv with all dependencies
source .venv/bin/activate

There is a manual route in the README for anyone who prefers to drive uv themselves, and it shows the ordering that matters, since torch is installed from the PyTorch index before the project itself:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
uv pip install -e .
source .venv/bin/activate

The dependency list in the pyproject.toml is worth reading closely because the pins are deliberate. The project metadata explains that versions are pinned where it matters for compatibility with the tuna-lmms-eval stack, and that torch is intentionally loose so users can pick the CUDA wheel that matches their host.

So transformers is pinned to 4.52.4, diffusers to 0.35.1, timm to 1.0.19, einops to 0.8.1, torchdiffeq to 0.2.5 and omegaconf to 2.3.0, while torch is only constrained to 2.4 or newer. numpy is held below 2. The dependency list even carries a comment explaining that setuptools is pinned between 68 and 81 because Python 3.12 environments no longer ship setuptools by default and setuptools 81 and later dropped the bundled pkg_resources module that torchtnt's version file still imports.

That last detail is a good signal about the state of the project. It is the kind of comment that only appears when someone has already hit the problem and fixed it for everyone else.

The package name is tuna at version 0.1.0, and it requires Python 3.10 or newer, with classifiers for 3.10 and 3.12. Two console scripts are declared, tuna-train and tuna-predict, pointing at the train and predict entry points.

One prediction script with a variant flag and a task flag

All inference goes through a single script, and the arguments carry the whole configuration:

bash scripts/launch/predict.sh --ckpt <PATH> --prompt <TEXT> [OPTIONS]

The flag table in the README is complete, and two of the entries shape everything else. The --variant flag selects none_encoder, siglip_pixel or vae, defaulting to none_encoder, which is Tuna-2, Tuna-R and Tuna respectively. The --task flag selects t2i or edit, defaulting to t2i. The --size flag takes 7b or 2b with 7b as the default, and the note in the table says the 2b size is only available for the vae variant, which is a constraint worth knowing before you plan a 2b Tuna-2 run.

The remaining flags are the usual generation controls: --ckpt and --prompt are required, --image is the source image and is required for the edit task, --resolution defaults to 512x512, --gpu selects the device index, --steps sets the number of diffusion steps with a default of 50, and --guidance and --negative come from the configuration, and --seed defaults to 42.

The resolution table separates two classes of output. The 512-class covers 512x512, 448x576, 576x448, 384x672 and 672x384, and the 1024-class covers 1024x1024, 896x1152, 1152x896, 768x1344 and 1344x768.

The examples show the two shapes of use. A Tuna-2 7B run at 512 pixels with a text prompt, and a Tuna 2B run on the VAE latent with a different prompt. The prompts themselves are unusually detailed, describing skin texture and pores in one case and a zero gravity scene in the other, which tells you the project expects long descriptive prompts rather than keyword strings. Sample prompts live in assets/prompts.txt.

Video: the code is there, the weights are not

The video situation is stated plainly and is worth reading carefully. Due to policy constraints, the video generation model is not being released. What is provided instead is the complete video training and inference codebase, described as a ready-to-use starting point for anyone who wants to train their own video model, with the training configuration at configs/train/video_t2v.yaml and the inference configuration at configs/predict/t2v_2b.yaml.

That distinction between releasing the code and withholding the weights runs through the whole project, and the README has a dedicated section on it. Due to organizational policy constraints, the full production-trained model weights are not being released. The plan is to release a foundation checkpoint with a small number of layers removed from both the LLM backbone and the diffusion head, which the README calls the flow head. The remaining layers and all other components, naming the vision encoder, projections and embeddings, are fully preserved.

The stated remedy is a short fine-tuning pass on your own data, after which the removed layers can be quickly re-learned and the model restored to full quality. Detailed fine-tuning instructions are in the training guide at tuna/README.md.

The TODO list records the same three gaps: release some of the Tuna-2 model weights, release some of the Tuna model weights, and release the fully restored model weights fine-tuned on external data to recover the missing layers. The README adds that the team is actively working on fine-tuning the removed layers using external data and plans to release the complete weights as soon as possible.

For a researcher this is workable, since a foundation checkpoint plus a fine-tuning pass is a normal research workflow. For anyone who wanted a finished checkpoint, it is a wait.

What the repository actually contains

The tree is larger than a research release usually is, and the top level tells you what kind of project this is. There is a tuna/ directory holding the model code, a configs/ directory, a scripts/ directory, a data_examples/ directory, an lmms-eval/ directory, and an assets/ folder for the teaser and sample images.

Two standalone files are worth naming. attention_visualization_simple.py sits at the top level, which suggests a tool for inspecting how attention is distributed in the model, and that is exactly the kind of script an architecture paper needs. The lmms-eval/ directory points at the evaluation stack, which matches the pyproject comment about compatibility with the tuna-lmms-eval stack, and it means the benchmark harness is vendored here rather than assumed to be installed from elsewhere.

The project uses hydra-core with omegaconf for configuration, which is why the inference flags that come from config, such as the guidance scale and the negative prompt, are described as coming from the configuration rather than from the command line. The dependency set also includes timm for vision components, torchdiffeq for the differential equation side of the diffusion head, safetensors and huggingface-hub for weight handling, accelerate, tensorboard, pillow, numpy and tqdm.

There is a requirements.txt alongside the pyproject, and it is looser, listing transformers at 4.52 or newer rather than the exact 4.52.4 pin. That is worth noticing, because it means the two install paths do not give you identical dependency sets, and the pyproject is the one the project explains in comments.

Other top level files include CODE_OF_CONDUCT.md, CONTRIBUTING.md, the LICENSE and a uv.lock, so the environment is intended to be reproducible.

Where Tuna-2 is the wrong tool

The first wrong case is anyone who needs to run the model today. The weights are not released, and the artefact that is planned is deliberately incomplete, with layers removed from the LLM backbone and the diffusion head. You can still train your own or fine-tune a foundation checkpoint, but there is no checkpoint to download and run, which makes this a research codebase rather than a product dependency.

The second is a team without the compute to fine-tune. The recovery path in the README is a short fine-tuning pass on your own data that restores full quality, and how short that is depends on hardware nobody has specified. Two of the model sizes are 7b and 2b, and the 2b is only offered for the vae variant, so the smallest configuration available in the inference table is not the smallest model in the project.

The third is a strict environment. The dependency pins are exact for transformers, diffusers, timm, einops, torchdiffeq and omegaconf, numpy is held below 2, and setuptools is capped below 81 for a specific reason. That is defensible for reproducibility, and it is friction for anyone whose own stack already sits on a different transformers or diffusers.

The fourth is a project that needs the video model today. The video code is complete and the video weights are not released at all, with no foundation checkpoint plan mentioned for video specifically. That is a clean split of code from artefacts, and it means the video path is a training project rather than an inference one.

Editorial conclusion

Adopt this codebase if you want to train or fine-tune a unified understanding and generation model with a single pixel embedding path, since the training, inference and configuration layout is complete and the pinned dependency versions make the stack reproducible. Do not plan a product on the released artefacts as they stand, because the model weights are not published and the intended release is a foundation checkpoint with layers removed from both the LLM backbone and the diffusion head. Before you start, read the note on model release in the README and the training guide at tuna/README.md, and check the pinned transformers 4.52.4 and diffusers 0.35.1 against your own environment, since the last push was on 2026-07-22 and there are no GitHub releases to fall back on.

Frequently asked questions

Official sources

1. facebookresearch/tuna-2 on GitHub
2. Issues
3. License: Apache-2.0
4. README