DreamLite: one 0.39B diffusion network for on-device image generation and editing

One network for generation and editing, sized at 0.39B parameters

DreamLite is described as a compact and unified on-device diffusion model with 0.39B parameters that supports both text-to-image generation and text-guided image editing inside a single network architecture. The paper title carries the same claim, a lightweight on-device unified model for image generation and editing, and the repository is the official implementation for an ECCV 2026 paper.

Unifying those two tasks is the interesting part of the design, because they usually arrive as separate models with separate weights and separate interfaces. Here they share a backbone and a conditioning path, so an application that wants both a generate button and an edit button loads one pipeline.

The parameter count is the headline constraint. At 0.39B the model is small enough to think about for a phone, and the demo claims it generates or edits a 1024 by 1024 image in about three seconds on an iPhone 17 Pro, with a 4-bit Qwen-VL text encoder and fp16 VAE and U-Net. It runs fully on device with no cloud dependency, which is the property that makes it different from most hosted generation APIs.

The README also mentions a project page with more visualisations, an arXiv paper, and a Hugging Face Spaces demo, so the artefacts around the model are fairly complete for a research release.

In-context spatial concatenation replaces a separate editing branch

The architecture is built on a pruned mobile U-Net backbone, and multimodal conditioning is unified through what the project calls in-context spatial concatenation, applied directly in the latent space.

The mechanism matters because conditioning in a latent diffusion model usually has a shape. Text conditioning arrives as embeddings that are injected into the network, while an image for editing has to enter somewhere. Doing the join in latent space through spatial concatenation means the model sees the conditioning as extra spatial content in the working representation, so the same network can be handed a text prompt for generation and a prompt plus an image for editing without a separate encoder branch or a second set of weights.

Two other design choices support the speed claim. The backbone is pruned, which reduces the work in the U-Net itself, and the model uses progressive step distillation to reach four-step inference. The variant naming reflects the split in emphasis: DreamLite-base is described as 28-step for high fidelity, and DreamLite-mobile as four-step for speed.

Distillation is the trade-off worth understanding. A distilled few-step sampler removes the iterative denoising schedule that gives diffusion models their quality, and recovers it with a model trained to imitate the full trajectory. The result is fast but not equivalent to the multi-step path, which is why the project ships both a high fidelity base model and an ultra fast mobile model rather than one compromise.

The weights are gated, and the access token is the first hurdle

Model weights are hosted on the diffusers branch of two Hugging Face repositories, carlofkl/DreamLite-base for the 28-step high fidelity variant and carlofkl/DreamLite-mobile for the four-step ultra fast one. Access is currently gated, and the project asks you to request access through an access request form first. Once approved you receive a Hugging Face access token, which you then use to log in locally.

huggingface-cli login   # paste the token we sent you when prompted
# or non-interactively:
huggingface-cli login --token <TOKEN>

The same token works for the download CLI, which lets you fetch the weights ahead of time into the directory layout the project expects.

hf download carlofkl/DreamLite-base   --revision diffusers --local-dir models/DreamLite-base   --token <TOKEN>
hf download carlofkl/DreamLite-mobile --revision diffusers --local-dir models/DreamLite-mobile --token <TOKEN>

The local layout is a models directory holding the two variant folders:

DreamLite/
├── models/
│   ├── DreamLite-base/
│   └── DreamLite-mobile/

With the weights present locally, the loading call takes a local path instead of a repository id. A pipeline call of DreamLitePipeline.from_pretrained on the models/DreamLite-base path with a torch_dtype argument does the job, and the diffusers branch revision is what makes from_pretrained fetch the right files on first use.

The gating is a real operational constraint for CI, since a build pipeline needs a token configured as a secret, and it also means the model cannot be evaluated by simply cloning the repository.

Diffusers support sits on the main branch, not in a stable release

The library integration is the second friction point. DreamLite has been merged into the official Diffusers library, which the project describes as pending the next release. The README is explicit that because the change is not yet in a stable Diffusers release, you should install the latest main branch directly from source.

pip install git+https://github.com/huggingface/diffusers.git

That instruction has consequences beyond a single pip command. A source install of the main branch means your environment tracks an unreleased state of Diffusers, so a change upstream can alter behaviour without a version bump on your side. In a production environment this is usually the part that needs a pin.

The repository itself is a separate matter. It contains infer.py and infer_mobile.py for command line use, a dreamlite/ package, a lora/ directory, a tools/ directory and a deploy/ directory, which is consistent with a release that supports both a scripting path and a deployment path. The requirements file lists torch, torchvision, numpy, Pillow, transformers pinned to 4.57.3, diffusers, accelerate, gradio, safetensors, datasets, peft, tqdm and einops, and the presence of gradio and peft points at a demo interface and at LoRA support in the toolchain.

The top-level LICENSE is Apache-2.0, and there is a separate WEIGHTS_LICENSE file for the model weights, so the code and the weights carry different terms and both are worth reading before shipping.

A few lines of Python cover both text-to-image and instruction editing

Once the environment and the token are in place, the Python path is short. The pipeline class is imported from diffusers, images are loaded with the load_image helper from diffusers.utils, and the same pipe object handles both tasks.

import torch
from diffusers import DreamLitePipeline
from diffusers.utils import load_image

model_id = "carlofkl/DreamLite-base"
device = "cuda"
dtype = torch.float16

pipe = DreamLitePipeline.from_pretrained(model_id, revision="diffusers", torch_dtype=dtype)
pipe.to(device=device)

Text to image is a single call with a prompt and a seeded generator, and the returned image list is written to disk. Image editing reuses the same pipe and adds an image argument, with the prompt phrased as an instruction such as making it snowy. The seed is set through torch.Generator with a manual_seed value, which matters for reproducibility when you are comparing output quality across settings.

The demo material in the README shows three editing categories on the phone: a human portrait with style transfer, a nature scene with a background swap, and a product with object replacement. Those three categories map onto what the conditioning mechanism is meant to handle, so they are a fair guide to the model's range.

One detail in the sample deserves attention. The edited example loads its input image from a URL rather than from disk, which means the sample as written expects network access at that point. If your deployment is fully offline, the image should be loaded locally instead, which the load_image helper can do from a file path as well.

Inference code is released, weights are gated, and no release is tagged

The news entries in the project notes tell the release story in order. DreamLite was publicly announced in 2026.03 together with the project page and the arXiv paper. The inference code was officially released in 2026.04. The Diffusers merge happened in 2026.06, and in the same month a community contribution, dreamlite-comfyui-lowvram, added ComfyUI custom nodes and a low-VRAM pipeline that runs the base model at 1024 by 1024 on 4 GB GPUs.

That last item is worth a note. A third-party project reached a configuration the official repository does not advertise, which tells you the base model is not restricted to phones and that a desktop GPU with 4 GB is a viable target once someone else does the memory work.

There are no GitHub releases in the repository, and the last push was on 2026-08-08. The code is therefore something you track at the commit level, and the checkpoints come from the Hugging Face repositories rather than from a tagged artefact.

For an on-device model the deployment story is separate from the training story, and the deploy/ directory in the repository tree points at the mobile path without describing its contents in the visible documentation. Anyone shipping this to devices will need to convert the weights into whatever format their app loads, and the README does not document that step, so it is the first thing to look for in the repository and in the project page.

Where the mobile claim holds and where it does not

The on-device numbers are specific and should be read narrowly. Four-step inference, 1024 by 1024 output, about three seconds on an iPhone 17 Pro, with a 4-bit Qwen-VL text encoder and fp16 VAE and U-Net. That is the configuration the README describes, and the on-device demo is presented as real-time generation and editing with no internet connection and no cloud processing.

Three limits are visible without testing anything.

The first is fidelity. Four-step distillation is what makes the speed possible, and the base variant at 28 steps exists precisely because the fast one gives something up. A project that needs maximum quality per image should use the base model and accept the step count, which is a different product from the mobile one.

The second is hardware coverage. The on-device claim is tied to one phone generation. A different mobile SoC, a different memory budget or a thermal limit will change the picture, and the README gives no data beyond that device. The community low-VRAM work for 4 GB GPUs is a sign that memory, not raw compute, is often the binding constraint.

The third is the packaging path. Nothing in the visible documentation describes converting the checkpoint for an app store build, choosing the on-device runtime, or handling the 4-bit text encoder inside a mobile app. The model is small enough to be interesting and young enough that those details are still being worked out by the community.

Editorial conclusion

DreamLite is worth evaluating if you are shipping an image feature to phones or edge devices, because one network handling both generation and editing changes the storage and integration story in a way a separate editing model does not. It is the wrong tool for a server-side pipeline that can afford larger models, since the design goal is a small mobile footprint rather than maximum fidelity. Before you build on it, verify three things yourself: that you can obtain the gated weights and a Hugging Face token, that your Diffusers installation can come from the main branch, because the merge is not yet in a stable release, and that the four-step mobile variant meets your quality bar, since the base model is documented as 28 steps for high fidelity and the two are not interchangeable.

Frequently asked questions

Official sources

1. ByteVisionLab/DreamLite on GitHub
2. Issues
3. License: Apache-2.0
4. README