OnnxStream: running Stable Diffusion XL in under 300MB of RAM
Lightweight inference library for ONNX files, written in C++. It can run Stable Diffusion XL 1.0 on a RPI Zero 2 (or in 298MB of RAM) but also Mistral 7B on desktops and servers. ARM, x86, WASM, RISC-V supported. Accelerated by XNNPACK. Python, C# and JS(WASM) bindings available.
At a glance
- What is it?
- OnnxStream is a C++ ONNX inference library built around one constraint: memory, not latency. The README claims SDXL 1.0 runs in less than 300MB of RAM on a Raspberry Pi Zero 2, at the cost of hours per image. Here is what that trade buys, and what it costs.
- Who is it for?
- Adopt OnnxStream if your binding constraint is memory rather than time: a Pi Zero 2 with 512MB of RAM, a browser tab via the WASM builds, or a machine where you cannot afford the 12GB of VRAM that SDXL normally wants.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 89 days ago.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The constraint OnnxStream was written against
Most ONNX runtimes optimize for latency or throughput, and spend RAM to do it. OnnxStream inverts that. The README states the original challenge plainly: run Stable Diffusion 1.5, a model with almost 1 billion parameters, on a Raspberry Pi Zero 2 with 512MB of RAM, without adding swap and without writing intermediate results to disk. The same document notes the commonly recommended minimum for SD 1.5 is 8GB of RAM or VRAM. That gap, roughly an order of magnitude, is the entire product thesis. The audience follows from it: people deploying to single-board computers, browser tabs, or any environment where the model weights are larger than the memory budget. The README claims SDXL 1.0 runs in less than 300MB of RAM, and the repository description mentions 298MB. Both numbers describe the same design point, and neither is a throughput claim.
WeightsProvider: the one abstraction that matters
The architecture hinges on decoupling the inference engine from whatever supplies the model weights. That supplier is a class derived from WeightsProvider, and the README describes it as able to implement any loading, caching and prefetching strategy. The consequence is concrete: a custom provider can pull parameters from an HTTP server and never touch disk, which is where the Stream in the name comes from. Three defaults ship with the library: DiskNoCache, DiskPrefetch and Ram. Those three names sketch the trade space without further documentation. DiskNoCache reads weights as needed and keeps nothing, so peak memory stays low and every tensor is re-read. DiskPrefetch reads ahead to hide some of that cost. Ram holds weights in memory, which is the mode you would use when memory is not the constraint and you want the latency back. The library is accelerated by XNNPACK, and the README lists ARM, x86, WASM and RISC-V as supported targets. Python, C# and JS bindings were added later, in September 2025 and via the WASM work before that.
Quantization is not uniform across the pipeline
Reading the SD 1.5 section carefully, the VAE decoder is the problem child. It is the only model in SD 1.5 that did not fit into the Pi Zero 2's RAM in single or half precision, and the README attributes this to residual connections plus very large tensors and convolutions. The fix was static quantization to 8-bit, and the README shows three output images at W16A16, W8A32 and W8A8 to let you judge the visual cost yourself. The W8A8 image is the one generated on the Pi Zero 2, in about 1.5 hours with the MAX_SPEED compile option (an earlier figure of 3 hours is struck through in the README). For SDXL the story changes shape. The README says UINT8 dynamic quantization is applied to the UNET but limited to a specific subset of large intermediate tensors, and that SDXL's VAE decoder is 4x the size of SD 1.5's, consuming 4.4GB in FP32 under OnnxStream. The README's SDXL VAE decoder paragraph then cuts off mid-sentence in the supplied material, so the exact resolution for that component cannot be confirmed here. Treat that as an open question to check in the repository, not as a solved detail.
Building and running the Stable Diffusion example
The README indexes a build section covering Linux, Mac, Windows, Termux and FreeBSD, and a separate page for converting a custom SD 1.5 model. The build produces the Stable Diffusion example implementation included in the repo, and MAX_SPEED is named as a compile option that changed a Pi Zero 2 generation from roughly 3 hours to roughly 1.5 hours. That is the only build flag the supplied material names, and it is worth understanding before you compile: the default build is not the fast one. Model files come from outside the repository. The README notes the SDXL ONNX files were exported from Hugging Face's Diffusers library, version 0.19.3, and links a contributor-written guide for converting SD 1.5. The LLM chat application covering TinyLlama 1.1B and Mistral 7B lives in assets/LLM.md, and the WASM demos live under examples/Whisper_wasm and at the YOLOv8 demo site. There is no package manager install path described in the material, so plan on a source build.
Where the design breaks down
The README is unusually honest about the cost. OnnxStream can consume up to 55x less memory than OnnxRuntime, with a 50% to 200% increase in latency, measured on CPU with a good SSD against SD 1.5's UNET. That 50% floor is the best case, and it assumes an SSD. Put the same weights on a slow SD card, which is exactly what a Pi Zero 2 has, and the DiskNoCache and DiskPrefetch providers are reading from the slowest storage in the system. The README's own SDXL figure makes the point: about 11 hours for a 10-step image on a Pi Zero 2, against 26 minutes for the same 10-step image on a 12-core PC with 32GB of RAM running Diffusers. OnnxStream is the wrong tool for anything interactive, for batch generation on a deadline, or for a workload where the model already fits comfortably in memory. If you have 12GB of VRAM, the memory-saving machinery is pure overhead.
What you give up against ONNX Runtime
ONNX Runtime is the obvious comparison and the README makes it directly, quoting the 55x memory figure against it. The difference in approach is not a feature gap, it is a different objective function. ONNX Runtime assumes the model and its activations fit in RAM or VRAM and optimizes execution from there, which is why it is the default choice for server-side inference and why it is the thing you reach for when latency matters. OnnxStream assumes they do not fit and pays in I/O and recomputation to keep peak memory down, with the WeightsProvider abstraction as the knob that lets you choose how much to pay. Neither is a superset of the other. If your deployment target is a normal server, ONNX Runtime will be faster and better supported. If your target is a Pi Zero 2 or a browser tab, ONNX Runtime's memory model is the thing standing in your way, and that is the gap OnnxStream was written to fill.
Maintenance, licensing and what to check before you build
The repository is not archived, the default branch is master, and the last push recorded is June 2026. Releases are sparse: v0.2 in January 2024 added the LLM chat app with optional cuBLAS support, and v0.1 in July 2023 shipped the win-x64 build with weights. The bindings and WASM work arrived as commits rather than releases, so if you pin to a release tag you will not get the Python, C# or JS bindings. That is a real upgrade-cost consideration: the library's useful surface has grown between releases, and tracking master is the only way to get all of it. On licensing, the GitHub page reports NOASSERTION, which means the platform could not classify the licence automatically. The supplied material does not name the licence text, so the terms you are actually agreeing to cannot be determined from what is here. Read the LICENSE file in the repository before you ship anything, and note separately that model weights such as SDXL, Mistral and Whisper carry their own licences that the library's terms do not cover.
Editorial conclusion
Adopt OnnxStream if your binding constraint is memory rather than time: a Pi Zero 2 with 512MB of RAM, a browser tab via the WASM builds, or a machine where you cannot afford the 12GB of VRAM that SDXL normally wants. Do not adopt it if you need interactive image generation, because the README itself puts a 10-step SDXL image at about 11 hours on a Pi Zero 2, and the memory saving is quoted as a 50% to 200% latency increase over OnnxRuntime even on a desktop CPU with a good SSD. Before committing, verify two things in the repository: whether the SDXL VAE decoder section of the README resolves the quantization path it cuts off mid-sentence, and which licence the NOASSERTION tag on the GitHub page actually corresponds to in the LICENSE file, since the API surface you will build against is small but the weights you stream are not covered by the library's terms.
Community notes