wllama: llama.cpp Compiled to WebAssembly for In-Browser Inference

The problem wllama solves: inference without a server

Most LLM integration paths assume a server. You run llama.cpp, vLLM or an API behind an endpoint, and the browser is a thin client. wllama removes that endpoint. It compiles llama.cpp to WebAssembly and exposes it as a TypeScript class, so the model file is fetched by the browser, the weights sit in the tab's memory, and tokens are generated on the user's machine. The README states the pitch directly: inference runs on the browser using WebAssembly SIMD, with no backend or GPU required. The package declares no runtime dependency, which matters if you are embedding this in a frontend bundle that already has enough to carry.

The audience is narrow but real. It suits developers building offline-capable tools, demos, or privacy-sensitive features where prompts and outputs should not leave the device. It also suits anyone who wants to avoid per-token API costs for small models. The demo on Hugging Face Spaces and the four example pages (basic, embeddings, multimodal, tools) are the intended entry points, and they map onto the four things the project claims to support. If your use case is none of those four, the examples will not tell you much.

Worker-based execution and the OpenAI-shaped API

The architecture is a worker plus a typed facade. The README lists as a feature that inference is done inside a worker and does not block UI render, which is the single most important design decision in the project. Long generation loops in the main thread would freeze the page, so the wasm module and the model buffers live off the main thread and the Wllama class communicates with it.

On top of that sits an API that mirrors OpenAI's shape. createChatCompletion takes a messages array with role and content fields, plus max_tokens, temperature, top_k and top_p, and returns an object where the text is at response.choices[0].message.content. That is a deliberate compatibility choice: code written against the OpenAI SDK can often be adapted by swapping the client. The README describes it as a fully-typed OpenAI-compatible API, and there are separate examples for completions, embeddings, multimodal and tool calling, so the surface is broader than chat alone.

One more mechanism worth noting is the automatic thread selection. The project ships single-thread and multi-thread builds and switches between them based on browser support. You do not choose at build time, and you can force single-thread by passing n_threads: 1 in the load config if you want deterministic behaviour.

Installing and loading a model with loadModelFromHF

The published package is @wllama/wllama on npm. The README gives npm i @wllama/wllama as the install command. If you want to build the wasm binaries yourself, they are not pre-built in the repository: you need Docker, then npm ci followed by npm run build:wasm && npm run build, and the README suggests cloning the repo as a git submodule for that path.

At runtime you construct a Wllama instance with a map of wasm paths, then load a model. The README's basic example uses loadModelFromHF with a repo and file pair, pointing at ggml-org/models and tinyllamas/stories260K.gguf, and passes a progressCallback that receives loaded and total byte counts so you can render a download bar. An alternative loadModelFromUrl exists for models that are not on the Hugging Face hub. There is also a wasm-from-cdn entry point, imported as @wllama/wllama/esm/wasm-from-cdn.js, but the README explicitly says this is not recommended and should only be used when you cannot embed the wasm files in your project.

For GPU offload, V3.1 enables WebGPU automatically and by default sends all layers to the GPU. The n_gpu_layers parameter on LoadModelParams controls this, and the README's example sets it to 4 with a comment that 0 disables GPU inference. There is also a compat mode, wllama.setCompat('default', 'firefox_safari'), which the README says allows WebGPU to run on Firefox at significantly degraded performance.

The 2GB ceiling and why you split models with llama-gguf-split

The hard constraint is file size. The README states the maximum file size is 2GB because of the ArrayBuffer length restriction, and points to a Stack Overflow thread for the underlying detail. This is not a wllama bug; it is a boundary of the platform the project builds on. If your GGUF exceeds 2GB, you must split it. The README's split section gives the command: ./llama-gguf-split --split-max-size 512M ./my_model.gguf ./my_model, using a pre-built binary from the llama.cpp releases page.

Splitting is also recommended below that threshold. The README advises chunks of maximum 512MB, on two grounds: multiple splits can be downloaded in parallel, which it says results in slightly faster download speed, and it prevents some out-of-memory issues. Those are the project's own claims, not measured numbers, and the README does not quantify either benefit. Treat 512MB as a starting point and test against your target devices.

Quantization guidance is equally blunt. The README recommends Q4, Q5 or Q6 as the balance among performance, file size and quality, and explicitly says IQ with imatrix is not recommended because it may result in slow inference and low quality. That is a useful piece of opinionated advice, since IQ quants are often chosen for size reasons and wllama is telling you the trade does not pay off here.

COOP and COEP: the deployment requirement that breaks multi-threading

Multi-threaded WebAssembly needs SharedArrayBuffer, and SharedArrayBuffer needs cross-origin isolation. The README states this plainly: to enable multi-thread, you must add Cross-Origin-Embedder-Policy and Cross-Origin-Opener-Policy headers, and links to a ffmpeg.wasm discussion for background. This is the failure mode most likely to bite in production. Because wllama auto-switches between builds based on browser support, a deployment that forgets those headers will not error out; it will quietly run the single-thread build. Your application works, and it is slower than the developer expected, with no exception to point at the cause.

Setting those headers is not free either. Cross-origin isolation changes how the page can embed third-party resources, so an app that relies on cross-origin iframes, certain CDN scripts or third-party embeds may find them blocked once isolation is on. The README does not discuss this consequence, and it is the kind of thing worth checking before you commit to the multi-thread path. If you cannot set headers at all, plan for single-thread and pass n_threads: 1 so the behaviour is explicit rather than incidental.

The WebGPU path has its own caveat. The README's compat mode for Firefox and Safari is described as significantly degraded in performance, which reads as a way to make the feature reachable rather than usable. If your audience is on Firefox, test that mode before assuming GPU offload helps.

Where wllama is the wrong tool, and what to use instead

wllama is the wrong tool when the model cannot be squeezed under the size budget, when you need throughput beyond what a browser tab can deliver, or when you need a shared cache across users. Every visitor downloads the weights independently. There is no server-side KV cache, no batching across requests, and no way to share one loaded model between sessions. For a public site with many concurrent users, that is a lot of duplicated bandwidth and duplicated compute.

The natural alternative is llama.cpp's own server mode, which the wllama README links to as the upstream project. The difference in approach is architectural rather than a matter of features: llama.cpp server runs the model once on a host with full access to system RAM and the GPU, and clients send HTTP requests to it. Memory is bounded by the host, not by the ArrayBuffer limit, so the 2GB per-file ceiling and the split-model workflow disappear. Concurrency is handled server-side. The cost is that you need a host, you pay for it, and prompts leave the user's device, which is exactly what wllama avoids.

A second reference point is the compat package. The README points to @wllama/wllama-compat for compatibility issues, which suggests the main package tracks a moving upstream and that older integrations may need the compat layer rather than the current one. If you are pinning an older API, that is the path to check first.

Release cadence, licence and what to verify before adopting

The release history shows a steady cadence: 3.5.1 in June 2026, 3.6.0 in August, 3.6.1 later in August, with the last push to master in early September 2026. The V3 line is a step change rather than a patch, adding WebGPU, multimodal input and tool calling together. The README directs readers to a V3 release guide and notes that WebGPU is enabled automatically from V3.1 onward. If you adopted before V3, upgrading is not a drop-in event: GPU offload becomes the default, which changes memory behaviour on machines with limited VRAM, and n_gpu_layers becomes a parameter you may need to tune.

The licence is MIT. That is permissive and imposes no copyleft obligation on your application, but it governs wllama's code, not the models you load or the llama.cpp components compiled into the wasm binary. Checking the licensing of your chosen GGUF and of the upstream project is your responsibility; this is not legal advice.

What to verify first is concrete. Load your actual production GGUF through loadModelFromHF with a progressCallback and watch the byte counts, because that tells you the real download size your users will absorb. Confirm your host can emit Cross-Origin-Embedder-Policy and Cross-Origin-Opener-Policy and that enabling them does not break third-party embeds already on the page. Then set n_gpu_layers deliberately rather than relying on the all-layers default, and test on the lowest-VRAM machine you intend to support. If any of those three checks fails, the llama.cpp server route is the honest fallback.

Editorial conclusion

Adopt wllama when the model can be quantized to Q4 to Q6, stays under the 2GB per-file ceiling or is split with llama-gguf-split, and the deployment target lets you set Cross-Origin-Embedder-Policy and Cross-Origin-Opener-Policy. Do not adopt it if you need models that cannot fit that budget, or if you cannot control response headers, because the multi-thread build will silently fall back to single-thread. Before committing, load your real GGUF through loadModelFromHF with a progressCallback and measure the download and first-token time on a mid-range laptop, then check whether n_gpu_layers needs to be lowered for your VRAM.

Official sources

1. License: MIT
2. ngxson/wllama on GitHub
3. Project website
4. README
5. Releases