Crane: a Rust and Candle inference engine for LLM, VLM, TTS and OCR workloads

What Crane is trying to replace, and for whom

The README positions Crane as an alternative to llama.cpp with the pitch that it is "much more clean and simpler" while compiling to a binary the same way. The stated goal is speed on both CPU and GPU through Candle kernels, with one codebase covering CPU, CUDA and Metal. The intended user is a Rust developer building an application that needs several AI abilities, not just text generation. The README lists chat, VLM chat, OCR through a VLM, VLA, TTS, ASR and VAD as abilities you can call without writing low-level inference code, and states plainly that Crane is not a low-level SDK. That is a different audience from the one that reaches for Candle directly to build a model implementation. Crane is for someone who wants the model already wired up and reachable over an API. The MIT license removes the redistribution question that a copyleft inference stack would raise for a closed product, though the licence of each downloaded model weight is a separate matter the README does not discuss.

The Candle dependency and what the kernels actually do

Crane's stated dependency policy is that Candle is the only dependency, and the engine layers its own kernels on top. The changelog is where the mechanism becomes concrete. On 2026.08.02 the project added a CRANE_PROF=1 forward-pass profiler that measures kernel submission time against wall-clock time after a device sync, a distinction the changelog notes rocm-smi's busy counter cannot make. That profiler found the CPU spending 21.8 ms of a 26.9 ms token merely enqueueing roughly 2000 kernel launches. The fix was fusing Qwen35RmsNorm and the GDN gated norm into single rms_norm launches and rewriting the Q/K L2 norm as one, using the identity x/sqrt(sum(x^2)+eps) = rms_norm(x, 1/sqrt(K), eps/K). Hoisting -exp(A_log) and dt_bias to load time brought submission down to 6.9 ms. The reported effect on an RX 7800 XT with Qwen3.5-2B-Q8_0 was decode rising from 35.7 to 63.0 tokens per second at depth 0 and from 28.8 to 49.2 at depth 4096, with the gap to llama.cpp narrowing from 3.0x to about 1.7x. These are the project's own numbers from its own changelog, not an independent measurement, and the hardware is a single AMD card.

GGUF loading and in-situ quantization

Crane reads community GGUF files directly with --model-path model.gguf, following the llama.cpp qwen35 layout including the hybrid GDN blocks. The architecture is auto-detected from the header, and the tokenizer and chat template are read from GGUF metadata, so no sibling files are required. Alternatively, a safetensors checkpoint can be quantized at load time with --quant q4k|q8_0 or the CRANE_ISQ environment variable, described as in-situ quantization through candle's QMatMul. The 2026.08.16 entry records a memory fix in this path: GGUF embedding tables now stay quantized and dequantize only the rows a forward pass gathers, instead of expanding all 248320 rows at load. The changelog claims 1772 MiB saved on Qwen 3.8-27B Q4_K_M, peak memory moving from 22007 to 20235 MiB, and describes the result as bit-exact on untied checkpoints with prefill logits cosine of 1.000000000. It also states that Qwen 3.8-27B Q4_K_M runs text-only on a single 24 GB RTX 3090. Those are README claims; nothing in the supplied material shows an external reproduction.

Serving: the OpenAI-compatible surface and thinking control

Crane ships a server, crane-serve, that the README describes as exposing OpenAI and SGLang interfaces. TTS is wired into /v1/audio/speech. Thinking control is passed per request through chat_template_kwargs with enable_thinking and reasoning_effort set to low, medium or xhigh, or through OpenAI's top-level reasoning_effort field. The scratchpad is separated out of content into reasoning_content, including during streaming. That separation matters for any client that renders content directly: a model that emits a long reasoning block will not pollute the visible answer, but a client that ignores reasoning_content will simply lose it. The README gives no example request body, no port number, and no startup command for crane-serve, so the exact invocation is not verifiable from the supplied material. If you are evaluating this for a gateway, that gap is the first thing to close.

Where the model matrix is thinner than it looks

The supported-model list mixes checked and unchecked entries in the same block. PaddleOCR-v6, Qwen 3.6 and 3.8, Qwen 3.5, Hunyuan Dense, Gemma 4, Qwen3 VL, PaddleOCR VL, Qwen3, Qwen 2.5, Moonshine ASR, Silero VAD, Qwen3-TTS, Voxtral-4B-TTS and MuScriptor are marked done. Qwen3.5-VLA, Qwen3.5-GR00T, Pi0.5 and Audio8-TTS are unchecked, and the README says VLA is "on the way" in one place while listing it as a future item in another. The Spark-TTS and Orpheus-TTS line is struck through and marked WIP. Kokoro-82M TTS is described as English only. Gemma 4 is text and vision with no audio. So the honest reading is that text, vision, OCR, ASR and several TTS models are implemented, while VLA and the audio8 TTS topic in the repository metadata are not. The repository topics list audio8-tts even though the model is unchecked, which is a small example of the README running ahead of the code.

Backend coverage and the ROCm caveat

The README claims a unified codebase for CPU, CUDA and Metal, and notes Metal acceleration on Apple Silicon. The 2026.08.01 entry adds ROCm: Crane's own kernels/cuda/*.cu files now compile with hipcc on first use and cache the code object, so the fused GDN recurrence, GPU top-k sampling and fused_silu_mul are no longer CUDA-only. The same entry says the Qwen 3.5 GGUF loader stopped forcing F32 side tensors on ROCm. Reported RX 7800 XT numbers for Qwen3.5-2B-Q8_0: prefill 183 to about 1600 tokens per second, decode at depth 2048 from 15.4 to 30.6, and peak VRAM on a 3800-token prompt falling from 99 percent to 69 percent of 16 GB. The first-use compile-and-cache step is worth noting as an operational cost: the first ROCm run pays a compilation that later runs do not. If your deployment needs a cold start with a predictable latency budget, that first run is a different animal from the steady state.

How this differs from llama.cpp and from raw Candle

Against llama.cpp, the difference is the scope of the runtime rather than the quantized text path. llama.cpp centers on GGUF text generation with a broad model zoo and a long release history. Crane centers on one Rust binary that also carries vision, OCR, ASR, VAD and TTS, and it reads llama.cpp's GGUF format rather than inventing one. The changelog's own framing puts Crane at roughly 1.7x behind llama.cpp on decode for one AMD configuration, so the case for Crane is not raw text throughput. Against using Candle directly, the difference is that Candle gives you tensor operations and model-building primitives while Crane gives you the assembled model, a tokenizer path, a chat template and an HTTP server. The cost of that convenience is that you inherit Crane's model list and its release cadence. If your model is not on the list, Candle is the lower-level tool and Crane is not a substitute for it.

Release artifacts, maintenance and what to check before adopting

The repository has no retrieved releases, so there is no tagged version to pin and no published changelog outside the README. The last push date is 2026-09-09. The changelog entries are dated and detailed, which suggests active work, but dated README entries are not the same as a released artifact with a version number and a checksum. The MIT license covers the code; it does not cover the model weights you download, and the README does not address weight licensing at all. Upgrade cost is hard to estimate from the supplied material because there is no version boundary to diff against: a user tracking main gets the kernel rewrites, the GGUF loader changes and the new model support as they land, which is fine for a lab and awkward for a product with a support matrix. The concrete first step is to check whether a tag exists at all, then confirm that your target model is a checked entry in the supported list rather than an unchecked box, then run the model on your own hardware rather than relying on the RX 7800 XT and RTX 3090 figures in the changelog.

Editorial conclusion

Adopt Crane if you want a single Rust binary that serves Qwen-family chat, vision, OCR, ASR and TTS behind an OpenAI-compatible HTTP surface and you are willing to read the changelog rather than the model checklist. Do not adopt it if you need a stable release artifact, a published benchmark you can reproduce independently, or a model family outside the Qwen, Gemma, Hunyuan and PaddleOCR set. Before committing, verify three things yourself: whether a tagged release exists, whether your target model appears as a checked entry rather than an unchecked box, and whether your GPU backend is one the changelog actually names.

Official sources

1. Issues
2. License: MIT
3. lucasjinreal/Crane on GitHub
4. README