NInfer: a from-scratch C++/CUDA engine that runs Qwen3.5 on exactly one RTX 5090

Specialization as the design, not a limitation to apologize for

NInfer is a from-scratch C++/CUDA inference engine for Qwen3.5 Dense and MoE architectures on a single NVIDIA GeForce RTX 5090. The README does not hedge about the narrowness: one GPU, one resident model, and a startup-fixed capacity of one to eight active requests. It runs text, image and video prompts through a local CLI or OpenAI- and Anthropic-compatible HTTP APIs.

The user is someone who owns exactly this hardware and wants to extract maximum inference performance from it, rather than portability. That is an unusual thing to build for, and it is the whole argument. General engines abstract over many GPUs and models and pay for it in generality; NInfer targets one card and a handful of checkpoints and can therefore assume things a portable engine cannot.

Five official artifacts are provided, covering Qwen3.6-27B, Qwen3.8-27B and Qwen3.6-35B-A3B in `groupwise-int` and `nvfp4` weight formats, each downloadable from Hugging Face with a model card. The README's quick-start uses Qwen3.8-27B NVFP4. Each v3 `.ninfer` artifact carries model configuration, encoded weights, logical bindings and frontend resources in one file, and the current engine requires v3, with a documented local upgrade path for existing v2 downloads that avoids re-downloading weights.

A build that rejects any GPU architecture but sm_120a

The specialization shows in the build requirements, which are exact. The README lists 64-bit Linux, an RTX 5090, a CUDA toolkit supporting `sm_120a`, CMake 3.28 or newer, a C++20 host compiler, Ninja, `pkg-config`, FFmpeg development libraries and `libcurl >= 7.85`. CUDA 13.1 is the validated toolkit, and the build rejects CUDA architectures other than `sm_120a`.

That last point is worth dwelling on: the engine will not build for a different GPU generation, by design. This is not code that happens to run best on a 5090; it is code that refuses to target anything else. The FFmpeg dependency is there because image and video prompts are first-class, not an afterthought.

The build itself is a standard CMake and Ninja flow:

git clone https://github.com/Neroued/ninfer.git
cd ninfer
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Tests and benchmarks are excluded from the default build; the README notes `cmake --preset release` configures the same product build while `cmake --preset dev` adds tests and benchmarks. There is deliberately no install target and no packaged binary, you run NInfer from its source build tree, which fits a project that assumes you are a developer on the one supported machine.

Serving a model with explicit memory tiers

Running NInfer means first fetching an artifact, then starting a server. The README uses the Hugging Face CLI for the download:

hf download neroued/Qwen3.8-27B-nvfp4-NInfer \
  qwen3_8_27b_nvfp4.ninfer \
  --local-dir models

The server command exposes the memory model directly rather than hiding it:

./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4.ninfer \
  --max-context 240000 \
  --kv-capacity 240000 \
  --max-concurrency 2 \
  --kv-dtype fp8 \
  --device-state-slots 2 \
  --host-state-slots 8 \
  --host-kv-mib 8192 \
  --spec mtp --draft-tokens 3

The README explains what those numbers buy: each request has a 240,000-token logical ceiling, a shared Device KV pool serves admitted requests, and two run concurrently when their combined reservations fit. The cache tiers give two Device checkpoint slots, eight pinned Host State slots and 8 GiB of pinned Host KV. This is an engine that hands you the knobs a datacenter server usually hides, because the operator is assumed to be tuning one machine. Once serving, an OpenAI-style `POST` to `/v1/chat/completions` works as expected, and a one-shot CLI request runs through `./build/apps/ninfer` with its own `--max-context` and `--max-new` allocation.

Speculative decoding and where the diagnostics go

NInfer's performance features are exposed as flags rather than buried. The serve and CLI examples both enable `--spec mtp --draft-tokens 3` and `--lm-head-draft`, which turn on multi-token-prediction speculative decoding with a three-token draft, and `--preserve-thinking` on the server keeps reasoning content. `--kv-dtype fp8` sets the KV cache precision. These are the levers that make single-GPU serving fast, and the README puts them in the first example rather than a footnote.

The output discipline is unusually well specified. The README says answer content goes to stdout, while human-readable startup and runtime diagnostics and the reasoning, timing, throughput, memory and speculative-decoding report go to stderr, with reasoning and the result report kept as unprefixed product output. On a terminal, weight materialization shows one transient progress line followed by a compact Engine-ready message. That separation matters for anyone scripting against NInfer: the answer stream is clean on stdout, and the noise is on stderr where it belongs.

This attention to stream hygiene, and the explicit memory tiers, are signs of an engine written by someone who runs it seriously, not a demo. The tradeoff is that understanding the flags is required; there is no zero-config mode that guesses good defaults for you.

The obvious limitation: it runs on one specific card

The limitation is the premise. NInfer targets a single RTX 5090 and refuses to build for other CUDA architectures, so if you do not have that card, this engine is not for you, and no amount of configuration changes that. It also serves one resident model at a time with a startup-fixed concurrency between one and eight, so it is not a multi-tenant server that scales elastically; you choose the capacity at launch and live with it.

The model set is fixed too. Five official Qwen3.5 checkpoints are provided, and while the README documents converting your own weights and reusing a recipe, this is not a general engine you point at any Hugging Face model. It runs the architectures it implements, in the formats it supports.

There is also operational friction by design: no install target, no packaged binary, run from the build tree. For a personal high-performance setup that is fine, but it means NInfer is not something you deploy as a service artifact. It is a tool you build and run on your workstation, and the README's framing throughout, one GPU, selected checkpoints, is consistent about that being the intended shape rather than a temporary state.

Against a general engine like vLLM

The natural comparison is a general-purpose serving engine such as vLLM, which runs a wide range of models across many GPU types and scales to multiple cards and tenants. That breadth is exactly what NInfer gives up. vLLM will run on your hardware whatever it is, serve models NInfer never implemented, and grow across a cluster.

NInfer's counter-argument is depth on one target. By assuming a single RTX 5090 and a small set of Qwen3.5 checkpoints, it can compile for `sm_120a` specifically, expose hand-tuned memory tiers, and ship speculative decoding tuned to those models, aiming for maximum single-GPU throughput rather than portable adequacy. The cost is everything vLLM offers: no other GPUs, no arbitrary models, no elastic scaling, and a source-tree run model instead of a packaged server. Choose vLLM when you need to serve many models or scale across hardware. Choose NInfer when you have precisely the supported card, want one of the supported Qwen3.5 checkpoints running as fast as that card allows, and are willing to build from source and tune the flags yourself.

Apache-2.0, v3 artifacts, and what to check first

NInfer is Apache-2.0, so the engine can be forked, modified and used commercially with attribution, and it carries the patent grant. The weight artifacts live on Hugging Face with their own model cards, so a user should check each card for the terms attached to that specific checkpoint rather than assuming the engine license covers the weights.

The artifact versioning is a maintenance detail that will bite if ignored: the engine requires v3 `.ninfer` artifacts, and existing official v2 downloads must be upgraded locally, which the README says can be done without re-downloading the weights. Anyone returning to NInfer after a gap should expect to run that upgrade before an old artifact loads.

The concrete first check is hardware and toolkit, because everything else is downstream of it. Confirm you have an RTX 5090 and a CUDA toolkit supporting `sm_120a`, ideally CUDA 13.1, since the build will reject other architectures outright, then do a `cmake --preset release` build and download the Qwen3.8-27B NVFP4 artifact the quick-start uses before trying your own weights. If the build rejects your CUDA architecture, that is the engine working as documented, not a bug to work around.

Editorial conclusion

NInfer is for one specific reader: someone with an NVIDIA RTX 5090 who wants a supported Qwen3.5 checkpoint running at maximum single-GPU speed and is comfortable building from source and tuning memory and speculative-decoding flags by hand. It is the wrong tool for anyone without that exact card, since the build rejects CUDA architectures other than sm_120a, for serving many models, since one model is resident with a startup-fixed one-to-eight concurrency, or for anyone wanting a packaged deployable, since there is no install target. Before anything else, confirm your GPU and a CUDA 13.1 toolkit supporting sm_120a, build with cmake --preset release, and load the Qwen3.8-27B NVFP4 artifact the quick-start uses to verify the path end to end.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. Neroued/ninfer on GitHub
4. README