Qwen3.8-27B on one RTX 3090 with vLLM: what syv-ai's serving stack actually buys you
Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok/s at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok/s single-user at default sampling / ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks
At a glance
- What is it?
- A patched vLLM 0.28.0 image, requantization scripts and a benchmark harness for running a 27B model on 24 GB of consumer VRAM. The throughput numbers come with a harness you can re-run, and the trade-offs are documented as trade-offs.
- Who is it for?
- Adopt it if you have exactly one RTX 3090, you want Qwen3.8-27B served through an OpenAI-compatible endpoint, and you are willing to let a prebuilt 9.5 GB image apply its own vLLM patches. Do not adopt it if you need multi-GPU tensor parallelism as a first-class path, or if you cannot accept that the quality tables are currently marked as reference baselines while the v0.28.0 GPU matrix is re-measured.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 16, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The 24 GB problem this repository is built around
A 27B model does not fit comfortably on a 24 GB card once you account for KV cache, activations and the CUDA context. The usual answers are to rent a bigger GPU, to quantize until quality visibly drops, or to accept a short context window. This project takes the third constraint seriously and attacks it with a specific combination: int8 tensor-core GEMMs, a 16-bit recurrent state, a calibrated int4 lm_head, and speculative decoding that drafts cheaply enough to pay for itself.
The stated target is a single RTX 3090 at a 250 W power limit, serving an OpenAI-compatible API with key authentication, at either 150k or up to 262k context depending on mode. Two profiles ship: single-user for one or a few people chatting, and batch for API backends and pipelines with many concurrent requests. The README is explicit that one GPU serves one profile at a time.
Who is this for? Someone who already owns the card. The repository is not trying to convince you that a 3090 is a data-center accelerator. It is trying to extract the most tokens per second from the hardware you have, and it publishes the harness so you can check whether it succeeded on your machine.
Speculation, split-KV verification and where the speed actually comes from
The default single-user path uses MTP speculation with four cheap drafts. The faster path, SPEC=dflash2, swaps Qwen's MTP head for a DFlash2 block drafter that proposes seven tokens in one pass instead of four chained ones. DFLASH_TOKENS=15 then lets the target verify sixteen tokens per step: the drafter still proposes the seven it was trained for, and the remaining positions are filled from the request's own context. That last detail is the interesting design choice. Drafting from the prompt costs nothing and is exactly right whenever the answer quotes the prompt, which is why the README reports 381 tok/s at 25k context on a document-reproduction workload versus 121 tok/s on ordinary chat.
Batch mode takes a different route. It keeps a 16-bit recurrent state and uses int8 tensor-core GEMMs, reaching roughly 1,035 tok/s steady-state decode at 64 concurrent with 128-token inputs and 512-token outputs. The README notes that enabling all layers in int8 raises that to about 1,222 tok/s. The two modes are not two configurations of one engine; they are two different bets about whether your bottleneck is latency or aggregate throughput.
One structural constraint deserves attention. The README states that a speculating request reserves recurrent-state pages the pool has few of, so on long independent sessions the crossover point where plain batching wins arrives much earlier than the roughly eight-concurrent-user figure that applies to short prompts. That is a real capacity cost, not a footnote.
Installing it with Docker Compose and serving a first request
The image is prebuilt and pushed to ghcr.io on every commit, and the build applies everything in patches/ and runs verify.sh as its gate. The first start pulls 9.5 GB, downloads and requantizes the model into ./models (about 20 GB, once), and serves on port 18020. Clone the repository and create the environment file first:
git clone https://github.com/syv-ai/qwen38-27b-rtx3090 && cd qwen38-27b-rtx3090
cp .env.example .envOn PowerShell the README gives Copy-Item .env.example .env instead. Then pick a profile. The single profile is for one or a few people chatting; batch is for many concurrent requests:
docker compose --profile single up -dThe example .env sets SPEC=dflash2 and PREFIX_CACHE=1, and leaves VLLM_WSL2_ENABLE_PIN_MEMORY=1 enabled, which the README says is required under Docker Desktop on WSL2 or the V2 runner aborts with RuntimeError: UVA is not available. On native Linux the variable is read only inside the WSL-detected branch, so leaving it on costs nothing.
The example also leaves API-key authentication disabled for local-only use. Before exposing the server beyond your machine, generate a key:
echo "VLLM_API_KEY=$(openssl rand -hex 24)" > .envIf you would rather skip Compose entirely, the README gives a plain Docker invocation with a named volume for the model and a separate volume for the compiled-kernel cache, so the FlashInfer JIT step and the torch.compile cache survive container replacement. To confirm the stack on your own hardware, the repository ships a verification script and a benchmark harness; the documented single-stream numbers were re-measured with bash bench/run_benchmarks.sh single against the eight prompts in bench/prompts_real.jsonl.
The WSL2 abort, the slot math, and when this is the wrong tool
The most concrete failure mode in the documentation is the WSL2 one. SPEC=dflash2 forces vLLM's V2 model runner, whose UVA buffers are refused by default under WSL2, and the container aborts with RuntimeError: UVA is not available before loading weights. The fix is a single environment variable, but you have to know it exists, and the README places it in three separate spots, which suggests it has bitten people.
The second limitation is capacity, not correctness. DFLASH_TOKENS=15 reduces request slots and context headroom compared with the default seven-token verify block: the README's own table shows 8 slots at 64k context for the default versus 4 slots at 56k with the longer verify block. If your workload is many independent short conversations, that is the wrong trade. The README says the crossover to plain batching on long independent sessions is much earlier than eight concurrent users for exactly this reason.
The third is context. The 262k figure is not a default. CTX=long gives about 150k and is slower than the speed profile. CTX=huge is documented as usable only after running bash kvarn/install.sh in a Linux shell with bash available, and the .env.example describes it as slower and not the speed winner. Reaching the largest window is a deliberate downgrade of decode rate.
Finally, the version note in the README states that the branch pins vLLM 0.28.0 and that the throughput and quality tables are retained as reference baselines while the v0.28.0 GPU matrix is re-measured. If your decision depends on the quality numbers rather than the throughput numbers, you are working with a baseline, not a current measurement.
How this differs from a plain vLLM setup or an off-the-shelf quantized checkpoint
The obvious alternative is stock vLLM plus a pre-quantized GGUF or AWQ checkpoint from a model hub. That path is simpler: you install vLLM, point it at a repository, and serve. What you give up is the whole reason this repository exists. Stock vLLM on the same card does not ship the int8 tensor-core GEMM path, the calibrated int4 lm_head and drafter, the split-KV verify attention, or the KVarN KV cache. The README's demo GIF is captioned as stock vLLM against this repo, same card, same prompts, which is the project's own framing of the comparison.
The second alternative is llama.cpp or another GGUF-based runtime. That approach quantizes the weights and runs a hand-written inference path, which tends to be easier to install and more portable across backends. The difference in approach here is that this project stays inside vLLM and patches it, which means you inherit vLLM's OpenAI-compatible server, its continuous batching and its paged attention, and you also inherit its version coupling. Pinning vLLM 0.28.0 is what makes the patch series tractable; it is also what makes an upgrade a project rather than a pip install.
The third comparison is the multi-GPU route. The compose file documents GPU_COUNT and EXTRA_ARGS="--tensor-parallel-size 2" for more than one card, and notes that GPU_COUNT defaults to 1, meaning the first GPU the runtime enumerates, which is not the card you meant on a box with mixed hardware. So multi-GPU is supported, but it is documented as an escape hatch with a caveat, not as the primary design. If you have two 3090s, a stack designed around two cards would be the more natural starting point.
Maintenance cost, patch series and the licence
The last push to main was on 2026-09-15, and the repository is not archived. The image is rebuilt and pushed on every commit, so the container tag tracks the current stack rather than a release cadence. There are no retrieved releases, which means there is no versioned artifact to pin against; you are tracking a branch.
The upgrade cost is concentrated in patches/series. The Dockerfile comments explain that patch apply order matters because a few patches carry hunk context that an earlier patch adds, so the glob order of patches/*.patch is wrong, and the build skips dflash2-backport.patch because DFlash2 is native in vLLM 0.28.0. Moving to a newer vLLM means re-deriving that series against a new tree, and the skip line is a reminder that upstream eventually absorbs some of this work. The README's version note about re-measuring the v0.28.0 GPU matrix is the visible cost of that coupling.
On licensing: the repository is Apache-2.0, and the LICENSE file is at the top level. The model it serves, Qwen3.8-27B, is hosted on Hugging Face under its own terms, and the base image is nvidia/cuda:13.0.3-base-ubuntu24.04, which carries NVIDIA's redistribution terms. The Docker image also installs vLLM 0.28.0, torch 2.13 and Triton 3.7.1 from their own distributions. Apache-2.0 on this repository does not relicense any of those. If you plan to redistribute the built image, check each component's terms; that is a question for your own review, not something the repository answers.
Editorial conclusion
Adopt it if you have exactly one RTX 3090, you want Qwen3.8-27B served through an OpenAI-compatible endpoint, and you are willing to let a prebuilt 9.5 GB image apply its own vLLM patches. Do not adopt it if you need multi-GPU tensor parallelism as a first-class path, or if you cannot accept that the quality tables are currently marked as reference baselines while the v0.28.0 GPU matrix is re-measured. Before committing, run bash bench/run_benchmarks.sh single on your own card and compare the decode rate against the harness described in the README, because a client with a different output length is not measuring the same thing.
Frequently asked questions
Is the RTX 3090 still good for AI?
This repository is a working answer for one specific case: a 27B model on a single 24 GB card, served through vLLM at 150k context in the documented profiles. It reports roughly 1,035 tok/s aggregate at 64 concurrent in batch mode and 121 tok/s single-stream at default sampling, measured with vllm bench serve at a 250 W power limit.
Is a 3090 still good in 2026?
The repository treats it as the target hardware rather than a compromise, and its optimization work is aimed at that card specifically: int8 tensor-core GEMMs, a 16-bit recurrent state for batch mode, and MTP or DFlash2 speculation for single-user. The limits it documents are context and slot capacity, not the card's ability to serve the model at all.
Is the RTX 3090 outdated?
The README documents one GPU serving one profile at a time, with 8 request slots at 64k context in the default single-user configuration, so the card constrains concurrency and context rather than model size. The repository also documents a multi-GPU path through GPU_COUNT and --tensor-parallel-size, which it presents as an option rather than the primary design.
Does the 3090 support FP16?
The repository relies on mixed precision throughout: batch mode keeps a 16-bit recurrent state while using int8 tensor-core GEMMs, and single-user mode uses a calibrated int4 lm_head and drafter. The README does not discuss FP16 support as a hardware question for the 3090.
Community notes