club-3090: Serving LLMs on RTX 3090s with vLLM, llama.cpp, and ik_llama

What club-3090 Actually Solves

Running modern LLMs on consumer GPUs is rarely a matter of just downloading a model and running it. Context windows, quantization, tensor parallelism, and engine quirks all interact with the 24 GB VRAM limit of an RTX 3090. club-3090 collects working Docker Compose configs, patches, and benchmark numbers for exactly that hardware class. It targets people with one or two RTX 3090s who want to serve models at home, in a homelab, or as a dev backend. The project is model-agnostic by design, but currently ships curated configs for Qwen3.6-27B, Qwen3.6-35B, and Gemma 4 variants (26B and 31B). The README explicitly positions it as a collection of recipes, not a framework. That distinction matters: you get tested configurations, not a new abstraction layer to learn.

The Two-Route Architecture: Throughput vs. Robustness

The core design choice is splitting serving into two complementary routes. The vLLM dual-card route is for maximum throughput, with the README citing up to 127 TPS for code generation (DFlash) and 4 concurrent streams at 262K context. The llama.cpp single-card route is for maximum robustness: full 200K context on one 3090, stress-tested with no prefill cliffs, 25K-token tool returns, and a 91K needle ladder pass. The measured speeds are lower, around 51 to 60 TPS with Q4_K_M and MTP, but the route does not crash on real-world tool-using agents. This is an explicit trade-off, not a hidden one. The project also supports ik_llama for best GGUF quants. The README notes that SGLang was evaluated but is currently blocked on Ampere, which shows the project is willing to report dead ends rather than just list supported engines.

Getting Started: Commands and Configuration

The quick start is straightforward. Clone the repository, ensure PyYAML is available, then run bash scripts/setup.sh. The script is an interactive hardware-aware picker that asks which model to use and where to store weights. You can skip the prompts by exporting MODEL_DIR and passing the model name. After setup, launch.sh calls switch.sh to swap the old container down and the new one up, then verify-full.sh to confirm the service is serving cleanly before you point a client at it. The default endpoint is an OpenAI-compatible API on localhost:8020. For those who prefer a terminal UI over the CLI, the c3 cockpit wraps the same flow. Install it with uv pip install -e tools/serve-cockpit, then run c3. The first run requires pressing S to set the Model Dir and HuggingFace token, saving with Ctrl+S, then pressing r to browse the catalog and serve a variant. The cockpit also has a lean mode (c3 --lean) that hides the producer lane for consumers only.

The c3 Cockpit and the Universal Pull Tool

The c3 cockpit is a lazydocker-style terminal UI that provides a keyboard-driven interface for the entire serving lifecycle: discover, serve, operate, and validate. It is more than a convenience wrapper; it is a separate Python package (tools/serve-cockpit) that depends on an in-repo core package (tools/tui-core). The README instructs users to re-run the install after a git pull to pick up new dependencies and UI changes. The second major tool is the universal pull flow, introduced in v0.8.0 and extended in v0.8.2. It evaluates any safetensors Hugging Face repository and gives an honest one-line fit verdict using the --recommend flag. If a pull hard-blocks, you can send a redacted diagnostic back with --submit-last, but only with explicit consent. This is a rare feature: most serving tools assume your model is already compatible. The pull tool is designed to be honest about confidence, which is a welcome change from the usual 'it should work' assumption.

Known Cliffs and the Retired Single-Card Path

The README is unusually candid about failure modes. It documents a 'Cliff 2' issue where GDN prefill OOMs at more than about 50K tokens for a single prompt on a 24 GB single-card vLLM setup. The workaround is to use the vllm/dual configuration with tensor parallelism (TP=2), which escapes the cliff. More importantly, the former single-card escape route, llamacpp/default, was retired on 2026-08-12 and is only available with a --force flag. The README states plainly: 'on one card there is no longer a cliff-immune qwen path.' This is a significant limitation. If you have a single 3090 and need to serve a Qwen model with long context, the project no longer guarantees a crash-free path. The full diagnosis is in docs/CLIFFS.md, and any user considering a single-card deployment should read that file before trusting a configuration.

Hardware Class Awareness and Cross-Rig Support

The project is calibrated for 3090s but explicitly supports 4090 and 5090 owners. The FAQ includes measured benchmark rows for these classes. The README notes per-class gotchas: the 4090 has tighter idle VRAM, and the 5090 has a 32 GB envelope. This class awareness is built into the tooling, not just the documentation. For example, the setup script is described as 'hardware-aware,' and the configs are 'class-aware.' The project also supports multi-GPU setups beyond two cards, including 4x 3090, 8x A6000, and mixed configurations. The multi-card documentation covers tensor parallelism scaling math and valid TP values, derived from the dual.yml file. This means the project is not just a set of static YAML files; it has a mental model of how hardware constraints affect serving decisions.

Maintenance, License, and the Cost of Upgrades

The repository is licensed under Apache-2.0, which is permissive for both personal and commercial use. The project is actively maintained, with releases v0.10.0, v0.10.1, and v0.10.2 pushed between June and July 2026. The README instructs users to re-run the install after a git pull, which implies that upgrades are not always backward-compatible. The c3 cockpit and the pull tool are separate packages that may change independently. The documentation is extensive, with dedicated files for WSL2 setup, quantization names, hardware questions, and a glossary. The maintenance cost is not zero: you need to track release notes and re-install tools after updates. However, the project's explicit documentation of cliffs and retired paths suggests that the maintainers are willing to make breaking changes when a configuration is proven unsafe. That is a trade-off worth accepting if you value correctness over stability.

Editorial conclusion

Adopt club-3090 if you have one or two RTX 3090s (or a 4090/5090) and want a tested path to serving modern LLMs locally, especially if you value robustness over raw speed. Skip it if you need a single-card solution for long-context tool-using agents, as the README notes the single-card cliff-immune path was retired; verify your exact model and context length against the CLIFFS doc before committing. Also check the FAQ for your GPU class, since 4090 and 5090 have different VRAM envelopes. The project is Apache-2.0 and actively maintained, but it is community-driven, so verify any config against your own hardware before production use.

Official sources

1. Official README
2. Project repository
3. Release notes