Surogate: a C++/CUDA training and serving stack that ships its own benchmarks

The problem Surogate targets: two toolchains for one model

Most teams fine-tune with one stack and serve with another. The training side produces an adapter or a merged checkpoint, and the serving side has to be re-tuned, re-quantized, and re-benchmarked before it is useful. Surogate's stated pitch is that training and serving live in one toolkit, with dedicated native engines rather than a Python wrapper over someone else's kernels. The README frames the audience directly: from a first fine-tune to hundreds of concurrent requests, scaling from a single workstation to multiple GPUs and training clusters, or using system RAM for models larger than the available cards. The topics list on the repository (cuda, deep-learning, fine-tuning, llama, llm, qwen, sft) points at the same buyer: someone with NVIDIA hardware who wants fine-tuning and inference on the same box. That is a narrower audience than a general deep learning framework, and the project does not pretend otherwise.

Two engines, one workflow: how the pieces fit

The architecture is split into a training engine and a serving engine. The training engine executes in native C++/CUDA but is configured from Python, and the README describes a Python DSL with ahead-of-time automatic differentiation, explicit graphs, and native kernel dispatch for extensible architectures. The serving engine is a native C++/CUDA HTTP server exposing OpenAI-compatible Chat Completions, Completions, and Responses endpoints plus Anthropic-compatible Messages. The bridge between the two is not just a file format. GRPO is documented as using native serving for reward environments, evaluation, and policy updates, with shared-weight single-GPU BF16 LoRA across supported training families except Nemotron. That exception is worth reading twice, because it is the kind of per-family carve-out that shows up in a project where the execution paths are hand-written rather than inherited. The data flow for training is conventional in outline (raw text to checkpoints) but unusual in its precision handling: configurable model, gradient, and adapter precision, with BF16, hybrid FP8, and Blackwell NVFP4 recipes, and LoRA over BF16 bases or FP8, NVFP4, and BnB/NF4 quantized bases. On the serving side, native GGUF decoding is the headline path, which means the serving engine is not simply loading the training checkpoint format.

The benchmark tables and what they actually compare

The README publishes four headline numbers: 136,200 tok/s training on four RTX 5090s with Qwen3-0.6B FP4 LoRA, 2.53x training throughput against Unsloth on one H100 with Qwen3-0.6B BF16 on both sides, 802 tok/s serving for one user on one RTX 5090 with Qwen3.5-0.8B native GGUF, and 7.0x serving throughput against llama.cpp on eight RTX 5090s with GLM-5.3-Flash at 16 users. The comparison baselines are named per row: Unsloth for training, vLLM and llama.cpp for serving. The tables also include a less flattering row, Qwen3.5-4B at 100 users, where the gain over vLLM is 1.19x. That row is the most informative one in the set, because it shows the advantage compressing as concurrency rises and the baseline gets closer. The latency figures point the same way: 40 ms median time to first token at 100 users against 230 ms for vLLM, and 1.59 seconds at 16 users for GLM-5.3-Flash against 55.13 seconds for llama.cpp. These are the repository's own measurements, published in docs/reference/benchmarks.md and surogate/serve/BENCHMARKS.md. I have not reproduced any of them. Treat them as a claim with a documented methodology behind it, not as an independent result.

Getting it running: the documented entry points

The README routes new users through a Quickstart section and an examples directory, with task-specific guides for training modes, precision recipes, memory management, GRPO, DPO, and distillation. The DPO path has its own document, docs/getting-started/quickstart-dpo.md, which suggests it is short enough to follow end to end. Configuration is Python-side, so a training run is described by a Python config rather than by shell flags, while execution happens in the C++/CUDA engines under csrc/src. The serving side is an HTTP server with OpenAI- and Anthropic-shaped routes; the README points at docs/inference/api.md for the supported fields and says plainly that the API guide lists what is supported. That phrasing matters. Compatibility here is field-level, not blanket, so an application that relies on a request field outside that list should expect to check the guide before assuming it works. Beyond the Quickstart, the repository does not spell out install commands in the material I have, so the exact build invocation is something you would read from the docs rather than from this description.

Where the design constrains you

The clearest limitation is hardware. Everything in the README assumes NVIDIA GPUs, and the precision story is generation-specific: NVFP4 is described as a Blackwell recipe, so the fastest training numbers are tied to a GPU generation rather than to the toolkit alone. The second constraint is the per-family exception for GRPO shared-weight BF16 LoRA, which excludes Nemotron. A third is the memory model. CPU offload covers weights, gradients, optimizer state, activations, and quants, and pipeline parallelism can stream frozen weights for LoRA across PCIe GPUs without NVLink or GPU-to-GPU P2P. Those are real capabilities, but they are also the paths where throughput expectations should be set by the hardware interconnect rather than by the headline tables, which were measured on RTX 5090s and an H100. Finally, the Python DSL with ahead-of-time automatic differentiation means new architectures are expressed in a project-specific graph form. If your model family is not among the supported ones, the cost of adding it is not a config change.

What it is not: the case for staying with Unsloth, vLLM, or llama.cpp

The README names its own comparisons, so the alternatives are not hypothetical. Unsloth is a Python fine-tuning library, and the difference is not only speed: it plugs into the PyTorch and Hugging Face ecosystem, so a model that exists there is generally trainable without touching a C++ engine. Surogate's advantage in its own table is 2.53x on one H100 for Qwen3-0.6B BF16, and 1.36x on one RTX 5090 for the same model. If your work is mostly small-model experimentation on one card, that gap is real but it is not the difference between feasible and infeasible. On serving, vLLM and llama.cpp are the baselines. llama.cpp is a broad C/C++ inference project with GGUF support across many architectures and a large set of quantization options; Surogate's claim against it is aggregate decode throughput at 16 users on eight cards, which is a concurrency story rather than a single-stream one. vLLM is the closer comparison for high-concurrency serving, and the 1.19x row at 100 users shows that the margin narrows exactly where vLLM is strongest. If your workload is already well served by vLLM at moderate concurrency, switching toolkits buys you a smaller improvement than the headline table suggests.

Maintenance, releases, and the Apache-2.0 terms

The release cadence visible in the repository is tight: v1.4.6 on 2026-07-31, v1.4.7 on 2026-08-03, and v1.4.8 on 2026-08-22, with the last push to main on 2026-09-10. Three releases in roughly three weeks, then a later commit, is the pattern of an actively developed project rather than a frozen one. That cuts both ways for operators. Frequent releases mean fixes arrive quickly, and they also mean the surface you pin can move. Pin a release tag rather than tracking main if you are deploying the serving engine behind an application. The licence is Apache-2.0, which permits commercial use and modification and includes a patent grant, with the usual obligations around retaining notices and stating changes. This is a description of the licence identifier, not legal advice; if you are redistributing a modified build or embedding the server in a product, have counsel read the LICENSE file. The README does not describe a separate commercial licence or an enterprise tier, so the terms in the repository appear to be the terms.

Editorial conclusion

Adopt Surogate if your hardware is NVIDIA and your workflow fits its documented path: LoRA or QLoRA fine-tuning, GRPO or DPO preference training, distillation, and serving through OpenAI- or Anthropic-shaped endpoints. Do not adopt it if you need a framework with a wide third-party model zoo, a CPU-only fallback, or a training API that does not require you to read docs/getting-started/training-modes.md before your first run. Verify three things first: that your target architecture appears in the supported-models list, that your precision recipe (BF16, hybrid FP8, or Blackwell NVFP4) is available on your GPU generation, and that the serving endpoints you depend on match the fields listed in docs/inference/api.md, because the README explicitly scopes compatibility to the fields documented there.

Official sources

1. invergent-ai/surogate on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases