dotLLM: a pure C# inference engine that trades ecosystem size for zero-GC control
LLM inference engine written in .NET
At a glance
- What is it?
- dotLLM reimplements GGUF loading, tokenization, sampling and CUDA kernels in .NET rather than binding to llama.cpp. It is pre-1.0 and GPL-3.0, so the decision is less about features than about whether you want the whole stack inside your own process.
- Who is it for?
- Adopt dotLLM if you are shipping a .NET application and want inference inside your own process, on an OpenAI-compatible HTTP surface, without a native llama.cpp dependency. Stay away if you need continuous batching, ROCm, or a stable API surface, because the README places continuous batching in Phase 9 and the project is still on 0.1.0 preview builds.
- Can I use it commercially?
- Yes, with conditions. GPL-3.0 is a copyleft licence: if you distribute software that includes it, you must release that software's source code under the same licence. Running it internally without distributing it does not trigger that obligation.
- Is it still maintained?
- Yes. The repository last received commits 47 days ago.
- What is it written in?
- Mainly C#, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem dotLLM addresses is dependency shape, not raw speed
Most .NET teams that want local inference end up with a process boundary. They run llama.cpp or a Python server next to their application and talk to it over HTTP or a socket. That works, but it means shipping a native binary, matching its build to the host platform, and accepting that the tokenizer, sampler and KV-cache live somewhere you cannot step through in a debugger. dotLLM's answer is to implement the entire stack in C#. The README is explicit that it is "not a wrapper around llama.cpp or Python libraries" and that orchestration, model loading, tokenization, sampling and CPU compute are all managed code. The target reader is a .NET engineer who wants inference in-process, with the same tooling, the same memory model and the same deployment artifact as the rest of their application. That is a narrower audience than "anyone who wants to run an LLM locally," and the project's design choices only make sense inside it.
Layered packages, GGUF on one side and an OpenAI-shaped HTTP surface on the other
The repository is split into five layers, and the README states that each ships as a separate NuGet package so consumers pull in only what they need. DotLLM.Core holds the abstractions: ITensor, IBackend, IModel, ISamplerStep. DotLLM.Models and DotLLM.Tokenizers handle GGUF and SafeTensors files plus BPE and SPM tokenizers. DotLLM.Cpu and DotLLM.Cuda are the backends, and DotLLM.Engine sits above them with the KV-cache, scheduler, samplers, constraints and speculative decoding. DotLLM.Server wraps the whole thing in an ASP.NET host exposing /v1/chat/completions and /v1/completions. The dependency direction is strictly downward, which is what makes swapping a backend plausible: IBackend is the seam, and the README lists CPU, CUDA and ROCm as separate packages, with ROCm listed as a planned backend rather than a shipped one. Sampling is a chain of ISamplerStep implementations applied in a fixed order: repetition penalty, then temperature, then top-k, top-p, min-p, and finally a categorical draw. That ordering is visible in the README rather than inferred, which matters because the order of those operations changes output.
Zero-GC tensor memory and PTX kernels are the two decisions that define the codebase
The performance section describes unmanaged memory via NativeMemory.AlignedAlloc with 64-byte alignment for all tensor data, and claims no managed heap allocations on the hot path. Whether that claim holds in practice is not something this review can confirm, but the mechanism is concrete and testable: aligned unmanaged buffers plus TensorPrimitives and hand-written System.Runtime.Intrinsics code for quantized matmul, RMSNorm, RoPE and softmax. Model loading uses MemoryMappedFile over GGUF, so the README's position is that multi-gigabyte models load in milliseconds because the OS demand-pages them. The GPU path is the more unusual one. CUDA acceleration comes from PTX kernels loaded through the CUDA Driver API, with no native shared library shipped alongside. That removes a deployment artifact and a version-matching problem, and it also means the kernel compilation path is entirely your problem to debug when it fails. Quantized inference covers FP16, Q8_0 and Q4_K_M, with fused scale-by-int dot-product kernels operating directly on quantized blocks rather than dequantizing first.
Two install paths, one CLI, and a model pull that needs a network
The README gives three ways in. The global tool path requires the .NET 10 runtime: dotnet tool install -g DotLLM.Cli --prerelease, then dotllm model pull QuantFactory/SmolLM-135M-GGUF, then dotllm run QuantFactory/SmolLM-135M-GGUF -p "The capital of France is" -n 64. The serve subcommand replaces run and starts the OpenAI-compatible API plus a built-in chat UI. The self-contained archives need no .NET install, and the README names them per platform: dotllm-<version>-win-x64.zip, dotllm-<version>-linux-x64.tar.gz, dotllm-<version>-osx-arm64.tar.gz. Unpacking and running ./dotllm model pull is the same sequence with a local binary. Native AOT builds are attached to each release as dotllm-<version>-aot-<rid>.{zip,tar.gz} and the README flags them as experimental, asking users to file an issue on a crash. The third path is referencing the libraries from your own .NET application, which is where the layered package split actually pays off. Note that model pull is a network operation against a model identifier, so the first run is not offline.
Speculative decoding and paged KV-cache exist; continuous batching does not
The serving feature list is where expectations need calibrating. Paged KV-cache is present, described as PagedAttention with block-level allocation, prefix caching and copy-on-write. Speculative decoding is present as draft-verify-accept with KV-cache rollback, but the README limits it to greedy mode today and points to issue #121 for non-greedy support. That is a real constraint: speculative decoding under greedy sampling is the easy case, and the sampling chain described elsewhere in the README (temperature, top-k, top-p, min-p) is exactly what the current implementation does not cover. Continuous batching is listed as planned for Phase 9, with iteration-level scheduling, preemption and priority queuing. Until that lands, throughput under concurrent load is not the same shape as a production serving stack, and the OpenAI-compatible endpoint will not behave like a vLLM deployment under many simultaneous requests. Constrained decoding via FSM and PDA is listed as shipped, guaranteeing valid JSON, JSON Schema, regex and grammar output, which is a stronger correctness story than the batching story.
The honest comparison is llama.cpp, and the difference is where the boundary sits
The obvious alternative is llama.cpp, and the README names it directly as the thing dotLLM is not a wrapper around. The practical difference is not which models run. Both consume GGUF, both support Q4_K_M and similar quantizations, both expose an OpenAI-compatible server. The difference is the boundary. llama.cpp is a C/C++ core with bindings, so a .NET consumer either P/Invokes it or runs it as a separate process; you get a mature, widely exercised kernel set and you give up the ability to step from your request handler into the attention kernel without crossing a language boundary. dotLLM inverts that: the entire path is managed code you can debug, at the cost of a much smaller implementation that is still on 0.1.0 preview releases and whose CUDA path is PTX-through-driver-API rather than a conventional compiled extension. If your team already has C++ build infrastructure and just needs tokens out, llama.cpp is the lower-risk choice. If your constraint is that the inference engine must be a NuGet package in a .NET solution, the trade flips.
GPL-3.0 is the constraint that decides more adoptions than any missing feature
dotLLM is GPL-3.0. For an internal tool or an open source project under a compatible licence, that is unremarkable. For a closed-source product that links the DotLLM.* packages, the copyleft obligation is the first thing your legal review will raise, and it is not a question this article can answer for you. The practical point is that the licence is not incidental to the packaging: the README's whole pitch is that you reference these libraries from your own .NET application, which is precisely the usage pattern where GPL-3.0 matters most. If you need permissive terms, the alternative is to run inference behind a process or network boundary, which is the architecture dotLLM was built to avoid. Maintenance cost is hard to estimate from the material available. The release history shows three preview builds in April 2026 and the last push in July 2026, with Phase 7 (diagnostics and interpretability) described as in progress and logprobs landed. LoRA adapters and OpenTelemetry observability are both listed as planned, the former for Phase 7 and the latter across Phase 7 and Phase 9. Any of those could shift the API surface you build against.
Editorial conclusion
Adopt dotLLM if you are shipping a .NET application and want inference inside your own process, on an OpenAI-compatible HTTP surface, without a native llama.cpp dependency. Stay away if you need continuous batching, ROCm, or a stable API surface, because the README places continuous batching in Phase 9 and the project is still on 0.1.0 preview builds. Before committing, verify one thing: that the CUDA backend loads on your target GPU, since the README describes PTX kernels loaded through the CUDA Driver API with no native shared library, and that path is the least conventional part of the stack.
Community notes