TensorSharp: a native .NET inference engine for GGUF models

The gap TensorSharp fills: GGUF inference without leaving .NET

Most GGUF runtimes are C++ or Python projects with a thin wrapper for other languages. TensorSharp inverts that. The engine itself is C#, and the README describes it as a "Native .NET LLM inference engine for GGUF models". The repository layout backs this up: TensorSharp.Core, TensorSharp.Models, TensorSharp.Runtime and TensorSharp.Backends.GGML are C# projects, and the solution file is TensorSharp.slnx. The native GGML library is still compiled, but it sits behind managed backends rather than being the application.

The intended audience is a .NET team that wants local inference inside its existing deployment and debugging story. If your services are already .NET, an inference engine in the same runtime means one toolchain, one process model, and stack traces you can read. The project also ships TensorSharp.AgentHost, described as adding "Agent Skills and a bounded, in-process model-to-tool loop for sandboxed file and shell work", and TensorAgent, which brings the same chat and agent experience to iPhone and iPad through the iOS ggml_metal backend. That combination of inference plus a sandboxed tool loop is a narrower target than a general-purpose server, and it is clearly the one the project is aiming at.

How the engine is layered: managed runtime, native GGML, pluggable backends

The architecture visible in the repository is a stack. TensorSharp.Core and TensorSharp.Models hold the managed implementation. TensorSharp.Backends.GGML, TensorSharp.Backends.Cuda and TensorSharp.Backends.MLX are separate projects, one per accelerator family. TensorSharp.GGML.Native is the bridge to the compiled native library, and TensorSharp.Distributed covers multi-GPU work. Above that sit the entry points: TensorSharp.Cli, TensorSharp.Chat, TensorSharp.Server and TensorSharp.Server.Host.

Backend selection is explicit at the command line. The README's quick start passes `--backend ggml_cuda` on Windows and NVIDIA, `--backend ggml_metal` on Apple silicon, and `--backend ggml_vulkan` for AMD, Intel or NVIDIA Vulkan. The native library is configured and built with CMake, and the build is switched on per backend through environment variables such as TENSORSHARP_GGML_NATIVE_ENABLE_CUDA and TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN. That is a compile-time choice as much as a runtime one: enabling CUDA changes what the native build produces.

For serving, the README lists continuous batching, paged and prefix-shared KV cache, speculative decoding, tensor parallelism and configurable security boundaries as available "when you need them". Tensor parallelism is documented for the direct cuda backend and for the GGML CUDA and Vulkan backends, with a notable wrinkle: the README states that Qwen 3.8 Flash Next and DeepSeek V4 use the same flag for a layer split rather than true tensor parallelism, and that GLM 5.x layer-splits by default when the flag is omitted. So `--tp N` does not mean the same thing across every model family, and anyone sizing a multi-GPU deployment should read that paragraph carefully before assuming a linear speedup.

Installing TensorSharp and running a first model

There are two paths. The Releases page provides self-contained CLI and Server archives for Windows x64 (CPU/CUDA), Linux x64 (CPU/CUDA) and macOS arm64, which avoids the build entirely. Building from source targets .NET 10, and the README is blunt that the runtime alone is not enough: "the .NET Runtime alone cannot build TensorSharp". Install the .NET 10 SDK, then confirm a 10.0.x SDK is visible.

dotnet --list-sdks

Beyond the SDK you need git, curl, CMake 3.20 or later for the native GGML library, and the toolchain for your chosen GPU backend. The README's verified fast path is Gemma 4 E4B, using the public file gemma-4-E4B-it-Q8_0.gguf (7.48 GiB); text-only inference needs no projector. On Windows with an NVIDIA card, the documented sequence clones the repository, downloads the model, writes a prompt file, enables the CUDA native build and runs the CLI.

git clone https://github.com/zhongkaifu/TensorSharp.git; Set-Location TensorSharp
New-Item -ItemType Directory -Force models | Out-Null
curl.exe -L --fail "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models\gemma-4-E4B-it-Q8_0.gguf
'Answer in one short sentence: what is TensorSharp?' | Set-Content prompt.txt
$env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA = 'ON'
dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true -- --model models\gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cuda

The README says this gets you running in roughly 30 seconds once the prerequisites are in place, though the model download and the first native build dominate that time on a cold machine. On macOS, drop the CUDA environment variable and switch to `--backend ggml_metal`. On Linux with NVIDIA, prefix the `dotnet run` with `TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON`. For Vulkan, set `TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON` and use `--backend ggml_vulkan`. The `--input prompt.txt` flag reads the prompt from a file; `--max-tokens 128` caps generation. What you should see is a one-sentence answer printed to the console.

Where TensorSharp is the wrong choice

The build cost is the first honest limitation. A prebuilt archive sidesteps it, but source builds require the .NET 10 SDK, CMake 3.20+, a C++ toolchain and the GPU vendor's toolkit. The README's Linux tensor-parallelism section even sets LD_LIBRARY_PATH to /usr/local/cuda-12.6/compat on RunPod Ubuntu 24.04 images, which tells you the CUDA version on your host can matter to whether the native library loads. None of that is unusual for a native inference engine, but it is a real difference from dropping a static binary on a box.

Second, the API surface is described as Ollama/OpenAI-compatible, not identical. Compatibility claims in this space usually mean the common endpoints work and the long tail does not. The README does not enumerate which endpoints or which request fields are supported, so if your client depends on a specific OpenAI feature, you have to check the server code or test it yourself.

Third, the model matrix is wide but uneven. The README lists text, vision and audio input, PDF, image editing and video generation, with image editing via Qwen-Image-Edit and video via MiniMax-H3 with 32 kHz stereo audio or Wan 2.1/2.2 for video alone. Multimodal paths carry more moving parts, and a model that is not in docs/models/README.md is not a supported target. If you need one specific model that is not listed, this is not the project to bend to your will.

Finally, TensorSharp is a poor fit if you have no .NET investment. The main reason to choose it over llama.cpp is the runtime and the managed tooling. Without that, you are paying the build cost for no benefit.

TensorSharp versus llama.cpp: same GGUF files, different centre of gravity

The README states that TensorSharp is benchmarked against llama.cpp on identical models and hardware, and points to docs/engine_comparison_report.md, with the caveat that results are specific to the measured model, backend and workload. Both projects consume GGUF files, so your model library is portable between them. The difference is what surrounds the inference loop.

llama.cpp is a C/C++ project with a broad set of language bindings and a long list of downstream tools; its centre of gravity is the engine and the ecosystem around it. TensorSharp's centre of gravity is the .NET application: a managed runtime, a CLI, a chat UI, an HTTP server, and an agent host with a sandboxed tool loop. If your team writes C#, the second shape is easier to debug and deploy. If your team writes Python or Go, or you want the widest possible set of bindings, llama.cpp and its derivatives are the more natural starting point. Neither is strictly better; they optimise for different things, and the benchmark report is the place to look if raw throughput on your specific model and GPU is the deciding factor.

Maintenance, licence and what an upgrade actually costs

TensorSharp is not archived and the last push was on 2026-09-16, so the repository is being worked on. Releases v3.4.0.0 on 2026-09-12 and v3.3.0.0 on 2026-08-30 suggest a cadence of weeks rather than months. The licence is BSD-3-Clause, which permits commercial use and modification provided the copyright notice and disclaimer are retained; that is a permissive licence, and it is the same family used by many inference projects. This is not legal advice, and if you redistribute a modified build you should read the LICENSE file in the repository root yourself.

The upgrade cost is the part that is easy to underestimate. Because the native GGML library is built with CMake and switched on per backend, a version bump can mean rebuilding native code, not just restoring a NuGet package. The README also notes that on older Ubuntu releases the .NET 10 SDK comes from the backports PPA, so the SDK itself may need attention on long-lived images. If you pin a prebuilt archive from the Releases page, upgrades are a download and a restart. If you build from source, budget for a native rebuild and a re-test of your chosen backend on every upgrade.

Editorial conclusion

Adopt TensorSharp if you already ship .NET services and want GGUF inference behind an Ollama/OpenAI-compatible HTTP surface without adding a separate Python or C++ runtime. Skip it if you need a one-command install with no SDK, CMake or GPU toolchain, or if you only run text models and llama.cpp already works for you. Before committing, verify three things: that `dotnet --list-sdks` reports a 10.0.x SDK, that your target GPU backend (ggml_cuda, ggml_metal, ggml_vulkan or the direct cuda path) builds on your machine, and that the model you intend to serve appears in docs/models/README.md with the modality you need.

Frequently asked questions

Official sources

1. License: BSD-3-Clause
2. Project website
3. README
4. Releases
5. zhongkaifu/TensorSharp on GitHub