Xinference 4.x: one serving API for open models, with torch and FastAPI strings attached

Swapping GPT out by changing one line of client code

Xinference, the Python package published from xorbitsai/inference, sits between your application and the models you host. The project pitches itself on one idea: code that already talks to GPT is pointed at an Xinference server instead, one line changes, and the requests land on an open-source model running on hardware you choose. The same API serves language models, speech recognition models and multimodal models, and the project says it runs on cloud machines, on-prem servers or a laptop. Deploying a built-in model is meant to take a single command. The package on PyPI is named xinference, a Docker image is published as xprobe/xinference, and the code is licensed Apache-2.0. A separate product, Xinference Enterprise, is linked from xinference.co; the self-hosted route is the open repository and its documentation at inference.readthedocs.io.

Version 4.0.0 added Xavier, heterogeneous P/D and runtime reconfiguration

The 4.x line is very new. v4.0.0 was tagged on 2026-10-10, v4.0.1 followed on 2026-10-16, and v4.1.0 on 2026-10-25. The 4.0.0 release headlines three changes: multi-engine Xavier, heterogeneous P/D, and runtime model reconfiguration. The front page names all three in a single line and sends readers to the GitHub release notes for the detail. That matters for planning: if your deployment design depends on Xavier spanning several engines, or on changing a loaded model's configuration without tearing the server down, the place to confirm how it behaves is the release note for the exact version you install, not the one-line summary. The main branch received a push on 2026-11-02 and the repository is not archived, so the 4.x code is still moving underneath those notes.

Auto batch, distributed workers and a vLLM KV cache shared across replicas

Several serving mechanisms trace back to specific pull requests. Auto batch (#4197) groups multiple concurrent requests into batches automatically; the project credits it with a significant throughput improvement, but the hot-topics line gives no number. Distributed inference (#2877) runs a model across workers instead of tying it to one machine. For the vLLM engine, #2732 added a KV cache shared across multiple replicas, so replicas of one model draw on a common cache rather than each building its own. None of these come with a published figure next to them on the front page, so the throughput you get is something to measure yourself. The repository has a benchmark/ directory and a monitor/ directory at the top level, which are the starting points for that measurement and for watching a running deployment.

Xllamacpp is a llama.cpp binding the Xinference team maintains itself

For llama.cpp models, Xinference uses Xllamacpp (xorbitsai/xllamacpp, brought in with #2997), a Python binding for llama.cpp that the Xinference team maintains. The stated reasons are that it supports continuous batching and is more production-ready. The consequence cuts both ways. Fixes to the binding can ship on the same schedule as the server, because the same people own both. But it also means llama.cpp support in Xinference is only as current as Xllamacpp: if a llama.cpp feature has not reached that binding, it has not reached Xinference either, and you cannot swap in a different wrapper to get it. Teams that already standardised on another llama.cpp binding elsewhere in their stack will be running two of them once Xinference is added.

torch is a base dependency, not an optional extra

The dependency list in pyproject.toml puts torch in the core requirements with no version constraint, beside pillow, huggingface-hub>=0.19.4 and modelscope>=1.21.0. Installing the xinference package therefore pulls torch even when the only model you plan to serve runs through llama.cpp, and it pulls both the Hugging Face and the ModelScope client libraries whether or not you use both hubs. Because the torch entry is unpinned, which torch build you end up with is decided by your package resolver and index, not by Xinference. On a GPU host, check which build landed before loading a model, or start from the xprobe/xinference image so the choice has been made for you. The core list also includes xoscar>=0.12.2 and psutil>=5.9.0, so the server has its own runtime layer underneath the HTTP API.

Python 3.10 or newer, CPython only, and FastAPI capped below 0.137

requires-python is ">=3.10", and the classifiers list CPython for Python 3.10 through 3.14 with no other implementation. FastAPI is pinned to >=0.110.3,<0.137 and click to >=8.1,<9. If you install Xinference into an environment that already runs another FastAPI app on a newer release, pip has to downgrade it or refuse, so a dedicated virtual environment or the container is the safer arrangement. Comments inside pyproject.toml show the dependency set shifting: python-multipart and orjson are now declared directly because they "used to come in transitively via gradio", and orjson is what serialises every REST API JSON response. websockets>=12 is there for the generic realtime API, and filelock>=3.30.0 is required so Wan MLX conversion locks stay stable on Windows. Upgrades can change this list, so re-read it when you move versions.

Three releases in fifteen days puts version pinning on you

Between 2026-10-10 and 2026-10-25 the project shipped v4.0.0, v4.0.1 and v4.1.0. That pace brings new built-in models in quickly. Recent additions include DeepSeek-V4.1-Flash and DeepSeek-V4.1-Flash-1017, GLM-5.3 in BF16 and FP8, the Qwen3.8-Flash-Next series, MiniCPM-V-4.7 in 1B and 35B-A3B sizes, EmbeddingGemma 2, the Ming-3.0 TTS and Confucius4-TTS speech models, OCR models such as Xiaomi-OCR-0 and jina-ocr-v1, and image models including FLUX 3 Image Klein and Qwen-Image-2.1-Turbo. The same pace means a server installed without a version pin can change behaviour on its next rebuild, including across a major version such as 4.0.0. The package version is dynamic and computed by setuptools-scm, so a build from a git checkout takes its version from the repository tags. Pin xinference to an exact version in production and read each release note before moving.

Integrations target RAG and agent platforms, and source builds use a custom backend

The integrations named on the front page are application platforms rather than client SDKs: Dify through a plugin on the Dify marketplace, FastGPT, RAGFlow, MaxKB, and Xagent, the Xorbits enterprise agent platform for planning, memory and tool use. Xinference calls the Xagent pairing agent-native serving, with dynamic planning, tool use and multi-step reasoning. For a team already on one of these platforms, Xinference is a backend to select rather than a client to write. Building from source is less plain than a stock setuptools package: pyproject.toml sets build-backend to build_backend with backend-path ".", and the repository holds build_backend.py, build_web.py and a frontend/ directory next to the xinference/ package. If you package Xinference yourself, that custom backend is part of what you have to run and audit.

Editorial conclusion

Xinference fits teams that want to host open language, speech, OCR and image models behind one API that existing GPT client code can call, and that can give it a dedicated Python 3.10+ environment or the xprobe/xinference container. It is a poor fit for a slim install that avoids torch, or for an app that must share its environment with FastAPI 0.137 or newer. Before adopting it, pin an exact 4.x release and read the 4.0.0 release notes on Xavier and runtime model reconfiguration.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. xorbitsai/inference on GitHub