MLServer: a V2-protocol inference server for Python model frameworks
An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more
At a glance
- What is it?
- MLServer packages a REST and gRPC inference server that speaks the KFServing V2 Dataplane spec, with pluggable runtimes for scikit-learn, XGBoost, LightGBM, MLflow and others. It is the right shape for teams already inside KServe or Seldon Core, and the wrong shape if you only need to expose one small model.
- Who is it for?
- Adopt MLServer if your models are already in scikit-learn, XGBoost, LightGBM, CatBoost, Spark MLlib, MLflow or HuggingFace format and you want a V2-protocol endpoint that KServe or Seldon Core can talk to without writing your own HTTP layer. Do not adopt it if you have a single model behind a simple internal API and no Kubernetes serving stack, because the runtime and protocol machinery buys you nothing there.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap MLServer fills between a trained model and a callable endpoint
Training a scikit-learn or XGBoost model produces a file. Turning that file into something another service can call over HTTP means writing request parsing, input validation, tensor encoding, concurrency handling and a model reload path. MLServer exists to remove that work for a specific class of users: teams that already run models on Kubernetes and want their endpoints to conform to the KFServing V2 Dataplane spec. The README frames the project as an inference server that is "fully compliant with KFServing's V2 Dataplane spec", and the topics list points at the ecosystem it targets: kfserving, seldon-core, mlflow, xgboost, lightgbm, scikit-learn. If your serving layer is KServe (formerly KFServing) or Seldon Core, MLServer is described as the core Python inference server those projects use. If you are building a small internal API for one model, the protocol compliance is overhead you will pay for and never use.
Runtimes are the backend glue, and they are separate installs
The architecture separates the server from the framework binding. The README calls inference runtimes "the backend glue between MLServer and your machine learning framework of choice", and they are distributed as distinct packages. Installing mlserver alone does not give you scikit-learn support; you also install mlserver-sklearn. The same pattern applies across the supported set: Scikit-Learn, XGBoost, Spark MLlib, LightGBM, CatBoost, Tempo, MLflow, Alibi-Detect, Alibi-Explain and HuggingFace each have their own runtime documentation page. This split is deliberate and it has a practical consequence: the dependency tree you deploy is only as large as the frameworks you actually serve. A CatBoost-only deployment does not drag in HuggingFace transformers. The cost is that version alignment across the main package and the runtime packages matters, which is why the repository ships a versioning script rather than letting each package drift. The README states that both mlserver and the runtime packages "try to follow the same versioning schema" and that ./hack/update-version.sh bumps the version across all of them. Runtime authors can also write their own, documented under docs/runtimes/custom.md, which is the escape hatch when your framework is not on the list.
Multi-model serving, worker pools and adaptive batching
Three capabilities distinguish MLServer from a thin model wrapper. First, multi-model serving lets multiple models run inside the same process, so a fleet of small models does not require a fleet of processes. Second, parallel inference across multiple models is implemented through a pool of inference workers, described in the user guide as a way to scale vertically. Third, adaptive batching groups incoming inference requests on the fly, which matters when individual requests are small and the underlying framework benefits from vectorised calls. These three features interact: batching changes latency characteristics, worker pools change memory footprint, and multi-model serving changes how a failure in one model affects its neighbours in the same process. The README lists all three as headline capabilities but does not quantify them, and the documentation for parallel inference and adaptive batching lives on separate pages. Anyone sizing a deployment should read those pages rather than assume defaults are appropriate. The design assumption underneath is that you have many requests and want to amortise framework call overhead, which is a different workload from a low-traffic internal endpoint.
Getting a model served: install, runtime package, then a V2 request
The install path is two commands. The base server comes from pip install mlserver, and the framework binding from a second package, for example pip install mlserver-sklearn for a scikit-learn model. The README is explicit that optional runtimes require their own install. Beyond that, the README points to a per-framework examples directory rather than inlining configuration: docs/examples/sklearn, docs/examples/xgboost, docs/examples/lightgbm, docs/examples/catboost, docs/examples/huggingface, docs/examples/alibi-detect, docs/examples/tempo, docs/examples/custom, plus docs/examples/mms for multi-model serving across frameworks and docs/examples/model-repository for loading and unloading models from a model repository. That last example is the one to read first if you intend to add and remove models at runtime rather than at startup, since it covers the load and unload path rather than just the initial serve. Because the endpoint conforms to the V2 Inference Protocol in both gRPC and REST flavours, the client side is not MLServer-specific: any client that speaks V2 can call it. That is the main interoperability argument for the project, and it is also the reason the examples matter more than the install commands.
Where MLServer is the wrong tool
The clearest limitation is the Python version window. The README's support table marks 3.7 and 3.8 unsupported, 3.9 through 3.12 supported, and 3.13 unsupported. A team standardised on Python 3.13 cannot run this today, and a team on 3.8 needs to plan an upgrade before adopting. That is a hard constraint, not a soft preference. The second limitation is scope: MLServer is an inference server for models, and the runtime list is the boundary of what it serves out of the box. If your model is a custom artefact format, you are writing a runtime, and the README directs you to docs/runtimes/custom.md rather than pretending the base install covers you. The third is the deployment assumption. The project's stated scalability story is deployment in Kubernetes native frameworks, Seldon Core and KServe. If you are not on those, you lose the main reason the V2 protocol compliance is worth the complexity, and you are running a server whose configuration surface was designed for a platform you are not using. None of these are defects; they are the shape of the tool. But a team that picks MLServer for a single low-traffic model and then discovers it must manage runtime packages, worker pools and protocol payloads has chosen the heavier option for no benefit.
How it compares to writing a thin FastAPI wrapper
The obvious alternative for a small deployment is a hand-written HTTP service, typically FastAPI or Flask, that loads the model at startup and exposes one or two endpoints. The difference in approach is not performance, it is contract. A FastAPI wrapper defines its own request schema, which means every client is bespoke and any change to the payload shape is a breaking change you coordinate yourself. MLServer implements the V2 Inference Protocol on both REST and gRPC, so the payload shape is fixed by an external specification and clients written against that specification work unchanged. The second difference is lifecycle. A FastAPI wrapper usually loads models once at process start; MLServer's model repository example covers loading and unloading models while the server runs, which is a capability you would otherwise build and test yourself. The third difference is batching and worker pools, which a hand-written wrapper does not have unless you implement them. The trade is control: with FastAPI you own every line and every dependency, and with MLServer you own the runtime package versions and the protocol version you target. For a single model behind an internal API, FastAPI is less machinery. For a multi-model serving platform with external clients, the protocol contract is the reason to pick MLServer.
Versioning, testing and the licence boundary you should read carefully
MLServer is licensed under Apache License, Version 2.0, which is permissive and unsurprising for a serving component. The README adds a caveat that matters more than the headline licence: software used in conjunction with, or alongside, MLServer may carry different terms. It names Alibi Detect and Alibi Explain specifically as licensed under the Business Source License 1.1. That is a source-available licence with usage restrictions, not an OSI-approved permissive one, so enabling those runtimes is a different decision from installing mlserver-sklearn. The README directs readers to the respective projects' documentation for the legal terms, which is the correct place to look; this is not legal advice and the licence text should be reviewed by whoever handles that in your organisation. On maintenance, the release cadence visible in the material is modest: 1.7.1 in June 2025, 1.7.0 in April 2025, and 1.6.1 in September 2024. The repository is not archived and the last push date is recent. The versioning script at ./hack/update-version.sh exists because the main package and the runtime packages move together, so an upgrade is not a single pip install but a coordinated bump across whichever runtimes you deploy. For contributors, the README gives two test entry points: make test for the full suite across MLServer and the runtimes, and tox -e py3 -- tests/batch_processing/test_rest.py for a single file. The fact that the example test path is under batch_processing is a small signal about where the project's own complexity sits.
Editorial conclusion
Adopt MLServer if your models are already in scikit-learn, XGBoost, LightGBM, CatBoost, Spark MLlib, MLflow or HuggingFace format and you want a V2-protocol endpoint that KServe or Seldon Core can talk to without writing your own HTTP layer. Do not adopt it if you have a single model behind a simple internal API and no Kubernetes serving stack, because the runtime and protocol machinery buys you nothing there. Before committing, verify three things: that your Python version is one of 3.9 through 3.12, that a runtime package exists for your framework rather than relying on the base mlserver install, and that the licence of any companion runtime you enable (Alibi Detect and Alibi Explain ship under the Business Source License 1.1) is acceptable to your organisation.
Community notes