LitServe: a Python inference server where you write the predict method
A minimal Python framework for building custom AI inference servers with full control over logic, batching, and scaling.
At a glance
- What is it?
- LitServe wraps a user-defined LitAPI class in a FastAPI-based HTTP server and handles batching, concurrency and worker scaling around it. It is a reasonable fit when your inference logic does not look like a single model behind a fixed schema, and the wrong fit when you want a turnkey LLM server.
- Who is it for?
- Adopt LitServe if your inference endpoint needs custom Python logic (multiple models, an agent loop, a RAG pipeline) and you are willing to own the server process, since the README shows a plain server.run(port=8000) rather than a managed runtime. Do not adopt it if you want a tuned LLM engine with paged attention, or if your team cannot read FastAPI-style Python.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap LitServe targets: serving code that is not a single model
The README opens with a direct claim about the problem: most serving tools are built for one model type and enforce rigid abstractions, which works until you need custom logic, multiple models, agents, or non-standard pipelines. LitServe's answer is to let you write the inference engine yourself in Python, while the framework owns performance, concurrency, scaling and deployment. The audience is therefore not someone who wants to download weights and get an OpenAI-compatible endpoint. It is someone who already has Python code that produces a prediction and now needs that code reachable over HTTP, with batching and more than one worker, without writing the queueing, process management and request parsing by hand. The README lists the intended shapes explicitly: inference APIs, agents, chatbots, RAG systems, MCP servers, multi-model pipelines. The two worked examples match that list. One is a toy pipeline that calls two lambdas, squares and cubes the same input, adds the results. The other is a news agent that fetches a URL, strips HTML tags with a regex, and sends the text to OpenAI's chat completions endpoint. Neither is a model server in the vLLM sense. Both are ordinary Python functions that happen to be expensive or remote.
LitAPI and LitServer: the two objects you actually write
The mechanism is small enough to describe completely from the README. You subclass litserve.LitAPI and implement two methods. setup(self, device) runs once per worker and is where the README puts model loading and client construction: the toy example assigns two lambdas to self.text_model and self.vision_model, the agent example builds an openai.OpenAI client. predict(self, request) runs per request and receives the parsed request body, which in both examples is a plain dict. It returns a dict, and the README's curl calls read the result as JSON. You then construct ls.LitServer(InferenceEngine(max_batch_size=1), accelerator="auto") and call server.run(port=8000). Note the max_batch_size argument in the README's example is passed to the LitAPI constructor rather than to LitServer, which is a small inconsistency in the snippet worth checking against the API reference before copying it. accelerator="auto" is the device selection hook that setup receives as its device argument. Around those two methods the framework supplies the HTTP layer (the repository topics list fastapi and rest-api, and the README's own comparison is against FastAPI), the batching described by max_batch_size, streaming, routing, multi-GPU autoscaling, and serverless support. The important structural point is that LitServe does not know what a model is. It knows how to move a dict from an HTTP request into a Python method and a dict back out, and how to do that across workers. Everything domain-specific stays in your class.
Getting a server running: pip, python, curl
The install path is one command, pip install litserve, with the README pointing to a separate install page for other options. From there the two documented launch routes are python server.py, which the README shows commented out next to the lightning deploy line, and lightning deploy server.py for the Lightning AI path, with --cloud as the flag that pushes it to managed infrastructure. The local run needs no config file and no YAML: the port is a Python argument, server.run(port=8000), and the batch size is a constructor argument. That is the whole configuration surface shown in the README, and it is the strongest argument for the project if you have ever maintained a serving config that drifted from the code it described. Verification is a curl against /predict with a JSON body. For the toy engine: curl -X POST http://127.0.0.1:8000/predict -H "Content-Type: application/json" -d '{"input": 4.0}'. For the agent: the same call with {"website_url": "https://text.npr.org/"}. The route name follows the method name predict, so the endpoint is not configurable in the material supplied. If you need a different path or a health endpoint, that is not documented here. The agent example also requires an OPENAI_API_KEY in the environment, and its predict method calls requests.get on a user-supplied URL, which means the server will fetch whatever address a caller sends unless you add validation yourself.
The batching and scaling claims, and what the README does not show
Batching is the feature most serving frameworks live or die on, and the README treats it as a bullet point plus a constructor argument, max_batch_size. What it does not show is the signature change. If requests are batched, predict presumably receives a list rather than a single dict, and the README's examples all set max_batch_size=1 or omit it, so a reader copying the quick start never sees the batched form. The same applies to streaming: it appears in the feature list and in the claim that you control batching, routing, streaming and orchestration, but no streaming example is included in the supplied material. The performance claim, 2x faster than FastAPI, is asserted in the README banner without a methodology, a hardware description or a workload. Treat it as a marketing line until you reproduce it on your own predict method, because the ratio will depend almost entirely on how much time your code spends outside Python. The scaling story is split between two places: multi-GPU autoscaling is listed as a LitServe capability, while autoscaling GPUs, monitoring and a free tier are described as properties of Lightning Cloud, the managed product. Those are not the same thing, and the README does not draw the line between what the open source package does on your own hardware and what the hosted platform adds. If autoscaling is the reason you are evaluating LitServe, that distinction is the first thing to resolve.
Where LitServe is the wrong tool
The README is unusually frank about this: vLLM and similar tools are built for a single model type, and LitServe exists for the cases they do not cover. Read the same sentence in the other direction and you get the limitation. If your workload is one transformer serving text generation, a purpose-built engine will bring optimizations that a general Python request loop does not have, and LitServe's own feature list mentions BYO model or vLLM, which suggests the intended pattern is to run vLLM inside a LitAPI rather than to replace it. The second failure mode is operational. Because LitServe hands you the process, you inherit the process: worker count, GPU placement, restarts, and the queue between the HTTP layer and your predict method are yours to reason about, and the README's examples are single-file scripts that do not address any of it. The third is that a general abstraction invites you to put things in predict that should not be there. The news agent example fetches a remote page and calls an external API inside the request path, with no timeout, no retry and no caching shown. That is fine as a demo and a liability in production, and nothing in the framework will stop you. Finally, the version cadence is uneven: v0.2.17 shipped in December 2025, v0.2.18 in August 2026, v0.2.19 in September 2026. Two releases eight months apart followed by two in a month is not a stability signal either way, but it does mean you should pin a version rather than track main.
LitServe against plain FastAPI and against vLLM
The honest comparison is with FastAPI, because LitServe is FastAPI plus a worker and batching layer. With FastAPI you write the route decorator, the Pydantic model, the startup hook that loads the model, and then you solve batching yourself: a queue, a background task that drains it on a timer or a size threshold, and a way to fan results back to the waiting requests. That is perhaps a hundred lines of concurrency code that is easy to get subtly wrong. LitServe replaces it with max_batch_size and a predict method, at the cost of adopting its request and response conventions. If your endpoint is a single call with no batching and no GPU, FastAPI alone is less machinery. The comparison with vLLM is different in kind rather than degree. vLLM is an inference engine for large language models with its own scheduler and memory management; LitServe is a hosting shell that does not care what runs inside predict. The README's own position is that you can put vLLM behind a LitAPI when you need that engine and still want custom routing or preprocessing in front of it. Choosing between them is therefore not a benchmark question. It is a question of whether your bottleneck is the model's token throughput or the shape of the pipeline around it.
Licence, maintenance and what to check before you commit
LitServe is Apache-2.0, shown as a badge in the README and listed as the repository licence. Apache-2.0 permits commercial use and modification and includes a patent grant, which matters if you are embedding the server in a product. It does not remove your obligations: you keep the licence and notice files with redistributed code, and the framework gives you no opinion about the licences of the models, datasets or third-party APIs you call from predict. The news agent example calls OpenAI, for instance, and that relationship is entirely outside the Apache grant. The repository is not archived and the last push is dated September 2026, so the project is active. Maintenance cost on your side is the part the README does not price. You are adopting a dependency whose API surface is LitAPI.setup, LitAPI.predict and LitServer's constructor, and those are the methods your code will break against on upgrade. Pin the version, read the release notes for v0.2.x before moving, and keep the server entry point thin so a signature change is a one-file fix. The managed Lightning Cloud path trades that maintenance for a vendor relationship and is a separate decision from adopting the library. For the licence itself, read the LICENSE file at the repository root rather than the badge, and take your own advice on legal review; nothing here is legal advice.
Editorial conclusion
Adopt LitServe if your inference endpoint needs custom Python logic (multiple models, an agent loop, a RAG pipeline) and you are willing to own the server process, since the README shows a plain server.run(port=8000) rather than a managed runtime. Do not adopt it if you want a tuned LLM engine with paged attention, or if your team cannot read FastAPI-style Python. Before committing, verify that the released version on PyPI matches the main branch you are reading, and check the licence file at the repository root rather than the README badge.
Community notes