Mosec: Rust HTTP layer, Python workers, and dynamic batching for ML serving

The gap between a trained model and an HTTP endpoint

A model that works in a notebook still needs a process that accepts requests, groups them, runs the forward pass on the right device, and returns results to the correct caller. Mosec targets exactly that layer. The README frames it as bridging the gap between a model you just trained and an efficient online service API, and lists dynamic batching and pipelined stages as its two structural features. The audience is an engineer who writes inference code in Python and does not want to hand-roll a request queue, a batching window, or a multi-process pipeline. The project is explicit that it does one thing: online serving. It does not attempt model optimization or business logic, which stays in your handler. That narrow scope is the useful part of the pitch, because it means the framework's job is bounded and your handler remains ordinary Python that you can also call from an offline test script. The README states that users serve models in an ML framework-agnostic manner using the same code as they do for offline testing.

Two layers: a Rust server and Python worker processes

The architecture splits into a web layer plus task coordination written in Rust, and a user interface written purely in Python. The Rust side handles HTTP and the async I/O that keeps CPU utilization high under concurrent connections. The Python side is where your model lives. You define a service as a class inheriting from mosec.Worker, initialize the model in __init__, and override forward with the signature forward(self, data) -> data, where a single item or a list is received and returned depending on whether dynamic batching is configured. When you declare multiple workers, mosec runs them as a pipeline: the first worker takes the request, the last worker sends the response, and data moves between workers over inter-process communication. The README gives downloading images, model inference, and post-processing as an example of such a split. This is the design decision that matters most. A single-stage service is one process doing everything; a multi-stage service lets you put a CPU-bound preprocessor and a GPU-bound model in separate processes so they overlap instead of serializing.

Dynamic batching and warmup, as the README describes them

Dynamic batching aggregates requests from different users into one batched inference call and distributes the results back to the right callers. The README points to a configuration section for the knobs, and the mechanism is visible in the example: the stable diffusion worker sets self.example to a list of four identical prompts, with the comment warmup (batch_size=4), and forward receives a List[str] and returns a list of memoryview objects. The batch size in that example is therefore tied to the length of the warmup example. Warmup has a second effect the README spells out: if an example is specified, the service is only marked ready after that example has been forwarded through the handler; if no example is given, the first real request pays the extra latency. That matters for GPU services, where the README says warmup usually helps allocate GPU memory in advance. For warming up with several different inputs, the README mentions a multi_examples attribute and links to a JAX example. The serialization side is handled by mixins: inheriting MsgpackMixin switches the protocol to msgpack, which the README recommends when returning binary payloads such as JPEG images, because JSON would require base64 and inflate the payload. Without the mixin, JSON is the default.

Getting a server running from the README

Installation is a single command on Linux or macOS: pip install -U mosec. The README also lists conda install conda-forge::mosec and pixi add mosec as alternatives. Building from source requires Rust and make package, which produces a wheel in the dist folder. The README states Python 3.7 or above is required. The worked example serves stable diffusion and needs diffusers and transformers first: pip install --upgrade diffusers[torch] transformers. The server file imports Server, Worker and get_logger from mosec, plus MsgpackMixin from mosec.mixin, defines the StableDiffusion class with the model loaded in __init__, and returns image buffers from forward. What the README does not show in the excerpt is the code that instantiates Server, appends the worker, and calls run. That call is the entry point, and it is the one piece a reader has to find in the documentation rather than in the README text above. Treat the example as a handler template, not a complete runnable file, until you have checked the linked docs.

Operational behaviour: warmup, shutdown, and metrics

Mosec is described as cloud friendly, with model warmup, graceful shutdown, and Prometheus monitoring metrics, and the README says it is easily managed by Kubernetes or any container orchestration system. The warmup semantics are the most consequential of these for deployment: because the service is not ready until the example has passed through the handler, a readiness probe will not pass early, which prevents traffic from hitting a process that has not yet allocated its GPU memory. Graceful shutdown is listed but not detailed in the material available here, so the exact drain behaviour under in-flight batched requests is something to confirm in the docs before you rely on it during a rolling deploy. Prometheus metrics are mentioned without a list of exported metric names, so you cannot plan dashboards from the README alone. The framework's scope statement is worth repeating in this context: it focuses on online serving, so scheduling, autoscaling, and model versioning are left to whatever orchestrator you already run.

Where mosec is the wrong choice

The clearest limitation is scope. If you need a model registry with versioned artifacts, A/B routing between model versions, or a scheduler that packs many models onto a fleet, mosec does not provide those; it serves the worker processes you start. The second constraint is the pipeline data flow. Because data between workers moves over inter-process communication, every stage boundary is a serialization and copy. Splitting a pipeline into many small stages will cost more in transfer than it saves in overlap, and the README gives no guidance on how many stages are reasonable. Third, the batching window is a latency trade: dynamic batching only helps if requests arrive close enough together, and the README does not state default values for the batching parameters in the excerpt, so you have to read the configuration reference to know what you are accepting. Fourth, the example's batch size is implicit in the warmup list length, which is easy to get wrong and produces a warmup that does not match production batch shapes. Finally, the README requires Python 3.7 or above, but the PyPI badge in the repository describes the supported Python versions; check that badge against your runtime before committing, because a 3.7 floor in prose does not guarantee a 3.13 wheel.

Compared with TorchServe

TorchServe is the natural comparison for anyone serving PyTorch models, and the difference is in where configuration lives. TorchServe takes a packaged model archive, a .mar file, and drives inference through its own handlers and config.properties, with model versioning and management APIs built in. Mosec takes a Python class and runs it. There is no archive format to build and no management API to learn; the handler is the same code you would call in an offline test, which is the property the README emphasizes. The trade is that everything TorchServe does around the model (registration, versioning, multi-model hosting) is absent from mosec and has to come from your container orchestration. If your models are PyTorch and you want the management surface, TorchServe gives you more out of the box. If you want the handler to stay plain Python and you already have Kubernetes handling deployment, mosec's smaller surface is the point. The framework-agnostic stance in the README, with PyTorch, TensorFlow, JAX and MXNet among the repository topics, is consistent with that: nothing in the worker interface assumes a particular tensor library.

Maintenance, licensing, and what to check before adopting

Mosec is licensed Apache-2.0, which permits commercial use and modification and includes an explicit patent grant. That is a permissive licence, but it is not legal advice; if you redistribute a modified mosec, read the NOTICE and attribution requirements in the licence text yourself. The release cadence visible in the material is roughly two releases a year, with 0.9.5 in June 2025, 0.9.6 in November 2025, and 0.9.7 in April 2026, and the repository is not archived. That is a slow but non-zero cadence, and the pre-1.0 version numbers mean minor releases can carry interface changes; pin the version in your requirements file rather than tracking latest. The upgrade cost is mostly on the Rust side, where a source build needs a working Rust toolchain, so teams that install from PyPI avoid that entirely. Before adopting, verify three things: that your Python version has a published wheel, that the Server instantiation and run call shown in the full documentation match your expected entry point, and that the batching parameters in the configuration reference produce acceptable tail latency at your request rate. The stable diffusion example is the fastest end-to-end check, since it exercises warmup, msgpack responses, and a batched forward pass in one file.

Editorial conclusion

Mosec fits teams that already have a Python inference function and want batching plus a real HTTP server without writing thread pools or a queue themselves. It is the wrong tool if you need a model registry, multi-model routing, or a scheduler that places models across a cluster; mosec serves the process you start. Before adopting, verify that your installed mosec version accepts your Python version, run the stable diffusion example from the README end to end, and confirm that your max_batch_size and max_wait_time settings produce the latency you can tolerate under your own traffic.

Official sources

1. License: Apache-2.0
2. mosecorg/mosec on GitHub
3. Project website
4. README
5. Releases