Infinity: A Multi-Backend Serving Engine for Embedding and Reranking Models
Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali
At a glance
- What is it?
- Infinity is a REST API server for text embeddings, rerankers, CLIP, and ColPali, built on PyTorch, ONNX, and CTranslate2. It targets teams that need to mix models and hardware, but its complexity and release cadence demand scrutiny.
- Who is it for?
- Adopt Infinity if you need to serve multiple embedding or reranking models behind one OpenAI-compatible API, especially across heterogeneous hardware like CUDA, ROCm, or CPU. Skip it if you only need a single model on one GPU, where lighter tools like TEI or a direct sentence-transformers service may suffice.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 176 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 16, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What Infinity Actually Serves
Infinity is a REST API server, not a library you call in-process. It exposes HTTP endpoints for text embeddings, reranking, and multi-modal models like CLIP, CLAP, and ColPali. The README positions it as a high-throughput, low-latency engine, and the design choices, dynamic batching and dedicated tokenization worker threads, support that claim. The target user is an engineer who runs model inference as a service, not someone who wants to embed a few strings inside a script. The project explicitly aligns its API with OpenAI's embeddings spec, so clients written for OpenAI can point at Infinity without rewriting. That alone makes it attractive for teams already on the OpenAI ecosystem but wanting open-weight models.
The Multi-Backend Architecture
Infinity does not run one inference stack. It builds on PyTorch, optimum (which wraps ONNX and TensorRT), and CTranslate2. FlashAttention is used where available. This is a deliberate bet: different models and accelerators perform best on different runtimes, and Infinity tries to abstract that choice behind a single API. The README lists NVIDIA CUDA, AMD ROCm, CPU, AWS Inferentia 2, and Apple MPS as supported accelerator targets. That breadth is rare. Most serving engines commit to one vendor or one runtime. Infinity's approach means you could run the same model on a CPU during development and on a GPU in production, or mix a CTranslate2-optimized model with a PyTorch-only model in one server. The cost is operational complexity. Each backend has its own dependencies, tuning parameters, and failure modes. The README does not explain how the server selects a backend for a given model, which is a gap you will need to fill by reading the source or the full docs.
Getting It Running: CLI and Docker
The simplest path is pip. The README shows `pip install infinity-emb[all]`, then `infinity_emb v2 --model-id BAAI/bge-small-en-v1.5`. The `v2` subcommand is the current CLI, and it accepts arguments via environment variables as well as flags. A `--help` flag lists all parameters. For production, the README recommends a pre-built Docker image `michaelf34/infinity`, which avoids dependency conflicts on the host. The image expects you to mount your accelerator, such as installing `nvidia-docker` for GPU access. The README also mentions that the CLI v2 supports launching multiple models with `--api-key`. That suggests you can serve several models from one process, each behind an API key. The exact syntax for multi-model launch is not in the excerpt, so you will need the docs. The project also ships a separate client package, `infinity_client`, installable via pip, which is useful if you do not want to hand-roll HTTP calls.
Multi-Modal and Multi-Model Orchestration
Beyond text embeddings, Infinity handles reranking models, CLIP, CLAP, and ColPali. CLIP and CLAP are multi-modal encoders, meaning the server must process images and audio as inputs, not just text. ColPali is a late-interaction model for document retrieval, which produces per-token embeddings. Supporting all these in one server is not trivial. Each model type has a different input schema and output shape. The README claims Infinity orchestrates them, meaning you can mix and match models in a single deployment. For a RAG pipeline that needs a bi-encoder for retrieval, a cross-encoder for reranking, and a ColPali model for document images, one Infinity server could replace three separate services. That is a concrete operational win, fewer containers, fewer ports, one API to secure. The trade-off is that the server becomes a monolith where a bug in one model's backend could affect the others. The README does not describe isolation guarantees between models, so you should assume they share the process.
Performance Features: Dynamic Batching and Tokenization Workers
The README states that Infinity uses dynamic batching and tokenization dedicated in worker threads. Dynamic batching means requests arriving at slightly different times are grouped into a single inference pass, which raises throughput on GPU. Tokenization in separate threads prevents CPU-bound text processing from stalling the GPU pipeline. These are standard techniques in serving engines like Triton or TEI, but their presence in Infinity is notable because the project is younger and smaller. The README also mentions experimental int8 (on CPU and CUDA) and fp8 (on H100 and MI300) support. That indicates a focus on squeezing more tokens per second from expensive accelerators. However, the word experimental is a warning. You should not adopt int8 or fp8 for a production workload without thorough validation of embedding quality, because quantization can degrade retrieval accuracy. The README does not provide benchmark numbers, so any performance claim you see elsewhere must be verified on your own models and hardware.
Licence and Maintenance Reality
Infinity is MIT-licensed, which is permissive. You can embed it in commercial products, modify it, and not share your changes. That is a clear advantage over GPL or AGPL alternatives. The project is actively developed. The latest release is 0.0.77 from August 2025, following 0.0.76 in March 2025 and 0.0.75 in January 2025. That is three releases in eight months, but the version number is still below 1.0. The README's news section lists features added in 2024 and 2025, including Blackwell support in July 2025 and AMD/CPU/ONNX Docker images in late 2024. This cadence suggests a project that is iterating quickly, but it also implies breaking changes are possible. The README does not mention a migration guide or a stable API promise. You should pin the exact version you deploy and read the changelog before upgrading. The project has a DOI and is cited in academic contexts, which suggests some level of community trust, but that is not a quality guarantee.
Limitations and Wrong-Tool Cases
Infinity is a server. If you need to embed text in a batch job that runs once a day, standing up a server is overkill. You would be better off calling a model directly with sentence-transformers or a similar library. The README does not claim to be a lightweight embedder; it is built for sustained, concurrent traffic. Another limitation is hardware support breadth. While the README lists many accelerators, not every backend works with every model. A model exported to ONNX may not support dynamic shapes that Infinity's batching requires, and a TensorRT engine is hardware-specific. The README does not explain these constraints. You will need to test your exact model on your exact accelerator. Also, the project's focus on embeddings and rerankers means it does not serve generative LLMs. If you need both embeddings and text generation from one server, Infinity is not the tool. Finally, the README mentions an API key option, but it does not discuss rate limiting, authentication beyond a static key, or multi-tenant isolation. For a public-facing service, you will need a reverse proxy.
Alternatives and How They Differ
The most direct alternative is Hugging Face's Text Embeddings Inference (TEI). TEI also serves embeddings and rerankers with dynamic batching and OpenAI-compatible endpoints. The key difference is that TEI is tightly coupled to Hugging Face's ecosystem and typically uses PyTorch or custom CUDA kernels. Infinity explicitly supports multiple backends, including CTranslate2 and ONNX, which can be faster on CPU or specific accelerators like AWS Inferentia. Another alternative is to run a model directly with FastAPI and sentence-transformers, which gives you full control but requires you to implement batching, queueing, and API compatibility yourself. Infinity packages those concerns. If you already use a vector database like Qdrant or Weaviate, some of them offer built-in embedding services, but those are usually less flexible about model choice. Infinity's multi-model orchestration is its strongest differentiator against TEI, which typically runs one model per container. For a deployment that needs CLIP and a reranker behind one port, Infinity has a clear edge.
Editorial conclusion
Adopt Infinity if you need to serve multiple embedding or reranking models behind one OpenAI-compatible API, especially across heterogeneous hardware like CUDA, ROCm, or CPU. Skip it if you only need a single model on one GPU, where lighter tools like TEI or a direct sentence-transformers service may suffice. Before committing, verify that your exact model and backend combination (e.g., ONNX, TensorRT, or int8) is supported in the current release, and check the changelog for recent breaking changes, given the rapid version bumps from 0.0.75 to 0.0.77 in under a year.
Community notes