FastEmbed: ONNX Embeddings Without the PyTorch Footprint
Fast, Accurate, Lightweight Python library to make State of the Art Embedding
At a glance
- What is it?
- FastEmbed is a Python embedding library from Qdrant that runs ONNX Runtime instead of PyTorch and exposes one interface for dense, sparse, late-interaction, image and reranking models. The core judgement: it fits CPU-only and serverless retrieval pipelines, but the model catalogue and the custom-model path are where the real decisions live.
- Who is it for?
- Adopt FastEmbed if you are embedding on CPU, in a serverless runtime, or inside a Qdrant retrieval stack, and you want dense, sparse, late-interaction and reranking models behind one import. Do not adopt it if you need a training loop, gradient access, or a model that is not in the supported list and has no ONNX export you can register.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 7 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The dependency problem FastEmbed is aimed at
Installing a sentence-transformer stack usually means pulling PyTorch, which the README describes as GBs of dependencies. FastEmbed replaces that with ONNX Runtime. The README states the library does not require a GPU and does not download those PyTorch packages, and names serverless runtimes such as AWS Lambda as a target. That is the whole pitch: same class of models, much smaller install, CPU execution. The audience is engineers building retrieval or RAG pipelines who care about cold-start size and per-invocation cost more than about training. If you are fine-tuning, this is not the library. FastEmbed generates embeddings and reranks; it does not train.
One import surface for five embedding shapes
The API is split by output type rather than by model family. TextEmbedding returns dense vectors, and the README's quickstart shows BAAI/bge-small-en-v1.5 producing 384-dimension float32 arrays. SparseTextEmbedding returns SparseEmbedding objects with parallel indices and values lists, which is what SPLADE-style models such as prithivida/Splade_PP_en_v1 emit. LateInteractionTextEmbedding covers ColBERT-style multi-vector output, where each document becomes a matrix rather than a single vector. ImageEmbedding takes file paths and returns one vector per image. LateInteractionMultimodalEmbedding is the ColPali case, and the README gives shapes explicitly: document images come back as (2, 1030, 128) and a text query as (1, 20, 128). Reranking lives in a separate module, fastembed.rerank.cross_encoder.TextCrossEncoder, which scores query-document pairs and returns floats. The split matters because the return types are genuinely different. A sparse embedding is not a numpy array, and a late-interaction embedding is not a single vector. Code that assumes one shape will break when you swap model classes.
Installation, GPU variant and the generator contract
Installation is pip install fastembed for CPU, or pip install fastembed-gpu when you want GPU support. The quickstart constructs TextEmbedding() with no arguments, which triggers a model download and initialization, then calls embed(documents). The README flags this twice: embed returns a generator, so you must wrap it in list() before indexing. The example calls embed twice, once to show the generator and once to materialize it, which is a slightly confusing illustration but makes the point. There is no explicit cache-path argument shown in the README, so where the downloaded ONNX files land is not documented in the material available here. Plan for a first-run download in any cold environment, and treat the model store as something your deployment has to accommodate.
Registering models the catalogue does not ship
The supported models page is described as ever-expanding, and the README provides an escape hatch for everything else. TextEmbedding.add_custom_model takes a model identifier, a pooling type (PoolingType.MEAN in the example), a normalization flag, a ModelSource, a dimension, and a model_file path. The example registers intfloat/multilingual-e5-small with sources=ModelSource(hf="intfloat/multilingual-e5-small") and model_file="onnx/model.onnx". The README notes two things about that source: it can carry a url instead of a Hugging Face id, for private storage, and model_file can point at a different ONNX export, such as a quantized onnx/model_O4.onnx, to change optimization level. TextCrossEncoder has the same mechanism with a smaller signature: model, model_file, sources. This is the most useful part of the library for anyone outside the default catalogue. It is also the part most likely to fail quietly, because you are asserting the pooling type and dimension yourself. Get pooling wrong and you get vectors that are the right shape and the wrong meaning.
Where FastEmbed is the wrong tool
Three cases stand out. First, training and fine-tuning: there is no optimizer, no loss, no gradient path in anything shown here. Second, models outside the catalogue without an ONNX export: add_custom_model expects a model_file inside the source repository, and if the upstream project only ships PyTorch weights you have to produce the ONNX file yourself, which is outside this library's scope. Third, workloads where the ONNX Runtime is not actually faster for your hardware. The README asserts ONNX Runtime is faster than PyTorch, but that is a general claim, not a number attached to a specific model and machine. Treat it as a hypothesis to check on your own hardware rather than a guarantee. The other constraint is the default model: BAAI/bge-small-en-v1.5 is English. Multilingual coverage exists, and the README mentions a few multilingual models, but you have to select one deliberately.
How this differs from Sentence-Transformers
Sentence-Transformers is the obvious comparison and the README names it directly as something FastEmbed is built to be lighter than. The difference is in what each library assumes. Sentence-Transformers is a PyTorch library: it trains, it fine-tunes, it exposes the underlying model, and it carries the PyTorch dependency. FastEmbed is an inference library: it wraps ONNX Runtime, exposes embed and rerank calls, and gives you a registration function when the catalogue falls short. If your workflow includes training a model on your own data, Sentence-Transformers is the natural home and FastEmbed is downstream of it. If your workflow is purely serving embeddings and you are paying for CPU time or cold starts, the ONNX path is the one the README argues for. The two are not mutually exclusive: you can fine-tune in one and export to ONNX for the other, which is exactly what the model_file parameter anticipates.
Maintenance, licence and what to check before adopting
The project is Apache-2.0 and is maintained by Qdrant, which the README states plainly. Releases are not on a tight cadence: v0.7.2 in August 2025, v0.7.4 in December 2025, v0.8.0 in March 2026. That is roughly a minor release every few months, with patch releases in between, and the repository is not archived. The upgrade cost is mostly model-level rather than API-level: the embed and rerank signatures shown in the README are small and stable-looking, while the model catalogue and the ONNX exports behind it are what change. Apache-2.0 covers the library code. It does not cover the model weights that get downloaded at runtime, and those carry their own licences. If you are shipping a commercial product, check the licence of each specific model you select, not just the licence of FastEmbed. The practical pre-adoption test is small: install fastembed, run the quickstart against your own documents, and confirm the vector dimension matches what your vector store expects. The README's own example asserts 384 for bge-small-en-v1.5, and that number has to line up with your collection configuration before anything else matters.
Editorial conclusion
Adopt FastEmbed if you are embedding on CPU, in a serverless runtime, or inside a Qdrant retrieval stack, and you want dense, sparse, late-interaction and reranking models behind one import. Do not adopt it if you need a training loop, gradient access, or a model that is not in the supported list and has no ONNX export you can register. Before committing, verify three things: that your target model appears on the supported models page, that the ONNX file path you pass to add_custom_model actually exists in the upstream repository, and that your deployment image can hold the model cache the first run downloads. The library is Apache-2.0, which covers the code, not the weights it fetches, so check each model's own licence separately.
Community notes