clip-retrieval: A Pipeline for CLIP Embeddings and kNN Search

The gap clip-retrieval fills between CLIP and a search box

CLIP gives you an embedding function. It does not give you an index, a filter, a query API, or a UI. The distance between a working CLIP model and a searchable collection of images is where most teams lose time, because each layer has its own conventions: how embeddings are stored, how approximate nearest neighbour indices are built, how a query is turned into a result list with captions and URLs attached. clip-retrieval exists to collapse that distance. The README describes the project as a way to "easily compute clip embeddings and build a clip retrieval system with them", and the component list backs that up: inference computes embeddings, index builds indices, filter prunes data using the index, back hosts the indices behind a Flask service, front is a UI querying the back. The target user is an engineer with a dataset of image URLs and captions who wants semantic search over it without writing the plumbing. The project also ships clip end2end, which chains img2dataset, inference, index, back, and front in one command. That is the clearest statement of intent: the maintainers expect people to start from an existing URL dataset, not from a folder of local images.

How the five components hand data to each other

The pipeline is linear and file-based. img2dataset downloads images from URLs and writes them to disk alongside captions. clip inference then reads that data and produces embeddings, at a rate the README puts at 1500 samples per second on a 3080. Those embeddings are what clip index consumes to build an efficient index. clip filter sits slightly to the side: it uses the index to filter out data, which is useful when you want to remove near-duplicates or unwanted content before serving. clip back loads the index and exposes it over HTTP, and clip front is the browser UI that queries that HTTP endpoint. The ClipClient class is the programmatic entry point to the same backend, accepting a backend_url and an indice_name and returning dictionaries with url, caption, id, and similarity fields. That result shape tells you what the index stores: enough metadata to reconstruct a result card, not the image bytes themselves. The architecture is deliberately boring, and that is the point. There is no message queue and no distributed coordination. Each stage reads files and writes files, which means you can rerun a single stage without rebuilding everything upstream of it.

Running end2end and what the flags actually control

Installation is one command: pip install clip-retrieval. The README points anyone targeting the laion5B index at a separate document, docs/laion5B_back.md, which is worth reading before assuming the default path applies to you. For a first run, the end2end command takes a parquet file of image URLs and captions plus an output directory. The README's example downloads a test file and runs it: wget https://github.com/rom1504/img2dataset/raw/main/tests/test_files/test_1000.parquet, then clip-retrieval end2end test_1000.parquet /tmp/my_output. After that, the UI is at http://localhost:1234. Two flags are documented. The first is --run_back False, which skips starting the backend, useful when you only want the index built. The second is an environment variable rather than a flag: export CUDA_VISIBLE_DEVICES= to avoid using your GPU when VRAM is insufficient. That is a telling detail. The default path assumes a GPU, and the escape hatch is to hide the GPU from the process entirely rather than to select a smaller model. On the client side, ClipClient exposes parameters that shape results: aesthetic_score and aesthetic_weight, use_mclip for a multilingual CLIP variant, modality to search images or text, num_images, deduplicate, use_safety_model, and use_violence_detector. The last three default to true, which means a default query is already doing deduplication and safety filtering before you see anything.

The GPU assumption and the VRAM escape hatch

The README's performance figures, 1500 samples per second and 100M text plus image embeddings in 20 hours, are stated for a 3080. Those numbers are the project's own claims, not independent measurements, and they describe embedding computation specifically, not index build time or query latency. The practical constraint is memory. The documented workaround for insufficient VRAM is to disable GPU use with export CUDA_VISIBLE_DEVICES=, which pushes the work onto the CPU and changes the throughput picture entirely. If you are processing a large collection and your GPU is small, the pipeline still runs, but the 20-hour figure no longer describes your situation and the README does not offer a CPU-side estimate. This is the kind of gap that matters during planning: you can size the GPU step from the README, but you cannot size the fallback from it. The index build step has its own resource profile, and the README does not quantify it. Anyone budgeting infrastructure should treat the index stage as unmeasured until they run it on their own data.

Where this is the wrong tool

clip-retrieval assumes your data starts as URLs and captions in a parquet file. If your images are already on local disk with a different metadata format, the end2end path does not fit, and you would be using inference, index, and back as separate pieces with your own glue. If you need a managed vector database with replication, sharding, and an SLA, this is not that. clip back is described as a simple Flask service, and simple is the operative word: there is no discussion in the README of authentication, rate limiting, or horizontal scaling of the service itself. Exposing clip back to the public internet without putting something in front of it is a decision the documentation does not address. There is also a metadata coupling worth noting. ClipClient returns url, caption, id, and similarity. If your search results need to carry additional fields, such as licence information or provenance, the index and the client result shape are not obviously built to carry them, and the README does not describe how to extend that payload. For a research prototype or an internal tool, none of this is disqualifying. For a product where result records must carry audit fields, it is a real constraint.

How it compares to assembling the stack yourself

The alternative is not a single competing product but a composition: open_clip or all_clip to load the model, autofaiss to build the index, img2dataset to fetch images, and your own service layer on top. The README itself lists all_clip, img2dataset, open_clip, and CLIP_benchmark as related projects, and notes that autofaiss uses clip-retrieval to display an example of use. That is the honest picture: clip-retrieval is partly a wrapper over tools from the same author and the same ecosystem. The difference in approach is integration versus control. Assembling the stack yourself means choosing each index type, each serving framework, and each storage format, and owning the interfaces between them. clip-retrieval means accepting its choices, in exchange for the end2end command working out of the box. If your requirements match the default path, the wrapper saves real time. If they diverge, for example if you need a specific index configuration or a serving layer with middleware, you will be reaching past the wrapper anyway, and the question becomes whether the wrapper's conventions help or hinder at that point.

Licence, releases, and what to verify before adopting

The repository is MIT licensed, which is permissive for the code. That licence does not extend to the CLIP model weights you pair with it, and the README does not make claims about model licensing. Since the pipeline is model-agnostic in principle, the licence terms of your chosen checkpoint are a separate question you need to answer yourself. The release cadence is uneven. Version 2.44.0 and 2.43.0 both landed on 2024-01-13, and 2.45.0 followed on 2025-08-15, with the last push to the repository dated 2026-03-28. That pattern, a burst of releases followed by a long quiet period and then a single release, suggests maintenance is intermittent rather than continuous. For a tool that sits in a data pipeline, intermittent maintenance is workable as long as the interfaces you depend on are stable, but it argues against building on undocumented internals. The primary language listed is Jupyter Notebook, which reflects the notebook-first documentation style rather than the shape of the shipped package. Before adopting, run one end2end pass on a small parquet file, inspect the on-disk layout the index step produces, and confirm that the ClipClient result fields cover what your application needs to display. Those three checks will tell you more about fit than any further reading.

Editorial conclusion

Adopt clip-retrieval if you already have image URLs and captions and want a local semantic search service without assembling the embedding, index, and serving layers yourself. Do not adopt it if you need a hosted vector database with managed scaling, or if your data does not fit the parquet-of-URLs shape that img2dataset and the end2end command assume. Before committing, verify that your GPU has enough VRAM for the chosen CLIP model, confirm the licence terms of the model weights you pair with the MIT-licensed code, and test one end2end run on a small parquet file to see the actual disk layout the index step produces.

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. rom1504/clip-retrieval on GitHub