clip.cpp: CLIP inference in plain C with a 4-bit model on disk

What clip.cpp is and the device it is aimed at

clip.cpp is a dependency free implementation of the well known CLIP model from OpenAI, and the credit goes to the work in GGML.

The stated motivation is deployment, not research. CLIP is deployed for several tasks, from semantic image search to zero-shot image labelling, and it is also a part of Stable Diffusion and of the recently emerging field of large multimodal models. The repository is aimed at powering useful applications based on such models on computation- or memory-constrained devices. The headline number is that 4-bit quantized CLIP is only 85.6 MB.

That number is the whole argument. A CLIP text and image encoder is a large model, and running one on a device with a few hundred megabytes of usable memory, or in a serverless function where the cold start has to be small, rules out the usual Python stack before any question of speed arises.

The features list names the reasons: dependency-free and lightweight inference thanks to ggml, 4-bit, 5-bit and 8-bit quantization support, support for text-only, vision-only and two-tower model variants, and a dependency free Python binding that relies on no large third-party packages, with no need for NumPy, TensorFlow, PyTorch or ONNX, nothing more than the standard Python library.

The short startup time compared with large ML frameworks is called out separately, and that is what makes it suitable for serverless deployments where the cold start is an issue. The users are people embedding a multimodal model inside a device or a service, and the C API is deliberately C-compatible so it can be called from almost anywhere.

Building the project and where the binaries land

The build is CMake, and the clone has to bring the submodules with it because ggml is vendored:

git clone --recurse-submodules https://github.com/monatis/clip.cpp.git

cd clip.cpp

mkdir build

cd build

cmake -DCLIP_NATIVE=ON ..

make

The binaries end up in the ./bin directory. The CLIP_NATIVE=ON option is the one worth noticing, since it is what turns on the CPU-specific paths rather than a portable fallback.

There is a known wrinkle on Apple hardware that the documentation is honest about. Some Mac devices report x86_64 instead of arm64 architecture, and the author points at a discussion in the whisper.cpp repository for that case, adding that they could not reproduce it on a Macbook M2 pro and so cannot help further, and inviting a solution to include in CMakeLists.txt. That is a real gap: a cross-platform project with an unresolved architecture detection issue on a popular platform.

The repository tree shows the scope of the code. There is clip.cpp and clip.h as the library, a ggml/ directory for the vendored tensor library, a models/ directory holding the conversion script, stb_image.h for image loading, a scripts/ directory, a tests/ directory and an examples/ directory. The examples include main.cpp, zsl.cpp, extract.cpp, a shared common-clip.cpp and common-clip.h, a simple.c file which is the C API example, an image-search/ example, and a python_bindings/ directory.

Getting a model: GGUF files and the conversion script

The model file format is the biggest constraint in the project, and it comes with an explicit breaking change warning.

As of 09/27/2023 clip.cpp uses a new model file structure in GGUF format, and the documentation states there will be no back compatibility support for older models, that the latest commit compatible with older .bin files is a specific hash, and that users are recommended to update to the new code and model files. The stated reason is that the new structure allows text-only and vision-only model files as well as 32-bit floating point precision, 5-bit and 8-bit quantization support in addition to the existing 16-bit floating point precision and 4-bit quantization.

That last point is the interesting one. The text-only and vision-only variants are the same architectural idea as the two-tower and single-modality support in the features list, and having them as separate files is what makes a device that only needs to embed text, or only images, able to skip the other tower entirely.

Preconverted models are published on Hugging Face in repositories tagged with clip-cpp-gguf, and the documentation says to download a model of your choice from that link and then jump to the building section. If you want to convert it yourself, the steps are: clone the model repo from the HF Hub with git lfs init and a git clone of a LAION repository, clone this repository with its submodules and change into the models directory, install pytorch and transformers with pip install -r requirements.txt unless you already have them, and run the conversion script:

python convert_hf_to_gguf.py -m ../../CLIP-ViT-B-32-laion2B-s34B-b79K

The output file lands in the model directory named in the command. Other options for the script are listed by running it with --help.

On which model to convert, the documentation has a recommendation: LAION's models outperform OpenAI's models in several benchmarks, so they are recommended.

Quantization types and the practical size trade

Quantization is the mechanism that makes the memory constraint work, and the supported types are named precisely.

The project supports q4_0, q4_1, q5_0, q5_1 and q8_0 quantization types. A model in f32, which is the recommended source format, or in f16, can be quantized into one of these with the ./bin/quantize binary, and the quantize usage output lists the type numbers: 2 for q4_0, 3 for q4_1, 6 for q5_0, 7 for q5_1 and 8 for q8_0.

The example given for converting to q5_1 is:

./bin/quantize ./CLIP-ViT-B-32-laion2B-s34B-b79K/ggml-model-f32.gguf ./CLIP-ViT-B-32-laion2B-s34B-b79K/ggml-model-q5_1.gguf 7

After that, the quantized model file is used the same way as the F16 model.

Two things follow from the design. The quantization is applied to a model you already hold locally, which is why the conversion path needs PyTorch and Transformers even though inference does not, and why preconverted files on Hugging Face are the fast route. And the 4-bit format named in the features list as 85.6 MB is the aggressive end of the range, where the accuracy cost is largest.

The two-tier structure is worth noting for a deployment. A conversion step that needs a full Python ML stack runs once, on a machine with the resources, and the result is a small file that the device itself only has to read.

The examples: API demo, zero-shot labelling and image search

The documentation says there are currently four examples, naming main, zsl and image-search.

main is for demonstrating the usage of the API and can optionally print out some verbose timings, and it calculates the similarity between one image and one text. It is the reference for how the C API is called.

zsl is the zero-shot labelling example, and it matters that it was updated to match Huggingface's zero-shot behavior in the zero-shot pipeline. That change came with the note that all functions are C-compatible now, so the same example serves the C audience.

extract is the fourth, and its presence in the tree suggests it is about pulling the embedding out of a model rather than comparing two things.

The image-search example has its own directory with a README, and the changelog entry that introduced it dates from 07/11/2023 and describes it as a semantic image search example directly in C++. That is the most complete demonstration of the library's purpose, since it is the task the whole design is aimed at.

The Python bindings live in examples/python_bindings/, and the announcement that introduced them is dated 09/11/2023. The claim is a dependency free binding without relying on any large third-party packages, with nothing more than the standard Python library, which means no NumPy, no TensorFlow, no PyTorch and no ONNX on the Python side either. There are also Clojure bindings available in a separate project, announced on 01/27/2024, which tells you the C API is genuinely portable rather than Rust-shaped.

Batch inference support for image encoding arrived on 07/12/2023, and given the device focus, that is a feature that matters as much as the quantization.

The image preprocessing caveat that decides your evaluation

There is one note in the documentation that deserves to be read before anything else, because it is where a project like this is judged.

PIL uses a two-pass convolutions-based bicubic interpolation when resizing with antialiasing applied. In Pytorch, antialiasing is optional. The documentation says this needs some extra attention to implement this preprocessing logic so that it matches their results numerically.

But then the author draws a practical conclusion: linear interpolation is also good enough both for comparison of different embeddings from this implementation and for comparison of an embedding from this implementation against another one from Transformers. So the decision is to use linear interpolation until a proper bicubic interpolation is crafted.

That is a deliberate, documented deviation, and it has two consequences. The first is that a zero-shot labelling pipeline built on this library may not produce the same labels as the reference implementation, because the image the encoder sees is not bit-identical to the image PyTorch would have produced. The second is that embeddings from this implementation and from Transformers are comparable to each other but are not identical to the reference, so any workflow that mixes the two has to tolerate a difference.

The author is refreshingly explicit that the match is approximate rather than exact, and the project publishes this rather than burying it. For a device deployment, where the reference stack is not present anyway, that is usually the right trade. For a research setting where you intend to reproduce published numbers, it is not.

Where clip.cpp is the wrong tool

The first limit is the model family. This is CLIP, and the project is about running CLIP in C. The hot topics list points at a separate project, ggmlc, described as an ML compiler to convert any PyTorch or JAX model to GGML and GGUF automatically and run it anywhere without manually implementing the model graph in C++. That is a pointer towards the general case, but clip.cpp itself is the CLIP implementation.

The second is the numerical difference described above. If your requirement is to match a reference implementation's output exactly, this is the wrong library, and the project says so itself.

The third is the currency of the model format. The move to GGUF in 2023 was a breaking change with no back compatibility, and the project is explicit that users are recommended to update code and model files together. A device fleet that was deployed on the old .bin format needs a coordinated migration.

The fourth is the platform question on Mac hardware, where the architecture detection issue is unresolved.

Compare that with using the Python stack directly. The difference in approach is that PyTorch with Transformers gives you the reference implementation, every architecture, and the exact preprocessing, and it costs you hundreds of megabytes of dependencies and a startup time measured in seconds. clip.cpp inverts that: one small model file, a C API, a fast start, and a documented deviation from the reference numbers. If you are shipping to a device, that inversion is the entire value. If you are doing research on a workstation, the Python stack is the better tool and this one is the wrong choice.

Licence, project state and what to check first

clip.cpp is released under the MIT licence, with the text in the LICENSE file at the repository root. The MIT terms are permissive, so embedding it in a commercial product carries only the obligation to keep the notice, and you should read the LICENSE file yourself for the exact wording.

The repository is not archived, and the last push was on 2026-08-24, so the tree is about two months old relative to today. It publishes no GitHub releases, so there is no tagged version to pin, and the version tracking is in the source tree and the commit history.

The most recent entry in the hot topics list is also the most recent push. On 08/24/2026 the author announced a new GGML-based project, ggmlc, an ML compiler that converts any PyTorch or JAX model to GGML and GGUF automatically and runs it anywhere without manually implementing the model graph in C++. That is worth knowing, because it suggests the direction of travel for the project: a general model runner rather than a CLIP-specific one, and a C and C++ implementation of a model graph in C++ is exactly the manual work that project aims to remove.

What to check first is concrete. Pull a preconverted model tagged clip-cpp-gguf from Hugging Face, build the project with cmake -DCLIP_NATIVE=ON, and run the main example against one image and one text to see the similarity output. Then run the image-search example, since that is the workflow the library is designed for. Only after that, read the preprocessing note and decide whether the linear interpolation deviation is acceptable for your use, because that decision, not the raw speed, is what determines whether the project is a good fit.

Editorial conclusion

clip.cpp fits a developer who needs CLIP embeddings on a device that cannot carry a Python ML stack, or who wants a serverless deployment without a cold start penalty, and the 4-bit model at 85.6 MB is small enough to embed. It does not fit a team that needs the newest architectures or the exact numerical output of the reference stack, because the image preprocessing deliberately differs from the reference implementation. Before you adopt it, read the note about image preprocessing and measure the embedding difference yourself, and check the GGUF model format requirement, since the change to that format is a breaking change with no back compatibility for older .bin models.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. monatis/clip.cpp on GitHub
4. README