Chinese-CLIP: a Chinese-native CLIP trained on roughly 200 million image-text pairs

The gap Chinese-CLIP fills: CLIP's text tower does not read Chinese

CLIP-style models map images and text into one embedding space, which lets you rank images against a caption or score a label against a picture without training a classifier. The original CLIP was trained on English text. Point it at Chinese captions and the text encoder is operating on tokens it barely saw during pretraining, so similarity scores degrade in ways that are hard to predict per query. Chinese-CLIP addresses that directly: the README describes it as the Chinese version of CLIP, trained on large-scale Chinese data of roughly 200 million image-text pairs, and aimed at Chinese image-text feature and similarity computation, cross-modal retrieval, and zero-shot image classification. The intended user is an engineer with a Chinese image or caption corpus who wants embeddings rather than a bespoke supervised model. The repository is built on the open_clip project, with changes aimed at Chinese-domain data and at getting better results on Chinese data. That provenance matters: if you already know open_clip's training loop and config style, the structure here will look familiar rather than novel.

Five checkpoints, and what the scale difference actually costs you

The README lists five released scales. chinese-clip-rn50 pairs a ResNet50 vision tower (38M parameters) with an RBT3 text tower (39M), 77M total, at 224 resolution. chinese-clip-vit-base-patch16 uses ViT-B/16 (86M) with RoBERTa-wwm-Base (102M), 188M total. chinese-clip-vit-large-patch14 uses ViT-L/14 (304M) with RoBERTa-wwm-Base, 406M total. chinese-clip-vit-large-patch14-336px is the same pairing at 336 resolution, 407M. chinese-clip-vit-huge-patch14 pairs ViT-H/14 (632M) with RoBERTa-wwm-Large (326M), 958M total. Two structural details are worth noticing. First, the text tower is not scaled in step with the vision tower: the three largest checkpoints all use the same 102M RoBERTa-wwm-Base text encoder, and only the huge model moves to a 326M text encoder. Second, the 336px variant differs from the 224px large model only in input resolution, which is a cheap way to trade inference cost for finer detail. Each checkpoint is published on both Hugging Face Hub and ModelScope, so you can pick the host that fits your network. The README says detailed per-scale metrics live in Results.md; only baseline and best-scale numbers appear in the README itself.

How the two towers meet: contrastive training, and the knobs the README exposes

The architecture is a dual-encoder contrastive setup. An image encoder and a text encoder each produce a vector, and training pulls matched image-text pairs together while pushing mismatched pairs apart. Chinese-CLIP keeps that shape and swaps in Chinese-capable text encoders (RBT3, RoBERTa-wwm-Base, RoBERTa-wwm-Large) plus Chinese training data. The repository's changelog shows which training-side controls were added over time, and those are the practical levers. Gradient accumulation support was added on 2023-03-20, described as simulating the effect of a larger batch size, which matters because contrastive training quality depends heavily on how many negatives sit in each batch; if your GPU cannot hold a large batch, accumulation is the documented workaround. FlashAttention support arrived 2023-02-16, described as improving training speed and reducing memory use, with details in flash_attention.md. The FLIP training strategy was implemented 2022-12-12 and can be activated during finetune. PyTorch 2.0 adaptation landed 2023-05-09. A knowledge-distillation finetuning feature based on the ModelScope library was added 2023-09-08. Read that list as the set of things the maintainers considered worth documenting; anything not on it (learning-rate schedules, augmentation policy, negative-sampling tricks) is not described in the supplied material, so treat the training recipe as partially documented.

Getting it running: the documented entry points

The README points at three code paths. For embeddings there is a fast API for image-text feature extraction, added 2022-07-13, which the README says lets you call the Chinese CLIP model in a few lines to compute features and similarity. For retrieval there is the cross-modal retrieval code, part of the original 2022-07-08 open-source release. For classification there is zero-shot image classification code, added 2022-11-22, which the README ties to the ELEVATER benchmark. Weights load through the Hugging Face Hub and ModelScope links in the model table, and the README notes that the model code and feature-extraction API were merged into Hugging Face transformers on 2022-12-01, so a transformers-based path exists alongside the in-repo code. Deployment is covered by separate documents: deployment.md for ONNX and TensorRT, including pretrained TensorRT models, added 2023-01-15; and a PyTorch-to-CoreML conversion script at cn_clip/deploy/pytorch_to_coreml.py, added 2023-11-30. The precise command lines and config keys are not reproduced in the README excerpt supplied here, so you will need to read those linked documents before you can write a working invocation. That is a real friction point, not a detail.

Where the evidence is thin, and where the project is the wrong tool

The published comparison tables are the strongest claim in the repository, and they are also the most narrow. On MUGE text-to-image retrieval, the README reports CN-CLIP at 63.0 R@1 zero-shot and 68.9 after finetuning, against Wukong at 42.7 and 52.7 and R2D2 at 49.5 and 60.1. On Flickr30K-CN, the visible portion of the table shows Wukong at 51.7 R@1 zero-shot for text-to-image, with the CN-CLIP row cut off in the supplied excerpt. These are retrieval benchmarks on specific Chinese datasets; they say nothing about your domain, and the README itself defers per-scale detail to Results.md. The repository is also primarily Jupyter Notebook, which shapes what you inherit: notebook workflows are good for reproducing an experiment and awkward for wiring into a service, and there are no releases retrieved for the project, so versioning is by commit rather than by tagged artifact. If you need a pip-installable library with semantic versioning and a deprecation policy, this is the wrong shape. If your corpus is predominantly English, the Chinese text encoders are overhead with no matching benefit, and you should use an English CLIP variant instead. And if you need a model that handles long captions or document-level text, nothing in the supplied material suggests the text tower was built for that.

Alternatives: Wukong and R2D2 as the comparison points the README itself uses

The README benchmarks Chinese-CLIP against Wukong and R2D2, so those are the honest alternatives to weigh rather than an invented competitor. The difference visible in the material is in the reported numbers, not in a described architectural divergence: on MUGE text-to-image retrieval, Chinese-CLIP's zero-shot R@1 of 63.0 sits well above Wukong's 42.7 and R2D2's 49.5, and the finetuned gap persists at 68.9 versus 52.7 and 60.1. What the README does not give is a mechanism-level explanation for why, beyond training on large-scale Chinese data and optimizations for Chinese data, so you cannot reason from first principles about which model will win on your corpus. The practical consequence is that the choice should be made empirically on your own sample, not from the table. A second alternative sits inside the ecosystem: the model code was merged into Hugging Face transformers, so if you already run transformers in production, loading the checkpoint through that path may cost you less integration work than adopting the in-repo code, at the price of depending on the transformers implementation rather than the repository's own scripts.

Maintenance, licence and what to check before you build on it

The project page lists the licence as MIT and the repository is not archived, with a last push of 2026-03-31. MIT is permissive: it allows commercial use and modification provided the copyright notice and permission notice are preserved, but this is a description of the licence identifier, not legal advice, and you should read the LICENSE file in the repository yourself, particularly because the pretrained weights are hosted separately on Hugging Face Hub and ModelScope and may carry their own terms. The changelog gives a rough sense of maintenance rhythm: activity clusters in 2022 and 2023, with the most recent entries in the visible list being the CoreML conversion script in November 2023 and the distillation finetuning feature in September 2023. There are no retrieved releases, so there is no tagged version to pin against and no changelog-driven upgrade path; upgrading means moving to a newer commit and re-reading the affected documents. The upgrade cost that the README does make visible is on the deployment side: ONNX and TensorRT export, plus a separate CoreML conversion script, means each target platform is a distinct path to validate, and the TensorRT route additionally assumes NVIDIA hardware. Budget for that work if you plan to serve the model rather than just embed it in a notebook.

Editorial conclusion

Adopt Chinese-CLIP if your content is Chinese text paired with images and you need embeddings for retrieval, similarity scoring or zero-shot labels, and you can accept a repository whose working examples live in notebooks. Do not adopt it if you need a single maintained Python package with a stable API surface, or if your corpus is mostly English, where the Chinese text encoder buys you nothing. Before committing, verify the exact checkpoint name against the table in the README, confirm the licence file in the repository matches the MIT identifier shown on the project page, and run the retrieval or zero-shot script on your own sample to see whether the published MUGE and Flickr30K-CN numbers transfer to your data.

Official sources

1. Issues
2. License: MIT
3. OFA-Sys/Chinese-CLIP on GitHub
4. README