FG-CLIP 2: a bilingual fine-grained vision-language model

Two models, two papers and two conferences

FG-CLIP 2 is the second generation of a text-image cross-modal model from 360CVGroup, and the repository holds the official implementation of both generations.

The first paper is FG-CLIP: Fine-Grained Visual and Textual Alignment, on arXiv as 2505.05071, accepted at ICML 2025, with the original code kept on a v1.0 branch. The second is FG-CLIP 2: A Bilingual Fine-grained Vision-language Alignment Model, on arXiv as 2510.10921, accepted at ICML 2026.

The author list is largely the same, with Chunyu Xie and Bin Wang marked as equal contributors, Dawei Leng as the corresponding author, and Yuhui Yin on both papers, along with Fanjing Kong, Jincheng Li, Dawei Liang, Ji Ao and Gengshen Zhang.

The artefacts are published rather than described. There is a model collection on Hugging Face for the second generation, the earlier fg-clip-large model is on the hub as well, the FineHARD dataset is published for evaluation, and there is a hosted API for the base model through the research site of the company. A DeepWiki link points at a third party source walkthrough, and the project site is 360cvgroup.github.io/FG-CLIP.

The claimed result is specific: across 29 datasets and 8 task types, in both languages, the model outperforms strong baselines including SigLIP 2 and MetaCLIP 2.

Why CLIP struggles with the details

The overview in the readme is commented out in the source file, which is a small hint about how the readme has been edited over time, but it is the clearest statement of the problem and it is best read.

Fine-grained vision-language understanding requires precise alignment between visual content and linguistic descriptions, and the text says this capability remains limited in current models, particularly in non-English settings.

The specific diagnosis is that models like CLIP perform well on global alignment but struggle to capture fine-grained details in object attributes, spatial relations and linguistic expressions, with limited support for bilingual comprehension.

That is a fair characterisation of the gap, and the three axes of it are the three things the model adds. Attributes are the case where a caption says red and the image shows a different shade of red, or says striped and the object is checked. Spatial relations are where the question is not what is in the picture but what is next to what. Linguistic expression is where the same visual fact can be described in several ways and the model has to match all of them.

The bilingual point is the second half of the diagnosis. A model trained mostly on English captions tends to align well with English descriptions and poorly with Chinese ones, even when the underlying visual content is the same, and a retrieval system serving Chinese users inherits that asymmetry.

Two stages, from global semantics down to the details

The architecture section describes a two stage hierarchical learning framework, going from global semantics to fine-grained details and strengthening the visual-language alignment step by step.

The first stage is global semantic alignment. Training starts from large-scale image-text pairs, and the readme notes that each pair contains a short text description, used for a concise scene-level description, and a long text description, used for richer contextual detail. Training on this bilingual corpus gives strong global alignment and lays the foundation for cross-modal understanding in both English and Chinese.

The second stage is fine-grained visual-language learning. On top of the globally aligned representation, region-level supervision signals and several fine-grained objectives are introduced to strengthen local correspondences.

The design logic is sound and it is the standard reason to do it this way. Global alignment is a comparatively easy objective with abundant data, and it produces a representation where the image and the caption agree about the scene. Fine-grained alignment is the hard objective, and it needs supervision that points at parts of the image, which is scarce. So you learn the coarse structure first, then train the fine structure on top of a representation that already knows what the scene is about.

The long captions in stage one deserve a note of their own, because they are the bridge between the two stages. A caption that describes a scene in detail carries phrase-level information that a short caption does not, so the model learns where the objects are before it is ever told to align regions to phrases.

Four objectives, and the loss that does the work

The second stage is described as four specific techniques, and they are the part of the paper a practitioner would want to read.

Fine-grained visual learning takes region features extracted with RoIAlign and aligns them with phrase-level descriptions. That is the objective that turns a global embedding into a spatially aware one.

Fine-grained text learning uses hard negatives produced by attribute perturbation to distinguish subtle differences in text. If the positive says a red mug and the negative says a blue mug, the model has to notice the colour, because the rest of the sentence is identical.

The cross-modal ranking loss with global threshold synchronization applies a ranking loss with a dynamic margin and uses globally synchronised thresholds to keep hard negative mining stable. That is the part that stops a training run from degenerating, because hard negative mining is self-reinforcing and a threshold that is computed per example makes the objective noisy.

The textual intra-modal contrastive loss does contrastive learning within a single language, so that region descriptions that are semantically close can be told apart. The readme calls it the Textual Intra-modal Contrastive loss and says it is introduced to distinguish semantically similar captions. It is the objective that addresses the linguistic expression problem directly, because the failure mode there is two captions that mean nearly the same thing being pushed apart or, worse, being treated as equivalent when the difference matters.

Bilingual data, a Chinese benchmark and 29 datasets

The performance section makes three claims and adds a contribution of its own.

The first is the supervision, which includes region-text matching, long-caption modelling and several discriminative objectives, plus the textual intra-modal contrastive loss described above. The second is the data, a carefully curated mixture of large-scale English and Chinese data, and the claim is that this mixture is what produces the bilingual performance rather than a bilingual evaluation applied to an English model.

The third is the performance itself. Experiments on 29 datasets across 8 tasks show the model outperforming existing methods, with state of the art results in both languages, and the baselines named in the readme are SigLIP 2 and MetaCLIP 2, which are the current reference points rather than a comparison against something older.

The fourth contribution is a Chinese multimodal benchmark, presented to enable rigorous evaluation, and it contains long-caption retrieval and bounding box classification. That is the part that matters for adoption, because the absence of a Chinese benchmark is exactly why bilingual claims have been hard to compare in the past. The dataset is published on Hugging Face as FineHARD.

The tasks themselves are the usual ones for a vision-language model, with the repository topics naming cross-modal retrieval, text-image retrieval and fine-grained classification, and the readme also mentions zero-shot image classification and the dense feature display demo, which shows the embedding space rather than a score.

Installation is a conda environment and an editable install

The installation instructions are short, and they assume you will train or fine-tune rather than only run the hosted model.

conda create -n FGCLIP2 python=3.10 -y
conda activate FGCLIP2
cd FG-CLIP && pip install -e .

Python 3.10 is the floor, and the project metadata agrees, with requires-python set to greater than or equal to 3.10.

The dependency list in pyproject.toml tells you what kind of project this is. Torch and Torchvision for the model, Transformers and Datasets for the data pipeline, Accelerate for the training loop, DeepSpeed for distributed training, and Weights and Biases for experiment tracking. The presence of DeepSpeed and WandB is the tell, because a pure inference package would need neither.

The packaging itself is a plain setuptools project with the build backend at setuptools.build_meta and a requirement of setuptools 61 or newer. The package is named fgclip2 and the version is 1.1.1. The package discovery excludes the assets, benchmark, docs, dist, playground, scripts and tests directories, which is a small sign of care about what ends up in the wheel.

The repository layout follows the same split, with the source in an fgclip2 directory, data in data, the scripts in scripts, the images used in the readme in use_imgs, and both a Chinese and an English readme at the root.

A year of published artefacts

The news list in the readme is a timeline, and it is best read because it shows what the project actually shipped rather than what it described.

In May 2025 the first paper was released, the repository was updated so the model could be tested, the model was uploaded to Hugging Face as fg-clip-large, and the paper was accepted at ICML 2025. In July the project published two demos as Hugging Face Spaces, one for fine-grained retrieval and one for a dense feature display, and later in the same month it opened API access to the base model of the second generation through the research site.

In September 2025 the team open-sourced an MCP server implementation for the model, in a separate repository. In October the same month came the second paper, together with the code and the model weights. And on 2026-05-01 FG-CLIP 2 was accepted at ICML 2026.

The last push to the repository was on 2026-06-16, and there are no GitHub releases, so the weights and the code are the distribution mechanism rather than a tagged archive.

That pattern, paper and weights and demo and API and an MCP server, is the shape of a research project that wants to be used rather than merely cited, and it is the strongest argument for looking at the second generation rather than the first, since the bilingual data, the Chinese benchmark and the newer objectives are all in it.

Editorial conclusion

FG-CLIP 2 fits a team that needs retrieval or classification where the difference between two visually similar items is the whole point, an aircraft variant in a photo or two similar products in a catalogue, because that is the capability the model was built for and the benchmark suite was built around. It is the wrong choice if you need a small embedding model you can run on a laptop, since the recipe assumes the deepspeed and transformer training stack and the weights are the large ones, and it is the wrong choice if you need a model in a language other than English or Chinese, because the bilingual claim covers exactly those two. Before you adopt it, decide whether you need the first generation or the second, since the repository keeps the original on a v1.0 code branch, and note that the fine-grained training objectives are only useful if your own data has region-level or phrase-level annotations.

Frequently asked questions

Official sources

1. 360CVGroup/FG-CLIP on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README