TIPS is a vision-language encoder built for spatial awareness, not just retrieval

Two papers, one repository, and a v2 that supersedes v1

The repository contains the implementation and models introduced in two papers. TIPSv2 is Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment, from CVPR 2026. TIPS is Text-Image Pretraining with Spatial Awareness, from ICLR 2025. The primary language of the repository is Jupyter Notebook, which is a signal about what kind of artefact this is: notebooks and model definitions rather than a library with an installable API.

The name expands to TIPS, which stands for Text-Image Pretraining with Spatial Awareness, and the series is described as foundational image-text encoders built for general-purpose computer vision and multimodal applications. The README states the models were validated on a comprehensive suite of 9 tasks and 20 datasets, displaying performance that matches or exceeds other recent vision encoders, with particularly strong spatial awareness.

That last clause is the differentiator. A general purpose contrastive image-text encoder learns that a photo of a dog matches the caption a dog, which is enough for retrieval and not much else. A model that also encodes where things are in the frame can support segmentation, depth and normals, and the numbers in the model tables bear that out: the TIPSv2 table reports a PASCAL segmentation column, an NYU depth column, an ImageNet-KNN column, two Flickr retrieval columns and an ADE150 zero-shot column.

The README recommends using the latest version, TIPSv2, but still provides the earlier TIPSv1 for completeness. Both have their own model tables, and the v1 table reports a slightly different benchmark set, including a UNED-KNN column that v2 does not list.

The project page, the two arXiv papers, a Hugging Face demo space and a Hugging Face model collection are all linked from the README, and the last push was on 2026-08-31 with the repository not archived and no GitHub releases.

Separate vision and text towers in four sizes

The model tables are the most informative part of the README, because they show that TIPS is not a single monolithic checkpoint but a family with separate vision and text towers at four sizes.

The TIPSv2 rows are given with the parameters of the vision tower and the text tower, and each row links to both a PyTorch and a Jax checkpoint for each tower. The g/14 model has 1.1B vision parameters and 389.1M text parameters, and it is the largest. The SO/14 model splits 412.4M vision and 448.3M text, which is the one size where the text tower is larger than the vision tower. The L/14 model has 303.2M and 183.9M. The smallest, B/14, has 85.7M and 109.6M.

That last pairing is worth a second look. At B/14 scale the text tower is bigger than the vision tower, and the ADE150 zero-shot column is highest for L/14 at 24.7 against 17.8 for g/14, so the largest model is not the best on every metric.

The four sizes give a real choice rather than a single default. A team running retrieval over a large image corpus will pick on the vision tower and the ImageNet-KNN and Flickr columns, where g/14 leads with 83.7 and 95.1 and 85.9. A team doing dense prediction will look at PASCAL segmentation, NYU depth and the zero-shot column, where L/14 has the edge. The smallest model at 85.7M and 109.6M parameters with 84.0 PASCAL and 79.8 ImageNet-KNN is the one to look at when the constraint is memory.

The v1 table has the same shape, with the HR and LR variants of g/14, SO/14 and L/14, and it shows the same pattern of the largest model leading on retrieval while the smaller models are competitive on the dense tasks.

Two implementations and the commitment they force

The repository provides both PyTorch and Jax implementations, and the README lists them as two directories under the project:

- `tips/pytorch/`: PyTorch inference for the model.
- `tips/scenic/`: Jax-based inference using the
[scenic library](https://github.com/google-research/scenic).

The Jax side is described as Scenic, which is Google Research's library for neural network models with structured data, and the inference Colab for Jax lives at scenic/notebooks/TIPS_Demo.ipynb.

That gives the project a dual-framework character, and it is a genuine strength when your organisation standardises on one of them. The checkpoint links in the tables are provided for both PyTorch and Jax model definitions, and the README says links are provided to all available checkpoints for both, together with representative evals.

The cost of the dual support is a decision you have to make early. The two implementations are not interchangeable at the code level, so a team that prototypes in one framework and deploys in the other is maintaining two inference paths. And the directory naming in the README is worth noting: the source directories are tips/pytorch/ and tips/scenic/, while the Colab links point at paths beginning with pytorch/ and scenic/ directly under the repository root. The top level of the repository does contain pytorch/ and scenic/ directories, so the two naming schemes in the README point at the same content in different ways, which is a small but real trip hazard for anyone scripting a path.

There is also a Hugging Face collection for TIPSv2 models, and the demo is a Hugging Face space, so a team that does not want to run the model code itself still has a route to the weights and to a hosted demo.

The task notebooks are the actual documentation

The demos and notebooks section is the most concrete part of the README, and it reads as a guided tour of what the encoder can be used for.

There is a Hugging Face demo space for feature visualisation, zero-shot segmentation, depth and normals estimation, and supervised segmentation. There is an inference Colab in PyTorch and an inference Colab in Jax. And then there are task-specific notebooks, each one showing a different way of using the same encoder.

The zero-shot segmentation notebook runs the PyTorch path. The foreground segmentation demo trains a linear head for foreground segmentation, which is the standard recipe for probing a frozen encoder: attach a small trainable head and see what the representation already knows. The decoder inference notebook runs inference with DPT heads for segmentation, depth and normals, which is the dense prediction path.

That last one is a notable choice. DPT heads are used for dense prediction, and running them on top of a frozen image-text encoder is the mechanism by which the spatial awareness claim becomes usable output. The encoder is not a segmentation model; it is a representation that a segmentation head can be attached to.

The repository also carries a docs directory with the overview image and a PCA visualisation image referenced in the README, which is the kind of artefact that tells you the model produces embeddings you can inspect rather than only consume.

The top level of the repository is small: a pytorch directory, a scenic directory, a docs directory, a LICENSE, a README and a CONTRIBUTING file. There is no setup.py, no pyproject and no requirements file, which confirms the shape of the project. This is checkpoints plus notebooks, not a package you pip install and call.

Where TIPS is the wrong tool

Three limits, and the first is the most consequential.

The first is that this is a frozen encoder, not a system. Everything the project publishes is a pretrained image-text encoder and, on top of it, heads you train yourself, such as the linear head for foreground segmentation or the DPT heads for depth and normals. If your requirement is an end to end model that takes text and produces an image, or a multimodal chat model, or a system that can be fine-tuned wholesale, this repository does not contain that. The TIPSv2 paper title is about advancing vision-language pretraining, and pretraining produces a starting point, not a finished application.

The second is that the evaluation claims are the authors' claims on the authors' chosen benchmarks. The README says the models were validated on 9 tasks and 20 datasets and that performance matches or exceeds other recent vision encoders with particularly strong spatial awareness. The tables give concrete numbers, and they are useful for choosing a size. But a benchmark is a benchmark, and the comparison set of other vision encoders is not enumerated in the visible text, so if the spatial advantage is the reason you are considering TIPS, the number to check is your own data.

The third is a packaging fact that decides the adoption question: there is no installable package. The repository is notebooks, model definitions and checkpoints, with no setup.py, no pyproject and no requirements. That is entirely reasonable for a research release and entirely awkward for a production service, where someone will have to decide how the model is packaged, pinned and served, and the answer is not in this repository. The Hugging Face collection and the demo space are the practical route for a team that wants the weights without the code, and the notebooks are the route for a team that wants to understand the code.

There is one more asymmetry worth weighing. The models are released under Apache-2.0 in the repository, and the Hugging Face collection carries the models separately, so the terms for the weights and the terms for the code are two documents to read rather than one.

Editorial conclusion

Adopt TIPS if you need a frozen image-text encoder with a genuine spatial signal, for retrieval, zero-shot segmentation, depth, surface normals or as a frozen backbone for a linear head, and if PyTorch or Jax is already in your stack. Do not adopt it if you need an end to end generative or multimodal model, because what is released is a pretrained encoder with separate towers and task specific heads trained on top, and the two implementations are not interchangeable, so picking one is a commitment. Verify first that the checkpoint size you want is published in both the PyTorch and Jax form for the version you intend to use, that the license and the model terms on the Hugging Face collection match your use, and that the evaluations in the README match your own data, since the numbers are the authors' on their chosen benchmarks and the spatial advantage is the claim you would most want to check yourself.

Frequently asked questions

Official sources

1. google-deepmind/tips on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README