# AnyUp: upsampling vision features without retraining per encoder

> AnyUp is the official repository for an ICLR 2026 oral paper that upsamples the feature maps of any vision encoder at inference time, so one trained model works with DINO, CLIP or ResNet features instead of a retrained upsampler per backbone.

**wimmerth/anyup** — [ICLR '26 Oral] Official repository of the paper "AnyUp: Universal Feature Upsampling".

- Repository: https://github.com/wimmerth/anyup
- Website: https://wimmerth.github.io/anyup
- Stars: 597 · Forks: 39
- Language: Jupyter Notebook
- License: CC-BY-4.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/wimmerth-anyup

## One upsampler per encoder is the problem

The paper's abstract starts with the limitation it targets. Learning based upsamplers for features such as DINO or CLIP have to be re-trained for every feature extractor, so they do not generalise to different feature types at inference time. AnyUp proposes an inference-time, feature-agnostic upsampling architecture to relieve that, and the experiments in the paper report a new state of the art for upsampled features, generalisation across feature types, preserved feature semantics and efficiency. The paper had an ICLR 2026 oral, and the repository is the official one, with the arXiv identifier 2510.12764, a project page, and a Colab notebook for example usage. The authors come from the Max Planck Institute for Informatics, ETH Zurich, Google and TU Munich, and the licence is CC-BY-4.0, which is worth noting for a research artefact that people build on in both code and weights.

## The upsampler runs at inference time, not as a training step

The mechanism is a two argument call. You hand the model a high resolution image and the low resolution features you want to expand, and you get features back at the image's resolution. The documentation puts it as: upsample features from any model, at any layer, without having to retrain the upsampler. The two notes that follow matter in practice. The high resolution image should be normalized to the ImageNet mean and standard deviation, as usual for most vision encoders, so the image is not a raw tensor but an encoder-ready input. The low resolution features can be any features from any encoder, with DINO, CLIP and ResNet given as examples, which is the whole point of the approach. The output has the same spatial resolution as the input image by default, and if a different output resolution is needed there is an output_size argument for a custom size.

## Three lines of code at inference

The usage example in the repository is short enough to read at a glance:

```python
import torch
# high-resolution image (B, 3, H, W)
hr_image    = ...
# low-resolution features (B, C, h, w) 
lr_features = ...
# load the AnyUp upsampler model (here we use the NATTEN-based version trained on multiple backbones)
upsampler   = torch.hub.load('wimmerth/anyup', 'anyup_multi_backbone', use_natten=True)
# upsampled high-resolution features (B, C, H, W)
hr_features = upsampler(hr_image, lr_features)
```

The model is loaded through torch.hub rather than from a local checkpoint file, which means the weights come from the hub entry points in hubconf.py in the repository, and there are two of them. The anyup entry point gives the model proposed in the original paper, without NATTEN and trained on a single backbone, DINOv2 ViT-S. The anyup_multi_backbone entry point gives the newer variant trained on multiple feature extractors. Two extra arguments control behaviour at the edges. output_size sets a custom output resolution, and q_chunk_size trades speed for memory when you hit out of memory issues at high resolutions, with a value of 10 given as the example.

## The NATTEN variant is not the same model as the paper

The 11/25 news entry adds a more efficient window attention module, based on NATTEN, in both memory and speed. You load it with use_natten=True, but the documentation is careful about what that means, because that model variant uses slightly different windows than the original AnyUp model, and the original was the one used for all the experiments in the paper. If you want to use it in your project you also have to install NATTEN in addition to PyTorch, following the official install instructions for your CUDA version. That caveat is the single most important line in the release notes for anyone reproducing the paper numbers, since the efficient variant and the published results are not the same system. The same date also brought multi-backbone training, which allows training a single AnyUp model with multiple different feature extractors, and the claim for it is that it improves generalisation to unseen backbones at inference time.

## Debugging the attention without a second model

There is an optional visualisation of the attention used to upsample the features, exposed through a flag in the forward pass:

```python
# matplotlib must be installed to use this feature
# upsampled features and display attention map visualization (B, C, H, W)
hr_features = upsampler(hr_image, lr_features, vis_attn=True)
```

The restriction is that it is only available if use_natten is False, so the paper-era model is the one you can inspect. For a method whose claim is that it preserves feature semantics, seeing where the attention lands is the natural first debugging step when an output does not look right, and having it in the forward pass rather than only in a notebook is a practical choice. The third news item points the other way, to reuse rather than debugging. On 03/26 AnyUp was used in Falcon Perception, and the authors there also implemented an efficient MLX-compatible version of AnyUp, available in their repository, which is the route to look at if you want to run the method on edge devices.

## Installing the package or only the dependencies

Two installation paths are documented, and the difference is whether you want the package importable locally. For local development as a package the sequence is a micromamba environment on Python 3.12, uv for the installer, then the PyTorch build that matches your CUDA setup, then the matching NATTEN wheel from the NATTEN wheel index, and finally an editable install of the repository itself:

```bash
micromamba create -n anyup python=3.12 -y
micromamba activate anyup
pip install uv
```

```bash
uv pip install -e .
```

The second path skips the package install and only pulls the dependencies you need to train, which is the list einops, matplotlib, numpy, timm, plotly, tensorboard, hydra-core, rich and scikit-learn, with the comment that the install should be called from the root of the repository. The project metadata backs this up. The package is named anyup at version 0.1.0, requires Python 3.10 or newer with classifiers through 3.13, is built with setuptools, and depends on matplotlib, numpy, timm, tensorboard, hydra-core, rich, scikit-learn, torch and torchvision, with optional extras for notebooks and for NATTEN, the latter marked to be installed manually to match the local CUDA and PyTorch version.

## Weights, checkpoints and a training entry point

The repository has released three artefacts and they are tagged in a way that tells you what each one is. The checkpoint tag holds the v1 paper model weights from 2025-10-14, the training-code tag released the v1 training code on 2025-11-09, and checkpoint_v2 covers the v2 NATTEN based implementation and multi-backbone training on 2025-11-25. The top level of the repository is small and readable, with the anyup package directory, a config directory, train.py as the training entry point, hubconf.py for the torch.hub entry points, the example_usage.ipynb notebook and the pyproject.toml. A Hydra based config directory alongside hydra-core in the dependency list is what tells you how the training runs are parameterised. The repository is not archived and the last push was on 2026-04-17, which is months after the last tagged artefact, so the working tree is ahead of the released checkpoints and the training code in the repository may differ from what produced the v2 weights.

## Conclusion

Adopt AnyUp if you need pixel-resolution features from a frozen encoder and do not want to train a separate upsampler for every backbone you use. Do not expect the published checkpoint to match your encoder exactly, since the paper model was trained on DINOv2 ViT-S, and check whether the NATTEN variant's different windows change your results before you deploy it.

## FAQ

### What problem does AnyUp solve?

Learning based feature upsamplers for encoders such as DINO or CLIP have to be retrained for every feature extractor, so they do not generalise to other feature types at inference time. AnyUp is an inference-time, feature-agnostic upsampler that works with features from any encoder.

### How do I run AnyUp on my own features?

Load the model with torch.hub.load('wimmerth/anyup', 'anyup_multi_backbone', use_natten=True) and call it with the high resolution image and the low resolution features. The image should be normalized to the ImageNet mean and standard deviation, and the features can come from DINO, CLIP, ResNet or any other encoder.

### How do I upsample to a resolution other than the image size?

Pass the output_size argument with the target size, for example upsampler(hr_image, lr_features, output_size=(H_prime, W_prime)). By default the output has the same spatial resolution as the high resolution image.

### What is the difference between the NATTEN version and the original AnyUp model?

The NATTEN based window attention module is more efficient in memory and speed, but that model variant uses slightly different windows than the original AnyUp model, which was the one used for all the experiments in the paper. Using it also requires installing NATTEN for your CUDA and PyTorch version.

### How do I install AnyUp for local development?

Create a micromamba environment on Python 3.12, install uv, then install the PyTorch build matching your CUDA setup, the matching NATTEN wheel, and finally run uv pip install -e . from the repository root. A second documented path installs only the training dependencies without installing the package.

## Sources

- [License: CC-BY-4.0](https://github.com/wimmerth/anyup/blob/main/LICENSE)
- [Project website](https://wimmerth.github.io/anyup)
- [README](https://github.com/wimmerth/anyup/blob/main/README.md)
- [Releases](https://github.com/wimmerth/anyup/releases)
- [wimmerth/anyup on GitHub](https://github.com/wimmerth/anyup)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/wimmerth-anyup