# VisionHOPE treats the backbone's memory and its update rule as one system

> VisionHOPE is a PyTorch implementation of a visual backbone built on self-referential nested learning, where five coupled memories and the rule that updates them evolve together inside a single image. It covers ImageNet classification, COCO detection and ADE20K segmentation, and the last push was on 2026-10-12.

**PSRben/VisionHOPE** — Official PyTorch implementation of VisionHOPE: Visual Backbones as Self-Modifying Learning Systems.

- Repository: https://github.com/PSRben/VisionHOPE
- Stars: 1,416 · Forks: 42
- Language: Python
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/psrben-visionhope

## The idea is that memory and learning rule co-evolve

VisionHOPE is described as a visual backbone based on self-referential nested learning, and the project name expands to Visual Backbones as Self-Modifying Learning Systems.

The framing in the README is that the model lets what it remembers and how it learns evolve together while processing an image. The concept figure is captioned as a progression from CNNs, ViTs, SSMs and TTT to nested learning, with the summary line reading from static local aggregation to self-modifying learning within an image.

That is the specific claim, and it is worth stating precisely because it distinguishes this from a new attention block. In a conventional architecture the weights are fixed during inference and the model composes a static function. Here, the memory contents and the mechanism that updates them are both part of the computation, and both are adjusted as the image is processed.

The academic lineage is stated too. VisionHOPE builds on the self-referential construction in Nested Learning: The Illusion of Deep Learning Architectures, and the paper is on arXiv as 2609.33325, with a Hugging Face paper page linked alongside it. The author list is Siran Peng, Tianshuo Zhang, Tianyu Fu, Weisong Zhao, Haoyuan Zhang, Jiankuo Zhao, Minghui Wu, Ping Jiang, Xiangyu Zhu, Chenxu Zhao and Zhen Lei, with equal contribution marked for the first two and corresponding authors marked for the last two.

The update timeline is short. The preprint appeared on 2026-09-27 and the pretrained checkpoints followed on 2026-09-29, and the last push was on 2026-10-12. This is a code release that tracks a recent preprint, and the repository is MIT licensed with a THIRD_PARTY_NOTICES.md file at the top level and a CITATION.cff file.

## The operator is three mechanisms working together

The VisionHOPE operator combines three things, and the README lists them precisely.

Coupled memories: five memories store content, generate key and value representations, and govern learning rate and retention, and they co-evolve as visual context accumulates. The important detail is that the memories are not just storage. Two of them influence the learning process itself, by setting the effective step size and how much is retained, so the model is partly deciding how it will learn as it goes.

Stable updates: a soft injection cap and a spectral clamp provide non-expansion guarantees for both token-wise and chunk-wise memory recurrences. Non-expansion is the property that keeps a self-modifying system from blowing up. A memory that can grow without bound is a memory that eventually overflows, and the two mechanisms named are the two places where the design bounds it. The guarantee is stated for both granularities, token-wise and chunk-wise, which matters because the operator uses chunk-wise recurrences in its scans.

Spatially aligned scans: four directional scans use row- and column-aligned chunks, followed by spatial restoration and learned channel-wise fusion. The architecture figure is captioned as four directional SRNL scans with five coupled memories, stability-matched step-size control and channel-wise output fusion.

The four scans are the spatial part. A row-aligned scan and a column-aligned scan in each direction gives the model a way to read the image in both axes, and the spatial restoration step puts the pieces back into a consistent arrangement after the scans have moved information around. The learned channel-wise fusion then combines the four results, which is where the model decides how much to trust each direction.

The repository exposes these pieces separately, with standalone SRNL, operator and block interfaces, so the mechanism can be used inside other architectures rather than only as the shipped backbone.

## One codebase, three tasks, three separate environments

The repository covers ImageNet classification, COCO detection and instance segmentation, and ADE20K semantic segmentation, along with the standalone SRNL and VisionHOPE operator modules.

The installation instructions are specific about the environment, and the specificity is the point. Use Linux with an NVIDIA GPU and the CUDA toolkit, including nvcc and a C++ compiler. Python 3.10 is recommended for all tasks, and classification also supports Python 3.11. The commands are run from the repository root with CUDA 12.1 installed:

```bash
python -m pip install torch==2.1.0 torchvision==0.16.0 \
    --index-url https://download.pytorch.org/whl/cu121
python -m pip install -r requirements/classification.txt
python -m pip install --no-deps -e .
```

The additional dependencies for COCO and ADE20K are given as a separate section, and the instruction is to use a separate Python 3.10 environment for each task, with the PyTorch and CUDA versions above, and to install MMCV first:

```bash
python -m pip install --only-binary=mmcv mmcv==2.1.0 \
    -f https://download.openmmlab.com/mmcv/dist/cu121/torch2.1/index.html
```

Then the task requirements follow, with requirements/detection.txt for COCO and requirements/segmentation.txt for ADE20K, each followed by the editable install.

The --no-deps flag on the editable install is worth understanding. It tells pip not to resolve the project's own dependencies, which is what makes the separately pinned environment work: the torch and mmcv versions are chosen deliberately, and letting pip try to reconcile them would break the pins.

The Python version constraint in pyproject.toml confirms the range, requires-python = ">=3.10,<3.12", with the ruff configuration targeting py310.

## The three backbone sizes and the checkpoint naming

The model table is small and practical. There is a hierarchical Tiny, Small and Base set of backbones that support classification and dense prediction, with the model names and checkpoint filenames listed for ImageNet-1K classification at 224 by 224.

VisionHOPE-T is visionhope_tiny with the checkpoint visionhope_tiny.pth, VisionHOPE-S is visionhope_small with visionhope_small.pth, and VisionHOPE-B is visionhope_base with visionhope_base.pth.

The naming convention extends to the other tasks, and knowing it saves time. COCO checkpoints are named visionhope_<size>_coco_<schedule>.pth, with visionhope_small_coco_3x.pth given as the example, and ADE20K checkpoints are named visionhope_<size>_ade20k.pth.

The train and test scripts look for checkpoints in a weights/ directory, and the README says to set WEIGHTS_ROOT to use another directory. That is a small but useful detail, because a script that hard codes a relative path is awkward in a scheduled job or a shared machine.

Downloading is done from Hugging Face with the CLI, from the repository root:

```bash
python -m pip install --upgrade huggingface_hub
hf download PSRben/VisionHOPE visionhope_small.pth --local-dir weights
```

The weights are also on the GitHub Releases page under the weights-v1 tag and on Baidu Netdisk under visionhope_weights with the access code 8866, so there are three routes depending on where you are and what your network allows.

The package metadata gives the rest of the dependency picture: torch at least 2.1, torchvision at least 0.16, timm pinned exactly to 0.6.13, numpy at least 1.24 and below 2, PyYAML at least 6, ninja at least 1.10 and setuptools in a bounded range. The timm pin is exact rather than a range, which tells you the timm API surface is relied on precisely.

## The package layout shows where the work is

The pyproject.toml packaging is unusual in a way that reveals the structure of the project. The package is named visionhope with a version of 0.1.0, and the setuptools configuration sets package-dir = {visionhope = "."}, so the repository root itself becomes the visionhope package. The setup.py is a small shim that finds packages and constructs the visionhope namespace from them.

The package data section then lists what has to travel with the code, and it is revealing. It includes the shell scripts, the docs including markdown, pdf and tex files, the docs/licenses/ directory, and the requirements text files. And two specific entries name CUDA source directories: visionhope.models.srnl_core includes csrc/*.cu, csrc/*.cpp and csrc/*.cuh, and visionhope.utils includes the same three patterns.

So the C++ and CUDA sources are shipped inside the Python package, and the README's note that CUDA extensions compile on first use explains why: they are not prebuilt, they are compiled on the target machine when the module is first used. That is why nvcc and a C++ compiler are hard requirements rather than optional extras.

The top level of the repository follows the task structure: models/ for the backbones, tasks/ for the per-task heads and training, train/ for training entry points, inference/ for inference, analysis/ for analysis, tools/ for utilities, utils/ for shared code, scripts/ for the shell drivers, requirements/ for the per-task requirement files, and assets/ for the figures. There is a run_inference.sh and a run_tasks.sh at the top, and a CITATION.cff for citation metadata.

One operational note from the README that is easy to skip: keep model parameters in FP32 and use torch.autocast for mixed precision. The distinction is that the parameters stay full precision while the compute is autocast, which matters for a model whose stability argument depends on the step size being controlled.

## Conclusion

VisionHOPE suits a vision researcher who wants to build on the nested learning formulation and needs a working implementation with pretrained checkpoints, and it is the wrong tool if you need a stable backbone you can drop into a production pipeline today, since the paper preprint is only weeks old and the project is a research code release. Before running it, read the installation section closely, because the CUDA extensions compile on first use and the task environments are deliberately separated, with a separate Python 3.10 environment for COCO and for ADE20K. Verify first that you have Linux with an NVIDIA GPU, the CUDA toolkit with nvcc and a C++ compiler, because the README states that requirement directly and offers no CPU path.

## FAQ

### What is VisionHOPE?

It is the official PyTorch implementation of VisionHOPE, a visual backbone built on self-referential nested learning, where what the model remembers and how it learns evolve together while processing an image. The repository covers ImageNet classification, COCO detection and instance segmentation, and ADE20K semantic segmentation.

### What are the three components of the VisionHOPE operator?

Coupled memories, five of which store content, generate key and value representations, and govern learning rate and retention. Stable updates, using a soft injection cap and a spectral clamp for non-expansion guarantees on token-wise and chunk-wise recurrences. And spatially aligned scans, four directional scans with row- and column-aligned chunks, spatial restoration and learned channel-wise fusion.

### What are the requirements to run VisionHOPE?

The README says to use Linux with an NVIDIA GPU and the CUDA toolkit including nvcc and a C++ compiler. Python 3.10 is recommended for all tasks and classification also supports 3.11, and the installation commands assume CUDA 12.1 with torch 2.1.0 and torchvision 0.16.0.

### Which VisionHOPE checkpoints are available?

Pretrained checkpoints for ImageNet-1K, COCO and ADE20K are on GitHub Releases under weights-v1, on Hugging Face at PSRben/VisionHOPE and on Baidu Netdisk. The model names are visionhope_tiny, visionhope_small and visionhope_base, with COCO checkpoints named visionhope_<size>_coco_<schedule>.pth and ADE20K ones visionhope_<size>_ade20k.pth.

### Why does the VisionHOPE install need separate environments per task?

The README instructs you to use a separate Python 3.10 environment for each task, installing MMCV 2.1.0 with the pinned PyTorch and CUDA versions first. The editable install then uses pip install --no-deps -e . so the deliberate version pins are not reconciled by the installer.

## Sources

- [Issues](https://github.com/PSRben/VisionHOPE/issues)
- [License: MIT](https://github.com/PSRben/VisionHOPE/blob/main/LICENSE)
- [PSRben/VisionHOPE on GitHub](https://github.com/PSRben/VisionHOPE)
- [README](https://github.com/PSRben/VisionHOPE/blob/main/README.md)
- [Releases](https://github.com/PSRben/VisionHOPE/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/psrben-visionhope