Taskonomy: pretrained task models, a dataset and the affinity data behind them

Four artefacts, not one package

The README lays out what the repository actually contains, and it reads as a list of deliverables rather than as a codebase. There are pretrained models in a task bank, a dataset, reference code, and task affinity analyses with results. The task bank folder holds information about the pretrained models plus scripts to download them, along with sample outputs and links to the live previews on the project page, and the models exist in both PyTorch and TensorFlow flavours. The data folder holds information and statistics about the dataset, some sample data, and instructions for downloading the full dataset, which lives outside the repository at the omnidata documentation site. The results folder contains the raw and normalised data used for measuring task affinities. The code folder is described as reference code. So a user gets three kinds of thing from this repository, downloadable weights, a dataset with a download procedure, and the measurements, and only the last of those is something no other source will give you.

The paper that produced all of it

Everything in the repository exists to support one paper, Taskonomy: Disentangling Task Transfer Learning, presented at CVPR 2018 and awarded a Best Paper Award. The author list is Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik and Silvio Savarese, and the project homepage at taskonomy.vision hosts the project page with links to assets and the interactive previews. The framing of the title is the framing of the field, since the paper is about separating or disentangling tasks from one another rather than assuming that one general purpose task transfers cleanly to another, which is the question a transfer learning practitioner faces when choosing a source task for a target task. The repository is therefore best read as a research artefact package: the code implements the experiments, the weights are the outputs of training, and the affinity data is the measurement the paper argues from.

Task affinity is the part you cannot regenerate

The results folder is where the lasting contribution sits. It contains the raw and normalised data used for measuring task affinities, and the image in the README that illustrates it is the affinity structure itself, a matrix of how strongly one task relates to another. Everything else in the repository can be produced again in principle: you can train models, and you can rebuild a dataset from the source images. Recreating the affinity measurements means re-running the whole transfer study and hoping the numbers come out the same, which is exactly the kind of result that shifts with training recipe and checkpoint. Publishing the raw and the normalised form in the same place addresses that, and it also means the affinity structure can be used without accepting the paper's conclusions. If you are building a task selection heuristic, or training a model to predict transferability, this directory is the input.

Getting the weights and the images

The download paths are described rather than scripted, and that is a real difference of experience from a typical library. For the models, the task bank folder contains information about the pretrained models and scripts to download them, so the route is clone the repository, read the task bank folder, and run the download scripts it provides, with sample outputs and links to the live previews available in the same place to check what a given model produces. For the data, the folder contains information and statistics, some sample data and instructions for downloading the full dataset, and those instructions point at a separate documentation site, which means the bulk of the images does not live in Git and that you are downloading a dataset rather than a file. The practical advice is to start with the sample data to understand the format and the statistics before pulling the full set, since a research dataset of this kind is not something you want to discover the size of after the download has started.

Reference code, and the absence of a setup guide

The code folder is the part with the least documentation, and the top level of the repository explains why. There is no setup section, no dependency list, no command line reference and no package manifest, and the whole tree is a LICENSE file, the README and the four directories. The word reference in the description of that folder is doing real work, since it tells you that the code is there to be read alongside the paper rather than installed and called. That is a legitimate choice for a paper artefact and it is also a genuine constraint for an engineer, because a research codebase from 2018 rarely builds against a current stack of PyTorch or TensorFlow without pinning old versions and patching small incompatibilities. If your plan is to run the training pipeline rather than to consume the pretrained outputs, budget time for that, and if your plan is to consume the outputs, the task bank scripts are the part that matters.

Citing it, licensing it, and where the repository stands

The README asks that you cite the paper if the code, models or data prove useful, and it gives the BibTeX entry directly:

@inproceedings{zamir2018taskonomy,
  title={Taskonomy: Disentangling Task Transfer Learning},
  author={Zamir, Amir R and Sax, Alexander and and Shen, William B and Guibas, Leonidas and Malik, Jitendra and Savarese, Silvio},
  booktitle={2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2018},
  organization={IEEE}
}

The licence is MIT, with a LICENSE file at the root. There is an interesting piece of repository archaeology in the raw README, because a commented out block mentions that more code, models and dataset of Taskonomy are coming soon and that the repository is under construction, which dates the ambition and explains the current shape of the tree. The licence field on the repository is recorded as a custom licence that GitHub cannot classify, so the LICENSE file is the text to read. On maintenance, the last push was on 2026-08-15 and the repository is not archived, while the repository publishes no GitHub releases at all, so there is no tag to pin and the version of what you download is whatever the download scripts currently point at.

Editorial conclusion

Taskonomy fits a researcher who wants pretrained models for a set of visual tasks, the dataset behind them, and the measured affinity structure that says which task transfers to which, and it fits a student who wants to see how a transfer learning study is laid out. It is the wrong choice if you need a maintained library with installation instructions and a supported API, because the repository calls its code reference code and documents no setup, and it is a poor fit for production inference, since the artefacts are research outputs rather than a packaged release. Before you build on it, check the download instructions in the data folder and the scripts in the task bank, since the full dataset lives outside the repository, and plan for the fact that the repository publishes no releases to pin against.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. StanfordVL/taskonomy on GitHub