TACO annotates litter where it is actually found, not in a studio

The images are of litter in situ, which is what makes them useful

TACO is a growing image dataset of waste in the wild. It contains images of litter taken under diverse environments, specifically woods, roads and beaches, and those images are manually labelled and segmented according to a hierarchical taxonomy so that object detection algorithms can be trained and evaluated on them.

The choice of environment is the dataset's whole argument. A model trained on studio shots of a plastic bottle on a white background learns about bottles; a model trained on a bottle half buried in sand on a beach learns about finding bottles. The scenes are the hard part of litter detection, and a dataset that omits them produces a detector that fails exactly where it is needed.

The dataset is explicitly unfinished. The README says TACO is still relatively small but growing, and that a server is collecting more images and annotations, with the site at tacodataset.org as the public face of that. The paper behind it is arXiv 2003.06975, Trash Annotations in Context for Litter Detection, by Pedro F Proença and Pedro Simões, and the README asks that any publication using the dataset and the API cite it.

The audience is computer vision researchers and municipal or environmental groups who need a detection benchmark for litter. It is not a general photo archive: the value is in the masks, not the pictures.

Annotations come in COCO format over a hierarchical taxonomy

For convenience, annotations are provided in COCO format, with the format reference pointing at the COCO dataset documentation. That single decision removes most of the friction of adopting the dataset, because the standard detection training code and the standard evaluation tooling can read it without a custom loader.

On top of the standard COCO structure, the labels follow a hierarchical taxonomy, so a piece of litter is not just a category but a set of nested attributes. The class names visible in the taxonomy cover the kinds of object that turn up in these scenes, and the taxonomy is what lets a user ask narrower questions than "is there litter in this image".

A hierarchy in the annotation scheme has a practical consequence. It multiplies the number of leaf classes, and with a dataset of this size most of them have very few annotations. The README states this plainly: most of the original classes of TACO have very few annotations, therefore these must be either left out or merged together.

That is the central limitation of the dataset, and it is stated in the same file that tells you how to use it, which is refreshing. It also shapes the detector section, because a detector that predicts the full taxonomy will be predicting labels it has almost no examples of.

Downloading is one script, and a Zenodo DOI is the alternative

The requirements are ordinary Python imaging and data tooling: pillow, requests, jupyter, numpy, pandas, matplotlib, seaborn, graphviz, Cython and pycocotools. Installing them is a single command:

pip3 install -r requirements.txt

The images themselves are fetched by a script in the repository root:

python3 download.py

Alternatively, the dataset is archived on Zenodo under the DOI 10.5281/zenodo.3587843, which is the route to take if you want a fixed, citable snapshot rather than whatever the server currently holds. For a dataset that is still growing, that distinction matters when you need to reproduce a result.

The repository root is small and tells you how the pieces fit: data/ holds the annotation files and the URL lists, download.py fetches images, demo.ipynb is the notebook for inspecting and visualising the data, and detector/ contains the training and inference code.

The same command shape is used for the unofficial annotation set, with a different dataset path:

python3 download.py --dataset_path ./data/annotations_unofficial.json

So switching between the reviewed and the contributed images is a matter of pointing the script at a different file, not of restructuring your workflow.

The demo notebook is the fastest way to see what the labels look like

The API contains a jupyter notebook, demo.ipynb, whose stated purpose is to inspect the dataset and visualise annotations. Given that the taxonomy is hierarchical and the class counts are uneven, that notebook is not a nice to have.

Looking at real annotations before training on them is the cheapest way to find out that your category of interest is spelled differently than you expected, that some classes are near empty, or that the segmentation boundaries in a beach scene are less tidy than a bounding box would suggest. A dataset page with a few rendered examples does not tell you how many of a given class exist, and it does not tell you how consistent the annotators were.

The README also points at the statistics page, tacodataset.org/stats, which is where the class distribution can be read directly. Together with the notebook, that gives you two views of the data before you commit a training run to it, and the README's own advice about leaving out or merging sparse classes only makes sense once you have seen the distribution.

The data directory additionally holds data/all_image_urls.csv, a list of URLs for both unlabelled and labelled images. Each row holds one URL for a VGA-resized version and one for the original, for images hosted by the project. The README asks that if you annotate those images with other tools, you make the results public and contact the maintainers so the annotations can be tracked, which is a reasonable way to grow a dataset that depends on outside labour.

The weekly unofficial annotations are contributed, not reviewed

Annotations submitted through the website are added weekly to data/annotations_unofficial.json, and the README is explicit about their status: they have not yet been reviewed by the maintainers, and some may be inaccurate or have poor segmentations.

That is a useful disclosure and it changes how the file should be used. A label produced by an automated or a rushed annotation tool, or by a volunteer working from a different interpretation of the taxonomy, is not the same kind of evidence as a reviewed annotation. Mixing the two in a single training set without tracking which is which means your evaluation numbers partly measure annotation noise.

The label collection itself opened in November 2019, when the README announced that TACO was officially open for new annotations at tacodataset.org/annotate. The growth log records a December 2019 addition of more than 785 images and 2642 litter segmentations, which is a useful indication of the order of magnitude a single annotation push contributes relative to the whole dataset.

So the intended workflow is a two tier one: train and evaluate on the reviewed annotations, and treat the unofficial file as a pool of extra training data to be filtered by your own criteria. The README does not specify such a filtering procedure, so that part is left to the user.

The detector ships with class maps because the raw taxonomy is too sparse

The repository includes an implementation of Mask R-CNN by Matterport, located in the detector/ directory, with a few modifications and the same requirements. Before using it, the dataset has to be split into train, validation and test subsets.

You can either download the published weights and splits from the project's release, or generate them from scratch with the split_dataset.py script, which produces N random train, validation and test subsets. Run from inside the detector directory:

python3 split_dataset.py --dataset_dir ../data

Further usage instructions are in detector/detector.py.

The class map question comes next, and it is the most useful thing in the detector directory. Because most of the original classes have very few annotations, they must be left out or merged. Depending on the problem, detector/taco_config contains several class maps to target classes, which keep the most dominant classes, for example Can, Bottles and Plastic bags, and the README invites you to build your own.

The result is a detector that predicts a smaller, better populated label set than the raw taxonomy, with the merging done by configuration rather than by code. The trade-off is explicit: the model can no longer distinguish the fine grained categories that make the taxonomy interesting, because the data to learn them is not there.

A segmentation model is the right fit for this data, since litter is often amorphous and overlapping, and a box around a crushed can tells you less about the area of waste than a mask does.

MIT licensed, weights from 2020, but the repository keeps moving

The project is licensed under the MIT License, with the terms in the LICENSE file at the root, which is the straightforward choice for a research dataset that people are expected to use in their own work.

The release history is sparse and old. There are two tags, a test release v0.1 in June 2019 and the 1.0 release of Mask-RCNN weights trained on TACO-10 in March 2020. There is no release for the dataset itself in that list, which is consistent with a dataset that is distributed through the download script and the Zenodo archive rather than as a versioned package.

The last push to the repository was on 2026-10-20 and the project is not archived, so the code and the data pipeline are still being worked on even though the published weights date from 2020. Anyone training a detector today is likely to retrain rather than use those weights, and the value of the 1.0 release is the splits as much as the model.

That gap is also the upgrade cost. The dependency list includes Cython and pycocotools and pins nothing, and the Mask R-CNN implementation it carries was written for an older era of the ecosystem. Building it in a current Python environment is where friction is most likely, and the README does not document a working version matrix, so the fastest way to find out is to install the requirements in a fresh environment and run the split script before committing to the training run.

Editorial conclusion

TACO fits a team training or evaluating litter detection on photographs taken outdoors, and that wants segmentation masks rather than boxes, with a taxonomy that separates material and object categories. It does not fit a team that needs a large, evenly balanced benchmark, because the class distribution is the acknowledged weakness of the dataset, and the weights published with it date from March 2020. Before you rely on it, look at the class statistics at tacodataset.org/stats for the categories you care about, decide whether you will use the shipped class maps or build your own, and remember that the annotations in data/annotations_unofficial.json are contributed weekly and have not been reviewed, so they carry a different reliability from the labelled set.

Frequently asked questions

Official sources

1. License: MIT
2. pedropro/TACO on GitHub
3. Project website
4. README
5. Releases