COCO-Stuff adds pixel-level stuff labels to the 164K COCO images

Things versus stuff, and why the second one matters

COCO, the dataset everyone uses, annotates countable objects. COCO-Stuff adds the other half of the picture. The README describes COCO-Stuff as augmenting all 164K images of the popular COCO dataset with pixel-level stuff annotations, and says these annotations can be used for scene understanding tasks like semantic segmentation, object detection and image captioning. The distinction between the two kinds of class is the whole idea. A thing is an object with a countable instance, a person, a car, a chair. Stuff is the material the world is made of, the sky, the grass, the wall, the water, the floor. Object detection datasets label things and ignore stuff entirely, so a detector trained on COCO sees a car and nothing at all about the road it is on. The highlights list makes the composition explicit: 164K complex images from COCO, dense pixel-level annotations, 80 thing classes, 91 stuff classes and one class named unlabeled, instance-level annotations for things inherited from COCO, the complex spatial context between stuff and things, and the five captions per image that COCO already provides. The authors are Holger Caesar, Jasper Uijlings and Vittorio Ferrari, and the paper is COCO-Stuff: Thing and Stuff Classes in Context, published at CVPR 2018, with the arXiv listing at 1612.03716 and a BibTeX file in the repository. The audience is computer vision researchers who need a training signal for the background, and who want it aligned with an existing detection and captioning dataset rather than having to reconcile two separate corpora.

Three distributions, and only one of them is the real thing

The versions section is where careful reading pays off, because the three releases are not the same data with different names. The final COCO-Stuff dataset is the version presented on the page, and it includes all 164K images from COCO 2017, split into train 118K, val 5K, test-dev 20K and test-challenge 20K. It covers 172 classes, being 80 thing classes, 91 stuff classes and one class for unlabeled, and the README says this dataset will form the basis of all upcoming challenges. The COCO 2017 Stuff Segmentation Challenge is a different thing. It is a semantic segmentation challenge on 55K images, train 40K, val 5K, test-dev 5K and test-challenge 5K, and to focus the task on stuff all 80 thing classes were merged into a single class named other. Results were presented at the Joint COCO and Places Recognition Workshop at ICCV 2017. COCO-Stuff 10K is the first dataset, annotated by 10 in-house annotators at the University of Edinburgh, covering 10K images from the COCO training set with a 9K and 1K train and validation split to make results comparable. It has the same 80 thing, 91 stuff and 1 unlabeled class structure, and the README notes it was initially presented with 91 thing classes before that was changed to 80, because 11 classes have no segmentation annotations in COCO. It is a subset of all the other releases. The practical rule: if you want the full labelled set, use the final dataset. The other two exist for reproducing older results.

Two annotation formats, and the indexing mismatch between them

The downloads table lists three annotation archives, and they are not interchangeable. The recommended one is stuffthingmaps_trainval2017.zip, 659 MB, which holds stuff and thing PNG-style annotations on COCO 2017 trainval. The README suggests using the stuffthingmaps because they provide all stuff and thing labels in a single .png file per image. There is a detail here that trips up nearly everyone who works with this dataset. The .png files are indexed images, so they store only the label indices and are typically displayed as grayscale images. To be compatible with most Caffe-based semantic segmentation methods, thing and stuff labels cover indices 0 to 181 and the value 255 indicates the unlabeled or void class. And that is a different indexing from the one used in labels.txt and labels.md. The pixel value in the stuffthingmaps equals the label id in labels.txt minus 1, except for the unlabeled class, which has id 0 in labels.txt but pixel value 255 in the stuffthingmaps. So there is an off by one between the two conventions, and one class that breaks the pattern entirely. A model trained on one indexing and evaluated on the other will produce a confusing result that looks like a bug in the model rather than a convention mismatch. The alternative is stuff_trainval2017.zip at 543 MB for stuff-only annotations in COCO format, and annotations_trainval2017.zip at 241 MB for thing-only annotations. Those are compatible with the COCO-Stuff API, and the README notes the stuff annotations contain a class named other with index 183 covering all non-stuff pixels. Two small files round out the set. labels.md holds the indices, names, previews and descriptions of the classes, and labels.txt is the machine readable version of the same list.

The setup steps are Linux oriented and start with 19 GB of images

The setup section gives instructions for downloading the dataset and creating the folder structure, and says they are for Ubuntu and require git, wget and unzip, with the note that on other operating systems the commands may differ. The first step is to get the repository.

git clone https://github.com/nightrome/cocostuff.git
cd cocostuff

Then the images, which is the part that catches people out in terms of time and disk.

wget --directory-prefix=downloads http://images.cocodataset.org/zips/train2017.zip
wget --directory-prefix=downloads http://images.cocodataset.org/zips/val2017.zip

The README is blunt about the size. To use this dataset you will need to download the images, 18 plus 1 GB, and the annotations of the trainval sets. The download table gives the same figures: train2017.zip with 118K images at 18 GB, val2017.zip with 5K images at 1 GB. That scale is the reason the setup instructions exist at all. There is no package manager entry and no streaming access in the documented path. The annotations live on the Edinburgh server at calvin.inf.ed.ac.uk, and the images come from the COCO server at images.cocodataset.org, so a full local copy means pulling from two hosts. For earlier versions the README redirects you to the COCO 2017 Stuff Segmentation Challenge download page or to the COCO-Stuff 10K repository.

Exploring the annotations without downloading anything

There is one piece of tooling the README points to that changes how you evaluate the dataset. You can use Scale's Nucleus platform to explore the COCO-Stuff dataset. The platform allows you to search for images with specific annotations or using textual image descriptions, and it also lets you identify potential annotation errors by comparing the predictions of an object detector to the ground-truth of COCO-Stuff. Registration is free. That last use is the one worth pausing on. A dense stuff annotation set is expensive to produce and hard to audit, and the README's own history shows the cost, with the 10K version annotated by 10 in-house annotators at Edinburgh. Being able to spot a mislabelled region by overlaying a detector's prediction on the ground truth is a genuinely useful workflow, and it does not require a 19 GB download. The Nucleus link carries the open_dataset campaign parameters, which suggests it is a promoted partnership rather than a community tool. The repository itself carries a models/ directory and a nucleus.gif at the root, plus a .gitmodules file, so there is reference material and a submodule beyond the three small documentation files at the top level. The paper and the BibTeX entry are also in the repository, so a citation is one file away.

A dataset with an unstated licence, which is a real problem for some users

The licence field on the repository says unknown, and there is no licence file among the top level entries, which are .gitignore, .gitmodules, README.md, cocostuff.bib, labels.md, labels.txt, the models/ directory and nucleus.gif. That is unusual for a dataset that is meant to be used by research groups and by companies building perception systems, and it is worth raising directly rather than glossing over. The paper is published through a venue that distributes it under a licence, and the COCO dataset it builds on has its own terms, so there is a path to using it. But the repository that hosts the download instructions does not itself state the terms. Anyone who needs to know what they are permitted to do with the annotations, especially in a commercial setting, should look at the COCO dataset terms and confirm the position rather than assume the permissive default. The homepage for the project is the arXiv listing for the paper, at arxiv.org/abs/1612.03716, which is where the authoritative description of the dataset lives. There is no separate project website and no GitHub releases, which is expected for a dataset whose artefacts are large hosted archives rather than versioned packages. On activity, the repository is not archived and the last push was on 2026-07-15. The data itself has not changed in that time, but the documentation, the labels files and the models directory can.

Editorial conclusion

Adopt COCO-Stuff when you need dense background labels aligned with the COCO images you already train on, for example for semantic segmentation, and use the stuffthingmaps archive rather than the separate ones. Do not adopt it if you need a small dataset or a quick download, because the full trainval set means 19 GB of images before annotations, and the 10K subset exists only for reproducing earlier results. Before you start training, confirm which label convention your pipeline expects, because the stuffthingmaps pixel values are the labels.txt ids minus one with 255 for unlabeled, and check the licence position for the annotations before commercial use, since the repository itself states no licence.

Frequently asked questions

Official sources

1. Issues
2. nightrome/cocostuff on GitHub
3. Project website
4. README