CutLER: learning object detectors without any human labels

The premise: no labels, still a detector

Cut and Learn, abbreviated CutLER, is described as a simple approach for training object detection and instance segmentation models without human annotations. The project documentation states that it outperforms the previous state of the art by 2.7 times for AP50 and 2.6 times for AR across 11 benchmarks, and that comparison is the paper's own claim rather than an independent measurement.

The mechanism is not a single trick but a two-stage pipeline. Cut-and-Learn has two stages: generating pseudo-masks with MaskCut, and then learning unsupervised detectors from the pseudo-masks of unlabelled data. The underlying idea is that a pretrained vision transformer already knows where the interesting regions of an image are, and that a detector can be trained to reproduce those regions using the vision transformer's own attention as the supervision signal.

What makes the result interesting is the range of the claim. The project states that CutLER can learn unsupervised object detectors and instance segmentors solely on ImageNet-1K, and that it exhibits strong generalisation to domain shifts when evaluated on 11 different benchmarks across domains such as natural images, video frames, paintings, and sketches. It also states that CutLER can serve as a pretrained model for fully or semi-supervised detection and segmentation tasks, which is a practical use that does not require believing the headline numbers.

The work comes from FAIR, Meta AI and UC Berkeley, with authors Xudong Wang, Rohit Girdhar, Stella X. Yu, and Ishan Misra, published at CVPR 2023. The repository is Python, and its licence is a custom one that GitHub cannot classify, so the LICENSE file is the text to read. The last push to the repository was on 2026-04-14 and there are no GitHub releases, so the model zoo and the checkpoints are the artefacts to track rather than version tags.

MaskCut, and the two knobs that shape the pseudo-masks

MaskCut is the first stage, and its job is to produce segmentation masks for multiple instances of each image. The project describes it as an approach to generate pseudo-masks for multiple objects in an image, and it is the component that carries the whole method, since everything downstream is trained on what MaskCut produces.

Two parameters control the output. N is the number of objects to segment, and tau is the threshold that determines what counts as a foreground region. The demo command shows the full set of options:

cd maskcut
python demo.py --img-path imgs/demo2.jpg \
  --N 3 --tau 0.15 --vit-arch base --patch-size 8 \
  [--other-options]

The vit-arch and patch-size options choose the vision transformer backbone and its patch resolution, and the demo images live in maskcut/imgs/. To run on a CPU you add the --cpu flag when running the demo script. The documentation also notes a practical case: for imgs/demo4.jpg you need to use N 6 to segment all six instances in the image, which is the clearest illustration of what N actually controls. Too low an N and you silently drop objects, which is the failure mode to watch for when you point this at your own images.

Two hosted options exist for trying the method without a local setup. There is a MaskCut demo on Colab that needs no GPU, and a web demo on Hugging Face Spaces. Both are the fastest way to build intuition about what the pseudo-masks look like before you commit a cluster to generating them for a dataset.

Generating pseudo-masks for a whole dataset

Running MaskCut over a real dataset is a batch job, and the project gives you the command with the parallelisation built in:

cd maskcut
python maskcut.py \
--vit-arch base --patch-size 8 \
--tau 0.15 --fixed_size 480 --N 3 \
--num-folder-per-job 1000 --job-index 0 \
--dataset-path /path/to/dataset/traindir \
--out-dir /path/to/save/annotations \

The shape of the command is the interesting part. The dataset path points at a training directory, and the job is sliced by folder: num-folder-per-job says how many folders this run handles and job-index says which slice. The documentation is explicit that generating pseudo-masks for all 1.3 million images in 1,000 folders takes a significant amount of time, and that multiple runs are recommended, with each run processing a smaller number of folders by setting those two options.

The outputs are then merged, and the merge step has to agree with the generation step on its parameters:

python merge_jsons.py \
--base-dir /path/to/save/annotations \
--num-folder-per-job 2 --fixed-size 480 \
--tau 0.15 --N 3 \
--save-path imagenet_train_fixsize480_tau0.15_N3.json

The documentation states that the num-folder-per-job, fixed-size, tau, and N of merge_jsons.py should match the ones used to run maskcut.py. That is a sharp edge. A mismatch in fixed_size or N between the two steps produces a merged annotation file that is internally inconsistent, and because the filename encodes the settings, the mistake is not obvious until training results look wrong.

For multi-node setups the project also provides a submitit script, run with bash run_maskcut_with_submitit.sh from the maskcut directory, and submitit appears in the requirements list.

The dataset preparation instructions are in datasets/README.md, and the installation instructions are in INSTALL.md at the repository root.

Running the trained detector, and the model zoo layout

Once you have the pseudo-mask annotations, the second stage trains the detector. The repository is split into directories that mirror the pipeline: maskcut/ for the pseudo-mask stage, cutler/ for the detector, videocutler/ for the video extension, datasets/ for the data preparation guides, docs/ for the images and figures, and third_party/ for vendored code, with a .gitmodules file at the root.

Inference is demonstrated from the cutler directory, and the procedure is to pick a model and its config file from the model zoo, for example model_zoo/configs/CutLER-ImageNet/cascade_mask_rcnn_R_50_FPN.yaml, and then run the provided demo.py against the builtin configs. The project also hosts a Colab demo with pretrained models that needs no GPU, a Hugging Face Spaces web demo, and a Replicate demo with an API.

The requirement list gives a picture of the software stack. It pins faiss-gpu at 1.7.2, opencv-python at 4.6.0.66, scikit-image at 0.19.2, scikit-learn at 1.1.1, shapely at 1.8.2, timm at 0.5.4, pyyaml at 6.0, fvcore at 0.1.5.post20220512, gdown at 4.5.4, and submitit, with colored for terminal output. The torch and torchvision lines are present but commented out, at 1.8.1 and 0.9.1, which means you install the PyTorch build that suits your own setup rather than the one the project pinned. That is a sensible choice and also a compatibility risk worth checking, since the rest of the stack, especially faiss-gpu and fvcore, comes from that era.

The cog.yaml file at the repository root is the configuration for the Replicate deployment, which tells you the inference path is packaged as a container.

VideoCutLER, and the part that needs no motion information

The same repository supports VideoCutLER, unsupervised video instance segmentation, and the claim in the project documentation is that video instance segmentation models can be learned without human annotations, without relying on natural videos since ImageNet data alone is sufficient, and even without motion estimations. The video code lives in its own directory with its own README, and the paper is VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation, by Xudong Wang, Ishan Misra, Ziyun Zeng, Rohit Girdhar, and Trevor Darrell, from UC Berkeley and FAIR, Meta AI, published at CVPR 2024.

The no-optical-flow part is the substantive claim. Most unsupervised video instance segmentation approaches rely on motion cues to separate foreground from background, which ties the method to video data and to a motion estimator. Dropping that dependency means the training data can be still images, which is a large practical difference because image corpora are far easier to obtain and to label-free prepare than video corpora.

The project page for VideoCutLER and its arXiv entry are linked from the main README, along with the arXiv entry for the CutLER paper itself. Both papers are hosted on arXiv, at 2301.11320 for CutLER and 2308.14710 for VideoCutLER, and the main README also links a Colab notebook and the project pages, so the method can be inspected without running anything.

Against a supervised baseline, and against a smaller pseudo-label tool

The comparison that has to be made first is with a supervised detector, because that is what you would actually use if you had labels. The difference in approach is that CutLER replaces an annotation budget with a compute budget. A supervised detector learns from human-marked boxes and masks, and the labelling is a one-off cost with a quality ceiling set by the annotators and the guidelines. CutLER replaces that with a vision transformer's own attention as the teaching signal, which is free but is also less precise, since attention is a proxy for saliency rather than a description of object boundaries.

The honest reading of the project's own claims is that this is a strong option when labels are unavailable and a weak one when they are cheap. The reported comparison is against previous unsupervised state of the art, not against a supervised model, and the stated use of CutLER as a pretrained model for fully or semi-supervised tasks suggests the authors see the combination as the realistic deployment, where you use the unsupervised result as a starting point and then fine-tune on whatever labels you can get.

The second alternative is building pseudo-labels with a different tool. You could run a pretrained supervised detector over your images and treat its output as ground truth, which is the standard self-training recipe and needs no vision transformer machinery. The difference in approach is that CutLER's pseudo-masks come from a saliency-style procedure that is not class-aware, so it does not need a detector in the loop, and it produces multiple instance masks per image rather than a set of class predictions. The cost is the same: you still run a heavy model over the whole dataset.

A third consideration is the practical one about where the labels come from. If your target domain is far from ImageNet, the 2.7 times claim is about the generalisation of the unsupervised result, and the project's own list of 11 benchmarks spanning natural images, video frames, paintings, and sketches is the evidence for that. For a domain none of those cover, the 11 benchmark figure tells you nothing, and the only way to know is to run the pipeline on a small sample of your own data first.

Editorial conclusion

Adopt CutLER if you need detection or segmentation models for a domain where labels are expensive or absent, and if you can afford to run the pseudo-mask generation over a large image set. Do not adopt it as a drop-in replacement for a supervised detector, since the reported gains are the paper's own comparison against previous unsupervised state of the art. Verify first that the compute budget for MaskCut over 1.3 million images fits your plan, because that stage is the expensive part of the pipeline.

Frequently asked questions

Official sources

1. facebookresearch/CutLER on GitHub
2. Issues
3. README