USB (microsoft/Semi-supervised-learning): A Benchmark Harness for Consistency-Regularization SSL

The problem USB addresses: SSL papers that cannot be compared

Semi-supervised learning research has a reproduction problem that is structural rather than accidental. Each method tends to ship as its own repository, with its own data split, its own augmentation pipeline, its own optimizer schedule and its own evaluation script. Two papers reporting FixMatch results on CIFAR-100 with 200 labels per class can differ by several points purely because one of them applies a different weight decay or a different number of warmup epochs. USB exists to collapse that variance. The README describes it as "a Pytorch-based Python package for Semi-Supervised Learning (SSL)" that is "easy-to-use/extend, affordable to small groups, and comprehensive for developing and evaluating SSL algorithms." The operative word is evaluating. The repository is a benchmark first, and an algorithm library second. It is aimed at graduate students and small research groups who want to place a new method next to established ones without reimplementing the baselines, and at engineers who need to know whether a semi-supervised approach is worth the labeling cost before they invest in one. The NeurIPS 2022 Dataset and Benchmark Track acceptance, noted in the news section, tells you which audience the authors were writing for.

One training loop, 14 algorithms, 15 tasks: how the unification actually works

USB's central design decision is that the algorithms differ only in a small number of places, so the codebase factors those places out and leaves everything else shared. Consistency-regularization methods all follow the same skeleton: take a batch of labeled examples and a larger batch of unlabeled examples, apply a weak augmentation to the unlabeled batch for the teacher pass, apply a strong augmentation for the student pass, compute a pseudo-label from the teacher, and add a consistency loss term weighted against the supervised loss. What varies between FixMatch, FlexMatch, FreeMatch, SoftMatch, DeFixMatch and the rest is how the pseudo-label is produced and how it is masked or reweighted. USB puts those variations into algorithm-specific modules and keeps the data pipeline, the optimizer, the learning-rate schedule, the exponential moving average of the teacher where applicable, and the logging in one place. The README states the package provides "the implementation of 14 SSL algorithms based on Consistency Regularization, and 15 tasks for evaluation from CV, NLP, and Audio domain." That domain spread is the unusual part. Most SSL repositories pick one modality. USB routes vision tasks through torchvision backbones, audio through torchaudio, and text through transformers, which is why the prerequisites list all three libraries alongside PyTorch. The cost of that breadth is that the abstraction has to be loose enough to accommodate a ResNet on CIFAR and a transformer on a text classification task, and loose abstractions leak. Expect to read the algorithm module you care about rather than trusting the config alone.

Getting a run started: conda, pip and one config path

The README's local setup path is short and concrete. Create an environment with conda create --name usb python=3.8, then install dependencies with pip install -r requirements.txt. The first training command given is python train.py --c config/usb_cv/fixmatch/fixmatch_cifar100_200_0.yaml. That single flag, --c, is the entire interface at the top level: it points at a YAML file that names the algorithm, the dataset, the number of labels per class, the backbone and the hyperparameters. The filename encodes the experiment: fixmatch is the algorithm, cifar100 is the dataset, 200 is the label budget per class, and the trailing 0 is a run index, which implies the benchmark repeats each configuration and reports a mean. If you do not want to clone the repository, the README offers a Python package called semilearn, installed with pip install semilearn, described as being for users "who want to start training/testing the supported SSL algorithms on their data quickly." The repository itself is cloned with git clone https://github.com/microsoft/Semi-supervised-learning.git for development of new algorithms. Dataset preparation is handled separately: the October 2022 news entry points to a preprocess directory holding download links and processing instructions, and the January 2023 entry notes that semilearn 0.3.0 added wandb support, so experiment tracking is configured rather than assumed. The distinction between the pip package and the cloned repository matters. The package gives you the algorithms and the training entry point; the cloned repository gives you the configs, the preprocess scripts and the results directory that the benchmark numbers live in.

Where USB is the wrong tool: label budgets, modalities and the imbalanced split

The most important limitation is in the name of the benchmark family. Every one of the 14 algorithms is a consistency-regularization method, and consistency regularization assumes you have a meaningful number of unlabeled examples drawn from roughly the same distribution as your labeled ones. If your unlabeled pool is small, or if it is drawn from a different domain than your labeled set, the pseudo-labeling loop will amplify the mismatch rather than correct it. USB will still run. It will produce a number. The number will be misleading. A second constraint is the label-budget framing. The configs are organized around fixed numbers of labels per class, which is a research convention, not a description of how labeling budgets usually arrive in practice. Real datasets have uneven class coverage, and the imbalanced algorithms added in the 0.3.0 release are the repository's acknowledgement of that, reported separately from the main results table. If your problem is class imbalance, use those variants and read their results, not the headline table. Third, the modality abstraction is not free. A text classification task routed through transformers has a different notion of weak and strong augmentation than an image task, and the shared training loop cannot hide that. Expect to modify the augmentation module for anything outside the supported task list. Finally, the benchmark results themselves are a snapshot of one hardware and software configuration. The README links to logs on Google Drive and to a wandb project, which is the right level of disclosure, but it also means the numbers are only directly comparable to runs made under the same environment.

The alternative you are actually choosing between: TorchSSL and single-method repositories

USB's own news section names its predecessor. The October 2022 entry links to TorchSSL logs as the source of the older classic results, which makes the relationship explicit: USB is the successor codebase to TorchSSL, extended from vision-only to vision, text and audio. If you are choosing an alternative, the real fork in the road is between a unified benchmark and a single-method repository. A single-method repository, the kind released alongside an individual paper, gives you the authors' exact implementation of one algorithm with the hyperparameters that produced their reported number. It is narrower, but it is also closer to the source. USB gives you that same algorithm reimplemented inside a shared loop, which is what makes cross-method comparison possible and what introduces the possibility that a reimplementation drifts from the original. That trade-off is the whole point of a benchmark, and it is also the thing to be suspicious of. If your goal is to reproduce one paper's number as faithfully as possible, the paper's own repository is the safer starting point. If your goal is to know whether method A beats method B on your task under matched conditions, USB is the only one of the two that can answer the question without you rebuilding the harness yourself.

Maintenance, upgrades and what the MIT licence does and does not cover

The repository is not archived, and the last push recorded is 2026-07-26, which is a long way from the 2022 release date and suggests the project is still receiving attention rather than being frozen. The release history is thin, however: the only tagged release listed is v.0.0.0 from July 2022, labelled "Release of Pretrained Models." Everything after that arrived as version bumps to the semilearn package, tracked in a CHANGE_LOG file, with 0.3.0 in January 2023 and 0.3.1 in July 2023, plus the March 2024 addition of EPASS, SequenceMatch and ReFixMatch. That pattern has a practical consequence. If you pin to the pip package, you are pinning to a semilearn version, and the algorithms added after that version will not be present. If you clone the repository, you get the current state of main but you also inherit whatever the configs and preprocess scripts currently assume. Neither path gives you a stable long-term API surface, so treat the version you pick as something to record in your own experiment metadata. The licence is MIT, which is permissive and imposes few conditions on redistribution or modification. The README's own licence badge is commented out in the source, which is a cosmetic inconsistency rather than a legal one, and the repository metadata confirms MIT. This is not legal advice; if you are redistributing the package or bundling it into a commercial product, read the LICENSE file in the repository and check the licences of the datasets and pretrained checkpoints separately, since those are not covered by the code licence and the README does not enumerate them.

What to verify before you build on USB

Three checks will tell you quickly whether USB fits. First, open the preprocess directory and confirm that a dataset structurally similar to yours is already handled. The repository's dataset support is implemented through those scripts, and if your data cannot be expressed as a labeled set plus an unlabeled set with a shared label space, you will be writing that layer yourself. Second, check the config directory for an algorithm and dataset combination close to your target. The path config/usb_cv/fixmatch/fixmatch_cifar100_200_0.yaml is the README's example, and the directory structure under config mirrors the algorithm and dataset taxonomy, so browsing it is the fastest way to see what is actually wired up versus what is described in prose. Third, run one of the existing configs end to end on the smallest supported dataset before touching your own data. That establishes that the environment, the dataset download and the logging all work, and it gives you a reference number you can compare against the results directory. If that run does not reproduce a number in the same range as the published table, the discrepancy is in your environment or your data pipeline, and you want to find that out before you have invested in a custom config. The repository's own results and logs are the baseline to check against, and they are linked directly from the news entries rather than buried in the documentation.

Editorial conclusion

Adopt USB if you need a reproducible baseline for a consistency-regularization SSL method on one of the 15 supported tasks, or if you are writing a paper that must be compared against FixMatch, FlexMatch, FreeMatch or SoftMatch under identical settings. Do not adopt it as a drop-in production trainer for a bespoke pipeline: the config system assumes one of the supported datasets and a single labeled/unlabeled split, and the algorithms are all in the same weak-strong augmentation family. Before committing, verify three things against your own data: that your dataset can be expressed through the preprocess scripts in the preprocess directory, that your backbone is one of the supported torchvision, torchaudio or transformers models, and that the algorithm you intend to use is not one of the imbalanced variants whose results the repository reports separately from the main table.

Official sources

1. License: MIT
2. microsoft/Semi-supervised-learning on GitHub
3. Project website
4. README
5. Releases