# vision-transformers-cifar10: a ViT training playground on 32 by 32 images

> A PyTorch repository that trains Vision Transformers and a dozen related architectures on CIFAR-10 and CIFAR-100 with a single script, carrying the accumulated training recipe fixes that make ViT behave on small images. It also exports to ONNX and TorchScript, and it has been used as a base by papers at CVPR, ICLR, NeurIPS and ICML.

**kentaroy47/vision-transformers-cifar10** — Let's train vision transformers (ViT) for cifar10/cifar100/Imagenet 

- Repository: https://github.com/kentaroy47/vision-transformers-cifar10
- Stars: 722 · Forks: 136
- Language: Python
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/kentaroy47-vision-transformers-cifar10

## A repository built around one script and one dataset

The stated purpose is direct: this is your go-to playground for training Vision Transformers and its related models on CIFAR-10 and CIFAR-100, a common benchmark dataset in computer vision.

The whole codebase is implemented in Pytorch, which the project says makes it easier to tweak and experiment. And using the repository is straightforward, since all you need to do is run the train_cifar10.py script with different arguments, depending on the model and training parameters you want to use.

The repository layout is small and tells you the scope. train_cifar10.py is the entry point, models holds the architecture implementations, randomaug.py is the RandAugment implementation, utils.py holds helpers, export_models.py handles deployment export, and there are log and requirements files alongside a CITATION.cff.

The audience is researchers and students who want a reference implementation to fork rather than a product to install. The readme notes the repo has been used in 40+ papers including CVPR, ICLR, NeurIPS and ICML, and it names specific papers that used it, such as Vision Transformer Pruning, Understanding why ViT trains badly on small datasets, an adaptive momentum paper, and Moderate coreset.

The citation block is a BibTeX misc entry naming Kentaro Yoshioka as author, and the project asks you to use it if you use this in your research, which is the contract for a research artefact that other people have built on.

## One flag changes the model, the dataset and the recipe

The interface is a single script with a small set of flags, and the readme gives a line per variant. The default run is:

```bash
python train_cifar10.py
```

which trains vit-patchsize-4. The dataset is a flag rather than a separate script:

```bash
python train_cifar10.py --dataset cifar100
```

The patch size and the input resolution are also flags, so a patch size of 2 or a 48 pixel input are one argument away:

```bash
python train_cifar10.py  --size 48
```

```bash
python train_cifar10.py --patch 2
```

The architecture is selected with --net. The listed values cover vit_small, vit_timm for training with a pretrained ViT, dyt for the Layernorm-less ViT, convmixer, mlpmixer, cait and swin, and res18 for a ResNet18 baseline with randaug. A typical longer run names the network, the epoch count and, where it matters, the learning rate:

```bash
python train_cifar10.py --net mlpmixer --n_epochs 500 --lr 1e-3
```

```bash
python train_cifar10.py --net swin --n_epochs 400
```

The learning rate being a flag rather than a constant is worth noticing, because the results table shows that the same architecture reaches very different accuracies depending on how long it trains. A ViT patch 4 is listed at 80% at epoch 200, 85% at epoch 500 and 89% at epoch 1000. The flag is what separates those three numbers, and it is also the flag that most affects whether your own run matches the table.

## The results table is the argument for using this repository

The readme publishes a results table for CIFAR-10 and a smaller one for CIFAR-100, and reading it tells you what this project actually is.

For CIFAR-10, the ViT patch 2 entry sits at 80%, and the patch 4 progression runs 80% at epoch 200, 85% at epoch 500 and 89% at epoch 1000. ViT patch 8 is listed at 30%, which is the striking number in the table: the patch size that works for 32 by 32 images is a small one, and a larger patch collapses the model. ViT small is at 80%, DyT at 74%, MLP mixer at 88%, CaiT at 80% and Swin-t at 90%.

The pretrained rows are the highest and they are labelled as such. ViT small with timm transfer is at 97.5% and ViT base with timm transfer at 98.5%. ConvMixerTiny without pretrain is at 96.3%. The ResNet18 baseline is at 93%, and ResNet18 with randaug at 95%.

That last pair is the most useful line in the table for anyone reading it carefully. The project did not add RandAugment to the Vision Transformer the same way it appears in the ResNet baseline; the 95% figure is the ResNet with augmentation, and the ViT patch 4 at 89% is without that boost, which is the whole point of the training recipe work described in the updates list.

For CIFAR-100 the table is much shorter: ViT patch 4 at epoch 200 is 52%, and resnet18 with randaug is 71%. Those are the only two entries, and they say clearly that this repository's headline results are CIFAR-10 results.

## The recipe changes are the actual contribution

The updates list is a record of two years of fixing a training setup, and it is the part of this repository that a from-scratch implementation would not have.

The order is telling. ConvMixer was added first, in October 2021, described as really simple. Then in March 2022 came wandb train log to reproduce results, then CaiT and ViT-small, then SwinTransformers, and in June 2022 MLP mixer. After the model zoo came the work that matters more: changed default training settings for ViT, then fixed some bugs and training settings in February 2024.

That ordering is the argument. Adding architectures is easy and anyone can do it. What is hard is that the default ViT recipe, which assumes ImageNet scale, does not work on 32 by 32 images, and that the fix is a set of hyperparameters, augmentation settings and bug fixes that took months of trial and error to arrive at. The released versions say the same thing: 0.3 changed default training recipe to fit ViT, and 0.4 fixed training recipe to use randaug as default.

The later entries extend the scope rather than repeat it. Onnx and torchscript model exports were added in December 2024, mobilevit in January 2025, CIFAR-100 support in April 2025, and Dynamic Tanh ViT in June 2025.

RandAugment getting a dedicated implementation file in randomaug.py, and eventually becoming the default, is the concrete thread running through that history. The repository now also has a log directory and the results table links to wandb runs for several entries, so a run can be traced rather than taken on trust.

## Exporting to ONNX and TorchScript, and the dependency list

The repository supports exporting trained models to ONNX and TorchScript formats for deployment purposes, and you export them with the export_models.py script. The basic usage is:

```bash
python export_models.py --checkpoint path/to/checkpoint --model_type vit --output_dir exported_models
```

The three arguments are the checkpoint to load, the model type, which has to match what you trained, and the directory to write into. This is the part of the repository that reaches beyond research, because a training script that only outputs a pickle is not deployable, and a deployment story that requires rewriting the model is a poor one.

The dependency list is unusually short. The requirements file contains vit-pytorch, einops, odach, wandb. That is four packages, and it tells you something about the design. einops is the tensor notation library that makes attention code readable. odach appears to be the author's own optimizer package. wandb is the experiment tracker, and its presence as a hard requirement rather than an optional extra is worth noting, since it means the training script is expected to log to a Weights and Biases project rather than to a local file.

That last point has a practical consequence for a reader deciding whether to use this repository. The stated purpose of the wandb integration was to reproduce results, and the results table links to wandb reports. If you want to run these recipes and compare against the published numbers, the intended path is to log to wandb. If you have no wandb account and no interest in one, the dependency is still declared in the requirements file.

## Where this repository is the wrong choice

There are clear boundaries, and they come from the scope rather than from any defect.

The first is scale. The whole repository is built around 32 by 32 images, and the patch sizes, the training recipe and the results table all assume CIFAR. If you are training on ImageNet, the argument for using this code weakens considerably, since a ViT at that scale is a different engineering problem with different augmentation and schedule decisions. The description mentions ImageNet, but the readme's own framing is CIFAR-10 and CIFAR-100.

The second is that it is a script, not a library. There is no package to install and no importable API; you clone it and run a script with flags. That is the right shape for a research playground and the wrong shape if you want ViT as a component in a larger system, where you would be writing your own training loop anyway and would take the model definitions as a starting point.

The third is the results table and the recipe moving together. The default training recipe changed in the 0.3 release, was fixed again in 0.4, and the readme's changelog entries for the 2024 bug fixes and the randaug default land in the same period. Anyone reproducing a number from an older fork should expect to get something different until they match the current defaults, and the table does not annotate which rows predate the recipe change.

The fourth is the dependency on a live experiment tracker, and the fifth is the download count of the four listed packages, where wandb being mandatory means the training script always wants a wandb project. A user who wants a quiet offline training run will have to look at how the script initialises the tracker.

And the last one is simply the pace. The last push was on 2026-05-12, and the released versions are 0.4 from September 2024, 0.3 from June 2022 and 0.2 from March 2022. The code has been touched more recently than the last tag, so the working tree and the released versions differ, and the version in use depends on which of the two you take.

## Against writing your own loop, and against a framework level library

Two comparisons put the value in place.

The first is against writing your own training loop. A ViT on CIFAR-10 is not a hard model to write, it is a hard model to make work at that resolution. The patch size has to be small, the augmentation has to be right, the learning rate and schedule have to suit the setting, and the result is 89% at epoch 1000 rather than 30% if you get the patch size wrong. The repository's value is the accumulated set of those decisions, and the 2022 and 2024 updates entries are the record of finding them. Whether that is worth forking depends on how much of your own time you would spend rediscovering them.

The second is against a framework level library. The timm model zoo is the obvious comparison, and the repository itself points at it, since the vit_timm option trains with a pretrained ViT and the results table lists timm transfer rows at 97.5% and 98.5%. The difference in approach is that a general library gives you the model and leaves the recipe to you, while this repository gives you the model and the recipe that is known to work on small images, at the cost of being a fixed script rather than a configurable library. A timm user with their own recipe and their own training loop does not need this repository. A student who wants to reproduce a published number does.

## Conclusion

Adopt vision-transformers-cifar10 if you need a working ViT training setup on small images and want the accumulated recipe fixes rather than writing your own, and if you want the option of exporting to ONNX or TorchScript. Skip it if you are training on ImageNet scale, since this repository is built around the CIFAR settings. Verify first that the result table in the readme reflects the recipe you are running, because the default training recipe changed in ways that moved these numbers, and check the requirements file, since it pins a small set of packages.

## FAQ

### How do I train a Vision Transformer on CIFAR-10 with vision-transformers-cifar10?

Install the dependencies with pip install -r requirements.txt, then run python train_cifar10.py for the default vit-patchsize-4. Use --dataset cifar100 for CIFAR-100, --patch 2 for a different patch size, --size 48 for a larger input, and --net to pick vit_small, vit_timm, dyt, convmixer, mlpmixer, cait, swin or res18.

### What accuracies does the vision-transformers-cifar10 results table report for CIFAR-10?

The table lists ViT patch 4 at 80% for epoch 200, 85% for epoch 500 and 89% for epoch 1000, plus ViT patch 2 at 80%, ViT patch 8 at 30%, ViT small at 80%, DyT at 74%, MLP mixer at 88%, CaiT at 80%, Swin-t at 90%, ConvMixerTiny without pretrain at 96.3%, resnet18 at 93% and resnet18 with randaug at 95%.

### Can I export the models trained with vision-transformers-cifar10?

Yes. The repository supports exporting trained models to ONNX and TorchScript formats using the export_models.py script, for example python export_models.py --checkpoint path/to/checkpoint --model_type vit --output_dir exported_models.

### What are the dependencies of vision-transformers-cifar10?

The requirements file lists four packages: vit-pytorch, einops, odach and wandb. The wandb dependency is what produces the train logs linked from the results table.

### What changed between the 0.3 and 0.4 releases of vision-transformers-cifar10?

The 0.3 release changed the default training recipe to fit ViT, and the 0.4 release fixed the training recipe to use randaug as default. Earlier releases in the series include 0.2 from March 2022, with 0.4 dated 2024-09-11.

### Is vision-transformers-cifar10 actively maintained?

The repository is not archived and the last push was on 2026-05-12, but the most recent tagged release is 0.4 from 2024-09-11, and the earlier tags are 0.3 from 2022-06-23 and 0.2 from 2022-03-13. The licence is MIT.

## Sources

- [Issues](https://github.com/kentaroy47/vision-transformers-cifar10/issues)
- [kentaroy47/vision-transformers-cifar10 on GitHub](https://github.com/kentaroy47/vision-transformers-cifar10)
- [License: MIT](https://github.com/kentaroy47/vision-transformers-cifar10/blob/main/LICENSE)
- [README](https://github.com/kentaroy47/vision-transformers-cifar10/blob/main/README.md)
- [Releases](https://github.com/kentaroy47/vision-transformers-cifar10/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/kentaroy47-vision-transformers-cifar10