# inplace_abn, BatchNorm rewritten so its input buffer can be thrown away

> In-Place Activated BatchNorm is a PyTorch layer that redefines a normalisation followed by a non-linearity as a single in-place operation, which lets the training loop discard one of the two large activation buffers a convolution stack normally keeps. The saving is up to half the GPU memory, and the price is a changed BatchNorm parametrisation that makes the weights incompatible with a standard BatchNorm network.

**mapillary/inplace_abn** — In-Place Activated BatchNorm for Memory-Optimized Training of DNNs

- Repository: https://github.com/mapillary/inplace_abn
- Stars: 1,333 · Forks: 183
- Language: Python
- License: BSD-3-Clause
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/mapillary-inplace-abn

## The two buffers that a convolution stack has to keep

The problem statement in the README is short and precise. When processing a BatchNorm, activation and convolution sequence in the forward pass, most deep learning frameworks need to store two big buffers, the input x of BatchNorm and the input z of the convolution. This is necessary because the standard implementations of the backward passes of BatchNorm and convolution depend on their inputs to calculate the gradients.

That is the whole difficulty. Autograd saves what a backward pass will need, and a BatchNorm followed by a convolution needs the input to the BatchNorm and the input to the convolution. In a deep network those are full-size activation tensors at every layer, and they are what makes training a large ResNet expensive in memory rather than in compute.

The approach is to replace the BatchNorm and activation sequence with a single in-place operation, which lets the framework safely discard x, saving up to 50 percent of GPU memory at training time. The README frames the result in terms of modern architectures specifically, naming ResNet, ResNeXt and Wider ResNet, which is the right framing because those are the families where the BatchNorm and convolution sequence repeats dozens of times and the buffers dominate.

The saving is a training-time saving, and that is worth being explicit about. Nothing about the inference graph changes size, because inference does not run a backward pass and does not need to save anything for later. The layer is a memory optimisation for the training loop, and a team that has already finished training and is now serving a model gets nothing from it.

The paper is In-Place Activated BatchNorm for Memory-Optimized Training of DNNs, published at CVPR 2018 by Rota Bulò, Samuel, Lorenzo Porzi and Peter Kontschieder, and the repository holds the PyTorch implementation plus the training scripts used to reproduce the ImageNet results.

## Rewriting the BatchNorm backward pass in terms of its output

How do you get rid of x if the backward pass needs it? The answer in the README is that the backward pass of BatchNorm is rewritten in terms of its output y, which is in turn reconstructed from z by inverting the activation function.

The chain is: the activation takes z and produces y, the layer stores y in place of x, and when the backward pass needs the BatchNorm input, it recovers it by applying the inverse of the activation to the output. That works only if the activation is invertible, and the usual suspects are not. ReLU is not invertible, and neither is a hard tanh, so the plain versions of these layers cannot be used at all.

That constraint is what shapes the rest of the design, and the README gives the second half of the answer. The parametrisation for the scaling factor of BatchNorm changed compared to standard BatchNorm, in order to ensure an invertible transformation, and the specific form of that scaling is given as an equation in the repository, shipped as an SVG so it renders in the readme.

So the layer is not simply a memory optimisation that happens to work. It is a different parametrisation of BatchNorm, chosen so that the whole composition can be inverted, and the memory saving is a consequence of that parametrisation rather than a trick layered on top of one.

For a reader this is the part to check against the paper, because it is where the design has teeth. A reviewer who wants to know whether the saving is real should look at whether the transformation is genuinely invertible, and the readme is unusually forthcoming in showing the equation rather than describing it in words.

## The weight incompatibility that the maths creates

The third note in the requirements section is the one that decides who can use this. In general, it is not possible to load weights from a network trained with standard BatchNorm into an InPlace-ABN network without severe performance degradation, due to the different handling of the BatchNorm scaling parameters.

This is a direct consequence of the previous section. The scaling factor is parametrised differently in order to make the transformation invertible, and a scaling factor is part of what a trained network stores. Convert a standard BatchNorm network's weights into the new parametrisation and the arithmetic is not the identity, so the network that comes out is not the network that went in. The result is a model that runs and predicts badly, which is the worst outcome of the three possible ones, because nothing errors out.

The practical implications are large for a modern workflow. There is no standard BatchNorm checkpoint you can convert, no state dictionary to reuse, and no way to compare a fine-tuned standard model against its InPlace-ABN counterpart on the same initialisation. Everything has to be trained from scratch, which is a cost in compute rather than in memory, and the cost of a training run is exactly what a team trying to save memory is usually trying to avoid.

It also means the technique is for training, not for adapting. If your plan is to take a released checkpoint, add a layer to fit a memory budget and carry on, this is the wrong tool. If your plan is to train a model on a dataset you control, with a budget that the memory saving makes possible, it is the right one.

The ImageNet results in the repository are therefore not only a demonstration of accuracy, they are also the only way to obtain a usable starting point, and the trained models are published with the weights, which is the practical answer to the compatibility problem.

## The extension builds itself, and you can force it on a machine without a GPU

The installation has two paths, and the first one is a trap for people who expect a pure Python package.

```bash
pip install inplace-abn
```

That is the documented route, and the readme immediately warns that some parts of InPlace-ABN have native C++ and CUDA implementations, meaning that the command will need to compile them. So a pip install is also a compiler invocation, with a CUDA toolchain when a GPU is present.

The alternative route also fetches the training scripts:

```bash
git clone https://github.com/mapillary/inplace_abn.git
cd inplace_abn
python setup.py install
cd scripts
pip install -r requirements.txt
```

The last command installs the extra libraries the ImageNet and Vistas scripts need, which is a separate concern from the layer itself and is easy to miss if you install the package and then try to run a training script.

The build logic in setup.py is worth reading, because it answers a question that comes up in containers. The sources are collected by walking the tree for .cpp and .cu files, and the extension is built as a CUDA extension when torch.has_cuda is true or when the environment variable IABN_FORCE_CUDA is set to 1. Otherwise it is built as a plain C++ extension with the same name, inplace_abn._backend, and the same include directory. Both variants compile with -O3, and the CUDA build defines WITH_CUDA.

That is why the readme documents a way to force the CUDA build on a system without a GPU, which is the situation in a container that only builds the wheel:

```bash
export TORCH_CUDA_ARCH_LIST="{archs}"
export IABN_FORCE_CUDA=1
```

where the architectures are a list such as Pascal;Volta, or the equivalent numbers 6.0;6.5. The first variable tells the build which GPU architectures to compile for, and the second tells the setup script to take the CUDA path even though torch reports no CUDA device. Without them, a Docker build of the package silently produces a CPU-only binary.

The rest of the repository is a normal research project layout, with inplace_abn/ for the Python layer, src/ and include/ for the native code, scripts/ for the experiments, a setup.cfg, a MANIFEST.in, a licenses.csv, and a requirements file for the training scripts.

## Reading the ImageNet table carefully

The README publishes ImageNet-1k results with trained models and md5 checksums, and the table is the interesting artefact because it lets you see both what the layer achieves and what it costs.

The ResNeXt101 standard BatchNorm row is trained with a batch of 256 and reports 77.04 top-1 and 93.50 top-5 at 224 pixels, with 78.72 and 94.47 at ten crops and 77.92 and 94.28 at 320. The ResNeXt101 InPlace-ABN row is trained with a batch of 512 and reports 78.08 and 93.79, with 79.52 and 94.66 and 79.38 and 94.67. WideResNet38 with InPlace-ABN at batch 256 gives 79.72 and 94.78. ResNeXt152 gives 78.28 and 94.04, DenseNet264 gives 78.57 and 94.17, and the synchronised InPlace-ABN variants of ResNet34, ResNet50 and ResNet101 at batch 512 give 73.27, 75.53 and 77.07 top-1 respectively.

There is a methodological point in that table that a reader should not miss. The standard BatchNorm ResNeXt101 uses a batch of 256 and the InPlace-ABN one uses 512, so the headline improvement of one point of top-1 accuracy is measured with different batch sizes. A larger batch is itself worth accuracy on ImageNet, so the difference between the two rows is not attributable to the layer alone. The table is honest in listing the batch column, and the trained models and checksums let somebody check the claims, which is more than most research readmes offer.

The models are made available under the licence attached to ImageNet, which is the reason they can be distributed at all, and each row links to an experiment configuration under scripts/experiments and to the model file with its md5.

The scripts also use torchvision.datasets.ImageNet for data preparation, which is where the pinned requirements file matters.

## Mapillary Vistas, and the inference code that came later

The project grew past classification. The README records that the inference code for semantic segmentation has also been released, together with the Mapillary Vistas trained model that led to the first position on the Mapillary Vistas Semantic Segmentation leaderboard.

The two problems are genuinely different in the way that matters for this layer. Classification training is where the activation buffers dominate, because the network is deep and the batch is large. Semantic segmentation training is dominated by something else entirely, the resolution of the output, since a segmentation map at full input resolution is much larger than a classification logit vector, and the memory pressure moves from the activations to the labels and the output tensors.

That context matters for reading the leaderboard result. A first position on a public leaderboard is a real result, but the leaderboard entry was produced with the released model and the released inference code, and the README does not claim that the segmentation training itself used the in-place layer to any particular degree. What the readme does claim is that the inference code exists and the model leads the board.

The organisation behind the project is worth noting too, because it explains the shape of the work. This is Mapillary's repository, and Mapillary Vistas is a large street-level segmentation dataset, which is the kind of data where memory during training is a real constraint rather than an abstract one. The setup.py carries a Facebook copyright, and the layer was published as a paper with authors from the research group, so the implementation is a research artefact with a production use behind it rather than a paper with a reference implementation.

For an adopter, the practical takeaway is that the layer is proven on ImageNet classification at batch sizes of 256 and 512, and that the segmentation work is a separate release with its own leaderboard entry.

## Where InPlace-ABN is the wrong tool

The first limit is the platform list, and the README states it in one line: only support for Linux platforms and CUDA versions of 10.0 or later is provided, and PyTorch 1.1 or later is required. That is a constraint written into a package whose requirements file now pins torch 2.7.0, and it means a researcher on a Mac, or on a CPU-only machine, is outside the supported set unless the C++ path is enough for their use.

The second is the build. The pip install compiles native code, so the package cannot be installed on a machine without a compiler, and cannot be installed at all in a locked-down environment where a compiler is not available. Container builds need the two environment variables, and the CUDA architectures have to be chosen deliberately, which is a detail that will be wrong on somebody's hardware eventually.

The third is the version story. The tagged releases are v1.0.11 for PyTorch 1.4 compatibility, v1.0.12 for 1.5, and v1.1.0 for feature parity with PyTorch's BatchNormNd plus a code clean-up, all in 2020. The branch has moved on since, with the last push on 2026-10-14 and the requirements file pinning torch 2.7.0, so the code is modern and the tags are not. The scripts requirements file also pins torchvision at 0.5.0, a version from the PyTorch 1.0 era, against torch 2.7.0, which is a combination that will not install cleanly and is worth checking before you plan a training run.

The fourth is the fundamental one, that the memory saving is only available if you can train from scratch. With no standard checkpoint loadable, the saving is paid for in compute. For a team with a fixed compute budget and a memory problem, that is a bad exchange. For a team with a fixed deadline and a memory limit, it is a good one.

The fifth is that the layer is a research implementation. It targets the ResNet family, the paper's evidence is ImageNet classification, and the segmentation release is a separate code path. A production training stack that needs general support, or that runs on hardware the CUDA path does not cover, needs a different answer.

## Conclusion

The layer fits a team training ResNet-family classifiers on GPUs it cannot afford to double, since the memory saving comes from the activation tensors rather than from a smaller model, and the ImageNet table in the repository shows the accuracy is preserved or improved. It does not fit a team that needs to start from standard BatchNorm checkpoints, that trains on macOS or on Windows, or that wants a maintained package with regular releases. Before adopting it, check the two constraints the README states plainly, that PyTorch 1.1 or later and Linux with CUDA 10 or later are required, and remember that the pip install compiles native C++ and CUDA code, so a build toolchain is a prerequisite rather than an optional extra. The licence is BSD-3-Clause, the last push was on 2026-10-14, and the tagged releases stop at v1.1.0 from 2020.

## FAQ

### What does InPlace-ABN actually save?

Up to 50 percent of GPU memory at training time, in architectures such as ResNet, ResNeXt and Wider ResNet. It redefines BatchNorm plus the non-linear activation as a single in-place operation, which allows the BatchNorm input buffer x to be discarded, since the backward pass is rewritten in terms of the output y.

### Can I load standard BatchNorm weights into an InPlace-ABN network?

Not without severe performance degradation, according to the README. The BatchNorm scaling parameters are handled differently, because the scaling factor is parametrised differently to keep the transformation invertible, so the conversion is not the identity and the resulting model predicts badly.

### How do I install inplace-abn?

With `pip install inplace-abn`, which compiles the native C++ and CUDA parts, or by cloning the repository and running `python setup.py install`, then installing the extra libraries the ImageNet and Vistas scripts need with `pip install -r requirements.txt` from the scripts directory.

### How do I build the CUDA extension on a machine with no GPU?

Set TORCH_CUDA_ARCH_LIST to the list of target architectures, for example Pascal;Volta or 6.0;6.5, and set IABN_FORCE_CUDA=1, which makes the setup script take the CUDA path even when torch reports no CUDA device. The resulting wheel then contains the CUDA kernels.

### What ImageNet results are reported for InPlace-ABN?

The table in the README lists top-1 and top-5 scores at 224 pixels, at 224 with ten crops, and at 320, for ResNeXt101 with standard BatchNorm, ResNeXt101, ResNeXt152, WideResNet38, DenseNet264 and synchronised InPlace-ABN variants of ResNet34, ResNet50 and ResNet101, together with trained models and md5 checksums.

## Sources

- [Issues](https://github.com/mapillary/inplace_abn/issues)
- [License: BSD-3-Clause](https://github.com/mapillary/inplace_abn/blob/main/LICENSE)
- [mapillary/inplace_abn on GitHub](https://github.com/mapillary/inplace_abn)
- [README](https://github.com/mapillary/inplace_abn/blob/main/README.md)
- [Releases](https://github.com/mapillary/inplace_abn/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/mapillary-inplace-abn