# bioinf-jku/SNNs is the three-ingredient recipe, written twice in two frameworks

> This repository reproduces the self-normalizing network setup from Klambauer et al. as TensorFlow/Keras scripts and PyTorch notebooks side by side, on the same three tasks. Its value is as a reference implementation to check a claim against, not as a library, and there is a SKILL.md at the root that suggests it is now being read by tools as much as by people.

**bioinf-jku/SNNs** — Tutorials and implementations for "Self-normalizing networks"

- Repository: https://github.com/bioinf-jku/SNNs
- Stars: 1,591 · Forks: 196
- Language: Jupyter Notebook
- License: GPL-3.0
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/bioinf-jku-snns

## The recipe is three lines long

Self-normalizing networks, as suggested by Klambauer et al., are associated with a very short recipe. The README states it as a list:

- SELU activations
- LeCun normal initialization
- AlphaDropout when dropout is used

That is the whole method. What the paper argues is that with those three choices the activations are self-normalizing, meaning they stay in a stable range through depth without the extra normalisation machinery that other architectures need. What a repository reproducing it has to show is that each framework actually offers these three things, which is where the second half of the README earns its keep.

The reference is the arXiv preprint 1706.02515, linked from the README, and the repository describes itself as tutorials and implementations rather than as a framework. That framing matters for what you can expect: there is no package to install, no API to import, and nothing that will keep working while you build something else with it. There is a set of scripts and notebooks that demonstrate a specific configuration on specific datasets.

The repository is GPL-3.0 and is hosted by the Institute of Machine Learning at Johannes Kepler University Linz, judging from the organisation name and the Tox21 data link pointing at bioinf.jku.at. It is not archived and the last push was on 2026-05-12, and there are no GitHub releases, so the master branch is the only thing to look at.

One entry in the root listing deserves attention before you plan around the project: there is a `SKILL.md`, alongside `AGENTS.md`-style tooling seen in other repositories of this kind. Whatever that file is, it means the repository is being written partly for automated readers, which is worth knowing if you were expecting a conventional research repository with a paper, a citation file and nothing else.

## Setting up means one Conda environment file

The environment section is short because there is only one file to use. The current examples target TensorFlow 2.x with Keras and PyTorch, and the environment is created from the root environment file:

```bash
conda env create -f environment.yml
conda activate snn
```

The environment name is `snn`, which is what the activation line expects, so if you want something else you have to change the name inside `environment.yml` as well.

There is one platform-specific note and it is the kind that saves an hour. On Linux and NVIDIA systems, the TensorFlow GPU dependencies are installed through the `tensorflow[and-cuda]` pip extra rather than being present already.

The README is explicit that the environment is intended for current TensorFlow and PyTorch versions, which is a promise about compatibility rather than about reproducibility over time. There is no pinned-old-environment file, so if a future version of either library changes an API these examples depend on, you have no fallback in the repository. For a teaching repository that is a reasonable trade, but for a reproduction it is a risk you should price in.

Note also what is not in the setup: no dataset download script. The three datasets are linked as raw archives in the Further Material section, and the tutorials assume you have fetched them yourself.

## Keras has all three pieces; PyTorch needs a manual translation

The implementation notes are the genuinely useful part of the README, because the two frameworks are not symmetric and pretending otherwise would waste your afternoon.

Keras provides all three directly. `tf.keras.activations.selu` is the activation, `tf.keras.initializers.LecunNormal` is the initialisation, and `tf.keras.layers.AlphaDropout` is the dropout, with each documented and linked.

PyTorch provides two of the three. There is `torch.nn.SELU` and there is `torch.nn.AlphaDropout`, both documented and linked. For LeCun normal initialisation, the README gives this PyTorch call:

```python
torch.nn.init.kaiming_normal_(weight, mode="fan_in", nonlinearity="linear")
```

The reasoning is worth reading rather than copying. This corresponds to LeCun normal because `fan_in` is used with a gain of 1. That is the whole derivation: PyTorch's Kaiming initialiser takes a mode and a nonlinearity, and the LeCun variant is the `fan_in` mode with linear gain. Getting this wrong is silent, since a network trains either way, it just does not self-normalise and the point of the architecture is lost.

That asymmetry is also the practical test of whether you understand the recipe. In Keras the three items are configuration. In PyTorch one of them is a call you make on every weight tensor of every linear and convolutional layer, and forgetting one layer breaks the property for everything downstream of it.

## Three datasets, two frameworks, six entry points

The tutorials are laid out as a matched pair, which is the point of doing the work twice.

For TensorFlow 2.x and Keras there are three Python scripts: a multilayer perceptron on MNIST, a convolutional network on MNIST, and a convolutional network on CIFAR10. They live under `TF_2_x/`.

For PyTorch there are three notebooks covering exactly the same three combinations: `SelfNormalizingNetworks_MLP_MNIST.ipynb`, `SelfNormalizingNetworks_CNN_MNIST.ipynb` and `SelfNormalizingNetworks_CNN_CIFAR10.ipynb`.

MNIST appears twice because it separates the architecture question from the data question. The MLP on MNIST has no convolution and no image structure, so if something behaves differently there, the cause is the network. The CNN on MNIST adds convolution with a trivial dataset. The CNN on CIFAR10 adds a dataset with real class difficulty. Read as a sequence, that is a controlled comparison rather than three unrelated demos.

The scripts rather than notebooks in the Keras half is worth noticing too. Scripts are easier to run repeatedly and easier to diff, which suits the framework where nothing had to be hand-implemented. Notebooks in the PyTorch half suit the framework where the initialisation call has to be visible in the cell that builds the layer.

Beyond the six tutorials there is more in the repository. `figure1/` holds notebooks and code to produce Figure 1 of the paper, and it builds on the biutils package, so it has an external dependency of its own. `Calculations/` holds Mathematica notebooks and a PDF for the calculations and numeric checks of the theorems, which is a different kind of artefact from the tutorials entirely: those are not neural networks, they are the mathematics.

The data directories are checked in as names, not as downloads: `UCI/`, `Tox21/` and `HTRU2/`. The README links the Tox21 archive from bioinf.jku.at, the HTRU2 set from the UCI archive, and the UCI data from a USC-hosted tarball.

## What this repository cannot tell you

The limitations of the documented approach are significant for anyone planning to use SELU in production.

There is nothing about when SELU is the wrong choice. The README presents the recipe and demonstrates it on three classic benchmarks. It does not compare SELU against ReLU variants on any task, it does not discuss the circumstances where the self-normalising property matters in practice rather than in theory, and it does not say what happens when the conditions the theorem assumes are violated, which is the usual reason architectures fail outside a paper.

There are no hyperparameter results. No learning rates, no widths, no epochs, no accuracy tables. The notebooks will have defaults in them and you can read them, but the repository does not tell you what configuration the authors used or whether they tuned anything.

There is no discussion of inference cost. SELU-based networks are not free to deploy, and the practical questions about serving them are absent.

The `SNN-successes/` directory at the root is named in a way that suggests the repository also tracks where this approach worked, and it is not explained in the README. That is exactly the material that would address the first gap, so it is worth looking at directly before concluding the recipe is broadly applicable.

What the repository is good for is narrower and still worth having: confirming that the three ingredients are available and how they map onto a specific framework, giving you working code to compare against, and providing the numerical checks behind the theorems in a form you can run rather than trust.

The honest framing is that this is a reproduction and teaching resource attached to one paper, not a maintained library and not a state-of-the-art benchmark. Use it to understand the method, not to decide what to deploy.

## Conclusion

Adopt this repository if you are implementing SELU-based networks and want a working reference for what LeCun normal initialization looks like in each framework, or if you want to reproduce the paper's figures to check a claim yourself. Do not adopt it as a dependency or a library, since it is a set of scripts and notebooks with no installable package, and do not expect it to describe when SELU is the wrong activation, because the README is silent on that. Verify first that the `environment.yml` at the root resolves against your TensorFlow 2.x and PyTorch versions, that on Linux with an NVIDIA GPU you install the `tensorflow[and-cuda]` extra as the README specifies, and whether `SKILL.md` and the `SNN-successes/` directory are files you intend to read before drawing conclusions.

## FAQ

### What does bioinf-jku/SNNs implement?

Self-normalizing networks as suggested by Klambauer et al., demonstrated with TensorFlow/Keras scripts and PyTorch notebooks. The recipe is SELU activations, LeCun normal initialization, and AlphaDropout when dropout is used.

### How do I set up the environment for these SNN examples?

Create the Conda environment from the root file with `conda env create -f environment.yml` and then `conda activate snn`. On Linux and NVIDIA systems the README notes that TensorFlow GPU dependencies come from the `tensorflow[and-cuda]` pip extra.

### How do I do LeCun normal initialization in PyTorch?

For PyTorch, the README uses `torch.nn.init.kaiming_normal_(weight, mode="fan_in", nonlinearity="linear")` on linear and convolutional layers. That corresponds to LeCun normal because `fan_in` is used with a gain of 1.

### Which examples are included?

Three in each framework: an MLP on MNIST, a convolutional network on MNIST, and a convolutional network on CIFAR10. The TensorFlow ones are Python scripts under `TF_2_x/`, the PyTorch ones are notebooks under `Pytorch/`.

### Does the repository provide the paper's calculations?

Yes. `Calculations/` contains a Mathematica notebook and a matching PDF for calculations and numeric checks of the theorems, and `figure1/` contains the notebooks and code that produce Figure 1, built on the biutils package.

### Are the datasets downloaded automatically?

No. The setup section covers only the Conda environment. The README links the archives for the UCI data, the Tox21 set from bioinf.jku.at, and HTRU2 from the UCI archive, and the repository has directories named `UCI/`, `Tox21/` and `HTRU2/` for them.

## Sources

- [bioinf-jku/SNNs on GitHub](https://github.com/bioinf-jku/SNNs)
- [Issues](https://github.com/bioinf-jku/SNNs/issues)
- [License: GPL-3.0](https://github.com/bioinf-jku/SNNs/blob/master/LICENSE)
- [README](https://github.com/bioinf-jku/SNNs/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/bioinf-jku-snns