Hysen Labsnninit
Weight initialisation schemes for Torch7 neural network modules
nninit: sensible weight initialisers for Torch7
Weight initialisation for Torch7 modules, built to work with nn and nngraph. An accessor picks the tensor, an initialiser fills it, and a gain table tunes it for the following nonlinearity.
How does the accessor argument work?
The accessor is the piece that pulls the tensor you want to touch out of a module. It can be a string, a table, or a function. With a string or a table, the tensor is fetched as a property of the module from the first element, then a subtensor is taken using Torch's indexing operator on the second element. A function returns the tensor directly from the module. The initialiser itself takes the module and tensor, adjusts the tensor, and hands the module back so calls can be chained.
Which initialisers ship in the box?
nninit carries a set of ready made initialisers. addNormal adds noise from a normal distribution to the current tensor. eye fills linear layers and lookup tables with an identity and convolutional filters with a Dirac delta. xavier, also called Glorot initialisation, sets the standard deviation from fan in and fan out and uses the uniform distribution by default. kaiming, or He initialisation, keys off fan in alone and defaults to the normal distribution. orthogonal fills a tensor of at least two dimensions with a random orthogonal matrix, and sparse zeroes out a chosen percentage of the tensor.
What do the gain mappings do?
Gains are worked out from the nonlinearity that follows. A numeric gain is used as is. A string gain looks up a mapping: linear and sigmoid map to 1, tanh to 5 over 3, relu to the square root of 2, and lrelu to a formula involving the leakiness. When a gain needs extra parameters, you pass a table with the string first and the named parameters after. Where gains apply they default to 1.
How do you install it for development?
To work on the library itself or to trial a new initialisation scheme, clone the repo and run luarocks make with the rockspec file to install nninit locally. The README keeps the developer setup to essentially these two steps, which is about as short as a Torch project gets.
Editorial conclusion
The README is a compact reference for initialising Torch7 modules: an accessor system for picking tensors, a stack of named initialisers, and a gain table for nonlinearities. A single luarocks command installs it for development.
Community notes