Memory Aware Synapses: forgetting by penalising important weights

The method is an importance score, not a memory buffer

The starting observation is about human learning and then about model capacity. People learn continuously, where old knowledge that is rarely used can be overwritten by new information while knowledge that is used often is protected from being erased. Lifelong learning research has mostly concentrated on accumulating knowledge across tasks and on catastrophic forgetting. The argument here is narrower and more uncomfortable: given limited model capacity and an unlimited supply of new information, knowledge cannot all be kept, so it has to be preserved or erased selectively. Memory Aware Synapses, or MAS, is the mechanism proposed for that decision. It computes the importance of the parameters of a neural network in an unsupervised and online manner, so no labels are needed to decide what matters.

Importance is how sensitive the output is to that weight

The definition of importance is what makes the method concrete. Given a new sample fed to the network, MAS accumulates an importance measure for each parameter based on how sensitive the predicted output function is to a change in that parameter. It is a running quantity, accumulated as samples arrive, rather than a one-off analysis over a trained model, which is what lets it adapt to what the network is currently doing. The use of that measure is the second half: when a new task is learned, changes to important parameters can be penalized, which is what prevents knowledge tied to previous tasks from being overwritten. So the mechanism is a regularizer whose weight per parameter is not a hand-set constant but a quantity the network itself maintains, and the repository contains the PyTorch implementation of exactly that, described as Memory Aware Synapses: Learning what not to forget.

A local version connects back to Hebb's rule

The biological framing is inspiration rather than decoration. The approach is described as inspired by neuroplasticity, and the paper draws an explicit connection between a local version of the method and Hebb's rule, the model of the learning process in the brain. That local connection is the part worth noticing: a restricted or simplified form of the accumulation rule resembles a Hebbian statement about which connections strengthen together. It also explains why the importance measure can be unsupervised, since the sensitivity signal comes from the forward pass rather than from any target. For a reader, this is the part of the paper that transfers beyond the particular code, because the Hebbian form is small enough to reason about by hand and does not depend on the object recognition or triplet experiments the repository ships.

Unlabelled data can shift what the network is told to forget

One claim in the paper is about adaptability rather than raw accuracy. The authors report state-of-the-art performance on their experiments, and then add what they describe as a first: the ability to adapt the importance of the parameters based on unlabeled data, towards what the network needs or does not need to forget. The qualification they attach matters as much as the claim, because what should be protected may vary depending on the test conditions. In practice this means the same accumulation run over a stream of unlabelled inputs can end up weighting parameters differently from a run over a different stream, without anyone re-tuning the penalty by hand. That is the feature that separates this approach from a fixed importance estimate computed once from a finished model, and it is the reason the online part is in the name of the method.

Two experiment families, and one demo file

The evaluation described in the README covers two kinds of task. The first is a sequence of object recognition tasks, which is the standard setting for measuring forgetting because a network is asked to learn several recognition problems in order. The second is the harder problem of learning an embedding for predicting subject, predicate and object triplets, where the representation itself has to carry relational structure rather than a class label. The repository ships a PyTorch implementation plus a demo file that shows a learning scenario using an MNIST split set of tasks, which is the smallest version of the sequential setup: digit classification split into stages, so you can watch what happens to earlier stages as later ones are trained. The primary language recorded for the repository is Jupyter Notebook, which fits a demo-led distribution.

PyTorch 0.3, Python 3.5 and CUDA 9.1 are the stated requirements

The requirements line is short and specific: the code was built using PyTorch version 0.3, Python 3.5 and CUDA 9.1. That is the single most consequential fact for anyone arriving today, because it dates the code to a generation of the framework well before the current releases and pins a CUDA runtime that no longer ships in the same form. It is worth knowing before you try to install anything. The repository is not archived, so the files are all still there, but nothing has been pushed to the master branch since 8 October 2018, and there are no published releases to install from. If you want to run the demo, plan on reading the code rather than on an install path, and treat any modern port as your own work.

One folder, named as if it were still unpublished

The tree has three entries and one of them explains a lot about how this was released. There is a directory called MAS_to_be_published, a README and a teaser image. So the implementation sits in a folder whose name says the release was pending, which is consistent with a paper artefact rather than a packaged library: there is no setup script, no module layout and no version file at the root. The README names the authors, Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach and Tinne Tuytelaars, and gives a single contact address for questions about the code rather than an issue tracker. The citation points at the ECCV 2018 proceedings in Lecture Notes in Computer Science, volume 11207, published by Springer in Cham.

Editorial conclusion

This repository fits someone who wants to read or reimplement the mechanism behind Memory Aware Synapses, and someone revisiting catastrophic forgetting who wants a small, readable PyTorch reference rather than a maintained framework, because the method is compact enough to read in one sitting and the demo shows the learning scenario on MNIST task splits. Three things to know first. The last commit on the master branch is 8 October 2018 and there are no releases, so nothing here tracks current PyTorch. The stated build environment is PyTorch 0.3, Python 3.5 and CUDA 9.1, which is the real obstacle to running it today rather than any missing documentation. And the licence statement is one sentence, that the package is freely available for research purposes, with no OSI identifier and no licence file at the root, so check that against your institution's rules before building on it.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. README
4. Releases