MAS-Memory-Aware-Synapses

Memory Aware Synapses is a lifelong learning method that keeps important knowledge intact while letting less used knowledge be overwritten. This repo carries a PyTorch implementation and a demo.

Humans keep learning for a lifetime because the brain does not treat every memory equally. Rarely used knowledge can be overwritten by new input, while frequently used knowledge stays protected. The paper that this code implements argues that artificial systems face the same constraint, limited model capacity against unlimited new information, so knowledge has to be preserved or erased selectively.

MAS computes, in an unsupervised and online way, how important each parameter of a network is. For a given sample it accumulates a measure based on how sensitive the predicted output is to a change in that parameter. When a new task arrives, changes to the important parameters are penalized, which keeps earlier knowledge from being overwritten. The authors also point out a connection between a local version of the method and Hebb's rule. Testing covered a sequence of object recognition tasks and the harder problem of learning an embedding for predicting triplets.

This directory holds a PyTorch implementation plus a demo file that walks through a learning scenario on a split MNIST set of tasks. The author list is Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars, with contact details for the main author. The citation is provided as the ECCV 2018 paper in Lecture Notes in Computer Science volume 11207, which makes the release easy to reference in later work. The README stops there: no training scripts, no benchmark tables, just the method's code and the demo.

• Source 1 · Project repository
• Source 2 · README