Abstract
sequential learning=lifelong learning=incremental learning = continual learning, look at the scenario with fixed model capacity, the learning process should account for future tasks to be added and thus leave enough capacity for them. (not selfish)
Introduction
the challenge situation of learning a sequence of tasks, without access to any previous or future task data and restricted to a fixed model capacity. 哺乳動物mammalian brain的大腦學習任務。用神經科學的內容來解釋motivation.首先，較少數量的神經元被激活用於表徵信息，然後被激活的神經元會減少該神經元周圍的神經元的活動（lateral inhibition）。This creates a powerful decorrelated and compact representation with minimum interference between different input patterns in the brain (Yu et al., 2014)
《Reducing overfitting in deep networks by decorrelating representations arxiv2015》 show that when the amount of overfitting in a neural network is reduced, the representation correlation is also reduced
Parameter sparsity or representation sparsity.
要理清幾個概念，disentangled representation.解耦的表達更不容易遭遇災難性遺忘
Sparse and decorrelated representation。Decorrelated representation=disentangled representation
EWC, MAS
後面講sparsity，the main idea of our regularizer is to penalize neurons that are active at the same time.
Method
Sparse coding through Local Neural Inhibition and Discounting (SLNID). 介紹了一種新的regularizer, which encourages sparsity in the activations for each layer.
Conclusion
sparsity should be imposed at the level of representation rather than at the level of the network parameters.
提出方法的motivation來自於lateral inhibition in the mammalian brain. 具體地，a new regularizer that decorrelates nearby active neurons.
Leaning a new task selflessly by leaving capacity for future tasks, avoid forgetting previous tasks通過考慮神經元的重要性neuron importance（之前的工作相似的insight，parameter importance）
Key points: 這篇文章motivation很好;包裝方法的解釋（神經科學）可以多學習