Memory Consolidation with Orthogonal Gradients for avoiding Catastrophic Forgetting

Abstract

Abstract The memory consolidation process enables the accumulation of recent and remote memories in the long-term memory store. In general, the deep network models of memory suffer from forgetting old information while learning new information, called catastrophic forgetting/interference, while the human brain overcomes this problem quite effectively. We propose a regularization-based model to solve the problem of catastrophic forgetting. According to the proposed method, the network parameters are constrained to vary in a direction orthogonal to the average error gradients corresponding to the previous tasks. We also ensure that the constraint used in parameter updating satisfies the locality principle. The proposed model’s performance is evaluated by comparing it with Elastic Weight Consolidation under various conditions, from simple to complex datasets and network architectures. The proposed model gives a new view of plasticity at the neuronal level. In the proposed model, the parameter updating is controlled by the neuronal level plasticity rather than synapse level plasticity as in other standard models. The biological plausibility of the proposed model is discussed by linking the extra parameters to synaptic tagging, which represents the state of the synapse involved in Long Term Potentiation.