Bayesian compression for dynamically expandable networks

Highlights

• A compact model structure with preserving the accuracy via sparsity inducing priors, which leads to fewer neurons at each hidden layer in the network, equivalently fewer parameters.

• Dynamically expands network capacity with only the necessary number of neurons by employing sparsity inducing priors for the added neurons, so as to increase the network capacity when necessary.

• Variational Bayesian approximation for the model parameters with parameter uncertainty.

Abstract

This paper develops Bayesian Compression for Dynamically Expandable Network (BCDEN), which can learn a compact model structure with preserving the accuracy in a continual learning scenarios. Dynamically Expandable Network (DEN) is efficiently trained by performing selective retraining, dynamically expands network capacity with only the necessary number of units, and effectively prevents semantic drift by duplicating and timestamping units in an online manner. Overcoming conventional DEN only giving point estimates, we providing the Bayesian inference under the principle framework. We validate our BCDEN on multiple public datasets under continual learning setting, on which it can outperform existing continual learning methods on a variety of tasks, and with the state-of-the-art compression results, while still maintaining comparable performance.

Introduction

Continual learning, also called lifelong learning and incremental learning, is an important topic in transfer learning where tasks arrive in sequence. The primary goal of continual learning is to perform well on the entire set of tasks in an incremental way that avoids revisiting all previous data at each stage. Since real world tasks continually evolve over time and the size of datasets often prohibits frequent batch updating which are the key problem in machine learning. To tackle this problem, we exploit the power of deep neural networks for continual learning. Fortunately, storing and transferring knowledge can be done in a straightforward manner through the learned deep network weights. By sharing these learned weights, a balance can be built between adapting to new task and retaining knowledge from existing tasks.

Considering continual learning as a special case of online or incremental learning in case of deep neural networks, multiple ways can be conducted to perform such incremental learning [1], [2]. The simplest way is to incrementally fine-tune the network to new tasks by continuing to train the network with new training data. However, such simple retraining of the network can lead to the catastrophic forgetting problem [3], [4], [5] and an inability to adapt to new tasks. For example, if the previous tasks are classifying images of animals and the new task is to classify images of cars, the features learned on the previous tasks may not be useful for the new one. On the other hand, the retrained representations for the new task could adversely affect the old tasks, as they may have drifted from their original meanings and are no longer suitable for them.

In order to ensure the knowledge sharing through the network is beneficial for all tasks in the online or incremental learning of a deep neural network, [6], [7], [8] prevent drastic changes in the parameters which have a large influence on prediction, but allows other parameters to change more freely. Nguyen et al. [9] merges online variational inference (VI) [10], [11], [12] with Monte Carlo VI for neural networks [13] to yield variational continual learning (VCL), and extends VCL to include a small episodic memory by combining VI with the coreset data summarization method [14], [15]. Yoon et al. [16] proposes a novel deep network model named as Dynamically Expandable Networks (DEN), with dynamically increasing the network capacity by adding in or duplicating neurons when necessary, DEN can maximally utilize the network learned on all previous tasks to efficiently learn to predict for the new task in the continual learning scenario.

While deep neural networks are a widely popular tool in the application of continual learning, they often have more parameters than the number of the training instances. As a result, running them on hardware limited devices remain difficult in many real world scenarios. Hence, compression and efficiency have become a topic of interest in the deep learning community. There is a variety of approaches to address these problem settings. However, most methods have the same strategy of reducing the neural network structure. A justification is the finding that neural networks suffer from significant parameter redundancy [17]. Methods in this line of thought are network pruning, where unnecessary connections are being removed [18], [19], [20], or student-teacher learning where a large network is used to train a significantly smaller network [21], [22].

In this paper, we develop our Bayesian Compression [23] for Dynamically Expandable Networks (BCDEN) which can get pruned network structure with preserved accuracy for each observed task in the continual learning scenarios. We will use the variational Bayesian approximation for the model parameters. By employing sparsity inducing priors for hidden units, not the individual weights, we can prune neurons including all their ingoing and outgoing weights. This avoids more complicated and inefficient coding schemes needed for pruning or vector quantizing individual weights. We train the network from task $t=1$ with sparsity inducing priors to promote group sparsity in the weights, which means fewer neurons to be connected at each hidden layer in the network. When a new task $t$ arrives at the model, BCDEN will first perform selective retraining for the new task. If the new task is highly relevant to the old ones, selective retraining alone will be sufficient for the new task. However, if the old network can not sufficiently explains the new task, BCDEN will dynamically add in the necessary number of neurons by employing sparsity inducing priors for them, to increase the network capacity. When the network shows degenerate performance for earlier tasks, BCDEN will perform network duplication to prevent such semantic drift or catastrophic forgetting.

Our main contributions in this paper include: (1) a compact model structure with preserving the accuracy via sparsity inducing priors, which leads to fewer neurons at each hidden layer in the network, equivalently fewer parameters; (2) dynamically expands network capacity with only the necessary number of neurons by employing sparsity inducing priors for the added neurons, so as to increase the network capacity when necessary; (3) variational Bayesian approximation for the model parameters with parameter uncertainty.