Incremental Task Learning with
Incremental Rank Updates

Deep neural networks have been extremely successful for a variety of learning and representation tasks (e.g., image classification, object detection/segmentation, reinforcement learning, generative models). A typical network is trained to learn a function that maps input to the desired output. The input-output relation is assumed to be fixed and input-output data samples are drawn from a stationary distribution [25]. If the input-output relations or data distributions change, the network can be retrained using a new set of input-output data samples. Since the storage, computing, and network capacity are limited, we may need to replace old data samples with new samples. Furthermore, privacy concerns may also force data samples to be available for a limited time [10, 25]. In such a training process, a network often forgets the previously learned tasks; this effect is termed catastrophic forgetting [21, 26].

Incremental task learning is a subcategory of continual learning or lifelong learning approaches aim to address the problem of catastrophic forgetting by adapting the network or training process to learn new tasks without forgetting the previously learned ones [23, 16, 3, 2, 4, 6, 28, 29, 12]. In this paper, we focus on task-incremental continual learning in which data for every task are provided in a sequential manner to train/update the network [8]. It has been a popular continual learning setup even in the very recent literature [7, 14, 11, 36, 31, 40]. ITL finds its application in setups where task id is available during inference; for instance, tasks performed under different weather/light/background conditions and we know the changes, or tasks learned on different data/classes where we know the task id.

Let us denote the network function that maps input $x$ to output for task $t$ as $f(x;\mathcal{W}_{t})$, where $\mathcal{W}_{t}$ denotes the network weights for task $t$. We seek to update the $\mathcal{W}_{t}$ for all $t$ as we sequentially receive dataset for one task at a time. Suppose the training dataset for task $t$ is given as $(\mathcal{X}_{t},\mathcal{Y}_{t})$ drawn from a distribution $\mathcal{P}_{t}$, where $\mathcal{X}_{t}$ denotes the set of input samples and $\mathcal{Y}_{t}$ denotes the corresponding ground-truth outputs. Our goal is to update network weights from the previous task ($\mathcal{W}_{t-1}$) to $\mathcal{W}_{t}$ such that

ITL setup above assumes that the task identity of test samples is known at the test time and the corresponding network weights are used for inference. Dynamic architecture approaches have the potential to achieve zero forgetting, using $\mathcal{W}_{t}$ for testing data for task $t$; however, this also requires storing the $\mathcal{W}_{t}$ for all the tasks. One of the main contributions of this paper is to represent, learn, and update the $\mathcal{W}_{t}$ using low-rank factors such that they can be stored and applied with minimal memory and computation overhead.

We propose a new method for ITL that updates network weights using rank-1 (or low-rank) increments for every new task. Figure 1 provides an illustration of our proposed method. We represent the network weights for each layer as a linear combination of several low-rank factors (which can be represented as a product of two low-rank matrices and a diagonal matrix). To update the network for task $t$ without forgetting the earlier tasks, we freeze the low-rank factors learned from the previous tasks, add a new trainable rank-1 (or low-rank) factor for every layer, and combine that with the older factors using learnable selector weights (shown as a diagonal matrix). We use a multi-head configuration that has an independent output layer for each task. As we are learning separate diagonal matrices for every task, we can achieve zero forgetting during inference. We present an extensive set of experiments to demonstrate the performance of our proposed method for different benchmark datasets. We observe that our proposed method outperforms the current state-of-the-art methods in terms of accuracy with small memory overhead.

The main contributions of this paper are as follows.

1. 1.

   Represent layers as low-rank matrices: We represent and learn network weights for each layer as a low-rank structure. We show that low-rank structure is sufficient to represent all the tasks in continual learning setup.

2. 2.

   Reuse old factors for better performance with a small memory overhead: We limit the number of parameters required for network update by reusing the factors learned from previous tasks. We demonstrate that a rank-1 increment suffices to outperform the existing techniques.

3. 3.

   Zero forgetting without replay buffer: Our method has zero forgetting that is achieved using incremental rank update or network weights. In contrast, most of the existing continual learning techniques require replay buffer or large memory overhead to achieve zero forgetting.

Incremental task learning (ITL) [10, 34] aims to train a single model on a sequence of different tasks and perform well on all the trained tasks once the training is finished. While training on new tasks, the old data from previous tasks will not be provided to the model. This scenario mimics the human learning process where they have the ability to acquire new knowledge and skills throughout their lifespan. However, this setting is still challenging to neural network models as a common phenomenon called ”catastrophic forgetting [21]” is observed during this learning process. Catastrophic forgetting occurs when the data from the new tasks interfere with the data seen in the previous tasks and thus deteriorating model performance on preceding tasks. To overcome this issue, different approaches have been proposed so far which can be divided into three main categories: regularization-based approaches, memory and replay-based approaches, and dynamic network architecture-based approaches. Some of these approaches are especially designed for ITL whereas others are designed for more general continual learning setup.

Regularization-based approaches [15, 23, 16] update the whole model in each task but a regularization term $\ell_{reg}$ is added to the total loss $\mathcal{L}=\ell_{current}+\lambda\ell_{reg}$ to penalize changes in the parameters important to preceding tasks thus preserving the performance on previous learned tasks. For example, Elastic Weight Consolidation (EWC) [15] estimates the importance of parameters using Fisher Information matrix; Variational Continual Learning (VCL) [23] approximates the posterior distribution of the parameters using variational inference; Learning without Forgetting (LwF) [16] regularizes the current loss with soft targets taken from previous tasks using knowledge distillation [13]. GCL [5] mixes rehearsal with knowledge distillation and regularization to mitigate catastrophic forgetting. A number of recently proposed methods force weight updates to belong to the null space of the feature covariance [37, 35].

Memory-based approaches [27, 28, 8, 9, 35] usually use memory and replay/rehearsal mechanism to recall a small episodic memory of previous tasks while training new tasks thus reduce the loss in the previous tasks. For example, iCaRL [27] is the first replay method, which learns in a class-incremental way by selecting and storing exemplars closest to the feature mean of each class; Meta-Experience Replay (MER) [28] combines experience replay with optimization-based meta-learning to optimize the symmetric trade-off between transfer and interference by enforcing gradient alignment across examples; AGEM [8] projects the gradient on the current minibatch by using an external episodic memory of patterns from previous experiences as an optimization constraint; ER-Ring [9] jointly trains new task data with that of the previous tasks.

Dynamic network architectures [30, 19, 39, 38, 33, 7, 41] try to add new neurons to the model at additional new tasks, thus the performances on previous tasks are preserved by freezing the old parameters and only updating the newly added parameters. For example, Progressive neural networks (PNNs) [30] leverage prior knowledge via lateral connections to previously learned features; PackNet [19] iteratively assigns parameter subsets to consecutive tasks by constituting binary masks. SupSup [39] also finds masks in order to assign different subsets of the weights for different tasks. BatchEnsemble [38] learns on separate rank-1 scaling matrices for each task which are then used to scale weights of the shared network. HAT [33] incorporates task-specific embeddings for attention masking. [24] also proposes task-conditioned hypernetworks for continual learning. [20] proposes nonoverlapping sets of units being active for each task. Piggyback [18] learns binary masks on an existing network to provide good performance on new tasks. [1] proposes task specific convolutional filter selection for continual learning. The mask-based methods listed above provide excellent results for continual learning, but they require a significantly large number of parameters to represent the masks for each task. A factorization-based approach was proposed in [22] that performs automatic rank selection per task for variational inference using Indian Buffet process. The method requires significantly large rank increments per task to achieve high accuracy; in contrast, our method uses a learning-based approach to find rank-1 increments and reuse old factors with the learned selector weights. ORTHOG-SUBSPACE [7] learns tasks in different (low-rank) vector sub-spaces that are kept orthogonal to each other in order to minimize interference.

Our proposed method falls under the category of dynamic network architecture approaches. Note that we can represent a low-rank weight matrix using two smaller fully-connected layers and increasing the rank of the weight matrix is equivalent to adding new nodes in the two smaller fully-connected layers.

We focus on the incremental task learning setup in which we seek to train a network for $T$ tasks. The main difference between incremental task learning and regular learning is that the training data for every task is only available while training the network for that task. The main challenge in incremental task learning is to not forget the previous tasks as we learn new tasks. Learning each task entails training weights for the network to learn the task-specific input-output relationship using the task-specific training data.

We seek to develop an ITL framework in which we represent the weights of any layer using a small number of low-rank factors. We initialize the network with a base architecture in which weights for each layer can be represented using a low-rank matrix. We then add new low-rank factors to each layer as we learn new tasks.

Let us assume the network has $K$ layers and the weights for the $k$th layer and task $t$ can be represented as $W_{k,t}$. Let us further assume that the weights for the $k$th layer and task $t=1$ can be represented as a low-rank matrix

where $U_{k,1},V_{k,1}$ represent two low-rank matrices and $S_{k,1,1}$ represents a diagonal matrix. To learn the network for task 1, we learn $U_{k,1},V_{k,1},S_{k,1,1}$ for all $k$. For task 2, we represent the weights for $k$th layer as

$U_{k,1},V_{k,1}$ represent the two low-rank matrices learned for task 1 and frozen afterwards. $U_{k,2},V_{k,2}$ represent two low-rank matrices that are added to update the weights, and these will be learned for task 2. $S_{k,1,2},S_{k,2,2}$ represent the diagonal matrices, which will be learned for task 2. We learn $S_{k,1,2}$, which is a diagonal matrix that assigns weights to factors corresponding to task 1, to include/exclude or favor/suppress frozen factors from previous tasks for the new tasks. We can represent the weights for the $k$th layer and task $t$ as

where $U_{k,i},V_{k,i}$ are frozen for all $i<t$ and $U_{k,t},V_{k,t}$ and all the $S_{k,i,t}$ are learned for task $t$. The entire network for task $t$ can be represented as $\mathcal{W}_{t}=\{U_{k,i},S_{k,i,t},V_{k,i}\}_{i\leq t}$. To update the trainable network parameters for task $t$, we solve the following optimization problem:

where we use $\text{loss}(\cdot,\cdot)$ to denote the loss function and $\mathcal{W}_{t}[U_{k,t},S_{k,i,t},V_{k,t}]$ to indicate the trainable parameters in $\mathcal{W}_{t}$, while the rest are frozen. We sometimes call $S_{k,i,t}$ a selector weight matrix/vector to indicate that its diagonal entries determine the contribution of each factor toward each task/layer weights.

Our proposed ITL algorithm works as follows. We train the low-rank factors for the given task using the respective training samples. Then we freeze the factors corresponding to the older tasks and only update the new factors and the diagonal matrices. In this manner, the total number of parameters we add in our model is linearly proportional to the rank of the new factors. To keep the network complexity small, we seek to achieve good accuracy using small rank for each task update and layer. We summarize our approach in Algorithms 1 and 2.

Note that we do not need to create the weight matrix $W_{k,t}$ for any layer explicitly since we can compute all the steps in forward and backward propagation efficiently using the factorized form of each layer. The size of each layer is determined by the choice of the network architecture. The rank of each layer for every task is a hyper-parameter that we can select according to the tasks at hand. To keep the memory overhead small, we need to use small values for rank increment. Let us denote the rank for $U_{k,t}$ as $r_{k,t}$, which represents the increment rank for $k$th layer and task $t$. At the time of test, we can use an appropriate number of factors depending on the task. For instance, if we want to predict output for task 1 then we use first $r_{k,1}$ factors and for task 2 we use $r_{k,1}+r_{k,2}$ factors. We can add new factors in an incremental manner as we add new tasks in the ITL setup. In the extreme case of rank-1 increments, $r_{k,t}=1$. In our experiments, we observed that rank-1 updates compete or exceed the performance of existing ITL methods (see Table 1) and the performance of our method improves further as we increase the rank (see Table 5). Any increase in the rank comes at the expense of an increased memory overhead.

We used different classification tasks on well known continual learning benchmarks to show the significance of our proposed approach.

We proposed a new incremental task learning method in which we update the network weights using low rank increments as we learn new tasks. Network layers are represented as a linear combination of low-rank factors. To update the network for a new task, we freeze the factors learned for previous tasks, add a new low-rank (or rank-1) factor, and combine that with the previous factors using a learned combination. The proposed method offered considerable improvement in performance compared to the state-of-the-art methods for ITL in image classification tasks. In addition, the proposed low-rank ITL circumvents the use of memory buffer or large memory overhead while achieving zero forgetting.

The need for task ID knowledge is a general limitation of our and other ITL methods. Such methods can be useful for incremental multitask learning where task ID is available during inference but training data is only available in a short window. Extending this method to class incremental learning (which does not require task ID) is an important problem for future work.