Partial Hypernetworks for Continual Learning

in contrast to the standard supervised learning, is a setting in which the entire dataset $\mathbf{D}$ is not available at once. Instead, data becomes available over time in the form of a data stream. Specifically, the model is exposed to a sequence of experiences $\mathcal{S}=\{\mathbf{e}_{1},\mathbf{e}_{2},...,\mathbf{e}_{N}\}$, where each $\mathbf{e}_{i},0<i<N$ represents an individual experience that contains a train and test set, i.e., $\mathbf{e}_{i}=\{\mathbf{D}^{(i)}_{tr},\mathbf{D}^{(i)}_{te},t^{(i)}\}$ with $\mathbf{D}^{*}_{tr}$, $\mathbf{D}^{*}_{te}$ and $t^{(i)}$ being the train set, test set and the task id of the experience. The datasets in each experience can be drawn from the same distribution or have different distributions.

A strategy refers to an algorithm that updates the model $\mathbf{f_{\theta}}$ over a sequence of experiences. When a strategy goes through each experience, it does so one after the other and does not revisit any previously observed experience. Once the strategy observes the $N^{th}$ experience $\mathbf{e}_{N}$, it obtains the corresponding converged parameters $\theta^{*}_{N}$. After each experience, the model is evaluated by calculating the Average Classification Accuracy (ACA) over the test set of all the experiences observed so far, i.e., $\text{ACA}=\sum_{i=1}^{N}\text{Acc}(\mathbf{f}_{\theta_{T}^{*}},\mathbf{D}_{te}^{(i)})$ with $\text{Acc}(\mathbf{f},\mathbf{D})$ being the accuracy of a function $\mathbf{f}$ over dataset $\mathbf{D}$. Strategies are typically compared in terms of their ability to preserve knowledge, which refers to the degree to which the model’s performance is affected after going through all experiences in the stream.

The type of scenario in CL determines what kind of information a new experience provides and how the input and output domains change over time. The most commonly encountered scenarios are domain-incremental, task-incremental, and class-incremental Van de Ven & Tolias (2019). In the domain-incremental scenario, the category space remains constant for all experiences, while only the ”domain” of the mapping changes. In both task-incremental and class-incremental scenarios, the category space changes as new classes are introduced. However, in the task-incremental scenario, the experience indicator is given during test time, while in the class-incremental scenario, the model is not presented with any form of task indicator at test time. It is possible to formulate the class-incremental scenario as a task-incremental learning problem with an additional task inference step as theoretically formulated in Kim et al. (2022). In this case, the mapping $\mathbf{f}$ becomes $\mathbf{f}:\mathbf{X}\times\mathbf{T}\rightarrow\mathbf{Y}$ where in the case of class-incremental, $\mathbf{T}$ has to be inferred. In our experimental setting, similar to Von Oswald et al. (2019), we assume that the task ID is either given or inferred, which converts our problem into a task-incremental learning problem.

is an operation that converts a function $\mathbf{f}$ into a composite function $\mathbf{f}_{P}(\mathbf{f}_{P-1}(...(\mathbf{f}_{1}(x))))$, where $P$ is the total number of consecutive chunks that result from splitting the neural network into $P$ parts. By decomposing a function into simpler parts, we can gain insights into its behavior and use it to solve problems more flexibly. For example, we often decompose functions in calculus to apply techniques like the chain rule or integration by substitution. A common form of decomposition in neural networks is to break them down into two simpler neural networks. In the case of LR with frozen layers, we decompose neural networks into two parts, where the parameters in the first half’s layers are frozen and used for feature extraction. In our problem setting, we perform depth-wise decomposition by always splitting a neural network based on a given depth. For depth-specific decomposition, we consider a model $\mathbf{f}_{\theta,l}$ with $l\geq 2$, where $l$ indicates the number of layers, and we define a decomposition for depth $k$ as below:

$O_{dec}$ denotes the decomposition operation , which is applied for depth $k$, and $\theta^{a:b}$ denotes the subset of parameters from consecutive layers $a$ to $b$. In the case of LR, $k$ is a hyper-parameter that should be set based on the model architecture, computational limitations, and the level of shared structure across tasks. An alternative notation to represent the model segments obtained after decomposition is $\mathbf{f}_{\theta^{1:k}}=\mathbf{g}_{\phi}$ and $\mathbf{f}_{\theta^{k:l}}=\mathbf{h}_{\omega}$. With this notation, the inference is performed by first computing $z=\mathbf{g}_{\phi}(x)$ and then using $\mathbf{h}_{\omega}$ to compute $\hat{y}=h_{\omega}(z)$. Figure 2 provides an example of such a decomposition.

is a technique utilized for task-conditional prediction in multi-task or continual learning problems. The primary goal is to generate task-specific weights for particular layers of a model $\mathbf{f}$, and HNs are mainly employed for this purpose. The hypernetwork $\mathbf{H}_{\Psi}$ generates the parameters $\theta$ of model $\mathbf{f}$ before making prediction. When predicting input $x$, $\mathbf{H}_{\Psi}$ first generates the weights conditioned on an additional conditioning vector $t$, which can be the input $x$ itself or a task-dependent indicator such as the task ID. This weight generation process can also be considered an adaptive approach for prediction since the predictor’s weights are initially adapted in a generative way for the given task ID and then used for prediction. In our setting, we always decompose $\mathbf{f}$ into two parts, i.e. $\mathbf{h}_{\omega}(\mathbf{g}\phi(x))$, as shown in Equation 1 and make prediction as below:

The layers corresponding to the first half are called learnable or stateful layers and the layers in the second parts whose weights are generated by the hypernetwork are referred to as stateless layers.

exploits a decomposed neural network $\mathbf{f(x)}=\mathbf{h}_{\omega}(\mathbf{g}_{\phi}(x))$ with decomposition depth $k$ to learn from a stream of experiences with minimal catastrophic forgetting. At each step of training, we are given a batch of data samples $(x,y,t)$ which is then used to update $\mathbf{f}$. For our experimental setup, we assume that we freeze the parameters $\phi$ after the first experience in the stream. Regarding used strategies, we mainly consider two sets of methods: LR and partial HNs. The LR methods are facilitated with an additional buffer $\mathcal{B}$ with a limited capacity which stores features from previous samples extracted from $\mathbf{g}_{\phi}$. The details of the HN architecture are given in Appendix A.

In the first experience $e_{1}$, we train the model with regular SGD updates. Using task indicator $t$, we first infer the parameters $\omega$ of the stateless layers of the main model via the hypernetwork: $\omega=\mathbf{H}_{\psi}(t)$. After generating the stateless weights in the main model, we make a prediction on the batch inputs $x$ to obtain $\hat{y}=\mathbf{h}(z=\mathbf{g}(x;\omega))$. The cross-entropy loss $\mathcal{L}_{CE}=CE(y,\hat{y})$ is used to compute the gradients of learnable (stateful) layers and the hypernetwork:

where $\alpha$ and $\beta$ are the learning rates for the main network and the hypernetwork, accordingly. The same process is repeated until convergence to obtain $H^{*(1)}_{\psi}$.

Before starting each new experience $e_{i}$, we first store a copy of the converged hypernetwork $\mathbf{H}^{*}_{\psi}\leftarrow\mathbf{H}^{*(i-1)}_{\psi}$ from the previous experience. Similar to Von Oswald et al. (2019), we use $\mathbf{H}^{*}_{\psi}$ to penalize the hypernetwork’s outputs for previous experiences:

A naive way to compute the gradients for each step would be to add the regularization term to the cross-entropy loss directly and backpropagate through the network. This would be problematic on the current batch as it will result in inaccurate gradients. We show a visualization of this effect in Appendix B.1. To fruther explain this effect, we give an example of why this happens:

Let’s assume the model is provided with batch $b=(x,y,t)$. In the first step of the experience, the regularization loss will be zero since $H_{\psi}$ only changes after making an update on the current batch $b$. Therefore the gradients computed via $\mathcal{L}_{H}$ will always be one step behind, and the final gradient obtained for each step will be inaccurate. This may not have a big impact in streams with similar tasks, however in streams with big shifts in the distributions of the tasks, it can results in a noticeable difference. Simply put, we aim to find a way to incorporate the impact of the current update made via $\mathcal{L}_{CE}$ on the hypernetwork’s outputs for previous tasks.

One solution is to use a different update method such as look-ahead to also obtain gradients for penalizing the changes of $H_{\psi}$ from the first step of the experience. Look-ahead is an optimization method, where instead of immediately updating the model on a given batch of data, it first virtually updates the model for one or more steps, then computes the gradients in the final step of the virtual model, w.r.t. the original parameters. This way, we can compute more accurate gradient $\psi$ by backpropagating through the optimization trajectory to measure the impact of the changes made by the new batch $b$ on the parameters that affect the previous tasks. This process is shown in Figure 11.

Thus, the gradient calculation is done in two steps:

Calculate loss $\mathcal{L}_{CE}$ on the current batch and compute gradients w.r.t. $\phi^{(0)}=\phi$ and $\psi^{(0)}=\psi$:

Use $\nabla_{\psi}\mathbf{H}_{1}$ to virtually update the model without changing the original model.

Calculate loss $\mathcal{L}_{\mathbf{H}}$ on the virtual model to compute the gradients w.r.t. the parameters in the first step:

where $J_{\mathbf{H}}$ and $J_{\mathbf{H}}^{\prime}$ represent the loss function used to calculate $\mathcal{L}_{H}$ as in Equation 7. To obtain $\nabla_{\psi}\mathbf{H}_{2}$, we need to compute the second-order derivatives as shown in blue. In Appendix B.2, we show that even a first-order approximation achieves similar performance, thus in practice, we can use a first-order approximation.

Finally, we need to compute the final gradients to update the main model. The gradients in the first step correspond to the changes that are useful for the current batch, and the gradients in the second step are used for the changes that can help to avoid forgetting the previous tasks. Therefore, we use the mean of the gradients:

$\nabla_{\psi}\mathbf{H}$ is used to update the parameters of the hypernetwork at each training step.

methods are mainly categorized into five groups Wang et al. (2023): regularization, replay and optimization, representation and architecture. In regularization methods, the goal is to control catastrophic forgetting by adding a regularization term to the loss function that penalizes changes to the model parameters that would cause a large shift in the outputs of the previously learned tasks Kirkpatrick et al. (2017); Zenke et al. (2017); Li & Hoiem (2017). In rehearsal methods, the goal is to mitigate forgetting by storing a subset of the original data for each task and replaying it during training on subsequent tasks, either by interleaving it with new data or by training the model on the replayed data after training on the new data Rolnick et al. (2019); Chaudhry et al. (2019); Rebuffi et al. (2017); Pellegrini et al. (2020). Optimization-based methods attempt to alleviate forgetting by manipulating the optimization process either via projecting gradients or customized optimizer Lopez-Paz & Ranzato (2017); Farajtabar et al. (2020). In representation-based methods, the main ideas is to learn representations that can boost continual learning for example by learning more sparse representations as in Javed & White (2019) or by learning richer representations for transfer learning via pre-training and self-supervised learning Gallardo et al. (2021); Mehta et al. (2021). In the architecture-based methods, the focus is on the role of the model’s architecture in CL with the assumption that model needs to learn task-specific parameters which is itself split to sub-groups: parameter isolation and dynamic architecture. In parameter isolation, the model architecture is fixed and different subsets of parameters are assigned to different tasks, for example by masking parameters Serra et al. (2018); Mallya et al. (2018) or model decomposition such as Miao et al. (2022); Yoon et al. (2019); Kanakis et al. (2020). In dynamic architecture methods, new modules are added over time for solving newer tasks such as Rusu et al. (2016); Aljundi et al. (2017); Xu & Zhu (2018). Our method is can be categorized into the parameter isolation sub-group in the architecture group.

are a type of neural network architecture that have gained significant popularity in recent years. The weight generation mechanism in HNs enables them to be applied to several applications such as neural architecture search Zhang et al. (2018) and conditional neural networks Galanti & Wolf (2020). Although the origin of hypernetworks dates back to the work of Hinton and Salakhutdinov in the 1990s Schmidhuber (1992), it was not until recently that they became a popular topic of research in deep learning Ha et al. (2016). In the context of CL, the work from Von Oswald et al. (2019) proposed to use hypernetwork to generate the full weights of a neural network. Morever, Ehret et al. (2020) proposed to use hypernetwork for more effective continual learning compared to the weight regularization-based methods. A probabilistic version of hypernetworks for CL for task-conditioned CL was proposed by Henning et al. (2021).

methods propose to store or generate activations from intermediate layers of a neural network, which are then replayed at the same intermediate layers to prevent forgetting when new tasks are learned during the CL process. Intuitively, Latent Replay methods assume that low-level representations can be better shared across tasks and, hence, require less adaptation. To this end, low layers are either regularized Liu et al. (2020), trained at a slower pace Pellegrini et al. (2020), or frozen, as in Van de Ven et al. (2020); Hayes et al. (2019). Freezing the low layers significantly reduces the computational cost of CL by reducing the number of trainable parameters. These methods rely on the assumption that feature extractors can produce meaningful features and should be preferred in applications with limited compute. In Ostapenko et al. (2022), the efficacy of pre-trained vision models for downstream continual learning is studied, exploring the trade-off between CL in raw data and latent spaces, comparing pre-trained models in benchmarking scenarios, and identifying research directions to increase efficacy.

is a highly popular paradigm in deep learning that enables sample-efficient learning by partially or completely fine-tuning a pre-trained model for a novel task. Many researchers have studied transfer learning to understand when and how transfer learning works. Yosinski et al. Yosinski et al. (2014) quantify the generality vs. specificity of deep convolutional neural networks, showing transferability issues and boosting generalization by transferring features. They demonstrate that the first layer of AlexNet can be frozen when transferring between natural and man-made subsets of ImageNet without performance impairment, but freezing later layers produces a substantial drop in accuracy. Neyshabur et al. Neyshabur et al. (2020) analyze the transfer of knowledge in deep learning and show that successful transfer results from learning low-level statistics of data. They also show that models trained from pre-trained weights stay in the same basin in the loss landscape. Kornblith et al. Kornblith et al. (2019a) examine methods for comparing neural network representations, show the limitations of canonical correlation analysis (CCA), and introduce a similarity index based on centered kernel alignment (CKA) that can identify correspondences between representations in networks trained from different initializations. The work by Kornblith et al. Kornblith et al. (2019b) evaluates the relationship between architecture and transfer learning by comparing various networks in different classification benchmarks. They find that pre-training on ImageNet provides minimal benefits for fine-grained image classification tasks. Moreover, He et al. He et al. (2019) show that transfer even between similar tasks does not necessarily improve performance, and random initialization can achieve a better solution given enough iterations.