DIRA: Dynamic Incremental Regularised Adaptation

Gradually assimilating new information from a continuously changing data stream, known as ‘continual learning’, poses a challenge for deep neural networks. Continual learning, however, is a fundamental aspect of evolving autonomous systems. In a continual learning setting the problem is broken down into several parts that need to be learned sequentially. In the continual learning literature, these parts are often called tasks. Thus, the term tends to have several meanings. These several connotations of the term task make it difficult to study the different challenges associated with continual learning. To overcome this problem, Ven et al.[15], proposed to brake down continual learning into three incremental learning scenarios: task-incremental, domain-incremental, and class-incremental learning (see Table 1). Each scenario describes the context of the parts required to be learned sequentially, formerly the three scenarios contexts were referred to using the term task. Braking continual learning into different scenarios makes it more convenient to study the different challenges associated with each scenario, and subsequently develop appropriate techniques to overcome the associated challenges [16, 17, 18, 19].

The first scenario (task-incremental learning), describes the case where the algorithm is required to learn incrementally a set of distinct tasks. For example, if a neural network model was to classify numbers from 0 - 9 in English (like in the MNIST dataset [20]), then a new task for the model can be to learn to classify samples in Permuted-MNIST [12] or Fashion-MNIST [21] i.e. the same number of classes but the pattern has changed distinctively. For more examples see [22, 23, 24]. In the second scenario (domain-incremental learning), the model needs to learn the same problem but in different contexts, because the domain or input distribution has shifted. Using our previous example of classifying digits 0 - 9, in domain incremental learning, the model is required to learn to classify digits 0 - 9 but with Gaussian noise or contrast noise added to the input samples. See [25, 26] for more examples. The third scenario (class-incremental learning), describes when the model needs to learn a growing number of classes. In the example of classifying digits 0 - 9, class-incremental learning is the model learning to classify digit ‘10’ as an additional class to the existing 0 - 9 classes. See examples [27, 28].

Since our focus is on dynamic distributional shifts during operation, we are interested in the second scenario of incremental learning i.e. domain-incremental learning. We focus on trying to achieve domain incremental adaptation using a limited number of samples.

There have been a number of introduced approaches in the literature that address the problem of domain-incremental adaptation. For a breakdown of categories for the different introduced approaches in the literature, we direct interested readers towards [29]. Here we will cover some state-of-the-art examples of these approaches relevant to our results discussed later in section 5.

One popular approach is using self-supervision to achieve domain adaptation. Test-time training (TTT) combine different self-supervised auxiliary contexts to achieve domain adaptation. They break down neural network parameters into three parts, such that pictorially the architecture has a Y-structure. The bottom section of the Y-structured architecture represents the input layer and the layers responsible for the shared feature extraction, whilst the other two sections contain layers for learning and outputting labels for the main and auxiliary tasks independently. An example of this auxiliary task is being able to tell the rotation of the input image. During training time the whole neural network is optimised using a combined loss function that aims to maximise performance on both the main and auxiliary tasks. During retraining to adapt to a new domain, only parameters of the shared feature extraction and the auxiliary task sections are allowed to change. By doing so the the shared feature extraction section of the network modifies to learn the new domain, so then the network may output correct predictions on the unchanged branch of the network responsible for the main task [30, 31].

Correcting domain statistics is another common approach to achieving domain adaption, e.g. [29, 32, 33, 34]. Some of these approaches rely on using a large number of samples to recalculate the running mean and standard deviation of batch normalisation layers for the target domain e.g. [33, 32]. Other approaches, like Dynamic Unsupervised Adaption (DUA) [29], combine the running mean and standard deviation for normalisation layers from the original domain and the target domain to achieve adaptation in an unsupervised fashion, whilst using significantly fewer samples (typically $\approx 100$ samples).

Our proposed DIRA method aims at achieving adaptation through the regularisation of new and old information.We retrain the model on samples from the target domain. Therefore, we require labels to be provided with the retraining samples, which makes our approach a supervised instead of an unsupervised method. We use regularisation techniques to avoid catastrophic forgetting and achieve adaptation using very few samples. By doing so, we benefit from transfer learning of information from the initial domain to the target domain. Our philosophy is that if humans use transfer learning to learn and adapt to different environments, we believe that neural networks can also achieve domain adaptation in a similar fashion. We see our approach can be combined with self-supervision methods, such as done in TTT, to overcome the need for providing labels. Exploring the use of self-supervision in our approach is left as future work. In this paper, we assume labels for samples from the target domain are available for retraining.

The concept of regularisation allows a neural network to learn new information whilst retaining previously learned information. This allows a neural network to learn new information without experiencing catastrophic forgetting and without needing access to training data of previously learnt information. Regularisation achieves this by presenting a penalisation term in the loss function of the optimisation problem. Several works in the literature have introduced different penalisation terms, some popular examples are Synaptic Intelligence (SI) [35], Learning without forgetting (LWF) [36], and Elastic Weight Consolidation (EWC) [37]. In DIRA, different regularisation techniques may be utilisable, however, based on surveys such as the one provided by Kemker [38], the EWC penalisation term results in state-of-the-art performance within regularisation techniques. Therefore, we developed our method predominantly based on EWC.

Kirkpatrick et al. [37] introduced Elastic Weight Consolidation (EWC) to overcome forgetting in task-incremental learning. EWC overcomes forgetting by introducing a penalisation term in the loss function when retraining. This penalisation term provides a sense of the importance of each weight in the trained model on the original classification task. Therefore, when retraining on a new task, the algorithm is guided to avoid making significant changes to weights with high importance to the initial task. In this paper, we are interested in adapting to new domains, instead of adapting to new tasks. In the rest of this section, we will discuss the derivation of EWC and outline the assumptions that need to be taken into account when using EWC for domain adaptation.

During training of a DNN, the goal is to minimise the loss function $\mathcal{L}(\theta)$, represented as the Log-Likelihood function $-log(P(\theta|D))$ [39]. This aims at estimating $\theta$, which is the set of weights and biases in a DNN, given $D$, the dataset representing the samples of the distribution of interest. $D$ can be split into two independent datasets such that $D=\{D_{A},D_{B}\}$, where $D_{A}$ and $D_{B}$ are datasets that are trained on sequentially and each of them may represent a different distribution. Using the chain rule in probability it can be shown that:

Considering the RHS of equation 1:

• •

  First term, using conditional independence $log(P(D_{B}|\theta,D_{A}))=log(P(D_{B}|\theta))$ and hence can be seen as the loss function, $\mathcal{L_{B}}(\theta)$ that needs to be minimised for the new distribution or dataset $D_{B}$ alone.

• •

  Second term, $log(P(\theta|D_{A}))$ is the loss function for training the neural network on distribution $D_{A}$ only. Thus can be denoted as $\mathcal{L_{A}}(\theta)$.

• •

  The Third term, is irrelevant as this term is constant with respect to $\theta$ and thus is lost when optimising using the stochastic gradient descent (SGD) i.e. does not need to be computed. We will neglect this term for the rest of the derivation.

Therefore, the overall loss function in equation 1 can be written as:

In continual learning, distribution $A$ would have been trained on initially, and later samples from distribution $B$ arise and must be learnt by the DNN. In this case, the term $\mathcal{L_{A}}(\theta)$ is considered to be intractable as it is assumed that access to training samples for distribution A is not available after initial training.

The underlying idea of EWC is to take a Bayesian approach to adapt the DNN model parameters, therefore learning additional distributions whilst avoiding catastrophic forgetting or minimising forgetting. However, due to intractable terms, it is not possible to maintain the full posterior $P(\theta|D))$. An inference technique is required to approximate these intractable terms. EWC can be seen as an online approximate inference algorithm [40]. An essential assumption for EWC to approximate $\mathcal{L_{A}}(\theta)$ is that the DNN has been optimised for $D_{A}$ such that $\theta$ has reached a local or a global minimum, $\theta^{*}_{A}$, for distribution $D_{A}$. This allows for $-log(P(\theta|D_{A}))$ to be approximated as a Gaussian distribution function at its mode using Laplace’s method [41]. Expanding $-log(P(\theta|D_{A}))$ using Taylor series around $\theta^{*}_{A}$:

The Hessian can be computed by approximating it to the empirical Fisher information matrix. Using Bayesian rule:

This section describes the algorithmic details of our method. In order to achieve successful domain-adaptation we have taken into consideration the two assumptions outlined in section 3.1 when developing DIRA. Let $M_{0}$ be the model trained on the original domain dataset $X_{0}$. The standard optimisation problem in training a neural network on the original domain with a loss function $\mathcal{L}_{0}$ solves:

The aim of our approach is to adapt the trained model to out-of-distribution target data $X_{T}$ using a few number of samples $S_{T}$ from the target domain. To achieve this goal we utilise the concept of transfer learning, aiming at reserving beneficial information learnt from the original domain to allow for successful adaptation to the target domain. Our hypothesis is that by using regularisation techniques one should be able to utilise this notion of transfer learning to achieve adaption with a limited number of samples from the target domain. Therefore, the problem we try to optimise for during adaptation becomes a combination of the loss function for the original domain $\mathcal{L}_{0}$ and the target domain $\mathcal{L}_{T}$:

The $\mathcal{L}_{0}$ is intractable during adaptation since we have no access to the original domain training data. Therefore, an approximation of the original domain is done using EWC which yields the optimisation problem:

To satisfy assumption 1, whenever we retrain we always start from the original model $M_{0}$. Practically this is achievable as a copy of $M_{0}$ can always be kept onboard of a system. Assumption 2 can be satisfied by calculating the Fisher matrix using the original training dataset during initial training and a copy of this calculated Fisher matrix would be saved on board of the system, omitting the need to keep a copy of the initial training data on board.

In each training step $t$, the model parameters are updated according to Equation 10, where $\eta$ is the learning rate.

The two hyperparameters critical for the success of our optimisation problem are $\eta$ and $\lambda$. Different numerical search methods can be used for finding values for these hyperparameters, e.g. grid search, Bayesian optimisation etc. From empirical testing, we found that a combination of $\eta=1$e-5 and $\lambda=1$ yields near optimum adaptation for datasets we used in our experimentation.

We utilise CIFAR-10C, CIFAR-100C, and ImageNet-C datasets in our experimentation. These are image classification benchmarks to evaluate a model’s robustness against common corruptions [46]. The benchmarks add different corruptions to the tests sets of CIFAR-10/CIFAR-100[47] and ImageNet [48] There are 20 corruptions in total with five different levels of severity, however, most SOTA domain-incremental retraining frameworks utilise 15 corruptions out of the 20 in their comparisons, e.g. [29, 30]. These are deemed the more common corruptions. We use the same 15 common corruptions used by other methods in the literature.

We list below the different baselines we assess against our DIRA approach:

1. 1.

   Source: Refers to results of the corresponding baseline model trained on the incorrupt data (i.e. $X_{0}$), without adaption to the target domain.

2. 2.

   SGD: Denotes retraining on samples of the corrupt data using only Stochastic Gradient Decent optimisation [39], i.e. without using any complimentary incremental learning frameworks, similar to how initial training on the incorrupt data is done.

3. 3.

   TTT [30]: Test-Time Training (TTT) adapts parameters in the initial layers of the network by using auxiliary tasks to achieve self-supervised domain adaption.

4. 4.

   NORM [32, 33]: Ignores the initial training statistics and recalculates the batch normalization statistics using samples from the target domain only (requiring a large number of samples).

5. 5.

   DUA [29]: Dynamic Unsupervised Adaptation (DUA), takes into account initial training statistics and updates batch normalization statistics using samples from the target domain (requiring few samples).

We used ResNets [49] in our experiments, utilising two versions of ResNet: ResNet-18 (18-layer) and ResNet-26 (26-layer). For CIFAR-10/CIFAR-100, we used ResNet-26. Initial training for the models was done locally. For ImageNet, we used a pre-trained off-the-shelf ResNet-18 model from PyTorch [45]. Experiments for CIFAR10 and CIFAR100 were done on an MSI GF65 THIN 3060 Laptop with 64GB RAM and a Linux Ubuntu 20.04.2 LTS (64-bit) operating system, whilst for ImageNet we used a Dell Alienware Desktop PC with 64GB RAM and a Linux Ubuntu 18.04.4 LTS (64-bit) operating system.

To achieve reliable comparisons against baselines the starting model parameters from which retraining is done must be the same. Otherwise, the accuracy improvements cannot be reliably attributed to the effectiveness of the retraining method and can be argued that it is due to varying starting model accuracies or parameters. Therefore, in our results for CIFAR10/100 on ResNet-26 we only compare against Source, as we do not have the initial models used in retraining by other SOTA methods. For ImageNet on ResNet-18 we compare against SOTA methods because the starting trained model used by other SOTA retraining methods is the same off-the-shelf ResNet-18 model from PyTorch.

We used Stochastic Gradient Decent (SGD) for optimisation during training and retraining in our work. For retraining DIRA, we found from empirical testing that $\eta=1$e-5 and $\lambda=1$ yields near optimum adaptation. The retraining is relatively quick as only a small number of samples are used and the retraining is done over 10 epochs. We use top-1 classification accuracy as our assessment metric in all experiments [50].

To investigate overall adaptation improvement using regularisation we plot Figure 1. Figure 1 shows how top-1 classification accuracy changes as the number of samples available from the target domain increases. The naive approach would be to retrain relying only on the Stochastic Gradient Decent (SGD) optimiser using a fixed learning rate ($\eta$). When using a learning rate of 1e-2, which is a common value used when training a model on a dataset, we can notice that the retrained model incurs catastrophic forgetting and does not improve on the target domain beyond the Source model, even when retraining samples are increased. Lowering the learning rate overcomes the issue of forgetting and allows the model to adapt gradually to the target domain eventually as the number of samples increases. Using DIRA improves the issue of forgetting further for low number of samples (i.e. 1 to 2 samples) and allows the model to reach higher accuracies when retraining using less than 10 samples. Eventually, the performance of retraining using SGD (with low $\eta$) converges with DIRA as the number of retraining samples increase.

Tables 2 and 3 shows the improvement DIRA achieves upon retraining on 100 samples for each type of corruption in CIFAR-10C and CIFAR-100C benchmarking datasets, respectively, compared to the Source accuracy.

In real-life scenarios, varying domains may occur during operation. To visualise how DIRA tackles such a scenario, we plot in Figure 2 the shift to four different domains consecutively from CIFAR-10C. The results depicted show that as soon as samples from the target domain are presented an abrupt improvement occurs in the accuracy of the model. This accuracy continues to grow as more samples from the target domain become present.

To assess how well our approach performs compared with SOTA domain adaptation frameworks we compare results with three domain adaptation frameworks from the literature: TTT, NORM and DUA, on ImageNet-C benchmarking dataset. Table 4 show top-1 classification accuracy for the highest severity level on dataset ImageNet-C. Our DIRA framework performs competitively with SOTA domain adaptation approaches. As can be seen from the table, we achieve SOTA overall performance averaged between the different corruptions the dataset. This is while using a limited number of samples from the target domain (100 samples).