Multi-Domain Multi-Task Rehearsal for Lifelong Learning

Fan Lyu¹, Shuai Wang¹, Wei Feng¹¹¹1Corresponding Author., Zihan Ye², Fuyuan Hu² and Song Wang^{1, 3}

Abstract

Rehearsal, seeking to remind the model by storing old knowledge in lifelong learning, is one of the most effective ways to mitigate catastrophic forgetting, i.e., biased forgetting of previous knowledge when moving to new tasks. However, the old tasks of the most previous rehearsal-based methods suffer from the unpredictable domain shift when training the new task. This is because these methods always ignore two significant factors. First, the Data Imbalance between the new task and old tasks that makes the domain of old tasks prone to shift. Second, the Task Isolation among all tasks will make the domain shift toward unpredictable directions; To address the unpredictable domain shift, in this paper, we propose Multi-Domain Multi-Task (MDMT) rehearsal to train the old tasks and new task parallelly and equally to break the isolation among tasks. Specifically, a two-level angular margin loss is proposed to encourage the intra-class/task compactness and inter-class/task discrepancy, which keeps the model from domain chaos. In addition, to further address domain shift of the old tasks, we propose an optional episodic distillation loss on the memory to anchor the knowledge for each old task. Experiments on benchmark datasets validate the proposed approach can effectively mitigate the unpredictable domain shift.

Introduction

Lifelong learning, also known as continual learning and incremental learning, aims to continually learn new knowledge from a sequence of tasks over a lifelong time. In contrast to traditional supervised learning, the lifelong setting helps machine learning work like a more realistic human learning by acquiring a new skill quickly with new training data. All the while, catastrophic forgetting (French 1999; Kirkpatrick et al. 2017) is the main challenge for lifelong learning, which happens when the learner forgets the knowledge of old tasks while learning a new task. To seek a balance between the old tasks and the new task, many methods have been proposed to handle the catastrophic forgetting in recent years. Following (De Lange et al. 2019), their methods can be categorized into Rehearsal (Lopez-Paz and Ranzato 2017; Chaudhry et al. 2018b; Guo et al. 2019), Regularization (Li and Hoiem 2016; Chaudhry et al. 2018a; Dhar et al. 2019) and Parameter Isolation (Mallya, Davis, and Lazebnik 2018; Yoon et al. 2017). Regularization-based and parameter isolation-based methods store no data from old tasks and highly rely on extra regularizers or architectures, resulting in their lower performance than the rehearsal-based methods. Rehearsal-based methods store a small number of samples in the training set, the model will retrain the saved data when training the the new task to avoid forgetting.

Refer to caption — (a) Traditional rehearsal

At each step of lifelong learning (see Fig. 1(a)), the most existing rehearsal-based methods (Rebuffi, Kolesnikov, and Lampert 2016; Lopez-Paz and Ranzato 2017; Chaudhry et al. 2018b; Guo et al. 2019) focus on training the new task while treating the stored data from old tasks as the constraints to preserve their performance. However, the old tasks in these methods may suffer from unpredictable domain shift that arises from two significant factors in the lifelong learning process: 1) The Data Imbalance between old and new task. The shrinkage of training data of old tasks leads to their domains will be prone to shift that manifests as the catastrophic forgetting. 2) The Task Isolation among all tasks (old and new), which makes such domain shift toward unpredictable directions and the boundary between any two tasks may become weak.

To address the unpredictable domain shift, in this paper, we propose a Multi-Domain Multi-Task (MDMT) Rehearsal method inspired by the multi-domain multi-task learning (Yang and Hospedales 2014) that considers both multiple tasks w.r.t. multiple domains and trains them equally. Specifically, as shown in Fig. 1(b), we first retrain the old tasks along with new task training parallelly rather than setting them as the constraints. We separate all these tasks by a Cross-Domain Softmax, which extends the softmax for each isolated task by combining the logits of all other seen tasks and separates them from each other. Then, to further alleviate the unpredictable domain shift, we propose to leverage a Two-level Angular Margin (TAM) loss to encourage the intra-class/task compactness and the inter-class/task discrepancy on the basis of Cross-Domain Softmax. In addition, we present an optional Episodic Distillation (ED) loss on all buffer memories for old tasks that suppress the domain shift by storing the latent representations of each sample in memories. We evaluate our MDMT rehearsal on four popular lifelong learning datasets for image classification and achieve new state-of-the-art performance. The experimental results show the proposed MDMT rehearsal can significantly mitigate the unpredictable domain shift. Our contributions are three-fold: (1) We propose a Multi-Domain Multi-Task Rehearsal method for lifelong learning, which parallelly and equally trains the old and new tasks and separate them by a Cross-Domain Softmax function. (2) We propose a Two-level Angular Margin (TAM) loss for lifelong learning to further boost the Cross-Domain Softmax for the sake of intra-class/task compactness and the inter-class/task discrepancy. (3) We build an optional Episodic Distillation loss to reduce the domain shift in lifelong progress.

Related Work

Lifelong learning. In contrast to static machine learning (He et al. 2016; Deng et al. 2018; Lyu et al. 2019; Lyu, Feng, and Wang 2020), Lifelong Learning (Ring 1997; Thrun 1998) is proposed to improve the self-learning ability of the machine that continually learns new knowledge. The previous solutions to the catastrophic forgetting (French 1999; Kirkpatrick et al. 2017) have been proposed in recent years and can be categorized into regularization-based, parameter isolation-based and rehearsal-based methods (De Lange et al. 2019). Regularization-based methods (Li and Hoiem 2016; Chaudhry et al. 2018a; Dhar et al. 2019) store no data but propose to use extra regularization terms in the loss function to consolidate previous knowledge. Parameter isolation-based methods (Mallya, Davis, and Lazebnik 2018; Yoon et al. 2017) freeze the task-specific parameters and grow new branches for new tasks to bring in new knowledge. Although many progress have been made on regularization-based and parameter isolation-based methods, their performance still has a big gap with rehearsal-based methods. Rehearsal-based methods store some knowledge of old tasks to remind the model. According to different saved forms, existing methods can be categorized into three groups: (1) by saving the raw data (Rehearsal, e.g., image) (Rebuffi, Kolesnikov, and Lampert 2016; Lopez-Paz and Ranzato 2017; Chaudhry et al. 2018b; Guo et al. 2019), the model can retrain the saved data with the training on the new task together, and these methods often construct single objective for the new task and set the saved data from old tasks as the constrains; (2) by saving the latent features for selected samples (Latent-rehearsal) (Pellegrini et al. 2019), the model slows down learning at the layers below the rehearsal layer and leaves the layers above free to learn at full pace; (3) by building generative model to synthesize data (Pseudo-rehearsal) (Shen et al. 2020; van de Ven and Tolias 2018; Lesort et al. 2019), the knowledge can be saved as parameters rather than data. In this paper, we only consider the native rehearsal by storing raw data in image classification.

Multi-domain multi-task learning. Multi-domain learning (Nam and Han 2016; Tang and Jia 2020) and multi-task learning (Lin et al. 2019; Sener and Koltun 2018) have been studied for years. Multi-domain learning refers to sharing information about the same problem across different contextual domains, while multi-task learning addresses sharing information about different problems in the same domain. By considering both multiple domains and multiple tasks, Multi-domain multi-task (MDMT) learning was first proposed in (Yang and Hospedales 2014), and has been applied to classification (Peng and Dredze 2016) and semantic segmentation (Fourure et al. 2017), etc.. The common method for MDMT learning is to construct parallel data streams and seek to build the correlations among tasks. Here, we explain why we decide to formulate the lifelong learning problem into a MDMT learning problem. (1) By storing some samples of a task into a memory, MDMT learning can significantly train them together, which helps mitigate the task isolation in the traditional rehearsal-based lifelong learning. (2) MDMT learning can help suspending the domain shift to some extent by making classifiers perceive each other.

Margin loss and distillation loss. The margin based Softmax explicitly adds a margin to each logit to improve feature discrimination. L-Softmax (Liu et al. 2016) and SphereFace (Liu et al. 2017) add multiplicative angular margin to squeeze each class. CosFace (Wang et al. 2018b, a) and ArcFace (Deng et al. 2019) add additive cosine margin and angular margin, respectively, for easier optimization. Based on ArcFace, we propose a Two-level Angular Margin loss to guarantee both inter-class/task compactness and intra-class/task discrepancy. The knowledge distillation (Hinton, Vinyals, and Dean 2015) from the teacher network to the student network transfers the knowledge about smoothed probability distribution of the output layer of the teacher network to the student network. Inspired by this, we propose to build distillation loss between the old and new models on old tasks by storing the latent representation of stored data.

Methodology

Multi-domain multi-task rehearsal

Suppose there are $T$ different tasks with respect to datasets $\{\mathcal{D}_{1},\cdots,\mathcal{D}_{T}\}$ . For the $t$ -th dataset (task), $\mathcal{D}_{t}=\{(x_{t,1},y_{t,1}),\cdots,(x_{t,N_{t}},y_{t,N_{t}})\}$ , where $x_{t,i}\in\mathcal{X}_{t}$ is the $i$ -th input data, $y_{t,i}\in\mathcal{Y}_{t}$ is the corresponding label and $N_{t}$ is the number of samples. $\mathcal{D}_{t}$ can be split into a training set $\mathcal{D}_{t}^{\text{trn}}$ and a testing set $\mathcal{D}_{t}^{\text{tst}}$ , and we denote $\mathcal{D}_{t}$ as $\mathcal{D}_{t}^{\text{trn}}$ in our presentation for simple denotation. Lifelong learning aims at learning a predictor $f_{t}:\mathcal{X}_{k}\rightarrow\mathcal{Y}_{k},~{}k\in\{1,\cdots,t\}$ , which can predict tasks that have been learned at any time. The rehearsal-based lifelong learning (Rebuffi, Kolesnikov, and Lampert 2016; Lopez-Paz and Ranzato 2017; Riemer et al. 2018; Chaudhry et al. 2018b; Guo et al. 2019) builds a memory buffer $\mathcal{M}_{k}\subset\mathcal{D}_{k}$ with small-size for each previous task $k$ , i.e., $|\mathcal{M}_{k}|\ll|\mathcal{D}_{k}|$ . Following (Lopez-Paz and Ranzato 2017), when training a task $t\in\{1,\cdots,T\}$ , for all $\mathcal{M}_{k}$ that $k<t$ , the rehearsal-based lifelong learning can be modeled as a single objective optimizing problem:

	$\displaystyle\mathop{\arg\min}_{\theta,\theta_{t}}$	$\displaystyle\ell(f_{\theta},f_{\theta_{t}},\mathcal{D}_{t}),\quad$		(1)
	s.t.	$\displaystyle\ell(f_{\theta},f_{\theta_{k}},\mathcal{M}_{k})\leq\ell(f_{\theta}^{t-1},f^{t-1}_{\theta_{k}},\mathcal{M}_{k}),\quad\forall k<t,$		(1)

where $\ell$ is the empirical loss. $\theta$ is the shared parameter across all tasks while $\theta_{k}$ and $\theta_{t}$ are the task-specific parameters. The constraints above are designed to prevent the performance degradation of previous tasks. Then, the problem can be reduced to find an optimal gradient that benefits all tasks. To inspect the increase in old tasks’ loss, (Lopez-Paz and Ranzato 2017; Chaudhry et al. 2018b; Guo et al. 2019) compute the angle between the gradient of each old task and the proposed gradient update on the current task.

However, such a single objective optimization on the current task for rehearsal-based lifelong learning over-emphasizes the new task while ignoring the difference among tasks. In other words, the old tasks can only play the role of source domain to be transferred into the current training model. The domain of old tasks will significantly shift because of the rectified gradient that the gradient norm of new task is much larger than the old tasks’, which may induce the domain overlap.

Algorithm 1 MDMT rehearsal based lifelong learning.

0: Train(

f_{\theta}

f_{\theta_{1:T}}

\{\mathcal{D}_{1}^{\text{trn}},\cdots,\mathcal{D}_{T}^{\text{trn}}\}

)

\mathcal{M},\mathcal{F}\leftarrow\{\}

for

t=1

T

for

(\mathbf{x},y)\in\mathcal{D}_{t}^{\text{trn}}

g,g_{1}\leftarrow\nabla_{\theta}\ell(f_{\theta}(\mathbf{x},t),y)

t=1

then

\tilde{g}\leftarrow g

else

g^{\text{ref}},g_{1:t-1}\leftarrow\nabla_{\theta}\ell(f_{\theta},f_{\theta_{1:t-1}},\mathcal{M})

g^{\text{ref}}\leftarrow g^{\text{ref}}+\nabla_{\theta}\tilde{\ell}(f_{\theta},\mathcal{F}_{\text{ref}})

\tilde{g}\leftarrow g+g^{\text{ref}}

end if

\theta\leftarrow\theta-\text{StepSize}\cdot\tilde{g}

\theta_{1:t}\leftarrow\theta_{1:t}-\text{StepSize}\cdot g_{1:t}

end for

\mathcal{M},\mathcal{F}\leftarrow\text{{StoreMem}}(\mathcal{M},\mathcal{F},\mathcal{D}_{t}^{\text{trn}},f_{\theta})

end for

StoreMem(

\mathcal{M}

\mathcal{F}

\mathcal{D}

f

)

for

i=1

|\mathcal{M}|/T

(\mathbf{x},y)\sim\mathcal{D}

\mathcal{M}\leftarrow\mathcal{M}+(\mathbf{x},y)

\mathcal{F}\leftarrow\mathcal{F}+f(\mathbf{x})

end for

Return

\mathcal{M}

\mathcal{F}

Eval(

f_{\theta}

f_{\theta_{1:T}}

\{\mathcal{D}_{1}^{\text{tst}},\cdots,\mathcal{D}_{T}^{\text{tst}}\}

)

a\leftarrow 0\in\mathbb{R}^{T}

for

t=1

T

a_{t}\leftarrow 0

for

(\mathbf{x},y)\in\mathcal{D}_{t}^{\text{tst}}

a_{t}\leftarrow a_{t}+\text{Accuracy}(f_{\theta_{t}}(f_{\theta}(\mathbf{x},t)),y)

end for

a_{t}\leftarrow a_{t}/|\mathcal{D}_{t}^{\text{tst}}|

end for

Return

a

In contrast, this paper treats the problem as a Multi-Domain Multi-Task (MDMT) learning problem to jointly and equally improve the current task as well as the old tasks:

$\displaystyle\mathop{\arg}_{\theta,\{\theta_{1},\cdots,\theta_{t}\}}$	$\displaystyle\{\min\ell(f_{\theta},f_{\theta_{t}},\mathcal{D}_{t}),\min\ell(f_{\theta},f_{\theta_{k}},\mathcal{M}_{k}),\cdots,$	(2)
	$\displaystyle~{}~{}~{}~{}~{}\min\ell(f_{\theta},f_{\theta_{1}},\mathcal{M}_{1})\},$
s.t.	$\displaystyle d(f_{i},f_{j})\geq d(f^{t-1}_{i},f^{t-1}_{j}),i,j\in[1,t],i\neq j,$

where $f_{i}=f_{\theta}(\mathcal{D}_{i})$ if $i=t$ and $f_{i}=f_{\theta}(\mathcal{M}_{i})$ if $i<t$ . $d$ means the distance between two domains. For the $t$ tasks w.r.t. datasets $\{\mathcal{D}_{1},\cdots,\mathcal{D}_{t}\}$ , a MDMT rehearsal model trains $t$ tasks parallelly and equally. The constraints above mean the domain distance between any two tasks should not be smaller than the model trained on the last task. Note that we only consider the situation that the tasks are irrelevant as the common lifelong learning.

We make two key operations to solve the Eq.(2) efficiently. First, we transform the multi-objective optimization as a single-objective optimization problem by ensembling all these objectives as the traditional solution to multi-task learning (Lin et al. 2019; Sener and Koltun 2018).

\mathop{\arg\min}_{\theta}\quad\ell(f_{\theta},f_{\theta_{t}},\mathcal{D}_{t})+\sum_{k=1}^{t-1}\ell(f_{\theta},f_{\theta_{k}},\mathcal{M}_{k}),

(3)

Second, it exists high memory-cost to calculate the distance between any two domains and store old predictors $f_{\theta}^{t-1}$ , but we can do this in a simple yet effective way by extending the softmax function for each task as

\ell_{k}=-\frac{1}{N_{k}}\sum_{n=1}^{N_{k}}\log\frac{e^{\left(W^{k}_{y_{n}}\right)^{T}x_{n}+b_{y_{n}}}}{\sigma_{n}},

(4)

where

\sigma_{n}=\sum_{j=1}^{C_{k}}e^{\left(W^{k}_{j}\right)^{T}x_{n}+b_{j}}+\sum_{i=1,i\neq k}^{t}\sum_{j=1}^{C_{i}}e^{\left(W^{i}_{j}\right)^{T}x_{n}+b_{j}}.

(5)

$N_{k}$ is the batch size for task $k$ and $W^{k}_{j}\in\mathbb{R}^{d}$ denotes the $j$ -th column of the weight $W^{k}\in\mathbb{R}^{d\times C_{k}}$ in the last fully-connected layer for task $k$ and $C_{k}$ is the class number. We name this extension as Cross-Domain Softmax (CDS), which combines the logits from other classifiers and is similar to a native softmax to a classification problem with total $\sum_{k=1}^{t}C_{k}$ class. Here, we discuss the difference. For MDMT rehearsal, different tasks never share a same classifiers as common classification, i.e., the classifiers for different tasks lack mutual perception. By combining the logits form other tasks, the tasks can perceive and separate from each other. The previous methods update the model by the optimal gradient that highly rely on the angle between the gradients of old and new tasks. In contrast, we directly obtain the hybrid gradient for the shared layers by ensembling the gradients from the new task and old tasks as $\tilde{g}\leftarrow\sum_{k=1}^{t}g_{k}$ .

We compare our MDMT rehearsal with some famous rehearsal-based lifelong works:

iCaRL (Rebuffi, Kolesnikov, and Lampert 2016) saves small number of samples to make the model not to forget old class, but they classify samples by the nearest prototype, which is not suitable for task-incremental lifelong learning because the task-specific parameters are ignored.

GEM/A-GEM (Lopez-Paz and Ranzato 2017; Chaudhry et al. 2018b) propose to solve forgetting by finding the optimal gradient that saves the old tasks from being corrupt, and they focus on training the new task with single objective optimization while ignore the domain shift of old tasks.

ER (Chaudhry et al. 2019a) extends Experience Replay (Rolnick et al. 2019) for reinforcement lifelong learning and be proven better than A-GEM. However, they never consider the relations among all tasks, which makes the domains of old task may significantly shift.

PRD (Hou et al. 2018) proposes to treat lifelong learning as a multi-task learning problem and proposes to build a distillation module with one saved CNN expert as teacher for each old task. Differently, we would like to build a MDMT rehearsal that leverage the expanded softmax without saving many extra models.

Two-level angular margin loss

The proposed MDMT rehearsal helps to jointly and equally train the new task and retrain the old tasks, making all tasks perceive each other. Nonetheless, the softmax loss is not efficient enough because it does not explicitly encourage intra-class compactness and inter-class discrepancy, in coping with which, large margin based softmax is widely used in recent discriminative problems (Deng et al. 2019; Liu et al. 2016). However, these methods cannot be directly applied to MDMT rehearsal based lifelong learning because these methods place the large margin only to single task and can not be applied to multiple tasks scenario.

In this paper, we propose two levels margin, i.e., class level and task level, on softmax for each task (Eq. (4)). Our work is based on the popular large margin based softmax method Arcface (Deng et al. 2019) where the large margin is added to the angle between weight and feature, which has been proven effective and efficient. Specifically, Arcface deletes the bias and transforms the logit fed into the softmax as $W_{j}^{T}x_{i}=\left\|W_{j}\right\|\left\|x_{i}\right\|\cos\theta_{j}$ , where $\theta_{j}$ is the angle between the weight $W_{j}$ and the feature $x_{i}$ , then an angular margin $m$ is placed between different classes

\ell=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{e^{s\cdot\cos\left(\theta_{y_{i}}+m\right)}}{e^{s\cdot\cos\left(\theta_{y_{i}}+m\right)}+\sum_{j=1,j\neq y_{i}}^{n}e^{s\cdot\cos\theta_{j}}},

(6)

where the individual weight $||W_{j}||$ is fixed to $1$ by $l_{2}$ normalization and the embedding feature $||x_{i}||$ is fixed to $s$ by $l_{2}$ normalization and rescale. The normalization on features and weights makes the predictions only depend on the angle between them. Such a geodesic distance margin between the sample and centers makes the prediction gain more intra-class compactness and inter-class discrepancy.

Based on Eq. (6), we propose our Two-level Angular Margin (TAM) loss for the task $k\in[1,t]$

\ell_{k}=-\frac{1}{N_{k}}\sum_{n=1}^{N_{k}}\log\frac{e^{s\cdot\cos\left((\theta^{k}_{y_{n}}+m^{\text{c}})+m^{\text{t}}\right)}}{\sigma_{n}},

(7)

where

	$\displaystyle\sigma_{n}=$	$\displaystyle e^{s\cdot\cos\left((\theta_{y_{i}}^{k}+m^{\text{c}})+m^{\text{t}}\right)}+\sum_{j=1,j\neq y_{i}}^{C_{k}}e^{s\cdot\cos(\theta_{j}^{k}+m^{\text{t}})}+$		(8)
		$\displaystyle\sum_{i=1,i\neq k}^{t}\sum_{j=1}^{C_{i}}e^{s\cdot\cos\theta_{j}^{i}}.$		(8)

In Eq. (7), we add class-level margin $m^{\text{c}}$ and task-level margin $m^{\text{t}}$ on the angular. $m^{\text{c}}$ is similar to $m$ in Eq. (6), which controls the intra-task class compactness and discrepancy and be proven effective on discriminative learning (Deng et al. 2019). $m^{\text{t}}$ controls the task compactness and discrepancy, which ensures the knowledge of each task not to mix up with others.

As shown in Fig. 3, the proposed TAM loss produces two advantages for MDMT rehearsal based lifelong learning. First, TAM helps the model to better discriminate into a task. Although the CDS has a better angle between feature and its target weight, TAM loss even reduce the angle to a smaller than CDS, which expresses the effect of $m_{\text{c}}$ . Second, TAM loss mitigates the domain overlap caused by the domain shift by forcing tasks to separate. We can also see that for the angles among weights center, TAM loss can significantly separate old and new tasks, which expresses the effect of $m_{\text{t}}$ . However, it is still difficult to omit the domain shift because of the extreme data imbalance between old tasks and new task. Thus, we construct an optional Episodic distillation loss for the MDMT rehearsal based lifelong process.

Table 1: Comparison with different state-of-the-arts. The numbers are averaged across 5 runs using a different seed each time.

Method	Permuted MNIST				Split CIFAR
Method	$A_{\text{T}}(\%)$	$F_{\text{T}}$	$LCA_{10}$	$LTR$	$A_{\text{T}}(\%)$	$F_{\text{T}}$	$LCA_{10}$	$LTR$
Joint	$95.30$	-	-	-	$68.30$	-	-	-
VAN	$47.55\pm 2.37$	$0.52\pm 0.026$	$0.259\pm 0.005$	$5.375\pm 0.194$	$40.44\pm 1.02$	$0.27\pm 0.006$	$0.309\pm 0.011$	$2.613\pm 0.174$
EWC	$68.68\pm 0.98$	$0.28\pm 0.010$	$0.276\pm 0.002$	$3.292\pm 0.135$	$42.67\pm 4.24$	$0.26\pm 0.039$	$0.336\pm 0.010$	$2.493\pm 0.427$
MAS	$70.30\pm 1.67$	$0.26\pm 0.018$	$0.298\pm 0.006$	-	$42.35\pm 3.52$	$0.26\pm 0.030$	$0.332\pm 0.010$	-
RWalk	$85.60\pm 0.71$	$0.08\pm 0.007$	$\mathbf{0.319}\pm 0.003$	-	$42.11\pm 3.69$	$0.27\pm 0.032$	$0.334\pm 0.012$	-
MER	-	-	-	-	$37.27\pm 1.68$	$\mathbf{0.03}\pm 0.030$	$0.051\pm 0.101$	-
GEM	$89.50\pm 0.48$	$0.06\pm 0.004$	$0.230\pm 0.005$	-	$61.20\pm 0.78$	$0.06\pm 0.007$	$0.360\pm 0.007$	-
A-GEM	$89.32\pm 0.46$	$0.07\pm 0.004$	$0.277\pm 0.008$	$0.716\pm 0.048$	$61.28\pm 1.88$	$0.09\pm 0.018$	$0.350\pm 0.013$	$0.643\pm 0.124$
ER	$90.47\pm 0.14$	$0.03\pm 0.001$	$0.184\pm 0.004$	$0.367\pm 0.013$	$63.97\pm 1.30$	$0.06\pm 0.006$	$0.349\pm 0.105$	$0.451\pm 0.333$
MEGA	$91.21\pm 0.10$	$0.05\pm 0.001$	$0.283\pm 0.004$	$0.524\pm 0.017$	$66.12\pm 1.94$	$0.06\pm 0.015$	$\mathbf{0.375}\pm 0.012$	$0.356\pm 0.114$
MDMT-R	$\mathbf{94.33}\pm 0.04$	$\mathbf{0.02}\pm 0.000$	$0.298\pm 0.003$	$\mathbf{0.247}\pm 0.009$	$\textbf{69.20}\pm 1.60$	$0.04\pm 0.010$	$0.334\pm 0.008$	$\mathbf{0.283}\pm 0.099$
Method	Split CUB				Split AWA
Method	$A_{\text{T}}(\%)$	$F_{\text{T}}$	$LCA_{10}$	$LTR$	$A_{\text{T}}(\%)$	$F_{\text{T}}$	$LCA_{10}$	$LTR$
Joint	$65.60$	-	-	-	$64.80$	-	-	-
VAN	$53.89\pm 2.00$	$0.13\pm 0.020$	$0.292\pm 0.008$	$0.976\pm 0.215$	$30.35\pm 2.81$	$0.04\pm 0.013$	$0.214\pm 0.008$	$0.202\pm 0.090$
EWC	$53.56\pm 1.67$	$0.14\pm 0.024$	$0.292\pm 0.009$	$1.021\pm 0.210$	$33.43\pm 3.07$	$0.08\pm 0.021$	$0.257\pm 0.011$	$0.675\pm 0.214$
MAS	$54.12\pm 1.72$	$0.13\pm 0.013$	$0.293\pm 0.008$	-	$33.83\pm 2.99$	$0.08\pm 0.022$	$0.257\pm 0.011$	-
RWalk	$54.11\pm 1.71$	$0.13\pm 0.013$	$0.293\pm 0.009$	-	$33.63\pm 2.64$	$0.08\pm 0.023$	$0.258\pm 0.011$	-
PI	$55.04\pm 3.05$	$0.12\pm 0.026$	$0.292\pm 0.010$	-	$33.86\pm 2.77$	$0.08\pm 0.022$	$0.259\pm 0.011$	-
A-GEM	$61.82\pm 3.72$	$0.08\pm 0.021$	$0.302\pm 0.011$	$0.456\pm 0.174$	$44.95\pm 2.97$	$0.05\pm 0.014$	$0.287\pm 0.012$	$0.178\pm 0.082$
ER	$73.63\pm 0.52$	$\mathbf{0.01}\pm 0.005$	$0.265\pm 0.004$	$\mathbf{0.001}\pm 0.001$	$54.27\pm 4.05$	$\mathbf{0.02}\pm 0.030$	$0.293\pm 0.009$	$0.014\pm 0.015$
MEGA	$80.58\pm 1.94$	$\mathbf{0.01}\pm 0.017$	$0.311\pm 0.010$	${0.002}\pm 0.002$	$54.28\pm 4.84$	$0.05\pm 0.040$	$\mathbf{0.305}\pm 0.015$	$0.070\pm 0.114$
MDMT-R	$\textbf{84.27}\pm 1.63$	$\mathbf{0.01}\pm 0.015$	$\mathbf{0.337}\pm 0.013$	$0.017\pm 0.014$	$\textbf{61.56}\pm 3.36$	$\mathbf{0.02}\pm 0.027$	$0.298\pm 0.008$	$\mathbf{0.002}\pm 0.002$

Episodic distillation

In this paper, we propose a simple yet effective solution to further mitigate the domain shift for old task named Episodic Distilllation (ED) loss. The main role the ED loss played is to reduce the feature distribution change along with the lifelong process as far as possible. First, apart from the sampled training data stored in memory, i.e., $\mathcal{M}_{k}=\{(x_{k,1},y_{k,1}),\cdots,(x_{k,|M_{k}|},y_{k,|M_{k}|})\}\subset\mathcal{D}_{k}$ , we also store the corresponding latent representations when they are first trained, denoted as $\mathcal{F}_{k}=\{\mathbf{f}_{k,1},\cdots,\mathbf{f}_{k,|M_{k}|}\}$ . Then, we train the model with an updated objective:

\mathop{\arg\min}_{\theta}\quad\ell(f_{\theta},f_{\theta_{t}},\mathcal{D}_{t})+\sum_{k=1}^{t-1}\left[\ell(f_{\theta},f_{\theta_{k}},\mathcal{M}_{k})+\tilde{\ell}(f_{\theta},\mathcal{F}_{k})\right],

(9)

where

\tilde{\ell}(f_{\theta},\mathcal{F}_{k})\triangleq\frac{1}{N_{k}}\sum_{i}\tilde{\ell}_{i}(f_{\theta}(\mathbf{x}_{k,i}),\mathbf{f}_{k,i}).

(10)

$\tilde{\ell}_{i}$ is the ED loss that can be in many formats, and we choose the Mean Square Error (MSE). By training with Eq. (9) in each step, we can ease the shift effectively.

ED loss is an optional loss function and builds extra memory buffers to save the latent representation for each sample in memories. The extra memory buffers do increase the memory cost to some extent, but still very small in compared with the whole training set. In our implementation, we save the representation from the fc layer before the last one, which is a vector with length from 256 to 2048 for different network. That means the cost of the representation memory is even smaller than the data memory.

Total algorithm

We follow A-GEM (Chaudhry et al. 2018b) that unite memory of all old tasks for efficient training. Let $\mathcal{M}=\cup_{k<t}\mathcal{M}_{k}$ and $\mathcal{F}=\cup_{k<t}\mathcal{F}_{k}$ be the united data and representation memory for old tasks. For each step, we will sample a batch of data from the united memory. In this way, the previous tasks will be optimized by an average gradient instead of all gradients for previous tasks, which speeds up the training.

We show the detailed process in Algorithm 1 including training and evaluation procedure. First, the storage of memory feature in StoreMem to be as the anchor of old task in current task training. Second, the gradient to be updated depends not only the gradient on old and current tasks using TAM loss, but the gradient on feature difference by ED loss. The evaluation procedure is similar with the previous works.

Table 2: Ablation study on Split CIFAR.

$m_{\text{t}}$	$m_{\text{c}}$	ED	$A_{\text{T}}(\%)$	$F_{\text{T}}$	$LCA_{10}$	$LTR$
-	-	-	$65.44\pm 1.13$	$0.052\pm 0.006$	$\mathbf{0.371}\pm 0.008$	$0.377\pm 0.076$
-	-	$\checkmark$	$66.44\pm 2.22$	$0.050\pm 0.009$	$0.370\pm 0.014$	${0.307}\pm 0.066$
0.0	0.0	-	$67.15\pm 2.02$	$0.053\pm 0.012$	$0.353\pm 0.006$	$0.411\pm 0.097$
0.1	0.0	-	$67.49\pm 1.55$	$\mathbf{0.049}\pm 0.010$	$0.354\pm 0.005$	$0.369\pm 0.096$
0.0	0.01	-	$67.45\pm 1.09$	$0.059\pm 0.008$	$0.354\pm 0.005$	$0.483\pm 0.064$
0.1	0.01	-	$67.68\pm 1.72$	$0.052\pm 0.008$	$0.350\pm 0.007$	$0.390\pm 0.062$
0.4	0.01	-	$67.28\pm 0.97$	$0.053\pm 0.012$	$0.347\pm 0.006$	$0.394\pm 0.106$
0.4	0.05	-	$66.68\pm 1.23$	$0.063\pm 0.005$	$0.333\pm 0.005$	$0.473\pm 0.077$
0.4	0.1	-	$64.97\pm 1.13$	$0.084\pm 0.009$	$0.324\pm 0.008$	$0.680\pm 0.086$
0.1	0.01	$\checkmark$	$\mathbf{68.64}\pm 1.35$	$0.059\pm 0.016$	$0.334\pm 0.008$	$\mathbf{0.297}\pm 0.103$

Experiments

Experimental Settings

We evaluate the proposed method on four image recognition datasets. (1)Permuted MNIST. (Kirkpatrick et al. 2017): this is a variant of standard MNIST dataset of handwritten digits with 20 tasks. Each task has a fixed random permutation of the input pixels which is applied to all the images of that task. (2)Split CIFAR. (Zenke, Poole, and Ganguli 2017): this dataset consists of 20 disjoint subsets of CIFAR-100 dataset (Krizhevsky, Hinton et al. 2009), where each subset is formed by randomly sampling 5 classes without replacement from the original 100 classes. (3)Split CUB. (Chaudhry et al. 2018b): the CUB dataset (Wah et al. 2011) is split into 20 disjoint subsets by randomly sampling 10 classes without replacement from the original 200 classes. (4)Split AWA. (Chaudhry et al. 2018b): this dataset consists of 20 subsets of the AWA dataset (Lampert, Nickisch, and Harmeling 2009). Each subset is constructed by sampling 5 classes with replacement from a total of 50 classes and the same class can appear in different subsets.

We leverage four existing metrics to evaluate the performance and catastrophic forgetting. (1) Average Accuracy ( $A_{t}\in[0,1]$ ) after the model has been trained continuously done till task $t\in\{1,\cdots,T\}$ . In particular, $A_{T}$ is the average accuracy on all the tasks after the last task has been learned. (2) Forgetting Measure (Chaudhry et al. 2018a) ( $F_{t}\in[-1,1]$ ) is the average forgetting after the model has been trained continuously with all the mini-batches for task $t\in\{1,\cdots,T\}$ . (3) Learning Curve Area (Chaudhry et al. 2018a). ( $LCA\in[0,1]$ ) is the area of the convergence curve for any average $b$ -shot performance after the model has been trained for all the $T$ tasks, where $b\in\left[0,\beta\right]$ . (4) Long-Term Remembering (Guo et al. 2019). ( $LTR\geq 0$ ) $LTR$ quantifies the accuracy drop on each task relative to the accuracy just right after the task has been learned. The detailed descriptions and the formulas can be shown in the supplementary materials.

Following the previous works (Lopez-Paz and Ranzato 2017; Chaudhry et al. 2018b; Guo et al. 2019), for Permuted MNIST we adopt a standard fully-connected network with two hidden layers, where each layer has 256 units with ReLU activation. For Split CIFAR we use a reduced ResNet18 (He et al. 2016). For Split CUB and Split AWA, we use a standard ResNet18.

Comparison with the state-of-the-arts

We compare the proposed method with the state-of-the-art methods including EWC (Kirkpatrick et al. 2017), MAS (Aljundi et al. 2018), RWalk (Chaudhry et al. 2018a), PI (Zenke, Poole, and Ganguli 2017), GEM (Lopez-Paz and Ranzato 2017), MER (Riemer et al. 2018), ER (Chaudhry et al. 2019b) A-GEM (Chaudhry et al. 2018b) and MEGA (Guo et al. 2019). Specifically, EWC, MAS, RWalk and PI are regularization-based methods that prevent the important weights from changing too much. GEM, MER, ER, AGEM and MEGA are rehearsal-based methods that rectifies the gradient guided by the stored data. VAN is a single supervised model trained continuously on the sequence of tasks. We also compare with the baseline that jointly trains all datasets with different classifiers together.

First, as shown in Tab. 1, the quantitative results of the proposed method outperform other state-of-the-arts. For $A_{T}$ , the performances of our method show the superiority on all four datasets. This indicates the less forgetting on old tasks and better learning on new tasks through the lifelong training by reducing unpredictable domain shift. $F_{T}$ evaluates the fine-grained batch-level forgetting on all tasks and never cares the Acc value. We get good $F_{T}$ except on Split CIFAR with slight worse ( $0.04$ vs. $0.03$ ) than MER. MER has a better $F_{T}$ but poor $A_{T}$ because it adopts a complex meta learning strategy. For $LCA_{10}$ , it evaluates the training speed on the first 10 training batches for each task, our method has the best $LCA_{10}$ only on Split CUB. This is because the TAM and ED losses may slow the early training to mitigate domain overlap, but the following training will be improved significantly. LTR focuses on long-term remembering and our method outperforms other methods on these datasets except Split CUB. We think this is because the dataset CUB contains similar classes of birds, which means less impact of TAM and ED losses because of similar representations. In Fig. 4, we show the average accuracy trends in the continual process (from $A_{1}$ to $A_{T}$ ), which also indicate the better performance of the MDMT-R.

In Tab. 2, we then analyze the importance of the main components including TAM and ED loss on Split CIFAR. The first row is the results with only vanilla softmax. By adding ED loss, the average accuracy gets a little improvement. By adding TAM loss, the performance obtains larger gains, and we select the best $m_{\text{t}}$ and $m_{\text{c}}$ as the hyperparameters where $m_{\text{t}}=0$ and $m_{\text{c}}=0$ means the Cross-Domain Softmax. By adding both TAM and ED loss, we obtain a dramatic improvement in performance compared to the vanilla softmax and the state-of-the-art methods, which means the TAM and ED loss can significantly reduce the catastrophic forgetting.

Domain shift observation

In this section, we would like to show some observations of domain shift using t-distributed Stochastic Neighbor Embedding (t-SNE) (Maaten and Hinton 2008) on Permuted MNIST. First, in order to intuitively reflect the task relation of the proposed method during the training process, we visualize the final feature distribution, i.e., trained after the task 17, of task 1, 9 and 17 in Fig. 5. A-GEM and MEGA cannot guarantee the task boundaries, which means generating some mix area and makes the task easy to misclassify. The proposed MDMT rehearsal separates each class in three task while obtain explicit task boundary, which means the proposed method is able to encourage the intra-class/task compactness and inter-class/task discrepancy. As shown in Fig. 6, we also show the domain shift of task 1 after the model trained on task 1, 9 and 17, respectively. The previous methods A-GEM and MEGA cannot reduce the domain shift at all, which makes them sustainable to forget. The proposed MDMT rehearsal method can significantly mitigate the unpredictable domain shift. Without ED loss, our MDMT rehearsal still gets some unpredictable domain shift (such as task 1 after 1 and 9) because of the shrink of training data.

Conclusion

In this paper, we address catastrophic forgetting, a major drawback of state-of-the-art lifelong learning study, by considering the unpredictable domain shift of old tasks in the training sequence. To this end, we proposed a Multi-Domain Multi-Task rehearsal method, which effectively makes all tasks perceive each other. Then we proposed a Two-level Angular Margin loss to further encourage the intra-class/task compactness and inter-class/task discrepancy. Finally, an optional Episodic Distillation loss was proposed to mitigate domain shift. We have tested the proposed approach on four image classification benchmark datasets. Extensive experiments show the superiority of our approach over state-of-the-art methods.

Acknowledgment

This work was supported by the Natural Science Foundation of China (Nos. 62072334, 61671325, 61876121, 61672376 and U1803264) and Jiangsu Provincial Key Research and Development Program (No. BE2017663). The authors would like to thank constructive and valuable suggestions for this paper from the experienced reviewers and AE.

References

Aljundi et al. (2018) Aljundi, R.; Babiloni, F.; Elhoseiny, M.; Rohrbach, M.; and Tuytelaars, T. 2018. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV).
Chaudhry et al. (2018a) Chaudhry, A.; Dokania, P. K.; Ajanthan, T.; and Torr, P. H. 2018a. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV).
Chaudhry et al. (2018b) Chaudhry, A.; Ranzato, M.; Rohrbach, M.; and Elhoseiny, M. 2018b. Efficient Lifelong Learning with A-GEM. In International Conference on Learning Representations.
Chaudhry et al. (2019a) Chaudhry, A.; Rohrbach, M.; Elhoseiny, M.; Ajanthan, T.; Dokania, P. K.; Torr, P. H.; and Ranzato, M. 2019a. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486 .
Chaudhry et al. (2019b) Chaudhry, A.; Rohrbach, M.; Elhoseiny, M.; Ajanthan, T.; Dokania, P. K.; Torr, P. H. S.; and Ranzato, M. 2019b. On Tiny Episodic Memories in Continual Learning. arXiv preprint arXiv:1902.10486 .
De Lange et al. (2019) De Lange, M.; Aljundi, R.; Masana, M.; Parisot, S.; Jia, X.; Leonardis, A.; Slabaugh, G.; and Tuytelaars, T. 2019. Continual learning: A comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv:1909.08383 .
Deng et al. (2018) Deng, C.; Wu, Q.; Wu, Q.; Hu, F.; Lyu, F.; and Tan, M. 2018. Visual grounding via accumulated attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, 7746–7755.
Deng et al. (2019) Deng, J.; Guo, J.; Xue, N.; and Zafeiriou, S. 2019. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Dhar et al. (2019) Dhar, P.; Singh, R. V.; Peng, K.-C.; Wu, Z.; and Chellappa, R. 2019. Learning without memorizing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Fourure et al. (2017) Fourure, D.; Emonet, R.; Fromont, É.; Muselet, D.; Neverova, N.; Trémeau, A.; and Wolf, C. 2017. Multi-task, multi-domain learning: Application to semantic segmentation and pose regression. Neurocomputing .
French (1999) French, R. M. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences .
Guo et al. (2019) Guo, Y.; Liu, M.; Yang, T.; and Rosing, T. 2019. Learning with Long-term Remembering: Following the Lead of Mixed Stochastic Gradient. arXiv preprint arXiv:1909.11763 .
He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Hinton, Vinyals, and Dean (2015) Hinton, G. E.; Vinyals, O.; and Dean, J. 2015. Distilling the Knowledge in a Neural Network. CoRR .
Hou et al. (2018) Hou, S.; Pan, X.; Change Loy, C.; Wang, Z.; and Lin, D. 2018. Lifelong learning via progressive distillation and retrospection. In Proceedings of the European Conference on Computer Vision (ECCV).
Kirkpatrick et al. (2017) Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences .
Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images .
Lampert, Nickisch, and Harmeling (2009) Lampert, C. H.; Nickisch, H.; and Harmeling, S. 2009. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition.
Lesort et al. (2019) Lesort, T.; Gepperth, A.; Stoian, A.; and Filliat, D. 2019. Marginal replay vs conditional replay for continual learning. In International Conference on Artificial Neural Networks.
Li and Hoiem (2016) Li, Z.; and Hoiem, D. 2016. Learning Without Forgetting. In European Conference on Computer Vision.
Lin et al. (2019) Lin, X.; Zhen, H.-L.; Li, Z.; Zhang, Q.-F.; and Kwong, S. 2019. Pareto Multi-Task Learning. In Advances in Neural Information Processing Systems.
Liu et al. (2017) Liu, W.; Wen, Y.; Yu, Z.; Li, M.; Raj, B.; and Song, L. 2017. SphereFace: Deep Hypersphere Embedding for Face Recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017.
Liu et al. (2016) Liu, W.; Wen, Y.; Yu, Z.; and Yang, M. 2016. Large-margin softmax loss for convolutional neural networks. In ICML.
Lopez-Paz and Ranzato (2017) Lopez-Paz, D.; and Ranzato, M. 2017. Gradient episodic memory for continual learning. In Advances in neural information processing systems.
Lyu, Feng, and Wang (2020) Lyu, F.; Feng, W.; and Wang, S. 2020. vtGraphNet: Learning weakly-supervised scene graph for complex visual grounding. Neurocomputing 51–60.
Lyu et al. (2019) Lyu, F.; Wu, Q.; Hu, F.; Wu, Q.; and Tan, M. 2019. Attend and imagine: Multi-label image classification with visual attention and recurrent neural networks. IEEE Transactions on Multimedia 1971–1981.
Maaten and Hinton (2008) Maaten, L. v. d.; and Hinton, G. 2008. Visualizing data using t-SNE. Journal of machine learning research .
Mallya, Davis, and Lazebnik (2018) Mallya, A.; Davis, D.; and Lazebnik, S. 2018. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV).
Nam and Han (2016) Nam, H.; and Han, B. 2016. Learning multi-domain convolutional neural networks for visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Pellegrini et al. (2019) Pellegrini, L.; Graffieti, G.; Lomonaco, V.; and Maltoni, D. 2019. Latent replay for real-time continual learning. arXiv preprint arXiv:1912.01100 .
Peng and Dredze (2016) Peng, N.; and Dredze, M. 2016. Multi-task Multi-domain Representation Learning for Sequence Tagging. CoRR .
Rebuffi, Kolesnikov, and Lampert (2016) Rebuffi, S.; Kolesnikov, A.; and Lampert, C. H. 2016. iCaRL: Incremental Classifier and Representation Learning. CoRR .
Riemer et al. (2018) Riemer, M.; Cases, I.; Ajemian, R.; Liu, M.; Rish, I.; Tu, Y.; and Tesauro, G. 2018. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 .
Ring (1997) Ring, M. B. 1997. CHILD: A first step towards continual learning. Machine Learning .
Rolnick et al. (2019) Rolnick, D.; Ahuja, A.; Schwarz, J.; Lillicrap, T.; and Wayne, G. 2019. Experience replay for continual learning. In Advances in Neural Information Processing Systems.
Sener and Koltun (2018) Sener, O.; and Koltun, V. 2018. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems.
Shen et al. (2020) Shen, G.; Zhang, S.; Chen, X.; and Deng, Z.-H. 2020. Generative Feature Replay with Orthogonal Weight Modification for Continual Learning. arXiv preprint arXiv:2005.03490 .
Tang and Jia (2020) Tang, H.; and Jia, K. 2020. Discriminative Adversarial Domain Adaptation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020.
Thrun (1998) Thrun, S. 1998. Lifelong learning algorithms. In Learning to learn. Springer.
van de Ven and Tolias (2018) van de Ven, G. M.; and Tolias, A. S. 2018. Generative replay with feedback connections as a general strategy for continual learning. arXiv preprint arXiv:1809.10635 .
Wah et al. (2011) Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. 2011. The caltech-ucsd birds-200-2011 dataset .
Wang et al. (2018a) Wang, F.; Cheng, J.; Liu, W.; and Liu, H. 2018a. Additive Margin Softmax for Face Verification. IEEE Signal Process. Lett. .
Wang et al. (2018b) Wang, H.; Wang, Y.; Zhou, Z.; Ji, X.; Gong, D.; Zhou, J.; Li, Z.; and Liu, W. 2018b. CosFace: Large Margin Cosine Loss for Deep Face Recognition. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018.
Yang and Hospedales (2014) Yang, Y.; and Hospedales, T. M. 2014. A Unified Perspective on Multi-Domain and Multi-Task Learning. In International Conference on Learning Representations.
Yoon et al. (2017) Yoon, J.; Yang, E.; Lee, J.; and Hwang, S. J. 2017. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547 .
Zenke, Poole, and Ganguli (2017) Zenke, F.; Poole, B.; and Ganguli, S. 2017. Continual learning through synaptic intelligence. Proceedings of machine learning research .

Appendix

Evaluation Metrics

Average Accuracy. ( $A_{t}\in[0,1]$ ) Average accuracy after the model has been trained continually with all the mini-batches up till task $t\in\{1,\cdots,T\}$ is defined as

A_{t}=\frac{1}{t}\sum_{j=1}^{t}a_{t,B_{t},j},

(11)

where $a_{k,B_{t},j}$ is the mean accuracy of task $j$ on $B_{t}$ mini-batches. In particular, $A_{T}$ is the average accuracy on all the tasks after the last task has been learned.

Forgetting Measure (Chaudhry et al. 2018a). ( $F_{t}\in[-1,1]$ ) Average forgetting after the model has been trained continually with all the mini-batches up till task $t\in\{1,\cdots,T\}$ is defined as

F_{t}=\frac{1}{t-1}\sum_{j=1}^{t-1}f_{j}^{t}

(12)

where $f_{j}^{t}$ is the forgetting on task $j$ after the model is trained with all the mini-batches up till task $t$ and computed as

f_{j}^{t}=\max_{l\in\{1,\cdots,k-1\}}a_{l,B_{l},j}-a_{t,B_{k},j}

(13)

Learning Curve Area (Chaudhry et al. 2018a). ( $LCA\in[0,1]$ ) $LCA_{\beta}$ is the area of the convergence curve for any average $b$ -shot performance after the model has been trained for all the $T$ tasks, where $b\in\left[0,\beta\right]$ .

\mathrm{LCA}_{\beta}=\frac{1}{\beta+1}\int_{0}^{\beta}Z_{b}db=\frac{1}{\beta+1}\sum_{b=0}^{\beta}Z_{b},

(14)

where

Z_{b}=\frac{1}{T}\sum_{t=1}^{T}a_{t,b,t}.

(15)

Intuitively, LCA measures the learning speed of different lifelong learning algorithms. A higher value of LCA indicates that the model learns quickly.

Long-Term Remembering (Guo et al. 2019). ( $LTR\geq 0$ ) $LTR$ quantifies the accuracy drop on each task relative to the accuracy just right after the task has been learned, which is defined as

\mathrm{LTR}=-\frac{1}{T-1}\sum_{j=1}^{T-1}(T-j)\min\left\{0,a_{T,B_{T},j}-a_{j,B_{j},j}\right\}

(16)

After the model is trained on all the $T$ tasks, LTR quantifies the accuracy drop on task $D_{j}$ relative to $a_{j,B_{j},j}$ .

Implementation details

Seed initialization. We implement our method with Tensorflow. The results are the average of 5 runs, where the seeds of Numpy and Tensorflow are both from 1234 to 1238 for all compared methods.

Selected classes. For Split CUB and AWA, AGEM has provided fixed classes selections. For Perm MNIST, each task has a fixed random permutation by shuffling the pixels in images, the seeds are the same with the training. Split CIFAR has 20 disjoint subsets by extracted each 5-classes in sequence from class 1 to 100. Train/test splits: For Perm MNIST and Split CIFAR, the data splits are the same as vanilla MNIST and CIFAR. For Split CUB and AWA, AGEM has provided fixed train/test list.

Hyper-Parameters selection. We report the hyper-parameters considered for different experiments of $m_{\text{t}}$ , $m_{\text{c}}$ and $s$ . The searching spaces for the hyper parameters are: $m_{\text{t}}$ : [0.01, 0.05, 0.1 (Permuted MNIST, Split CIFAR), 0.2 (Split AWA), 0.3, 0.4 (Split CUB), 0.5]; $m_{\text{c}}$ : [0.01 (Permuted MNIST, Split CIFAR), 0.02 (Split AWA), 0.03, 0.04, 0.05 (Split CUB), 0.06, …, 0.1]; $s$ : [12, 16, 20 (Split CUB), 24 (Split CIFAR), 28, 32 (Permuted MNIST, Split AWA), 36, …, 64]. Note that we do not conduct exhaustive hyper-parameter search on learning rate and regularization factors and reuse the same as A-GEM (Chaudhry et al. 2018b). The learning rates for each dataset are: Perm MNIST: $0.1$ , Split CIFAR and CUB: $0.03$ , Split AWA: $0.01$ .

Table 3: Computational cost and memory complexity. The memory cost consists of several parameters: (1) the number of task

T

; (2) the total number of parameters

P

; (3) the size of the mini-batch

B

; (4) the total size of the network hidden state

H

(assuming all methods use the same architecture); (5) the size of the episodic memory

M

per task; (6) the size of the

l

-th layer hidden state

\tilde{H}

w.r.t. the latent representation.

Method	Inference time (ms)				Memory
Method	MNIST	CIFAR	CUB	AWA	Training	Testing
VAN	-	-	-	-	$P+B*H$	$P+B*H$
EWC	-	-	-	-	$4P+BH$	$4P+BH$
GEM	-	-	-	-	$PT+(B+M)H$	$P+B*H$
A-GEM	32.58	28.74	81.83	97.47	$2P+(B+M)H$	$P+B*H$
MEGA	37.63	26.38	84.96	109.67	$2P+(B+M)H$	$P+B*H$
MDMT-R	35.16	30.09	90.47	97.51	$2P+(B+M)H+M*H_{l}$	$P+B*H$

Evolution of LCA

We show the evolution of $LCA$ during the first ten mini-batches on the four datasets, where $LCA_{k}$ represents the learning speed of new task in the first $k$ mini-batches. As shown in Fig. 7, the proposed MDMT rehearsal based lifelong learning outperforms other methods on Split CUB, however a slight worse than MEGA. In our option, this is because the ED loss tries to make the model not fall into catastrophic forgetting at the training beginning. Thus, the proposed MDMT rehearsal based lifelong learning do not achieve fast improvements at the beginning of each task.

Computational cost and memory complexity

We show the computational cost and memory complexity in Table 3 by evaluating the training time on single RTX 2080Ti GPU card. Easy to see, the inference time of the proposed method is comparable with A-GEM and MEGA on four datasets. We can also see the memory cost of training and testing procedure in the right of Table 3. Compared to the previous baselines based on EpsMem, the cost of the proposed method has a slight increase $M*\tilde{H}$ , which arises from the saving of latent representation of old tasks. However, such an increase is very small compared to the total training cost because in most situations the latent representation (a vector) has smaller size than the image (a 3-channel tensor).

Raw Record of Learning Process

In this section, we report the raw record of learning process for the proposed method, MEGA and A-GEM. The entry $(i,j)$ of the matrix for each method is the test accuracy of the $j$ -th task after the model is trained on the $i$ -th task.

PERMUTED MNIST

MDMT-R:

0.9658 0.1207 0.1267 0.0846 0.1393 0.0835 0.1005 0.0883 0.1218 0.1004 0.0952 0.1267 0.1263 0.1041 0.0929 0.0910 0.0995

0.9638 0.9676 0.1059 0.0995 0.1204 0.0740 0.0849 0.0955 0.1126 0.1292 0.1146 0.1180 0.1302 0.0945 0.0867 0.1021 0.0981

0.9574 0.9638 0.9612 0.0842 0.0935 0.0925 0.0793 0.1035 0.1002 0.1226 0.1073 0.1043 0.1086 0.1024 0.0962 0.0706 0.0822

0.9548 0.9596 0.9652 0.9554 0.0954 0.0790 0.1026 0.1120 0.0935 0.1198 0.0961 0.1186 0.1247 0.0833 0.1099 0.0753 0.0821

0.9498 0.9578 0.9607 0.9660 0.9679 0.0854 0.0889 0.1347 0.1134 0.1258 0.1014 0.1410 0.1396 0.1091 0.0994 0.0822 0.1033

0.9477 0.9544 0.9595 0.9627 0.9655 0.9662 0.0998 0.1316 0.1121 0.1262 0.1203 0.1498 0.1303 0.0968 0.1044 0.0817 0.0903

0.9451 0.9532 0.9568 0.9607 0.9630 0.9636 0.9669 0.1310 0.1004 0.1341 0.1004 0.1286 0.1438 0.1127 0.1124 0.0834 0.0990

0.9415 0.9503 0.9550 0.9559 0.9610 0.9588 0.9644 0.9612 0.0977 0.1129 0.0831 0.1223 0.1353 0.1027 0.1154 0.0748 0.0992

0.9359 0.9466 0.9483 0.9543 0.9574 0.9557 0.9600 0.9609 0.9625 0.1141 0.1028 0.1318 0.1331 0.1005 0.1128 0.0909 0.1003

0.9326 0.9424 0.9466 0.9529 0.9536 0.9563 0.9572 0.9590 0.9632 0.9621 0.0958 0.1082 0.1196 0.1095 0.1116 0.0762 0.1005

0.9313 0.9426 0.9465 0.9499 0.9512 0.9528 0.9556 0.9557 0.9598 0.9627 0.9660 0.1218 0.1369 0.1405 0.1088 0.0821 0.1195

0.9273 0.9397 0.9438 0.9450 0.9468 0.9513 0.9506 0.9540 0.9571 0.9605 0.9640 0.9630 0.1645 0.1249 0.0974 0.0684 0.1202

0.9224 0.9363 0.9376 0.9446 0.9469 0.9466 0.9498 0.9496 0.9541 0.9588 0.9600 0.9619 0.9630 0.1373 0.1011 0.0633 0.1328

0.9178 0.9326 0.9377 0.9418 0.9419 0.9453 0.9481 0.9484 0.9518 0.9562 0.9586 0.9594 0.9618 0.9649 0.1074 0.0798 0.1247

0.9167 0.9296 0.9341 0.9349 0.9356 0.9439 0.9417 0.9460 0.9495 0.9535 0.9553 0.9558 0.9596 0.9623 0.9629 0.0775 0.1267

0.9132 0.9269 0.9314 0.9338 0.9346 0.9418 0.9414 0.9443 0.9478 0.9511 0.9533 0.9544 0.9559 0.9597 0.9628 0.9663 0.1293

0.9111 0.9221 0.9277 0.9303 0.9341 0.9387 0.9398 0.9409 0.9451 0.9491 0.9495 0.9520 0.9555 0.9566 0.9595 0.9625 0.9621

MEGA:

0.9613 0.1091 0.1229 0.0832 0.1374 0.0708 0.0907 0.1017 0.1165 0.1286 0.0979 0.1182 0.1188 0.0886 0.0968 0.0854 0.0928

0.9535 0.9645 0.0895 0.0997 0.1191 0.0685 0.0803 0.1022 0.1165 0.1472 0.1054 0.1112 0.1264 0.1027 0.0872 0.0979 0.0993

0.9391 0.9556 0.9596 0.0996 0.1020 0.0900 0.0807 0.1083 0.0959 0.1400 0.1001 0.1012 0.1096 0.1085 0.0977 0.0716 0.0768

0.9295 0.9473 0.9527 0.9477 0.1113 0.0725 0.0856 0.1033 0.0884 0.1209 0.0847 0.1149 0.1285 0.0939 0.1193 0.0867 0.0824

0.9206 0.9405 0.9437 0.9569 0.9611 0.0785 0.0884 0.1113 0.0926 0.1189 0.0936 0.1337 0.1544 0.1154 0.1282 0.1010 0.0994

0.9119 0.9347 0.9378 0.9481 0.9547 0.9594 0.0916 0.1200 0.1013 0.1042 0.0908 0.1380 0.1415 0.1199 0.1210 0.0908 0.0847

0.9083 0.9261 0.9348 0.9419 0.9478 0.9556 0.9575 0.1092 0.1040 0.1083 0.0863 0.1202 0.1286 0.1177 0.1250 0.0801 0.0931

0.9100 0.9192 0.9291 0.9332 0.9419 0.9462 0.9527 0.9598 0.1152 0.1132 0.0945 0.1054 0.1248 0.1228 0.1187 0.0945 0.0934

0.9022 0.9133 0.9215 0.9271 0.9344 0.9381 0.9430 0.9476 0.9551 0.1187 0.1162 0.1119 0.1364 0.1249 0.1108 0.1012 0.1059

0.8974 0.9074 0.9147 0.9242 0.9289 0.9348 0.9367 0.9404 0.9519 0.9571 0.1102 0.1032 0.1536 0.1261 0.1122 0.1036 0.1093

0.8957 0.9042 0.9146 0.9193 0.9229 0.9313 0.9321 0.9318 0.9409 0.9545 0.9591 0.1144 0.1368 0.1373 0.1143 0.1092 0.1169

0.8863 0.8981 0.9056 0.9127 0.9148 0.9220 0.9270 0.9249 0.9335 0.9444 0.9528 0.9564 0.1517 0.1103 0.1051 0.1002 0.1390

0.8840 0.8992 0.9054 0.9085 0.9149 0.9179 0.9238 0.9198 0.9304 0.9419 0.9441 0.9523 0.9570 0.1292 0.1083 0.0926 0.1301

0.8808 0.8901 0.8994 0.8986 0.9084 0.9113 0.9168 0.9185 0.9239 0.9360 0.9382 0.9453 0.9522 0.9589 0.1017 0.0941 0.1277

0.8770 0.8850 0.8957 0.8926 0.9012 0.9101 0.9090 0.9132 0.9188 0.9301 0.9358 0.9368 0.9430 0.9508 0.9521 0.0946 0.1334

0.8752 0.8806 0.8911 0.8854 0.8965 0.9070 0.9062 0.9059 0.9145 0.9265 0.9286 0.9338 0.9374 0.9434 0.9462 0.9601 0.1291

0.8732 0.8765 0.8824 0.8809 0.8945 0.9024 0.9016 0.9007 0.9088 0.9202 0.9228 0.9276 0.9279 0.9376 0.9408 0.9521 0.9556

A-GEM:

0.9613 0.1091 0.1229 0.0832 0.1374 0.0708 0.0907 0.1017 0.1165 0.1286 0.0979 0.1182 0.1188 0.0886 0.0968 0.0854 0.0928

0.9509 0.9645 0.0956 0.0991 0.1304 0.0696 0.0840 0.1033 0.1219 0.1454 0.1064 0.1133 0.1314 0.1043 0.0883 0.0979 0.0973

0.9410 0.9545 0.9615 0.0995 0.0964 0.0921 0.0710 0.1126 0.1176 0.1402 0.1112 0.1026 0.1185 0.1204 0.1077 0.0779 0.0761

0.9299 0.9450 0.9540 0.9546 0.1046 0.0788 0.0959 0.1033 0.1096 0.1266 0.1015 0.1152 0.1476 0.0885 0.1375 0.0984 0.0831

0.9151 0.9361 0.9425 0.9551 0.9588 0.0803 0.0809 0.1143 0.1063 0.1227 0.1066 0.1253 0.1436 0.1154 0.1131 0.1079 0.0915

0.9068 0.9312 0.9401 0.9450 0.9566 0.9590 0.0892 0.1189 0.1285 0.1086 0.1007 0.1433 0.1279 0.1179 0.1097 0.0892 0.0865

0.9015 0.9228 0.9339 0.9385 0.9473 0.9548 0.9586 0.1063 0.1073 0.1102 0.1048 0.1164 0.1291 0.1284 0.1341 0.0854 0.1024

0.8980 0.9155 0.9248 0.9280 0.9356 0.9403 0.9539 0.9580 0.1015 0.1231 0.1129 0.1125 0.1267 0.1133 0.1220 0.0921 0.0985

0.8952 0.9055 0.9201 0.9182 0.9273 0.9310 0.9447 0.9435 0.9512 0.1098 0.1374 0.1166 0.1264 0.1064 0.1183 0.0986 0.1048

0.8846 0.8996 0.9083 0.9154 0.9189 0.9267 0.9363 0.9339 0.9513 0.9558 0.1243 0.1095 0.1179 0.1137 0.1126 0.0945 0.0979

0.8764 0.8977 0.9011 0.9073 0.9086 0.9167 0.9292 0.9274 0.9386 0.9481 0.9631 0.1116 0.1099 0.1417 0.1100 0.0975 0.1166

0.8710 0.8882 0.8937 0.8922 0.9043 0.9077 0.9151 0.9174 0.9279 0.9324 0.9518 0.9572 0.1346 0.1240 0.0964 0.0930 0.1283

0.8625 0.8822 0.8847 0.8855 0.9013 0.8990 0.9093 0.9088 0.9189 0.9295 0.9411 0.9458 0.9533 0.1309 0.1059 0.0987 0.1139

0.8581 0.8784 0.8774 0.8817 0.8954 0.8938 0.8986 0.9003 0.9082 0.9225 0.9307 0.9350 0.9435 0.9603 0.1048 0.1023 0.1048

0.8492 0.8674 0.8732 0.8697 0.8828 0.8826 0.8930 0.8898 0.8962 0.9098 0.9184 0.9267 0.9290 0.9425 0.9542 0.1012 0.1070

0.8322 0.8700 0.8644 0.8493 0.8765 0.8787 0.8904 0.8848 0.8883 0.8979 0.9110 0.9158 0.9177 0.9299 0.9403 0.9609 0.1179

0.8438 0.8603 0.8555 0.8488 0.8864 0.8785 0.8798 0.8702 0.8916 0.8968 0.9076 0.9094 0.9092 0.9228 0.9228 0.9463 0.9551

SPLIT CIFAR

MDMT-R:

0.6828 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6704 0.6292 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6912 0.6136 0.6596 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6732 0.5828 0.6316 0.7064 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6604 0.5944 0.6288 0.7012 0.7120 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6736 0.6140 0.5988 0.6792 0.7064 0.7584 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6680 0.5944 0.6068 0.6752 0.6976 0.7800 0.6996 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6824 0.6036 0.6096 0.6780 0.6780 0.7236 0.6952 0.7236 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6716 0.6028 0.6084 0.6812 0.6840 0.7396 0.6772 0.7120 0.7744 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6872 0.6136 0.6140 0.6848 0.6864 0.7432 0.6728 0.6916 0.7492 0.7052 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6604 0.6040 0.6104 0.6908 0.6788 0.7368 0.6764 0.6980 0.7628 0.6700 0.7592 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6548 0.5888 0.6136 0.6784 0.6832 0.7136 0.6836 0.6744 0.7440 0.6472 0.7460 0.6820 0.0000 0.0000 0.0000 0.0000 0.0000

0.6484 0.5964 0.6100 0.6640 0.6744 0.7224 0.6540 0.6804 0.7224 0.6504 0.7264 0.6728 0.7424 0.0000 0.0000 0.0000 0.0000

0.6628 0.5936 0.6152 0.7008 0.6940 0.7272 0.6624 0.6788 0.7372 0.6504 0.7232 0.6712 0.7256 0.7772 0.0000 0.0000 0.0000

0.6648 0.5928 0.6204 0.6936 0.6848 0.7172 0.6624 0.6900 0.7308 0.6484 0.7168 0.6424 0.7336 0.7672 0.6684 0.0000 0.0000

0.6704 0.5920 0.6152 0.6984 0.6988 0.7260 0.6732 0.6764 0.7260 0.6464 0.7064 0.6292 0.7216 0.7520 0.6780 0.7612 0.0000

0.6736 0.6140 0.6168 0.6928 0.6996 0.7228 0.6792 0.6908 0.7284 0.6736 0.7136 0.6452 0.7208 0.7496 0.6492 0.7416 0.7920

MEGA:

0.6472 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6260 0.5824 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6324 0.5700 0.6300 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6236 0.5496 0.5624 0.6452 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6132 0.5612 0.6048 0.6736 0.6960 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6140 0.5628 0.5692 0.6632 0.6792 0.7688 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5780 0.5500 0.5864 0.6364 0.6792 0.7420 0.6868 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5756 0.5308 0.5764 0.6292 0.6500 0.6820 0.6580 0.6580 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6212 0.5704 0.5876 0.6376 0.6600 0.7056 0.6636 0.6916 0.7376 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5992 0.5580 0.5828 0.6212 0.6528 0.6512 0.6496 0.6700 0.7276 0.6732 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6104 0.5552 0.5804 0.6396 0.6700 0.6960 0.6568 0.6752 0.7412 0.6752 0.7432 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5880 0.5516 0.5816 0.6248 0.6552 0.6568 0.6412 0.6284 0.7044 0.6304 0.7196 0.6660 0.0000 0.0000 0.0000 0.0000 0.0000

0.6020 0.5628 0.5792 0.6164 0.6444 0.6636 0.6356 0.6536 0.7008 0.6132 0.7000 0.6336 0.7108 0.0000 0.0000 0.0000 0.0000

0.6124 0.5692 0.5924 0.6420 0.6516 0.6912 0.6352 0.6492 0.6848 0.6400 0.6872 0.6312 0.7280 0.7596 0.0000 0.0000 0.0000

0.6012 0.5468 0.5908 0.6128 0.6552 0.6852 0.6288 0.6428 0.6704 0.6176 0.6988 0.6268 0.7088 0.7348 0.6324 0.0000 0.0000

0.6244 0.5588 0.5960 0.6432 0.6448 0.6868 0.6388 0.6540 0.6896 0.6056 0.6920 0.6192 0.7124 0.7344 0.6560 0.7604 0.0000

0.6088 0.5896 0.5840 0.6552 0.6716 0.6904 0.6584 0.6372 0.7032 0.6300 0.6900 0.5864 0.6832 0.7140 0.6348 0.7264 0.7780

A-GEM:

0.6772 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5948 0.5764 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6324 0.5828 0.6432 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5980 0.5384 0.5396 0.6456 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5864 0.5404 0.5576 0.6436 0.7004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5728 0.5392 0.5068 0.5940 0.6344 0.7180 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5572 0.5404 0.5308 0.6224 0.6116 0.6520 0.6688 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6064 0.5356 0.5492 0.5872 0.6164 0.6532 0.6296 0.6724 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6060 0.5472 0.5528 0.6236 0.5920 0.6348 0.6076 0.6348 0.6972 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6004 0.5080 0.4960 0.6128 0.5656 0.6356 0.5752 0.6140 0.6580 0.6792 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5992 0.5408 0.5332 0.5964 0.5928 0.6520 0.5928 0.6304 0.6764 0.5916 0.7364 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5748 0.5020 0.5104 0.5936 0.6016 0.6184 0.5772 0.6164 0.6408 0.5688 0.6776 0.6436 0.0000 0.0000 0.0000 0.0000 0.0000

0.6056 0.5100 0.5200 0.5916 0.6012 0.6056 0.5816 0.6060 0.6236 0.5808 0.6288 0.5768 0.7332 0.0000 0.0000 0.0000 0.0000

0.6184 0.5344 0.5308 0.5888 0.6116 0.6188 0.6012 0.6248 0.6136 0.5836 0.6428 0.5688 0.6524 0.7392 0.0000 0.0000 0.0000

0.6012 0.5220 0.5488 0.6008 0.5828 0.6048 0.5728 0.5884 0.6356 0.5740 0.6476 0.5540 0.6520 0.6804 0.6460 0.0000 0.0000

0.5984 0.5360 0.5520 0.5808 0.5704 0.6184 0.6068 0.6108 0.6452 0.5404 0.6520 0.5256 0.6624 0.6512 0.5864 0.7388 0.0000

0.6232 0.5356 0.5412 0.6104 0.6080 0.6248 0.5944 0.5900 0.6492 0.5872 0.6468 0.5352 0.6336 0.6520 0.5908 0.6444 0.7508

SPLIT CUB

MDMT-R:

0.4895 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6745 0.5760 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7401 0.7400 0.6662 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7442 0.7663 0.7934 0.7369 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7583 0.7702 0.7941 0.8119 0.6626 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7710 0.7837 0.7941 0.7424 0.7669 0.7327 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7811 0.7697 0.8204 0.8044 0.7630 0.8067 0.7581 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7971 0.7810 0.8303 0.8311 0.8019 0.8095 0.8337 0.7485 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7703 0.8008 0.8249 0.8081 0.7980 0.8369 0.8662 0.7686 0.7511 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.8047 0.8215 0.8316 0.8165 0.8089 0.8049 0.8463 0.7734 0.7968 0.7640 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7924 0.8011 0.8303 0.8435 0.8126 0.8358 0.8531 0.7990 0.7823 0.8107 0.8219 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7886 0.7963 0.8114 0.8250 0.7875 0.8222 0.8249 0.8164 0.7612 0.8091 0.8305 0.7917 0.0000 0.0000 0.0000 0.0000 0.0000

0.8000 0.8034 0.8516 0.8224 0.8052 0.8355 0.8185 0.7993 0.7778 0.8177 0.8461 0.8107 0.8405 0.0000 0.0000 0.0000 0.0000

0.8276 0.8085 0.8549 0.8120 0.8289 0.8487 0.8130 0.8041 0.7675 0.8183 0.8329 0.8060 0.8332 0.7764 0.0000 0.0000 0.0000

0.7938 0.8199 0.8154 0.8069 0.8141 0.8358 0.8395 0.7959 0.8041 0.8286 0.8365 0.8043 0.8527 0.7462 0.8142 0.0000 0.0000

0.8215 0.7984 0.8517 0.8471 0.8237 0.8352 0.8538 0.8115 0.7959 0.8302 0.8366 0.8304 0.8494 0.8036 0.8071 0.8064 0.0000

0.8342 0.8331 0.8544 0.8695 0.8449 0.8628 0.8789 0.8328 0.8235 0.8601 0.8938 0.8434 0.8538 0.8096 0.8195 0.8166 0.7844

MEGA:

0.4050 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6876 0.5088 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6844 0.7244 0.6263 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7071 0.7445 0.7753 0.6553 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7294 0.7483 0.7941 0.7787 0.6857 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7049 0.7266 0.7712 0.7538 0.7528 0.6366 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7271 0.7488 0.7877 0.7893 0.7861 0.7905 0.6830 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7316 0.7737 0.7920 0.7867 0.7851 0.7956 0.8179 0.6816 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7503 0.7525 0.7860 0.7947 0.7838 0.7818 0.8148 0.7912 0.6934 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7445 0.7475 0.7819 0.7599 0.7680 0.7834 0.7901 0.7858 0.7883 0.7085 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7506 0.7732 0.8053 0.8129 0.8049 0.8099 0.8275 0.7938 0.8041 0.8146 0.7211 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.7475 0.7670 0.7860 0.7982 0.7818 0.7892 0.8133 0.7881 0.8014 0.8145 0.7958 0.7144 0.0000 0.0000 0.0000 0.0000 0.0000

0.7393 0.7572 0.7907 0.8006 0.7821 0.8043 0.8123 0.7629 0.7905 0.7902 0.8161 0.7622 0.7262 0.0000 0.0000 0.0000 0.0000

0.7492 0.7501 0.8031 0.8098 0.7956 0.8020 0.8053 0.7811 0.7873 0.8081 0.8159 0.7672 0.7981 0.6794 0.0000 0.0000 0.0000

0.7488 0.7661 0.7958 0.7830 0.7845 0.7873 0.8007 0.7759 0.7902 0.8139 0.8142 0.7746 0.7984 0.7651 0.6831 0.0000 0.0000

0.7538 0.7857 0.8116 0.8170 0.8050 0.8052 0.8277 0.8024 0.8070 0.8220 0.8301 0.7767 0.8210 0.7839 0.7777 0.7505 0.0000

0.7728 0.7846 0.8203 0.7995 0.8141 0.8219 0.8201 0.7973 0.8109 0.8366 0.8390 0.7880 0.8300 0.7948 0.8078 0.8112 0.7418

A-GEM:

0.4263 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4383 0.5243 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4642 0.5220 0.6064 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4850 0.5420 0.6057 0.6765 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4935 0.5378 0.6328 0.6042 0.6621 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4678 0.5071 0.6369 0.5585 0.5966 0.6137 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4909 0.5630 0.6435 0.6369 0.6140 0.6278 0.6825 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4928 0.5607 0.6157 0.5992 0.6025 0.6037 0.6572 0.6617 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4850 0.5401 0.6090 0.5830 0.6004 0.5980 0.6384 0.6520 0.6850 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4901 0.5542 0.6086 0.5858 0.6080 0.5985 0.6206 0.6533 0.6607 0.6964 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4909 0.5455 0.6465 0.6220 0.6241 0.6186 0.6094 0.6197 0.6181 0.6569 0.7128 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5075 0.5675 0.6266 0.6126 0.5999 0.6037 0.6073 0.6214 0.6004 0.6424 0.6547 0.6673 0.0000 0.0000 0.0000 0.0000 0.0000

0.4909 0.5588 0.5943 0.5852 0.5898 0.6018 0.6188 0.5974 0.6266 0.6228 0.6792 0.6281 0.7035 0.0000 0.0000 0.0000 0.0000

0.5297 0.5535 0.6220 0.6281 0.6383 0.6037 0.6234 0.5967 0.6277 0.6051 0.6393 0.5950 0.6965 0.6630 0.0000 0.0000 0.0000

0.5104 0.5677 0.6446 0.6198 0.6135 0.6093 0.6100 0.5710 0.5912 0.6314 0.6312 0.5951 0.6924 0.6422 0.6584 0.0000 0.0000

0.5196 0.5527 0.6350 0.6043 0.6384 0.6009 0.6226 0.5971 0.6028 0.6210 0.6507 0.6318 0.6757 0.6003 0.5899 0.7272 0.0000

0.5345 0.5519 0.6395 0.6064 0.6335 0.6000 0.5879 0.5899 0.5869 0.6509 0.6486 0.6332 0.6914 0.6130 0.5983 0.6806 0.7216

SPLIT AWA

MDMT-R:

0.3792 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5119 0.4335 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5147 0.4985 0.4556 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5382 0.5230 0.5662 0.3475 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5292 0.5307 0.5874 0.4767 0.4884 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5725 0.5134 0.5924 0.5043 0.5974 0.4749 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5690 0.5398 0.6020 0.4712 0.5798 0.6075 0.4768 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5645 0.5055 0.6122 0.4804 0.5792 0.5996 0.5433 0.5065 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5561 0.5574 0.6216 0.5301 0.6216 0.6023 0.5638 0.5725 0.4626 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5941 0.5437 0.6306 0.5239 0.6218 0.6206 0.5343 0.5575 0.5019 0.5462 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.6117 0.5522 0.6058 0.5104 0.6036 0.6073 0.5348 0.5694 0.5112 0.6453 0.5159 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5784 0.5168 0.5892 0.5313 0.6071 0.5785 0.5446 0.5509 0.5224 0.6316 0.5863 0.5257 0.0000 0.0000 0.0000 0.0000 0.0000

0.5837 0.5408 0.6202 0.5282 0.6289 0.6234 0.5625 0.5494 0.5276 0.6285 0.6178 0.6038 0.4910 0.0000 0.0000 0.0000 0.0000

0.5962 0.5716 0.6666 0.5407 0.6461 0.6104 0.5786 0.5918 0.5276 0.6605 0.6201 0.6079 0.5582 0.5378 0.0000 0.0000 0.0000

0.6177 0.5444 0.6474 0.5467 0.6466 0.6326 0.5556 0.6075 0.5505 0.6552 0.6383 0.6123 0.6076 0.6551 0.5770 0.0000 0.0000

0.5900 0.5438 0.6428 0.5373 0.6405 0.5971 0.5792 0.5991 0.5428 0.6470 0.6332 0.5935 0.5679 0.6485 0.6678 0.5730 0.0000

0.6184 0.5875 0.6696 0.5620 0.6434 0.6194 0.5766 0.6226 0.5566 0.6739 0.6475 0.6349 0.5714 0.6531 0.6640 0.6051 0.5604

MEGA:

0.4101 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4791 0.4377 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5061 0.5174 0.4116 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5184 0.5219 0.4962 0.4741 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4931 0.5219 0.4828 0.5142 0.4026 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5122 0.5357 0.4894 0.5293 0.5053 0.4680 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5242 0.5226 0.5038 0.5219 0.5215 0.5605 0.4820 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5383 0.5359 0.5216 0.5323 0.5204 0.5551 0.5774 0.4682 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5291 0.5179 0.4850 0.5367 0.5043 0.5380 0.5565 0.5111 0.4698 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5386 0.5106 0.4995 0.5245 0.5150 0.5470 0.5457 0.4862 0.5380 0.4619 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5612 0.5569 0.5260 0.5566 0.5387 0.5765 0.5847 0.5483 0.5444 0.5622 0.5542 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.5593 0.5593 0.5416 0.5611 0.5345 0.5348 0.5539 0.5358 0.5418 0.5363 0.5720 0.4725 0.0000 0.0000 0.0000 0.0000 0.0000

0.5471 0.5522 0.5333 0.5383 0.5159 0.5410 0.5638 0.5234 0.5234 0.5342 0.5614 0.5611 0.5026 0.0000 0.0000 0.0000 0.0000

0.5557 0.5561 0.5391 0.5368 0.5093 0.5442 0.5743 0.5026 0.5484 0.5126 0.5788 0.5548 0.5641 0.5386 0.0000 0.0000 0.0000

0.5240 0.5563 0.5366 0.5465 0.5102 0.5659 0.5688 0.5071 0.5320 0.5183 0.5865 0.5705 0.5295 0.6206 0.5483 0.0000 0.0000

0.5541 0.5575 0.5219 0.5559 0.5310 0.5490 0.5983 0.5121 0.5466 0.5204 0.5929 0.5626 0.5466 0.6064 0.6291 0.4661 0.0000

0.5542 0.5469 0.5093 0.5677 0.5136 0.5471 0.5581 0.4954 0.5326 0.5169 0.5857 0.5771 0.5345 0.5964 0.6081 0.5187 0.4655

A-GEM:

0.4127 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4256 0.4422 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4436 0.4445 0.4058 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4371 0.4784 0.4334 0.4463 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4339 0.4795 0.4236 0.4258 0.3963 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4226 0.4674 0.4311 0.4505 0.3864 0.4495 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4279 0.4462 0.4268 0.4217 0.3854 0.4254 0.4239 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4621 0.4733 0.4421 0.4563 0.4239 0.4356 0.4489 0.4299 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4672 0.4774 0.4363 0.4402 0.4265 0.4387 0.4503 0.4129 0.4431 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4659 0.4417 0.4385 0.4386 0.4188 0.4419 0.4655 0.4075 0.3971 0.4286 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4555 0.4657 0.4495 0.4645 0.4133 0.4374 0.4717 0.4083 0.4289 0.4144 0.5037 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000

0.4425 0.4463 0.4277 0.4532 0.4301 0.4319 0.4865 0.4288 0.4043 0.4021 0.4172 0.4478 0.0000 0.0000 0.0000 0.0000 0.0000

0.4427 0.4582 0.4438 0.4527 0.4345 0.4638 0.4895 0.4327 0.4181 0.4295 0.4745 0.4303 0.4889 0.0000 0.0000 0.0000 0.0000

0.4538 0.4519 0.4066 0.4696 0.4099 0.4459 0.4859 0.4080 0.3931 0.3827 0.4382 0.3854 0.4055 0.4736 0.0000 0.0000 0.0000

0.4476 0.4786 0.4148 0.4879 0.4269 0.4576 0.5014 0.4532 0.4264 0.4137 0.4535 0.4178 0.3955 0.4809 0.4941 0.0000 0.0000

0.4482 0.4635 0.4114 0.4720 0.4231 0.4527 0.5082 0.4076 0.4231 0.4262 0.4436 0.4122 0.3915 0.4809 0.4762 0.4286 0.0000

0.4530 0.4605 0.4294 0.4519 0.4388 0.4649 0.4885 0.4514 0.4395 0.4236 0.4760 0.4495 0.4105 0.4621 0.4730 0.4039 0.4646