Adversarial Training of Variational Auto-encoders for Continual Zero-shot Learning(A-CZSL)

The main problem in continual learning is catastrophic forgetting. McCloskey and Cohen first introduced the term catastrophic forgetting or catastrophic interference in the 1980s [17]. They claimed that catastrophic interference is a fundamental limitation of neural networks and a downside of its high generalization ability. While the cause of catastrophic forgetting has not been studied analytically, it is known that the neural networks parameterize the internal features of inputs, and training the networks on new samples causes alteration in already established representations. There are three types of continual learning to tackle forgetting: regularization-based, memory-based, and structure-based methods.

Here, the learning capacity is fixed [13, 27], and continual learning is performed by penalizing a network’s parameters. Researchers use a new regularizer for a novel method. In [27], the essential parameters are computed online. They keep track of how the loss function changes due to a specific parameter change and accumulate this information during training. There should be a weight importance process for parameters selection to prioritize parameters usage—the way elastic weight consolidation(EWC)[13] gives importance to parameters based on the Fisher information matrix. The usages of these methods are limited because they cannot perform well for a large number of tasks.

Methods in this category try to prevent forgetting by either storing [14, 15] or synthesizing data from previous classes. The first one needs memory for rehearsal, whereas the latter is a generative model like GAN[8] or VAE [12], or both synthesize data of previous tasks to perform pseudo-rehearsal. The number of examplers stored decreases with the increase in classes if the memory budget is limited. Researchers have recently proposed using a tiny memory[14] to store a few examples per class for old tasks.

The third approach to mitigate forgetting is structure-based methods[19]. The size of a network grows with each task to prevent catastrophic forgetting. Previous tasks’ performance is maintained by freezing the learned module while accommodating new tasks by augmenting the network with new modules. The computational cost for this method is inevitable if the number of tasks is high.

Recently, zero-shot learning(ZSL)[20, 5] has attracted a lot of attention because the model can distinguish unseen classes during testing. ZSL models are able to do so by transferring knowledge from seen to unseen levels through a semantic relationship between classes and their attributes. We can transform a ZSL problem into a supervised machine learning problem using generative models like GAN or VAE, or both. Once a generative model gets trained, it can synthesize data for unseen classes, and the data is useful for training a classifier like a conventional supervised problem. Another modification of ZSL is GZSL, a more practical approach, where data come from both seen and unseen classes.

Reference [7] proposed a continual zero-shot learning model called Generalized continual zero-shot learning(GCZSL), a single head CZSL where the task identity is revealed during training but not during testing. To mitigate forgetting, they used knowledge distillation and stored few actual data per class using a small episodic memory. The GCZSL and reference [22] used the single-head setting, where the task identity is disclosed during training but not at test time. There are a few works[3], [26] that used the multi-head setting. Reference [3] proposed an average gradient episodic memory(A-GEM) based CZSL, and [26] offered a generative model-based CZSL.

Adversarial learning has usages in many domains such as generative models[8], object composition[2], representation learning[16], domain adaptation[23], active learning[21] etc. In adversarial training, a model learns the parameters through the minimax game, where a module wants to maximize the cost function, and another wants to minimize the same. This paper shows shared play the minimax game with discriminator, where shared tries to minimize the loss function, and the discriminator wants to maximize.

Autoencoders can effectively learn feature space and representation[27, 2]. A variational Autoencoder(VAE) is a generative model that follows an encoder-latent vector-decode architecture of classical autoencoder, which places a prior distribution on the feature space and uses an expected lower bound to optimize the learned posterior. Conditional VAE is an extension of the VAE, where data are fed to network with class properties such as labels, attributes, etc. The VAE is a fundamental building block of our approach. Variational distribution aims to find a true conditional probability distribution over the latent variables z through minimizing their distance using a variational lower bound limit. The loss function for a VAE is:

Where the first term is the reconstruction loss, and the second one is the KL divergence between $q({z}|{x})$ and p(z). The encoder predicts $\mu$ and $\sum$ such that $q_{\phi}({z}|{x})=\mathcal{N}(\mu,\sum)$, from which a latent vector is synthesized via reparametrization process.

The final objective function of the model for the $t^{th}$ task is:

Where, $\lambda_{1},\lambda_{2},\lambda_{3},$ and $\lambda_{4}$ are regularizers to control the effect of each component. The full algorithm of the model is given in Algorithm [1].

Catastrophic forgetting occurs because of the imbalance between old and new classes that results in a bias of the network towards the newest ones. One insight of our approach is to decouple the single representation learned for all tasks continually into two parts: private and shared. Though knowledge is transferred for ZSL and GZSL mostly from the shared module from seen to unseen classes. The critical approach is experience replay that gets concatenated to the current task’s data during training of the model with the same task to avoid forgetting sequentially.

We evaluate our model on four benchmark datasets used for ZSL: Attribute Pascal and Yahoo(aPY)[20], Animals With Attributes(AWA1, AWA2)[20], and Caltech-UCSD-Birds 200-2011(CUB)[25]. Statistics of the datasets are presented in Table [I].

The dataset we use follows the setting used in[22]. It explains whether a class is seen or unseen is decided based on the number of tasks a model has been trained so far. If a model goes trained continually up to the $t^{th}$ task, the classes are assumed to be seen till the $t^{th}$ task, and the rest of the whole dataset’s classes are accepted unseen for the model while training.

We evaluate the resulting model on all previous tasks similar to[4, 15] after training for each new task. We use ACC as the average test classification accuracy across all classes for GZSL, seen classes, and unseen classes for GSL to measure our model’s performance. To measure forgetting, we calculate backward transfer, BWT that says how much learning new tasks have influenced previous tasks’ performance. While $BWT>0$ indicates catastrophic forgetting and $BWT<0$, learning new tasks has helped improve performance on previous tasks. We calculate forgetting measure for seen classes only.

mSA is the mean seen classification accuracy across all tasks.

mUA is the measure of zero-shot learning performance for the model.

mOA is the measure of generalized zero-shot learning performance.

mH is the hermonic mean classification accuracy.

Where $R_{j,i}$ is the test classification accuracy on task i after sequentially finishing learning the $j^{th}$ task.

The research on continual zero-shot learning(CZSL) has been less explored. References[22, 7] has investigated the work before on a single-head setting that we represent in this paper. Reference [7] used the following baselines, so we do the same in this paper.

• •

  AGEM + CZSL[22]: It is an average gradient episodic memory-based continual zero-shot learning. The authors of [22] have mentioned the harmonic mean of the CUB dataset only.

• •

  SEQ + CVAE[7]: The authors train CVAE sequentially without considering any continual learning strategy. After SEQ+CVAE is trained on the current task, synthetic are generated using noise and class embeddings for all classes to train a separate classifier.

We also compare our results with CZSL-CV+mof[7], CZSL-CV+rb[7], and CZSL-CV+res[7].

We use Pytorch as our framework. We train our model with a hundred epochs and a classifier for thirty epochs for each task on all datasets except CUB. We use the same number of epochs for the CUB dataset till task fifteen and then reduce to fifty for the model and ten for the classifier till task eighteen and again decrease to twenty for the model and five for the classifier. The Adam[11] optimizer is used in all experiments, and the learning rate for the classifier and others is 0.0001 and 0.001, respectively. We use weight decay 0.0001 as a regularizer for the classifier. We use 500 hidden units for both shared and private modules. Latent dimension is 50, and batch size for both model and classifier is 61. We take $\lambda_{1}=\lambda_{2}=\lambda_{3}=1$, and $\lambda_{4}=0.5.$

We divide the CUB dataset into 20 tasks, where each consists of ten classes. We compare our results with several methods in Table [II]. Generalized CZSL[7] used memory to store real data as the replay, and it achieved mH = 20.15. In contrast, our model achieves mH = 17.41 when each task gets trained for 100 epochs till the $15^{th}$ task, then from task $16^{th}$ to $18^{th}$ for 50 epochs and rest of the tasks for 20 epochs. Similarly, the classifier gets trained for 30 epochs till the task $15^{th}$, then from task $16^{th}$ to $18^{th}$ for 10 epochs and rest of the tasks for 5 epochs. The performance of our model per task is given in Figures [4, 4].

We divide the aPY dataset into 4 tasks, where each consists of eight classes. We compare our results with several methods in Table [III]. Generalized CZSL[7] used memory to store real data as the replay, and it achieved mH = 23.90. In contrast, our model achieves mH = 23.05 when each task gets trained for 100 epochs. Similarly, the classifier gets trained for 30 epochs for all tasks. The performance of our model per task is given in Figures [6, 6].

We divide the AWA1 dataset into 5 tasks, where each consists of ten classes. We compare our results with several methods in Table [IV]. Generalized CZSL[7] used memory replay to alleviate the catastrophic forgetting and used memory to store real data, and it achieved mH = 35.51. In contrast, our model achieves mH = 35.75 when each task gets trained for 100 epochs. Similarly, the classifier gets trained for 30 epochs for all tasks. The performance of our model per task is given in Figures [8, 8].

We divide the aPY dataset into 5 tasks, where each consists of ten classes. We compare our results with several methods in Table [V]. Generalized CZSL[7] used memory to store real data as the replay, and it achieved mH = 38.34. In contrast, our model achieves mH = 37.19 when each task gets trained for 100 epochs. Similarly, the classifier gets trained for 30 epochs for all tasks. The performance of our model per task is given in Figures [10, 10].