Learning Aligned Cross-Modal Representation
for Generalized Zero-Shot Classification

GZSL is a more realistic than ZSL because it not only learns information which can be adapted to unseen classes but also apply to the testing data from seen classes. The classical GZSL methods learn a mapping function and can be roughly divided into three categories.

One type of methods directly extracts visual features for mapping to semantic space. DAP and IAP (Lampert, Nickisch, and Harmeling 2009) establish the rigid correspondence between original visual features and attribute vectors for GZSL. The popular attention mechanisms (Zhu et al. 2020; Hou et al. 2021) also have been adopted to distinguish informative local visual features for GZSL, such as (Xie et al. 2019; Niu et al. 2019; Huynh and Elhamifar 2020; Liu et al. 2021; Ge et al. 2021). Considering the distribution consistency of semantic features in all categories, some methods aims to extract semantic features from textual descriptions for mapping to visual space. DeViSE (Frome et al. 2013) maps the visual features into word2vector embedding space learned by Skip-gram Language Model (Mikolov et al. 2013). Moreover, the generative models (Felix et al. 2018; Wan et al. 2019; Vyas, Venkateswara, and Panchanathan 2020; Feng and Zhao 2021; Yue et al. 2021) try to generate synthetic images of unseen classes for GZSL.

Due to the above methods often conduct mapping in a unidirectional manner, which tend to easily cause the domain shift problem. Therefore, the common-space based methods try to map different features into common spaces for connecting visual features with semantic features (Akata et al. 2015b, a; Sung et al. 2018; Keshari, Singh, and Vatsa 2020). CVAE (Mishra et al. 2018) utilizes conditional VAE to encode the concatenation feature of visual and semantic embedding vectors. OCD (Keshari, Singh, and Vatsa 2020) designs an over-complete distribution of seen and unseen classes to enhance class separability. Although these methods extract the effective features on the common space, they often neglect explicitly aligning different modalities.

Recent cross-alignment based methods of GZSL generally focus on establishing a shared latent space for visual and semantic features via generative adversarial network (GAN) (Goodfellow et al. 2014) or variational auto-encoder (VAE) (Kingma and Welling 2013). The pioneer GAN-based method, SJE (Akata et al. 2015b) learns a convolutional GAN framework to synthesize images from high-capacity text via minimax compatibility between image and text. LiGAN (Li et al. 2019) employs GAN to map features into synthetic sample space. f-VAEGAN-D2 (Xian et al. 2019) learns the marginal feature distribution of unlabeled images via a conditional generative model and an unconditional discriminator. cycle-GAN (Felix et al. 2018) and EPGN (Yu et al. 2020) strengthen the alignment of latent features in their respective modalities by cross-reconstruction. However, GAN-based models are difficult to optimize parameters during the training process.

VAE is an autoencoder for learning the conditional probability distribution from training data, which helps to generate samples having certain desired properties. CADA-VAE (Schönfeld et al. 2019) combines cross-modal and cross-reconstruction, and leverages two independent VAEs to align features via minimizing distance between the latent distributions of visual and semantic features. Based on CADA-VAE, Li et al. (Li et al. 2020) learns modality-invariant latent representations by maximizing mutual information and entropy on latent space. DE-VAE (Ma and Hu 2020) adopts a deep embedding model to learn the mapping from the semantic space to the visual feature space. Disentangled-VAE (Li et al. 2021) decouples the latent features by shuffling classification-irrelevant information to obtain discriminative representations. Although the VAE-based methods achieve promising performance, the aligned relationships across different modalities tend to be confused on the common embedding space without structural restriction since the lack of fine-grained instance-wise annotations.

Suppose that $X$$=$$\{X^{S},\ X^{U}\}$ denotes the visual space of images, $A$$=$$\{A^{S},\ A^{U}\}$ denotes the semantic space of attributes, and $Y$$=$$\{Y^{S},\ Y^{U}\}$ denotes the corresponding label set. Notably, $Y^{S}$ and $Y^{U}$ are disjoint, i.e., $Y\cap Y^{U}$$=$$\emptyset$. Consequently, we can individually collect training dataset $D_{s}$$=$$\{x_{i}^{s},a_{i}^{s},y_{i}^{s}\}_{i=1}^{N}$ and testing dataset $D_{u}$$=$$\{x_{i}^{u},a_{i}^{u},y_{i}^{u}\}_{i=1}^{M}$, where $x_{i}^{s},x_{i}^{u}\in X$ is the $i$-th visual feature, $a_{i}^{s},a_{i}^{u}\in A$ is the $i$-th semantic feature, and $y_{i}^{s},y_{i}^{u}\in Y$ is their corresponding labels, of seen/unseen classes, respectively. The task of ZSL aims to learn a classifier $f_{ZSL}:X\rightarrow Y^{U}$ for recognizing a testing instance $x$ of unseen classes. For adapting to both seen and unseen classes, GZSL adopts a more realistic setting and learn a classifier $f_{GZSL}:X\rightarrow Y^{U}\cup Y^{S}$.

Let $Z_{x},Z_{a}$ respectively denote the feature vectors of visual and semantic modalities, the joint distribution $P(Z_{x},Z_{a})$ can assess the degree of cross-modal alignment. For example, if two modalities are completely aligned $P(Z_{x},Z_{a})=$ 1 , otherwise $0\leq P(Z_{x},Z_{a})<1$. Optimizing cross-modal alignment can be mathematically formulated as:

		$\displaystyle 2\log p(z_{x},z_{a})\geq\log p(z_{x}|z_{a})+\log p(z_{a}|z_{x})+\mathcal{H}(X)$		(1)
		$\displaystyle+\mathbb{E}_{p(x)}[\log p(z_{x},X)]+\mathbb{E}_{p(a)}[\log p(z_{a},A)]+\mathcal{H}(A)$

where $\mathbb{E}(\cdot)$ represents expectation, $\mathcal{H}(\cdot)$ represents entropy.

For conducting cross-modal alignment, we can train a deep learning model for mapping image features and attributes into a common space. According to the optimization objective (Equation 1), the common space can be utilized to align the features by the first two items, the features on the common space can contain discriminative information corresponding inputs by the middle two items, and the features can be utilized to reconstruct original modalities by the last two items. Consequently, we summarize Equation 1 into three parts. The first two log-likelihood items $\log p(\cdot)$ can be realized by a mutual alignment constraint $C_{MA}$, the middle two expectation items $\mathbb{E}$ can be realized by a representation constraint $C_{Rep}$, and the last two entropy items $\mathcal{H}$ can be realized by a reconstruction constraint $C_{Rec}$. Hence, the total optimization objective can be simplified as:

$$\mathcal{O}=C_{MA}+C_{Rep}+C_{Rec}$$

(2)

Variational Auto-Encoder. Since the excellent ability of fitting data distribution, VAE is widely used in cross modal alignment. We employ two independent VAEs (shown in Figure 2) composed of multilayer perceptron (MLP) to respectively establish visual and semantic modality. In VAE, the encoders are used to generate the latent variables of each modality, and the decoders are used to reconstruct the latent vectors. Therefore, we can estimate the reconstruction constraint $C_{Rec}$ with VAE loss, which can be formulated as:

	$\displaystyle\mathcal{L}_{Rec}$	$\displaystyle=\mathbb{E}_{q_{\phi}(z_{x}|x)}[\log p(x|z_{x})]-\alpha D_{KL}(q_{\phi}(z_{x}|x)\|p_{\theta}(z_{x}))$		(3)
		$\displaystyle+\mathbb{E}_{q_{\phi}(z_{a}|a)}[\log p(a|z_{a})]-\alpha D_{KL}(q_{\phi}(z_{a}|a)\|p_{\theta}(z_{a}))$

where $q_{\phi}(z_{x}|x)$ denotes latent feature distributions of visual modality, $q_{\phi}(z_{a}|a)$ denotes latent feature distributions of semantic modality, $p(x|z_{x})$ and $p(a|z_{a})$ denote the observation distributions of corresponding modalities, $\alpha$ weights the KL Divergence. The priors $p_{\theta}(z_{x})$ and $p_{\theta}(z_{a})$ are both standard Gaussian distributions.

Because VAE models the distribution of hidden variables, we can transform the mutual alignment constraint $C_{MA}$ of maximizing log likelihood into minimizing the Wasserstein distance between the latent Gaussian distributions of semantic modality and visual modality. Hence, the mutual alignment constraint can be formulated as:

$$\mathcal{L}_{MA}=\left(\|\mu_{x}-\mu_{a}\|_{2}^{2}+\left\|\Sigma_{x}^{\frac{1}{2}}-\Sigma_{a}^{\frac{1}{2}}\right\|_{\text{Frobenius }}^{2}\right)^{\frac{1}{2}}$$

(4)

where $\mu_{x}$ and $\Sigma_{x}$ are all predicted by visual encoder, while $\mu_{a}$ and $\Sigma_{a}$ are all predicted by semantic encoder.

Information Enhancement Module. Because of the “latent variable collapse” problem (Alemi et al. 2018), the distribution of latent vector $p(Z|I)$ generated by VAE can easily degrade into a standard normal distribution $\mathcal{N}(0,1)$, which is detrimental to learn discriminative representations of latent variables for classification. From this key observation, we propose a novel Information Enhancement Module (IEM, shown in Figure 3) to magnify the joint probability between observation $I$ and the inferred latent variables $Z$.

According to (Chen et al. 2019), we randomly shuffle the features of encoder to extract discriminative features. For obtaining the joint probability, we develop a MLP with one hidden layer to learn the marginal probability of $I$, and multiply it with the conditional probability of $Z$. Mathematically, our IEM can be formulated as:

	$\displaystyle p(z_{x},X)$	$\displaystyle=p(E(X;\varphi_{x})\mid X)p(\varPsi(X;\psi_{x}))$		(5)
	$\displaystyle p(z_{a},A)$	$\displaystyle=p(E(A;\varphi_{a})\mid A)p(\varPsi(A;\psi_{a}))$

where $p(z_{x},X)$, $p(z_{a},A)$ denote latent variable distributions with enhanced mutual information of visual modality and semantic modality; $E(X;\varphi_{x}),E(A;\varphi_{a})$ respectively denote visual modality encoder and semantic modality encoder; $\varPsi(X;\psi_{x}),\varPsi(A;\psi_{a})$ are marginal probability perceptrons of corresponding modalities.

When optimizing the vanilla VAE with joint probability, KL divergence can be re-defined as:

	$\displaystyle D_{KL}$	$\displaystyle=\int q_{\phi}(z,i)\log\frac{q_{\phi}(z,i)}{p_{\theta}(z)}dz$		(6)
		$\displaystyle=q_{\phi}(i)D_{KL}(q_{\phi}(z|i)\|p_{\theta}(z))+q_{\phi}(i)\log q_{\phi}(i)$

According to Equation 3, the latent variable will collapse with the decrease of $\mathcal{L}_{Rec}$. Then, $D_{KL}(q_{\phi}(z|i)\|p_{\theta}(z))$ will be approximate to 0. But with $q_{\phi}(i)\log q_{\phi}(i)$ containing distribution of observation $i$ in re-defined $D_{KL}$, $z$ will not degrade to $\mathcal{N}(0,1)$ and tend to contain more effective information about true data distribution.

Vision-Semantic Alignment. Vision-Semantic Alignment (VSA, shown in Figure 4) is proposed to align the multi-modal latent variables $Z$ after information enhancement. Since the lack of fine-grained instance-wise annotation, images of each class share a pre-class attribute vector, the alignment between different modalities tend to be confused on the latent space without any structural restriction. To tackle this issue, we further constrain the latent features on latent subspace guided by a learned classifier to strengthen the alignment. We introduce classification information $Y$ into the optimization function and maximize the joint distribution $p(Z,I,Y)$. Then the optimization function can be obtained by replacing $p(z_{x},X),p(z_{a},A)$ in the middle two items of Equation 1 with $p(z_{x},X,Y)$, $p(z_{a},A,Y)$. Then the two expectation items of Equation 1 can be respectively formulated as:

	$\displaystyle\mathbb{E}$	$\displaystyle{}_{p(x)}[\log p(z_{x},X)]=$		(7)
		$\displaystyle\mathbb{E}_{p(y)}\mathbb{E}_{p(x)}[\log p(Y|z_{x},X)]+\mathbb{E}_{p(y)}\log p(z_{x},X)$
	$\displaystyle\mathbb{E}$	$\displaystyle{}_{p(a)}[\log p(z_{a},A)]=$
		$\displaystyle\mathbb{E}_{p(y)}\mathbb{E}_{p(a)}[\log p(Y|z_{a},A)]+\mathbb{E}_{p(y)}\log p(z_{a},A)$

where $p(Y|z_{x},X),p(Y|z_{a},A)$ denote classification probability of corresponding modalities in latent space, the last $\mathbb{E}(\cdot)$ represent joint expectation. This formula highlights that maximizing the optimization objective is equivalent to maximize the conditional probability $p(Y|z_{x},X)$ and $p(Y|z_{a},A)$ and to maximize the expectation of the joint distribution $\mathbb{E}(\cdot).$ Hence, $\mathbb{E}_{p(y)}\mathbb{E}_{p(x)}[\cdot]$ item can be approximated by a classification loss based on latent variables, $\mathbb{E}(\cdot)$ item can be approximated by a reconstruction loss based on latent variables after IEM. In this way, we can constrain the generation of latent variables by the classifier to improve its discriminability.

By replacing the expectation items in representation constraint $C_{Rep}$ with latent variable representation after information enhancement (shown in Equation 7). The new $C_{Rep}$ are implemented by IEMs and VSA modules, and the entropy items as reconstruction are merged into VAE. Hence we can denote the representation loss as:

	$\displaystyle\mathcal{L}_{Rep}=$	$\displaystyle\frac{1}{N}\sum_{i=1}^{N}L_{cls}(y_{i}^{s},f(z_{i}^{x};W_{x}))+$		(8)
		$\displaystyle\frac{1}{N}\sum_{i=1}^{N}L_{cls}(y_{i}^{s},f(z_{i}^{a};W_{a}))$

where $z_{i}^{x}$, $z_{i}^{a}$, $y_{i}^{s}$ are visual latent variable, semantic variable, and label in training set, $L_{cls}$ is the classification loss, $f$ is the classifier composed of two independent block.

Loss function. The overall loss of our method can be formulated as:

$$\mathcal{L}=\mathcal{L}_{Rec}+\beta\mathcal{L}_{MA}+\lambda\mathcal{L}_{Rep}$$

(9)

where $\mathcal{L}_{Rec}$ is the reconstruction loss of intra-modalities, $\mathcal{L}_{MA}$ is the mutual alignment loss of inter-modalities, and $\mathcal{L}_{Rep}$ is the classification loss of latent representations. $\beta$ and $\lambda$ are two hyper-parameters training by an annealing schedule (Bowman et al. 2016).

We conduct experiments on four benchmark datasets: CUB (Wah et al. 2011), SUN (Patterson and Hays 2012), AwA1 (Lampert, Nickisch, and Harmeling 2009), and AwA2 (Xian et al. 2018a). The detailed information of the datasets is summarized in Table 1. Specifically, CUB is a fine-grained categorized dataset collected from professional bird websites, which covers 200 categories and each category has a 312-dimensional attribute vector. SUN is a fine-grained scene understanding dataset, which is widely used in objective detection, image classification, and so on. In SUN, images are annotated with 102-dimensional attribute. AwA1 is an animal image dataset that uses 85-dimensional attribute to describe 50 categories. AwA2 is a more sophisticated variant of AwA1, which collects 37,322 images from public web sources. AwA1 and AwA2 have no overlapping images.

For a fair comparison, we adopt the setting as in (Xian et al. 2018a) for training and testing. The evaluation metrics in generalized datasets setting include the average classification accuracy (ACA) of the samples from seen/unseen classes, the harmonic mean of $ACA^{S}$ and $ACA^{U}$, details are listed as followings:

• •

  $ACA^{U}$: the average accuracy of per-class on test images from unseen classes, which represents the capacity of classifying unseen classes samples.

• •

  $ACA^{S}$: the average accuracy of per-class on test images from seen classes, which is used to represent the capacity of classifying seen classes samples.

• •

  $H$: the harmonic mean value, which is formulated as

  $$H=\frac{2\times ACA^{U}\times ACA^{S}}{ACA^{U}+ACA^{S}}$$

  (10)

Following the setting in other methods (Xian et al. 2018a; Schönfeld et al. 2019), ResNet-101 pre-trained on ImageNet is used to extract the 2048 dimensional visual features. In IEM, the hidden layer of MLP has 99 units. We respectively utilize 1560, 1680 hidden units for visual encoder and decoder, and 1450, 665 hidden units for semantic encoder and decoder. The size of our aligned cross-modal representation in latent space is 64 for all datasets. The final softmax classifier includes a fully connected layer and a non-linear activation layer. The input size of the fully connected layer is 64, and the output size is equal to the total number of all categories. Our approach is implemented with PyTorch 1.5.0 and trained for 100 epochs by the Adam optimizer (Kingma and Ba 2015).We set learning rate as 1.5e-04 for training VAEs, 3.3e-05 for training IEMs, 7.4e-03 for training VSA, 0.5e-03 for training softmax classifier. For all datasets, the batch size of ACMR is set to 50 and the batch size of final softmax classifier is set to 32.

Baseline models. We compare our model with 15 state-of-the-art (SOTA) models. Among them, SP-AEN (Chen et al. 2018), AREN (Xie et al. 2019), DAZLE (Huynh and Elhamifar 2020) are deep embedding based methods, cycle-CLSWGAN (Felix et al. 2018), f-CLSWGAN (Xian et al. 2018b), LiGAN (Li et al. 2019), f-VAEGAN-D2 (Xian et al. 2019), DASCN (Ni, Zhang, and Xie 2019), LsrGAN (Vyas, Venkateswara, and Panchanathan 2020), E-PGN (Yu et al. 2020) are GAN based methods, and SE (Verma et al. 2018), CVAE (Mishra et al. 2018), CADA-VAE (Schönfeld et al. 2019), OCD (Keshari, Singh, and Vatsa 2020), DE-VAE (Ma and Hu 2020), Disentangled-VAE (Li et al. 2021) are VAE-based methods.

The detailed results are listed in Table 2. In our method, the value of $H$ can reach 55.3% on CUB, 43.8% on SUN, 67.5% on AwA1, and 68.7% on AwA2, respectively. Specifically, our method respectively outperforms the original benchmark CADA-VAE by 2.9% on CUB, by 3.2% on SUN, by 3.4% on AwA1, and by 4.8% on AwA2 in terms of $H$. Compared with deep embedding based methods, our ACMR outperforms DAZLE on three datasets. Especially, our ACMR significantly increases $H$ by 18% and 29.5% on SUN and AwA2, respectively. Moreover, our method outperforms APN (Xu et al. 2020) by 3.1% (H) on AwA2, by 0.1% (H) on SUN, and outperforms RGEN (Xie et al. 2020) by 7% (H) on SUN. Compared with GAN-based methods, our ACMR outperforms LsrGAN by 2.3% on CUB, by 2.9% on SUN, by 4.5% on AwA1, and by 5.7% on AwA2 in terms of $H$, respectively. We can respectively increase $H$ by 2.8% on CUB, by 2.2% on AwA1, and by 7.7% on AwA2, compared with E-PGN whose $H$ are 52.5%, 65.3% and 61% corresponding datasets. Compared with VAE-based methods, our ACMR outperforms Disentangled-VAE by 0.9% on CUB, by 2.4% on SUN, by 1.3% on AwA1, and by 2.1% on AwA2 in terms of $H$, respectively. Compared with DE-VAE, we can increase $H$ from 54.3% to 55.3% on CUB, from 40.9% to 43.8% on SUN, from 66.9% to 67.5% on AwA1 and from 67.4% to 68.7% on AwA2. Obviously, our method achieves SOTA performance and outperform other common space based methods with a great margin.

In this section, we conduct ablation study to evaluate the effectiveness and necessity of our proposed components, including backbone, IEM, VSA, and the hyper-parameters of weighting loss.

About the IEM and VSA. We present the results under different combinations of our proposed modules in ACMR, including Dual-VAE (only two VAEs), IE-VAE (two VAEs and two IEMs), VSA-VAE (two VAEs and VSA module) and our original ACMR. The detailed results are listed in Table 3.

From Table 3, we can see that VSA-VAE outperforms Dual-VAE by 21.4% on CUB, by 5.2% on SUN, by 13.3% on AwA1, and by 1.8% on AwA2 in terms of $H$. It means that discriminative representations can be learned by aligning cross modal features guided by classifier. In addition, compared with Dual-VAE, IE-VAE increases $H$ from 32.7% to 36.1% on CUB, from 38.3% to 39.0% on SUN, from 52.4% to 65.1% on AwA1, and from 66.1% to 66.6% on AwA2, respectively. Compared with VAS-VAE, ACMR increases $H$ from 54.1% to 55.2% on CUB, from 43.5% to 43.8% on SUN, from 65.7% to 67.3% on AwA1, and from 67.9% to 68.7% on AwA2, respectively. It means that the information enhancement module can effectively strengthen the discriminative ability of latent variables.

Inference about weighting coefficient $\alpha$, $\beta$ and $\lambda$. These three hyper-parameters are the weighting coefficient of our loss. In this experiment, we vary $\alpha$ from 1 to 5, $\beta$ from 1 to 12 for all datasets, and $\lambda$ from 285 to 305 for CUB, SUN, AwA1, from 70 to 90 for AwA2, the detailed results are shown in Figure 5. From $\alpha$-curve of all datasets, we can see that $H$ generally keeps rising along with the increase of $\alpha$. From $\beta$-curve of all datasets, we can see that $H$ rises smoothly on CUB, SUN, and fluctuates greatly on AwA1, AwA2 with the increase of $\beta$. From $\lambda$-curve of all datasets, we can see that performance fluctuates greatly with the increase of classification weighting coefficient $\lambda$, we can conclude that extracting classification subspace features is more helpful to GZSC than extracting common space features.

Visualization. According to the method of qualitatively measuring the extent of two domains (Saito et al. 2019), we leverage t-SNE (Van der Maaten and Hinton 2008) to visualize the features distribution of visual modality and semantic modality in the latent space on CUB and SUN, to visually show that ACMR our proposed can learn well aligned cross-modal representation, as shown in Figure 6. We extract the latent features of seen and unseen classes on testing datasets by the trained ACMR respectively, and use blue dots to represent the features of the visual modality, and yellow dots to represent the features of the semantic modality. From Seen Classes column of CUB and SUN, we can observe that visual features and semantic features overlap almost completely with each other. From Unseen Classes column, although some semantic features have slightly shifted, we can observe that features of two modalities also coincide well. Hence, it verifies the effectiveness of our proposed ACMR.