Entropy-Based Uncertainty Calibration for Generalized Zero-Shot Learning

Recent state-of-the-art approaches to GZSL with generative models have achieved promising performance. Generative models can synthesise an unlimited number of fake features from side information on the novel classes, e.g., semantic attributes. With these synthesised features, ZSL problems become a relatively straightforward supervised classification task. The two most commonly used generative models are generative adversarial networks (GANs) [24] and variational autoencoders (VAEs) [25]. Often, both models are jointly used to form generative architectures for ZSL tasks. GAZSL [18] leverages Wasserstein GANs (WGAN) [26] to synthesise vivid visual features. Currently, GAZSL models are the state-of-the-art for ZSL tasks with noisy text. CANZSL [chen2019canzsl] is a cycle-consistent generative framework for solving domain shift problems and enhancing denoising operations on natural language input. SAE [21] is a semantic autoencoder that learns mappings between the semantic space and the visual space for the encoder to use to translate visual features into the semantic space. The decoder can then synthesise unlimited visual features from the semantic information. GDAN [20] takes a different approach with a novel structure consisting of a conditional variational autoencoder (CVAE) and a GAN that generate various visual features from class feature embeddings as input. SE-ZSL [19] adopts an autoencoder followed by an attribute regressor to train a model with three alignments: visual-to-attribute, attribute-to-visual and visual-to-attribute. The latter two feedback alignments help to improve the effectiveness of the decoder, which works as a visual feature generator.

However, in these methods, the feature generation between the semantic and visual spaces is constrained by asymmetric information. For example, suppose that the wing colour in a bird image is not annotated, so the visual features generated from the semantic labels do not include this information. However, the visual feature extractor can and does extract the wing colour. The two visual representations are now asymmetric, so training a classifier from the generated visual features but testing with extracted visual features may result in sub-optimal performance.

With this motivation, our framework incorporates two variational autoencoders with triplet metric regulation to learn latent features that best discriminate between the visual and semantic spaces.

One strategy for mitigating the distribution overlap issue is to simply distinguish between seen and unseen classes [27, 28, 29]. This strategy is a challenging yet rewarding approach in transfer learning tasks, such as GZSL, few-shot learning and domain adaptation. Gidaris and Komodakis [28] proposed a few-shot classification weight generator in a recent work to differentiate seen from unseen classes. LisGAN [29] presented a technique for GZSL to leverage unseen samples given a high level of confidence that the samples had been correctly classified. The high-confidence unseen samples are then regarded as references for future classification. OSBP [30], an approach to domain adaptation, trains the feature generator on representations so as to separate unknown target samples from known ones.

Another strategy to balance performance between seen and unseen classes is to apply a softmax probability calibration [31, 13, 32]. From a realization that performance with unseen class in GZSL was unsatisfactory, Chao et al. [13] proposed a shifted-calibration approach to improve performance. DCN [31] incorporates temperature calibration to enable simultaneous calibration of deep networks based on confidence in the source classes and uncertainty in the target classes. Tuning the temperature for softmax layer is conducted through knowledge distillation [33], which produces softer probability distributions over the classes. Temperature calibration in DCN, however, has another exciting purpose, which is to mitigate the uncertainty for unseen classes in GZSL. RAC [32] involves scaled multiplicative calibration of the classification scores of the seen classes, which changes the effective variance of the seen categories.

Unlike the methods above that directly calibrate the probability distribution of the seen classes, our framework is based on a probability entropy over the seen classes as a measure of the confidence that a class is seen.

When used in conjunction with distance or similarity-based techniques, such as nearest-neighbour approaches, the promise of metric learning methods becomes very clear. Learning an effective distance metric among visual pairs is a significant paradigm in visual analysis, with a number of recent studies in this area [34, 35, 36]. The two commonly used networks for metric learning are the Siamese network and the triplet network. Inspired by DeepRanking [37], triplet networks [38] have been shown to learn better representations than Siamese networks. Hence, triplet networks have become widely used in distance metric learning tasks, like image retrieval and face verification [34, 35, 36]. FaceNet [36] uses triplets of roughly aligned matching and non-matching face pairs selected with an online triplet mining approach. The result is state-of-the-art face recognition performance. DAML [35] generates hard negative samples from observed samples in an adversarial learning manner for both image retrieval and clustering. Weifeng Ge [34] developed a novel hierarchical triplet loss (HTL) for image retrieval tasks that collects informative training samples with the guide of a global class-level hierarchical tree with favourable performance over vanilla triplet loss. We have also taken advantage of triplet loss in metric learning and incorporated it into our generative model to synthesise discriminative latent space features.

Metric learning has also been seen in recent approaches to ZSL. For example, GAZSL [18] involves a technique called visual pivot to force the generated visual features to approximate the cluster centres in each class, which means nearest-neighbour approaches perform quite well. However, the fixed visual pivots are only able to pull samples with the same label into nearby points; they lack the ability to push samples of different classes apart. GDAN [20] uses a regressor in additional to a generative model to learn a flexible distance with which to evaluate the closeness of the visual features and class embeddings. Experiments show, however, that the model performs extremely well with seen classes, but unseen objects are prone to bias toward seen categories. Conversely, our metric learning regulation is explicitly applied to the latent feature space to achieve good discrimination.

Let $\mathcal{T}$ be the set of all possible triplets $\{\tau_{a},\tau_{p},\tau_{n}\}$ from the seen classes $S$, where $\tau_{a}$ and $\tau_{p}$ are from the same category, and $\tau_{n}$ is a negative sample randomly chosen from a different category.

Each $\tau$ contains a triplet $(x,s,y)$, where $x$ denotes the visual features extracted from an image, $s$ represents the corresponding semantic attributes and $y$ is the associated one-hot class label, respectively. During the evaluation phase, visual features $x^{u}$ and the corresponding semantic representations $s^{u}$ are drawn from the unseen classes $U$. In ZSL, the aim is to predict the class label $y^{u}$. By comparison, the aim with GZSL is to predict the class label for the image $x$ from both $S$ and $U$.

The basic module in our framework is a variational autoencoder (VAE) [25] that includes an encoder, a decoder and an objective function. The VAE models the probabilistic generation of data, and the encoder and decoder networks are probabilistic. Specifically, we can compute the mean $\mu$ and the diagonal covariance $\Sigma$ of the latent conditional distribution $z$ from $x$ with the encoder Q. The latent data $z$ is then sampled from $\mathcal{N}(\mu_{z|x},\Sigma_{z|x})$. Further, the reconstructed or generated data is synthesised by the decoder $P(x|z)$. The objective function of a VAE has a tractable lower bound. Therefore, we can optimize the gradient of this bound with the following formula:

where the first term is the reconstruction loss of the input data, and the second term is the Kullback-Leibler (KL) divergence between the encoder model and the simple Gaussian prior. Here, the functions $Q_{\phi}$ and $P_{\theta}$ are the encoder and decoder networks with the corresponding weights $\phi$ and $\theta$, respectively.

As shown in Figure 2, the synthesised features are regularised in the latent space according to the labels by applying a triplet loss. The latent space is formed between the visual and semantic spaces through a dual VAE framework, which includes a visual variational autoencoder $v$-VAE and a semantic variational autoencoder $s$-VAE. In terms of the implementation of the dual VAEs, we followed the CADA-VAE approach set out in [39].

The conventional role of VAEs as a generative model leverages the decoder as the generator. By contrast, both the encoders in our dual generative framework work as generators to synthesise embedding vectors in the latent space, while the decoders translate the latent vectors back into visual or semantic features. In other words, the VAE has a cycle structure, providing supervision information to improve the generation ability of the encoders.

The VAE loss for $v$-VAE and $s$-VAE can be formulated as:

where $Q_{v}$, $Q_{s}$ denote the encoder networks in $v$-VAE and $s$-VAE respectively, and $P_{v}$, $P_{s}$ represents decoder networks. The first terms are the reconstruction loss and the second ones are the KL divergence. $\beta_{1}$ and $\beta_{2}$ are the coefficients of KL divergence for the two VAEs, respectively. $z_{v}$ and $z_{s}$ are the latent vectors synthesised from the visual and semantic spaces, respectively. In more detail, the Gaussian distributions $\mathcal{N}_{v}(\mu_{x},\Sigma_{x})$, $\mathcal{N}_{s}(\mu_{s},\Sigma_{s})$ are the direct output of $Q_{v}$ and $Q_{s}$. $z_{v}$ and $z_{s}$ sampled with a reparametrization trick [25] from the two Gaussian distributions.

To make the latent space and intermediate region between the visual and semantic spaces, we use a multi-distribution loss to allow the two Gaussian distributions to approximate each other. In our experiments, we found that a Wasserstein distance gave better performance than other distance functions, such as mean squared error (MSE). Hence, the cross-modal loss is formulated as follows:

Further, the latent vectors $z_{v}$ synthesised from the visual space can be decoded into semantic embeddings $\hat{s}\leftarrow P_{s}(z_{v})$ and vice versa for the latent vectors $z_{s}$ encoded from the semantic space, $\hat{v}\leftarrow P_{v}(z_{s})$. The objective function for optimizing the multi-modal reconstruction is given below:

where we use an L1 norm because is it insensitive to outliers. With an L2 norm, too many outliers in the dataset will degrade performance.

To regularize the latent features for optimal discriminativeness, we have incorporated a triplet loss into the method in addition to the above constraints. As shown in Figure 2, a triplet loss is enforced in the latent space to ensure a small distance between all objects of the same category and a large pairwise distance between objects different categories. Training is performed in batches, with selected triplet batches, including an anchor batch, a positive batch and a negative batch. The samples in the positive batch have the same corresponding labels as the anchor batch, whereas the negative batch samples all have different random labels. Formally, the triplet loss can be written as follows:

where $x_{a}$, $x_{p}$ and $x_{n}$ are the anchor images, the positive samples and the negative samples, respectively. $\alpha$ represents the margin between the positive and negative pairs, which varies from dataset to dataset. Similar triplet losses $\mathcal{L}_{s\text{-}triplet}$ with the same $\alpha$ are also applied to the latent features $z_{s}$ generated from the semantic embeddings.

To further improve multi-modal reconstruction and regularize the margin between classes, we propose the following multi-modal triplet objective function:

where i, j and m represent the modality M, which can be either visual or semantic. Note that i, j and m cannot be the same modality. This objective function specifies six more triplet loss terms in the latent space.

The superiority of the soft margin in triplet loss over fixed visual centres (aka. visual pivots) as proposed in GAZSL[18] is worth discussing. A visual centre is simply calculated by the mean value of all image features in the same class, and the synthesised visual features from the same class are pushed towards the corresponding visual centre. However, while optimizing the distance between the visual data points, the model may converge into a situation where the samples from different categories overlap. The triplet loss in our framework not only pulls the samples of the same category towards each other, it also pushes negative category objects backwards.

The overall objective function for the proposed approach GML is as follows:

where $\lambda$ denotes the weight of Wasserstein distance.

To mitigate catastrophic distribution overlap issues, we further develop a novel entropy-based uncertainty calibration technique to predict the labels of test samples. Once the dual VAE model has been trained, arbitrary instances can be generated in the latent space based on semantic embeddings of the classes. In GZSL, a softmax classifier is trained over both the seen and unseen classes with 200 latent representations for each seen class. For datasets that do not have enough samples per class, the available instances are simply repeated until 200 samples have been trained. For example, if only 50 samples were available, the training set would consist of four duplicates of the 50 samples. Plus, there are fewer unseen categories in the dataset than seen ones. To avoid an imbalance of knowledge to be learned between the seen and unseen classes, more samples of the unseen classes can be synthesised, say 400. Note that the general classifier training data of seen classes are synthesised by the visual encoder $Q_{v}$, whereas the latent features for the novel classes are all synthesised from semantic embeddings.

There are three networks involved in predicting test samples: a general classifier $f$, a seen classifier $g$ and a visual encoder $Q_{V}$, as shown in Figure 3. Specifically, the general classifier is a softmax classifier that recognizes both seen and unseen samples. Latent features from seen classes and unseen classes need to be provided to train this general classifier. The seen latent features are mapped from the visual samples in the training set by the visual encoder $Q_{v}$. Unlimited novel latent features for the unseen classes can be synthesised with the trained semantic encoder $Q_{s}$ given class embeddings. To predict the labels in conventional GZSL, latent representations are first synthesised by the visual encoder given test the visual feature of the sample. And, once the general classifier is trained, recognizing which class a sample belongs is a straightforward task. When classifying objects into seen classes, almost all unseen objects will have high entropy, whereas approximately half of the seen samples will have lower entropy than the unseen ones. Therefore, entropy is calculated with unnormalized log probabilities from the softmax output. The insight here is that our latent feature generator is trained on the seen classes so whether an object is seen or unseen is more certain as determined by the probability entropy.

Nevertheless, this approach does not prevent overlaps between the seen domain and the unseen domain in the same latent space and, in turn, performance degradation. Therefore, we incorporated an entropy threshold to judge whether an object is likely to be seen; thus, reducing the risk of the general classifier classifying an object into the wrong unseen class. After excluding the uncertain objects, a simple softmax classifier is trained in the seen domain to recognize the low entropy objects. Note that, instead of latent representations, this classifier is trained with visual features, which result in greater than 90% accuracy with seen objects in the testing stage. However, a side effect of the proposed uncertainty calibrator is that performance with unseen classes can degrade slightly since a small portion of novel class samples are improperly classified into seen classes. Nevertheless, this tiny sacrifice of accuracy in the unseen domain is more than offset by the substantial improvement in accuracy in the seen domain. A more detailed comparison of merit with and without the uncertainty calibrator is provided in the ablation study.

To demonstrate the robustness of our method, we conduct experiments on six benchmark ZSL datasets, including three coarse-grained datasets (aPaY [40], AWA1[41], AWA2[42]) and three fine-grained medium-sized datasets (CUB[43], SUN[44] and FLO[45]). The dataset statistics are reported in Table 1. The image features and class embeddings used are provided by [42] and are publicly available. Caltech-UCSD Birds-200-2011 (CUB) consists of 11,788 images from 200 different bird species annotated with 312 attributes. We follow previous zero-shot split in this dataset with 150 classes as seen and 50 as novel classes.

Attribute Pascal and Yahoo (aPY) consists of 18,627 images from the 42 classes and annotated with 64 attributes. It combines datasets a-Pascal and a-Yahoo together, with 30 and 12 classes respectively. Images in a-Yahoo dataset is selected from the Yahoo image search engine to supplement a-Pascal dataset, and the chosen objects are very similar to a-Pascal, e.g. ”wolf” vs. ”dog”.

Animals with Attributes1 (AWA1) is a larger course-grained datasets than aPY in terms of the scale of image number, with 30,475 images of 50 animal species. Each species in the dataset is annotated with 85 attributes, and there are 40 species categorized as seen classes, and 10 as unseen classes.

Animals with Attributes1 (AWA2) collected 37,322 images for the 50 classes of AWA1 dataset from public web sources, e.g. Flickr, Wikipedia, etc. The split scheme and attribute dimension of AWA2 is same as AWA1.

SUN attributes (SUN) is built on top of SUN categorical database to ensure diversity of categories and study the relationship between attribute-based and category-based representations. There are 14,340 images spans 717 categories, in which 645 are split as seen categories and 72 as novel classes. Each category is annotated with 102 attributes.

Oxford Flowers (FLO) includes 8,189 images from 102 flowers categories. The image number of each class is not fixed, ranging from 40 to 258. Also, the images vary in pose, scale and light variation. Since there is no annotated attributes availble for this dataset, we use 2048-dimensional features extracted from semantic descriptions with the RNN model [46].

The dataset statistics is reported in Table 1. We use publicly available image features and class embeddings provided by [42].

We evaluated performance against representative methods proposed over the last few years as well as recent state-of-the-art frameworks. These include DAP [41], CONSE [47], SSE [48], SJE [49], LATEM [50], ESZSL [51], ALE [52], SYNC [53], SAE [21], DEM [54], GAZSL [18], GDAN [20], CADA-VAE [39]. Of these methods, GAZSL and GDAN learn to synthesise artificial visual representations, then convert the ZSL task into a conventional supervised classification problem. CADA-VAE uses a common embedding space between the visual and semantic modalities to conduct supervised classification. The generative architecture of this model is very similar to our own but does not include the metric learning technique or the uncertainty calibrator mechanism. The classic methods DAP and CONSE first predict the attributes of unseen objects, then infer the attribute labels according to similarity. SSE, SJE, LATEM, ESZSL and ALE all adopt a linear compatibility function or some form of similarity-based metric to compare the embedded visual and semantic features. SAE uses a semantic autoencoder to learn two mappings from the semantic to the visual space, and vice versa from the visual to the semantic space. DAP, ESZSL and SAE perform well on conventional ZSL problems but degrade dramatically when migrating to the GZSL paradigm.

Our framework is implemented with Pytorch¹¹1http://pytorch.org. The encoders and decoders are implementated as two-layer fully-connected (FC) networks. The visual, semantic encoders and visual, semantic decoders have 1560, 1450, 1660 and 665 hidden units, respectively. The dimensionality of the latent space is set as 64. All experiments are conducted on a server with 16 Intel(R) Xeon(R) Gold 5122 CPUs and 2 GeForce RTX 2080 Ti GPUs

The metric used to evaluate both the generalized and conventional ZSL tasks was the widely-used average per-class top-1 accuracy, defined as follows:

where $\mathcal{Y}$ denotes the total class count, and $\mathit{c}$ is the specific class. It is more difficult to achieve a high average per-class top-1 accuracy than the standard accuracy since the former requires good accuracy across all classes.

We use harmonic mean as the evaluation criteria to calculate the joint accuracy of the source and target domains because our goal is to ensure good performance in both domains, and an arithmetic mean could be high simply due to stellar performance in one domain. The formula used to calculate the harmonic mean $\mathcal{H}$ is provided below:

where $\mathcal{Y}^{ts}$ and $\mathcal{Y}^{tr}$ denote the accuracy of seen and unseen classes, respectively.

The results of the GZSL task for all methods are provided in Table 2, showing that our method outperforms the others on most of the datasets. The confusion matrix for the aPaY dataset, which shows classification performance at the class level, is presented in Figure 5. An analysis of these results confirms that the uncertainty calibrator improved model performance with seen objects to a statistically significant degree. At the same time, the results for the unseen classes equal the state-of-the-art. For example, on the aPaY dataset, the visual classifier for the source domain yielded 98% accuracy on the training set of seen objects, and 91% accuracy on the test set of seen objects. For the samples certainly from seen classes, the visual classifier is applied to classify them into seen classes. In conventional GZSL settings, seen class accuracy is merely 51.8% with a general classifier but, with the uncertainty calibrator, it surges dramatically to 62.7%. The minor trade-off in performance with the novel classes (  0.2%) is entirely worthwhile for an improvement of 41.0% to 44.9% in terms of the harmonic mean.

Figure 4 shows the statistics of the classification entropy for the test samples from the aPaY dataset. Notably, the entropy of samples in the seen and novel classes have different distributions. So, when we set the entropy threshold to 2.7, a large amount of the seen test samples fall below the threshold, which means they were confidently classified into the source domain. Conversely, almost all of the novel test samples fall above the threshold are were easily classified into unseen classes. Even with the few misclassifications that slipped through, conventional GZSL approaches struggle in these circumstances as the results show.

As a further analysis, we are curious to see how our framework compared with conventional ZSL, where only objects in unseen classes need to be classified. We use our dual VAE model to synthesise a fixed amount of latent representations (400 was optimal in this case), given novel semantic embeddings in n classes, then train an n-way classifier on the supervised data. Again, average per-class top-1 accuracy is the evaluation metric. The results, shown in Table 3, confirm that our approach not only performs well with GZSL tasks but also delivers state-of-the-art performance in conventional ZSL. It is also worth mentioning that the proposed model outperformed all state-of-the-art methods on the CUB, SUN and FLO datasets. The result with the SUN dataset is particularly promising since this dataset has more than 700 categories with only 20 images per category, and accuracy still reached 63.5%.

Our ablation study covers the effects of the basic dual VAE framework, the triplet loss and the entropy-based uncertainty calibrator. We train a variant of our model without the entropy-based uncertainty calibrator (EUC), and another without either the uncertainty calibrator or triplet loss and compared performance with the complete framework. The results appear in Table 4. It is clear that each component positively and significantly contributes to performance. The triplet loss improves accuracy by around 2%, while the uncertainty calibrator improves performance on the seen classes significantly as discussed above.

A comprehensive class-wise accuracy comparison before and after applying the uncertainty calibration is shown in Figure 6. The 20 categories on the left are from the seen classes, while the 12 on the right are the unseen classes. The blue histogram indicates the accuracy for each class without the uncertainty calibrator; the yellow is with. As illustrated, most of the seen classes witnessed a major improvement after applying the entropy-based uncertainty calibration. The object categories that most benefitted are bicycle, monkey, zebra, mug, building and bug. The seen class zebra has the highest boost from an original 0 recognition rate to 90.2% accuracy.

We use the FLO and aPaY datasets to explore the most appropriate hyper-parameter values for the proposed method with GZSL. The entropy-based uncertainty calibrator is, arguably, the most significant component in our overall architecture, and the results in Figure 7(a) indicate that the best thresholds were 2.78 on aPaY and 2.25 on FLO. We observe that performance is very sensitive to this threshold because it controls the percentage of novel samples that are likely to be misclassified into seen classes. Figures 7(b) and 7(c) show the impact of the triplet loss. The optimal weight and margin for the FLO and aPaY datasets is 0.1 and 5, respectively. Further, we vary the number of visual samples generated for training the general classifier, finding 400 samples per class to be the optimal number, as shown in Figure 7(d).

To further verify the triplet loss regularisation, we produce a t-SNE visualization with the AWA dataset. Figure 8 visualizes the unseen visual samples from the test set and the corresponding latent features generated by the visual encoder. The distributions of the visual features and the encoded latent features are approximate to each other, which proves that the visual encoder preserves the most visual distribution information for the unseen visual samples. Apart from avoiding the information loss problem, the encoded visual features are more discriminative. For example, the visual features of Bat (marked in green) are mixed with Rabbit (marked in red) and Persian cat (marked in gray). This problem is alleviated by the visual feature encoder. It is particularly noticeable in Figure 8(a) that the data points from Bat are more centralized than the original visual features. Moreover, since the negative class samples are forced to separate during training with triplet loss, the distribution is more inter-class discriminative. To compare the latent features from both modalities, we encoded the semantic vectors as rounded dots shown in Figure 9. The latent features generated from visual features are shown as cross marks. Different colours represents different classes. It can be seen that the visual latent features are centered around the semantic latent features of the same classes.

Following [18], we validate our proposed method with natural languages as the semantic information. Considering that annotating attributes is expensive and impractical, we cannot be always annotate attributes for the new categories. Hence, directly using the existing Wikipedia articles as the semantic information is much more practical. Elhoseiny et al.[55] collected raw Wikipedia articles for CUB and NAB [56] datasets. They first tokenized the Wikipedia articles into words and got rid of the full stops. In order to weight the terms in the dataset appropriately, Term Frequency-Inverse Document Frequency (TF-IDF) is adopted to extract text feature vectors. The dimensionality of TF-IDF feature for dataset CUB and NAB are 7,551 and 13,217 respectively. There are two splitting designs according to the relationship between seen categories and unseen categories: Super-Category-Shared splitting (SCS) and SuperCategory-Exclusive splitting(SCE). In SCS, unseen categories are chosen to share same super-class with the seen categories. In result, the relevance in this design between seen categories and unseen categories is relatively high. Intuitively, SCE is more difficult to achieve good performance. As for the visual features, Elhoseiny et al.[55] prepared visual features from images in the two bird datasets with Visual Parts CNN Detector/Encoder (VPDE), yielding 3,582 and 3,072 dimensional visual features for CUB and NAB respectively. We compare to other eight state-of-the-art approaches: MCZSL [57], WAC [58], ESZSL [51], SJE [49], ZSLNS [59], SynC [53], ZSLPP [55] and GAZSL [18]. In Table IV We report the performance of our proposed approach and the existing methods, which are the reported from GAZSL[18]. It can be seen from the table that the proposed method consistently outperforms all the counterparts on both datasets and both splits.

We also conduct zero-shot image retrieval on CUB and NAB datasets. The same semantic information as zero-shot learning from natural language are leveraged in this task. In zero-shot retrieval, given the semantic information only, the images of unseen classes are to be classified and returned. The commonly used evaluation metric mean average precision (mAP) is used, where the precision is the percentage of correct images retrieved for the class. Table VI reports the accuracy of each method when retrieving 100%, 50% and 25% of the images from all unseen classes. The retrieval process begins by generating a fixed amount of latent features for the given semantic vectors. We then calculated the mean point among the generated latent features. In the latent space, all the images are embedded as the latent features. We search for the image points that are nearest to the mean point. Since all the images in the datasets are embedded as the latent features with a much lower dimensionality than visual features, the retrieval complexity is relatively lower.

Zero-shot learning in video classification is a promising research direction. Our proposed method is also applicable to Zero-shot video classification. We validate our proposed method on video classification under conventional ZSL and GZSL settings. Following [60], we use datasets HMDB51 and UCF 101, which includes 6,700 and 13,000 videos respectively with 51 and 101 categories. we extract spatio-temporally pooled 4096-d I3D features from pre-trained RGB and Flow I3D networks and concatenate them to obtain 8192-d video features [61]. As for the semantic information, in HMDB51, a skip-gram model [62] is used to generate semantic embeddings of size 300, using action class names as input. In UCF101, we use semantic embeddings of size 115, provided with the dataset. We compare to existing state-of-the-art methods for zero-shot video classification in Table VII. They are HAA [63], ST [64], TZWE [65], Bi-dir [66], SJE [49], ConSE [47] and GGM [67]. we report the performance comparison to these approaches in Table VII.