Class Knowledge Overlay to Visual Feature Learning for Zero-Shot Image Classification

Semantic attributes refer to express a class or an instance using attributes. ZSL uses attributes as side information and consists of two steps: 1) to train the seen classes: gain knowledge about attributes; 2) to inference the unseen classes: classify some unseen objects via known knowledge. This is the first and most basic method of ZSL. In 2009, a pioneering study on ZSL, Ref. [25], proposed direct attribute prediction(DAP) and indirect attribute prediction(IAP). They are the main forms of attribute-based learning which learns the attribute classifier first and then seeks the most promising unseen class. Ref. [50] proposed an author-topic model to describe the attribute-specific distributions of image features. Ref. [41] has proposed a weighted version of DAP based on the observation probability of the attributes. However, attribute-based learning ignores the associations between different attributes, and it is more accurate in predicting attributes than classes. Furthermore, attributes need a large number of experts to label, which is inefficient. On the contrary, our approach does not depend on any prior attributes.

Semantic embedding is a text-to-vector technique that can be used for mapping the visual feature to semantic space. The semantic embedding-based learning is one of the most widely used methods [11, 12, 32, 40, 52]. Attribute label embedding(ALE) [2], proposed a label embedding framework to solve the prediction of classes aiming at the attribute learning directly. It not only takes attribute as side information but also takes word vector and hierarchy label embedding(HLE) as side information. Besides, inspired by ALE, Ref. [3] proposed structured joint embedding(SJE), a structured joint framework and used various side information to replace the era of artificial annotation attributes in ZSL tasks. In 2016, LatEm [46], a nonlinear model of SJE, was proposed. It has a stronger expressive ability and can be adapted to different types of samples. The semantic similarity embedding (SSE) [52] not only maintains semantic consistency but also ensures the accuracy of classification. The above studies directly transfer the visual feature space to the semantic space, which leads to the problem of the large semantic gap problem. In 2017, Ref. [24] introduced semantic autoencoder(SAE), a bidirectional encoding and decoding method that significantly reduces the semantic gap. However, the experimental results of SAE are not optimistic because the feature space transferring technique cannot eliminate the semantic gap.

Different from semantic embedding, semantic-to-visual mapping is designed to learn the mappings from semantic space to visual space. Currently, most approaches follow the idea of semantic-to-visual mapping [4, 33, 48] and lead a new era of ZSL. The Ref. [53] combined the generative adversarial network (GAN) and ZSL to transform the ZSL problem into an ”imagination” problem. The method implements semantic-to-visual mapping using ”imagining” visual features from semantic features. Other studies [1, 3, 26, 36, 39, 49, 51, 53] show that these approaches yielded optimistic results. However, these approaches cannot ”imagine” the visual features of the unseen classes if the corresponding semantics have not appeared. In this study, we use both semantics from one class and the ”class knowledge overlay” to obtain more semantics from other similar classes. This approach significantly enriches the semantics for semantic-to-visual mapping.

Suppose there is a series of data points $({d,y})$ from the original image dataset $D$ and label $Y$ respectively. We use subscripts $u$ and $s$ to represent datasets of unseen and seen classes after splitting the dataset, respectively. The visual features $x\in V$ can be extracted by using original images $d$. The semantics of seen and unseen categories are represented as $t_{s},t_{u}$, which come from the semantic space $T$. For the $i$-th class, the representation of the class name is $E_{i}\in E$, where $E$ is the sets of all class names. The goal of ZSL is to predict $y_{u}$ based on $x_{u}$ and $t_{u}$. Generator $G$ and discriminator $D$ are represented as $R^{T}\times R^{M}\to R^{V},R^{V}\to\{0,1\}\times L_{cls}$ where $R^{M}$ represents the mapping relationship of semantic features into visual features, and $L_{cls}$ represents the corresponding class labels in visual features $V$. We converted the parameters of $G$ and $D$ into $\theta$ and $\omega$.

The visual features are extracted by the visual feature extraction methods described below: the fast-RCNN framework and the VGG16 architecture are used as the backbones to detect seven parts of the birds. First, the features of the input images $d\in D$ are extracted by VGG16. The proposed region of interest(ROI) pooling layer in [15] is input into an n-ways softmax layer and a boundary box regression. Then, it is regarded as a detected visual feature when the proposed area is larger than a confidence threshold; otherwise, it is regarded as a missing part. Finally, the detected region is input into the visual encoder subnet and eventually encoded into 512-dimensional feature vectors for each part. The visual features of these seven parts are concatenated together to form 3584-dimensional visual features $x\in V$.

Class Knowledge Overlay: The overall flow of the CKO algorithm is shown in the Algorithm 1. First, the word2vec is applied to transform each class to a vector. Second, a cosine similarity is used to calculate the similarity scores among the class vectors and the top-k similar classes of each class are ranked. Finally, the Wikipedia text of the category is represented by concatenating its own Wikipedia text and the Wikipedia text of the top-k similar classes. Fig.2 shows the similarity results of Logger-Head Shrike and other classes. Obviously, Logger-Head Shrike has a high similarity score to Great-Grey Shrike, which demonstrates that Great-Grey Shrike is likely to contain the knowledge of Logger-Head Shrike.
Semantic Embedding: The Wikipedia texts are tokenized into words, firstly. Then, some necessary preprocesses, such as removing stop words, porter stemmer, and tokenization [34], are applied to reduce inflected words to their word stem. Finally, the text encoder, TF-IDF, is used to extract and embed the semantic features.

Visual Feature Generation: Text encoder $\phi$ is used to embed texts. The embedded texts $\phi(T_{c})$ are used as input to a generator ($G$ for short), which is a multi-layer perceptron with random noise $z$. Through this process, visual features $\widetilde{x}$ can be generated by $G_{\theta}(T_{c},z)$.

Because of the sparsity of training data (about 60 pictures per class of CUB datasets, and the distribution of visual features has about 3500 dimensions), it is difficult for the generator to achieve good results in transforming class knowledge into visual features. Ref.[53] reported that classes have the following characteristics in the visual space: the distance of intra-classes is short, the distance of inter-classes is long, and an overlap rarely occurs. Therefore, a new constraint can be added to the knowledge-to-visual features generation to make the synthetic visual features have the same visual distribution as the seen classes. The constraint is defined as follows:

where $C$ is the number of seen classes, $x_{s}^{c,i}$ is the $i$th visual feature of class $c$, $\widetilde{x}_{s}^{c}$ denotes the synthetic-visual features of category $C$ in the seen class, $\bar{c}$ denotes a class that does not belong to class $c$, $margin$ represents the minimum distance between two different class clusters, and $dist$ represents any measure. In this study, Euclidean distance is used as a measure. Finally, the loss of generator is defined as:

where the first two terms approximate Wasserstein distance of the distribution of real features and fake features, the third and forth terms are classification losses of real and synthesized features. $\lambda_{t}$ is a regularization coefficient.

Discriminator: The discriminator ($D_{\omega}$ for short) accepts two inputs: fake visual features from $G$ or real visual features from images. Then it propagates them forward to a full connection layer with a ReLu activator. Next, two subnetworks are used to distinguish whether features are real or fake and classify the category label of these features. The loss function of $D_{\omega}$ is the same with the previous work[53].

During each SSL iteration, a conventional classifier are trained by using examples from $(G(t_{u},z),y_{u})$. In this paper, the conventional classifier is k-NearestNeighbor model. Then, the classifier predicts pseudo-labels, which have highest class probability in all classes, for each unseen class sample in $d_{u}$. Those samples whose class probability is above a certain threshold are stored in a set of $D_{p}=\{(D_{s},\widetilde{y_{u}})\}$. In the next training, the training set $D_{s}$ is updated to $D_{s}\cup D_{p}$. Because at the beginning of training, the model only trains the seen classes data. After the semi-supervised learning, the unseen classes with pseudo labels will be added to the training set. If the pseudo label is marked as unseen class, then a new class is introduced in the training set. So we need to dynamically add new neurons to the subnetwork in the discriminator, which are used to classify new classes, and include this new category when calculating the triplet loss. The detailed training process of GAN with SSL is shown in Algorithm 2.

Training: Semantic features are extracted using the proposed class knowledge overlay(CKO), and visual features are extracted. through real images and generators. Then, ACGAN is trained with $n_{iter}$ iterations, including the training generator’s ability to generate visual features using semantic features with tripletloss, and the the training discriminator to judge visual features as fake or real and predict the class labels. After the ACGAN training is completed, the generator uses the visual features generated by the semantic features of unseen classes and the corresponding semantic labels to train the traditional classifier(eg. Decision Tree, SVM,…). The trained classifier will give the label probability for visual features of the unseen class. For labels with a probability higher than a certain threshold, their visual features are added to the training set. Repeat the above process until the $n_{ssl}$ semi-supervised process is executed.

Testing: After training, we obtain the generation model $G$, which can transform semantic features of classes into synthetic visual features. In the testing process, the model compares the real visual features (from the new coming image) with the synthetic visual features (from the text of class). Then, the model decides the class of the new coming image.

In the ZSL domain, it is not sufficient to only consider the performance of the unseen classes. A more generalized evaluation criterion is needed. In [53], a generalized evaluation metric, which considers the accuracy of the seen and unseen classes, was proposed for ZSL. A balance parameter was used to draw the curves of the seen and unseen classes(SUC, the accuracy of the seen classes is the vertical axis and the accuracy of the unseen classes is the horizontal axis), and the area under SUC (AUSUC) was used to represent the generalization ability of the ZSL model. Table 4 shows the AUSUC scores between our method and the other methods. The AUSUC score of our method increased by 1.6% and 3.95%, respectively, on two benchmark datasets with SCE splitting compared to the other methods. In the SCS-split, our method is slightly lower than CorrectionNet and CANZSL, only 1.4% and 0.7%, but still surpasses a large number of the state-of-the-art methods.

We also evaluate the AUSUC scores of each component in our method. Fig.4 shows that the effect of triplet loss on the result performance is relatively stable, while the performance of CKO and SSL methods changes greatly. This is because the CKO and SSL sometimes introduce some noise that affects the training of the model. However, the generalization of each combination reached the state-of-the-art standard.

In addition, we use another GZSL setting that emerges recently to evaluate the proposed method on AwA1 and AwA2 datasets. These two datasets are based on attribute and respectively contain 30,475 and 37,322 images of 200 animals with 40 seen and 10 unseen classes with 85-dimensional attributes. In this setting, test set includes data samples from both the seen and unseen classes. We follow the same setting in [48], which adopt the average per-class top-1 accuracy S and U, as well as their harmonic mean to evaluate the performance of the model and combines the seen and unseen classes as the search space. The Table 5 shows that the proposed method compared with seven latest methods. The results show our GAN-CST exceeds a large number of the latest method. Especially in the S of AwA1 dataset and S and H of AwA2 dataset, the best performances are achieved, which are 97.2%, 94.0% and 85.6%, respectively. An obvious rule can be observed: our GAN-CST improves the U accuracy while ensuring a high S accuracy. Although AwA1 has achieved better performance on U and H compared to our method, the performance of our method on S far exceeds it, and the accuracy on S is almost 100%. This shows that our method can well retain the discriminative features of seen classes while improving the performance of unseen classes.

The task of zero-shot retrieval means to retrieve the relevant images from unseen classes giving the semantic representation of the specified class in unseen class set. We use mean average precision (mAP) to evaluate the performance. For comparing with other methods fairly, we report the performance of different settings in Table 6: retrieving 25%, 50%, 100% of the number of images for each class from the whole dataset are ranked based on their final semantic similarity scores. The precision is defined as the ratio of the number of correct retrieved images to that of all retrieved images.

Table 6 presents the comparison results of different approaches for mean accuracy precision (mAP) on CUB and NABird datasets. We note that the proposed approach has achieved consistent improvement compared with GAN-ZSL and beats all the competitors.

We also visualize some qualitative results of our approach on two datasets, shown in Fig. 5. Each row is a class, and the class name and precision are shown on the left. The first column is the benchmark. The following five columns are Top-5 without considering the instances in the first column. Some instances are hard to distinguish even for humans, but the model can recognize. For example, the top-5 retrieval images of class ”Northern Waterthrush” are all from their ground truth class since their visual features are similar. However, the query ”Mountain Bluebird” retrieves some instances from its affinal class ”Florida Scrub Jay” since their visual features are too similar to distinguish.

In this section, different hyperparameters were set to observe their impact on the performance of our model. Three groups of experiments were conducted. The hyperparameter settings are shown in Table 7. Fig.6 and 7 show the generalized accuracy curves with different splitting methods and different hyperparameters in two benchmark datasets. The horizontal axis represents the values of the hyperparameters, while the vertical axis represents the generalization accuracy of the seen classes (calculated by formula 3). The values of the hyperparameters corresponding to the highest generalization accuracy are set as the parameters of the model. Table 1 summarizes the values of the hyperparameters in all groups. Our method is more robust than the other methods because of the gaps in the accuracy of the unseen classes with different hyperparameter settings.