Robust Region Feature Synthesizer for Zero-Shot Object Detection

In ZSD, we have two disjoint sets of classes: seen classes in $\mathcal{Y}^{\rm s}$ and unseen classes in $\mathcal{Y}^{\rm u}$, where $\mathcal{Y}^{\rm s}$ $\cap$ $\mathcal{Y}^{\rm u}$ = $\varnothing$. The training set contains seen objects, where each image is provided with the corresponding class labels and bounding box coordinates. In contrast, the test set may contain unseen objects only (i.e., the ZSD setting) or both seen and unseen objects (i.e., the GZSD setting). During learning and testing, the semantic word-vectors $\mathcal{W}=\{\mathcal{W}^{\rm s},\mathcal{W}^{\rm u}\}$ are provided for both seen and unseen classes. The task of ZSD is to learn detectors (parameterized by $\theta$) that can localize and recognize unseen objects corresponding to semantic word-vectors.

Figure 2 shows the proposed overall framework for ZSD. As can be seen, it contains an object detection module and a domain transformation module. The object detection module is a Faster-RCNN model [33] with the ResNet-101 as the backbone [17]. First of all, we train the Faster-RCNN model with seen images and their corresponding ground-truth annotations. Once the model is obtained, we can use it to extract region features using RPN for seen classes. Second, we train the region feature synthesizer to learn the mapping between semantic word-vectors and the visual features. Then, we use the learned feature synthesizer to generate region features for unseen classes. With these synthesized unseen region features and their corresponding class labels, we can train the unseen classifier for unseen classes. Finally, we update the classifier in the Faster-RCNN model to achieve a new detector for the ZSD task. The overall training procedure is also elaborated in Algorithm 1.

Notice that the core of our method is how to learn a unified generative model to learn the relationship between visual and semantic domains. Specifically, we design a unified region feature synthesizer for feature synthesizing in real-world detection scenarios, which contains an intra-class semantic diverging component and an inter-class structure preserving component.

Given the object feature collection $\mathcal{F}^{s}$, the corresponding label collection $\mathcal{Y}^{s}$, and the semantic vector $\mathcal{W}^{s}$ for seen training data $\mathcal{X}^{s}$, the goal is to learn a conditional generator ${G}:\mathcal{W}\times\mathcal{Z}\mapsto\mathcal{F}$. That is to say, when we take a class embedding $\textbf{w}\in\mathcal{W}$ and a random noise vector $\textbf{z}\sim\mathcal{N}(0,1)\in\mathbb{R}^{d}$ sampled from a Gaussian distribution as inputs, we can generate the visual feature $\tilde{\textbf{f}}\in\mathcal{F}$ for the object regions belonging to this class. Then, with the synthesized region features for the unseen classes, we can learn the classifiers for unseen objects. In other words, the generator $G$ learns the mapping between the semantic vectors and the corresponding region features. To learn such a region feature synthesizer, we propose the following learning objective function:

where $\mathcal{L}_{WGAN}$ is the Wasserstein GAN loss [3] used to enforce the generator to synthesize region features that are aligned well with the distribution of the real region features:

where $\mathbf{f}^{s}$ is the real visual features of the object regions from the seen classes, $\mathbf{\tilde{f}}^{\rm s}={G}(\textbf{w}^{s},\textbf{z})$ denotes the generated visual features conditioned on class semantic vector $\textbf{w}^{s}\in\mathcal{W}^{s}$, $\mathbf{\hat{f}}^{\rm s}=\mu\mathbf{\tilde{f}}^{\rm s}$ + (1-$\mu$)$\mathbf{\tilde{f}}^{\rm s}$ with $\mu$ sampled from the uniform distribution $\mu\sim\mathcal{N}(0,1)$, and $\lambda$ is the penalty coefficient. The discriminator $D$: $\mathcal{F}^{\rm s}$ ($\mathcal{\tilde{F}}^{\rm s}$) $\times$ $\mathcal{W}^{\rm s}$ $\rightarrow$ $[0,1]$, takes a real region feature $\textbf{f}^{\rm s}$ $\in$ $\mathcal{F}^{\rm s}$ or a synthetic visual feature $\mathbf{\tilde{f}}^{\rm s}$ with the corresponding class-semantic embedding $\textbf{w}^{\rm s}$ as input. It tries to accurately distinguish real visual features from synthetic visual features. In $\mathcal{L}_{\rm WGAN}$, the first two terms calculate Wasserstein distance, while the third term constrains the gradient of the discriminator $G$ to have unit norm along with the line connecting pairs of real feature $\textbf{f}^{\rm s}$ and synthesized feature $\mathbf{\tilde{f}}^{\rm s}$. $\mathcal{L}_{C_{s}}$ ensures the generated visual features aligned with the pre-trained classifier on the seen data, which refers to [16].

To improve the robustness of the region feature synthesizer, we explore two new learning terms, including $\mathcal{L}_{S_{d}}$ and $\mathcal{L}_{S_{p}}$. Specifically, $\mathcal{L}_{S_{d}}$ is the proposed intra-class semantic diverging loss, which diverges the semantic word-vector of one object category into a set of region visual features. $\mathcal{L}_{S_{p}}$ is the proposed Inter-class Structure Preserving loss, whose goal is to constrain the separability of the synthesized visual features. $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are the weighting hyper-parameters to balance each component.

To enable the model to synthesize diverse visual features, we consider the IntraSD component to diverge the semantic vector of one semantic word-vector into a set of visual features. Specifically, we conjecture that the above issue could be alleviated by enhancing the influence of noise vectors on the synthetic visual features while preserving the fitness of the synthetic visual features to the class-semantic embeddings. To this end, we propose a novel Intra-class Semantic Diverging loss, where the visual features synthesized from adjacent noise vectors will be pulled closer while those synthesized from distinct noise vectors will be pushed away.

The key design underlying the Intra-class Semantic Diverging loss is how to select the “positive” and “negative” sample pairs. We design positive samples and negative samples by manipulating the input noise vectors [24]. Specifically, given a query noise vector z $\sim$ $\mathcal{N}(0,1)$, we define a small hyper-sphere with radius $r$ centered at the query noise vector z. We random sample a positive query noise vector $\textbf{z}_{+}$ as a vector with the sphere $\textbf{z}_{+}$ $=$ z $+$ $\bm{\rho}$, where $\bm{\rho}$ is a randomly sampled vector from a uniform distribution $\bm{\rho}$ $\sim$ $\mathcal{U}[-r,r]$. We sample the negative noise vectors as random vectors outside the sphere within a latent space, i.e., $\textbf{z}_{i-}\sim$ $\{\textbf{z}_{i-}|\textbf{z}_{i-}\sim\mathcal{N}(0,1)\cap|\textbf{z}_{i-}-\textbf{z}|\succ r\}$ for $i=1,\ldots,N$, where $\succ$ is the element-wise greater-than operator. Once these noise vectors are determined, we can define the “positive” and “negative” samples. For a query visual feature $\mathbf{\tilde{f}}^{\rm s}$ $=$ $G(\mathbf{z},\mathbf{w}^{\rm s})$ synthesized from the noise vector $\mathbf{z}$, we define its “positive” sample as $\mathbf{f}^{\rm s}_{+}$ $=$ $G(\mathbf{z}_{+},\mathbf{w}^{\rm s})$ synthesized from the noise vector $\mathbf{z}^{+}$. The $N$ “negative” samples synthesized from the sets of noise vectors $\{\mathbf{z}_{i-}\}$ can be defined as $\mathbf{f}^{\rm s}_{i-}$ $=$ $G(\mathbf{z}_{i-},\mathbf{w}^{\rm s})$. The Intra-class Semantic Diverging loss is given by

where “$\cdot$” is the dot product between two visual feature vectors to measure the cosine similarity and $\tau$ is a temperature scale factor.

In order to make the synthesized visual features approximate distributions of the real data, meanwhile improving the discrimination of the learned visual features, we further introduce the Inter-class Structure Preserving component into the learning framework. In this learning component, we not only consider the synthesized visual features of different categories, but also pay attention to the real region features extracted by the window proposals, which contain both the positive object proposals, i.e., proposals with the same class as the synthesized feature, and many negative and background proposals.

By doing so, the proposed Inter-class Structure Preserving component has the following merits: 1) The proposed method surpasses the reconstruction error-based loss in the conventional WGAN as it can force the synthesized visual features to be close to other different real visual features of the same category in the window proposal pool. By this way, the synthesized visual features can well approximate the distribution of the real data, facilitating robust one-to-many projection from the semantic word vector to the synthesized region features. 2) By pushing away the visual features from different categories (both in real and synthesized feature space), this learning component can effectively enhance the discrimination of the synthesized visual features.

From the above description, we can observe that the proposed method uses both the synthesized region features and real proposal features to implement the learning process, which essentially constructs a hybrid visual feature pool denoted as $\mathbf{g}=\{{\tilde{\mathbf{f}}^{\rm s},\mathbf{f}^{\rm r},\mathbf{f}^{\rm bg}}\}$, where $\mathbf{f}^{\rm r}$ denotes the real features of the window proposals for different object categories and $\mathbf{f}^{\rm bg}$ indicates the background visual features extracted from the training images. Then, the learning objective function of the proposed Inter-class Structure Preserving component can be written as:

where $\Phi=\{{\mathbf{g}}_{\rm j}\}$ indicates the collection of visual features satisfying $y(\mathbf{g}_{j})\neq y(\mathbf{\tilde{f}}^{\rm s})$ in the hybrid visual feature pool, $y(\cdot)$ is the category indicator function, i.e., $y(\tilde{\textbf{f}^{s}})$ is the class label for visual feature $\tilde{\textbf{f}^{s}}$. $\mathbf{g}_{\rm+}$ is the positive examples corresponding to the current synthesized visual feature $\tilde{\textbf{f}^{s}}$. It can be selected from the synthesized visual features or the object proposals generated by the detector which share the same category label with the current synthesized visual feature $\tilde{\textbf{f}^{s}}$. Therefore, this Inter-class Structure Preserving loss enables the synthesized visual feature $\tilde{\textbf{f}^{s}}$ to be close to both the synthesized and real object proposals of the same category, while far apart from all other visual features from different class labels in the hybrid visual feature pool.

In Table 1, we compare with state-of-the-art methods on the PASCAL VOC dataset on ZSD and GZSD settings. We can observe that our method outperforms all the comparison methods in terms of the ZSD setting. Compared with the second-best method SU [16], our method improves the mAP of ZSD performance from 64.9 $\%$ to 65.5 $\%$. Our method achieves the best performance on unseen classes denoted as “U” among all the comparing methods in terms of GZSD setting. Although our seen performance denoted as “S” is lower, our method achieves the best performance in terms of “HM”, which reveals that our method maintains a good balance between seen and unseen classes. This benefits from the robust region feature synthesizer trained with the Intra-class Semantic Diverging component and the Inter-class Structure Preserving component. We also report the class-wise mAP performance in terms of ZSD setting on the PASCAL VOC dataset in Table 2. Our method achieves the best performance on 3 out of 4 classes, which further demonstrates the superiority of our method on ZSD.

In Table 3, we compare our method with the state-of-the-art methods on MS COCO dataset over two splits. For the 47/17 split, our method outperforms all the compared methods by a large margin in terms of both Recall@100 and mAP measurements. Compared with the second-best method BLC [42], our method improves the Recall@100 by 9.6 $\%$ and mAP by 26.4 $\%$ at IoU=0.5. For the 65/15 split, we can observe that our method also achieves a significant performance gain, which improves the Recall@100 and mAP of method SU [16] from 54.0 $\%$ and 19.0 $\%$ to 62.3 $\%$ and 19.8 $\%$ at IoU=0.5.

In Table 4, we compare our method with other methods under the GZSD scenario, which is more realistic and challenging. Our method outperforms all the comparison methods over two splits in terms of all the metrics. Compared with the second-best method BLC [42], our method improves the “HM” performance in Recall@100 and mAP from 51.4 $\%$ and 8.2 $\%$ to 59.2 $\%$ and 20.4 $\%$ under the split 48/17. For the 65/15 split, our method improves the “HM” performance achieved by the second-best method SU [16] from 55.8 $\%$ and 25.1 $\%$ to 60.2 $\%$ and 26.0 $\%$. This “HM” performance gain proves that our region feature synthesizer can synthesize robust features for unseen classes.

We report clas-wise AP of our method in Table 5 for 65/15 split. Our method achieves the best performance on 9 out of 15 classes and comparable performance on other classes. Since other methods did not report their class-wise AP results on the 48/17 split, we show our class-wise AP in the supplementary material, individually.

To further verify the effectiveness of our method, we conduct the experiment on the DIOR dataset, which is the first attempt for implementing zero-shot object detection in remote sensing imagery. We re-implement the state-of-the-art zero-shot object detection methods based on their released codes on the DIOR dataset for comparison in Table 6. Compared with the second-best method SU [16], the “ZSD”, “U”, and “HM” are improved from 10.5 $\%$, 2.9 $\%$, and 5.3 $\%$ to 11.3 $\%$, 3.4 $\%$, and 6.1 $\%$, respectively. We also achieve the same S performance as method SU [16]. Due to space limitation, the concrete class-wise AP scores will be reported in the supplementary material.

To provide further insight into our method, we conduct ablation studies on the PASCAL VOC dataset to analyze the contributions of each component in our method. In Table 7, We report the ZSD and GZSD performance in terms of mAP metric at IoU 0.5. $\mathcal{L}_{\rm b}$ contains the $\mathcal{L}_{\rm WGAN}$ and $\mathcal{L}_{\rm C_{s}}$, which can be regarded as our baseline method. $\mathcal{L}_{\rm S_{ps}}$ means the hybrid visual feature pool $g$ only contains the synthesized visual features $\tilde{\mathbf{f}}_{\rm s}$. “$\checkmark$” denotes the model with the corresponding component.

IntraSD Component Analysis. We first analyze the effectiveness of the proposed IntraSD component. To verify its contributions, we compare the performances of our baseline model and the variant by adding the IntraSD during training. We can observe that the “ZSD” performance and the “U” performance of GZSD all have been improved significantly from 62.1 $\%$ and 45.9 $\%$ to 64.0 $\%$ and 48.3 $\%$, respectively. The HM performance is improved from 46.5 $\%$ to 47.7 $\%$. These large performance gains demonstrate the effectiveness of the proposed IntraSD component in the ZSD model, which can encourage our generator to synthesize more diverse visual features for unseen classes. The seen performances of GZSD do not obtain performance gains since the parameters of the classifier for seen class is fixed.

InterSP Component Analysis. To verify the effectiveness of the InterSP component, we first add the $\mathcal{L}_{\rm S_{ps}}$ component on top of the $\mathcal{L}_{\rm b}$ and $\mathcal{L}_{\rm S_{d}}$. As a result, the mAP measurements of “ZSD”, “U”, and “HM” are improved from 64.0 $\%$, 48.3 $\%$, and 47.7 $\%$ to 64.7 $\%$, 48.7 $\%$, and 47.9 $\%$. Secondly, we further add the $\mathcal{L}_{\rm S_{p}}$ component on top of the $\mathcal{L}_{\rm b}$ and $\mathcal{L}_{\rm S_{d}}$. The mAP measurement of “ZSD”, “U”, and “HM” are improved from 64.0 $\%$, 48.3 $\%$, and 47.7 $\%$ to 65.5 $\%$, 49.1 $\%$, and 48.1 $\%$. These two comparisons demonstrate that our InterSP component can improve the discrimination of the learned visual features. Compared with the method variant $\mathcal{L}_{\rm S_{ps}}$, the variant with $\mathcal{L}_{\rm S_{p}}$ gains an absolute improvement of 1.2 $\%$, 0.8 $\%$, and 0.4 $\%$. This phenomenon indicates that the real features of the window proposals play an important role in our InterSP module, since it contains the ground-truth positive object proposals and many background negative proposals.

Feature visualization In Fig 3, we conduct the t-SNE [35] visualization for the synthesized region features of the unseen classes on the PASCAL VOC dataset. The visual feature distributions corresponding to our baseline model and the proposed model have been illustrated in Fig 3(a) and Fig 3(b). Features from similar classes (car and train in Fig 3(a)) are confused with each other due to high similarity in their semantic space, which may lead to miss-classify these similar classes. The synthesized features in Fig 3(b) have obvious segregated clusters. This verifies that our synthesizer is robust in synthesizing intra-class diverse and inter-class discriminative region features, which benefits learning a more discriminative classifier to improve the detection performance for ZSD.

Detection Results. To further show the effectiveness of our method, we show the detection results of our method on PASCAL VOC, MS COCO, and DIOR datasets in Fig 4. For the ZSD setting, the images only contain unseen objects. For the GZSD setting, the images may contain seen and unseen objects together. The qualitative results prove the effectiveness of our method in detecting seen and unseen objects simultaneously in the challenging scenarios.