Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection

Most zero-shot detection (ZSD) methods [4, 5, 6, 7] align the visual embeddings to the text embeddings of the corresponding base category generated from a pre-trained text encoder. Several methods introduce GCN [17], contrastive learning [18], and a novel ‘polarity loss’ [19] to bridge the gap between the visual embeddings and text embeddings. Inspired by the success of VLMs [10, 20], several methods attempt to perform ZSD by training with image captions or directly leveraging a pre-trained VLM. [21] propose to pre-train the CNN backbone and a Vision to Language Module (V2L) via grounding task on an image caption dataset, and then the whole architecture is fine-tuned with an additional RPN module. However, it still suffers from a large performance gap with the SOTA, and the domain of upstream and downstream datasets must remain similar to maintain performance [11]. Based on a pre-trained VLM, [22] distills the learned image embeddings of the cropped proposal regions to a student detector, [23] further introduces an inter-embedding relationship matching loss to instill inter-embedding relationships. While each proposal needs to be fed forward into the image encoder of VLM. which requires huge computation costs.  [11] proposes a basic self-training pipeline based on ALBEF [20], which firstly utilizes Grad-CAM [24] to obtain dense activation region of specific words in the caption and then generate the pseudo bounding-box label by selecting the proposal that has the largest overlap with the activation region. However, most VLMs lack the ability to perform pixel-wise classification since it is pre-trained with the correspondence of text embedding and global visual token instead of pixel embeddings. Besides, the reliance on caption data limits the application of this generalization to some datasets without caption. Inspired by [25], our method can obtain the pseudo annotations with the pixel-alignment patches.

The pre-training tasks of Vision-and-Language Pre-training (VLP) models can be divided into two categories: image-text contrastive learning tasks and Language Modeling (LM) based tasks. The first category, e.g., CLIP [10] and ALIGN [8], aims to align the visual feature with textual feature in a cross-modal common semantic space. The other category, e.g., VisualBERT [26], UNITER [9], M6 [27], DALL-E [28], ERNIE-ViLG [29], employs LM-like objects, include both autoregressive LM (e.g., image captioning [30], VQA [31]) and masked LM (e.g., Masked Language/Region Modeling). Different from previous works that only focus on global feature alignment, our proposed model can achieve pixel-aware alignments via prompts and perform better in downstream tasks of pixel-wise dense prediction.

Freezing the pre-trained models with only tuning the soft prompts can benefit from efficient serving and matching the performance of full model tuning. Prompt tuning has been verified as an effective method to mitigate the gap between pre-training and fine-tuning. As a rapidly emerging field in NLP [14], prompt tuning is originally designed for probing knowledge in pre-trained language models [32] and now applied in various NLP tasks, e.g., language understanding [33], emotion detection [34] and generation [35]. Prompt tuning has now been extended to vision-language models. Instead of constructing hand-crafted prompts in CLIP [10], CoOp [15] proposes tuning soft prompts with unified context and class-specific context in downstream classification task. CPT [36] proposes colorful cross-modal prompt tuning to explicitly align natural language to fine-grained image regions. The main differences between our usage of learnable multi-modal prompts with the previous prompt tuning works lie in three aspects: 1) Unaligned upstream and downstream tasks; 2) Multi-modal; 3) Learnable prompts for self-training-based open-vocabulary object detection.

Fig. 3 illustrates the overall pipeline of the proposed VTP-OVD. We use different colors to indicate the parameter fixing or requiring training as clarified in the upper right corner of Fig. 3. Firstly, at the fine-grained visual-text prompt adapting stage, by employing the learnable text prompt module $prompt_{t}$ and visual prompt module $prompt_{v}$ into the text encoder $\Phi_{t}$ and image encoder $\Phi_{v}$, we can fine-tune the whole network under dense supervision of pseudo dense score map with other parameters fixed. Secondly, for pseudo labeling stage, we obtain the pseudo labels of non-base classes by leveraging the dense classification ability of the fine-tuned VL model and the location ability of a pre-trained (on base classes) Region Proposal Network (RPN) $\Phi_{r}$. Finally, in the final self-training stage, an open-vocabulary detector $\Phi_{d}$ is further trained with the combination of the ground truth of base classes and the pseudo labels of non-base classes to fulfill the self-training pipeline. The complete training procedure can be found in Algorithm 1.

The crucial part of the self-training pipeline for open-vocabulary detection (OVD) lies in the pseudo labeling stage that determines the final detection performance of novel classes. To generate better pseudo labels for the detection task, our method first aligns each pixel with a category and then adopts a pre-trained RPN for bounding-box localization. Since most vision-language methods (VLM) are trained via the alignment between the whole image and the corresponding caption, they lack the dense alignment ability between pixels and categories. Thus in this section, we focus on modifying the pre-trained vision-language model (VLM) to enhance the current self-training paradigms with fine-grained adapting stage via a newly designed dense alignment loss function and learnable text/visual prompts. Note that to obtain the dense pixel-level visual embeddings instead of the global visual embedding, following [37], we modified the image encoder by removing the query and key embedding layers, which is denoted as $\Phi_{v}$ in Fig. 3 (a).

Based on the more precise dense score map generated by the fine-tuned VLM with learnable text and visual prompts, we can obtain the non-base classes’ pseudo labels by additionally leveraging the location ability of a pre-trained region proposal network (RPN) $\Phi_{r}$ (shown in Fig. 3(b)). Note that $\Phi_{r}$ is only trained on base classes, and the experiments in [22] already demonstrate that training only on the base categories can achieve comparable recall to average recall of the overall categories. Previous pseudo labeling generation strategy [11] used the objects of interest in the caption, which not only harms the transfer ability to the detection datasets without the caption but also is limited to the uncompleted description of caption data. For example, such a strategy cannot generate the pseudo labels of classes that are not included in the caption.

To address these issues, We directly use non-base classes’ names as the input of fine-tuned text encoder $\Phi_{t}$. Besides, to better distinguish the background classes, instead of directly using the word "background", we treat all the base classes as background since we already have the ground-truth annotation of these classes. Then we obtain the pseudo dense score map of non-base classes $\hat{S}_{\overline{B}}\in\mathbb{R}^{HW\times|C_{\overline{B}}|}$ from the fine-tuned VLM with the procedure same to Eq. (3). For each image $x_{i}$, we firstly compute the connected regions $r_{j}\in\mathbb{R}^{HW}$ for $j$-$th$ non-base category on $\hat{S}_{\overline{B}}$ by setting a similarity threshold $\delta$, then we adopt the intersection of union (IoU) of $k$-$th$ proposal $p_{i}^{k}$ from RPN and $r_{j}$ as confidence score:

where $p_{i}=\Phi_{r}(x_{i})$ and the $score_{i,j,k}$ denotes the confidence score that the proposal $p_{k}$ belongs to the $j$-${th}$ non-base category for image $x_{i}$. Finally, a hard score threshold $\gamma$ is adopted on the confidence score to filter out pseudo boxes with less confidence.

Where the ${L}_{CLS}$ and ${L}_{REG}$ denote the cross-entropy loss for classification head and L1-loss for regression head, $\hat{Y}$ and $Y$ represent the model prediction $\Phi_{d}(X)$ and annotations combined with pseudo labels of non-base classes and the ground-truth of base classes. The trained detector $\Phi_{d}$ can then transfer to other detection datasets by providing the class names.

Benchmark Setting: We benchmark on the widely-used object detection dataset COCO 2017 [2]. Following the well-known settings [4] adopted by many OVD methods, we divide the categories used in COCO into 48 base (seen) classes and 17 novel (unseen) classes. Each method can only be trained on annotations of the base categories and then predicts the novel categories without seeing any annotations of these categories. We report the detection performance of both the base and novel classes during inference, as the generalized settings used in [21]. We also evaluate the generalization ability of the trained detector by directly transferring it to three other object detection datasets, including PASCAL VOC [1], LVIS [3] and Objects365 [16].

Evaluation Metric: We use mean average precision (mAP) with IoU threshold 0.5 as the evaluation metric. We pay more attention to the novel class performance since we aim to build an open-vocabulary detector and the annotations of base classes are already provided.

We utilize a pre-trained CLIP (RN50$\times$16) [10] model as the VLM. Note that our method is compatible with conventional VL models. For simplicity, we take CLIP as an example. All the detection models are implemented on the mmdetection [45] codebase and follow the default setting as Mask-RCNN (ResNet50) [46], 1X schedule unless otherwise mentioned. At the adapting stage, the prompt-enhanced CLIP is trained for 5 epochs, with the text prompt learning rate set to 1e-1 and the visual prompt set to 1e-5 separately. For the pseudo labeling stage, the RPN is trained only on the base classes in the 2X schedule. The objectness score threshold for RPN is set to 0.98. The similarity threshold $\delta$ and score threshold $\gamma$ are set to 0.6 and 0.4. For the self-training stage, following the default setting, we adopt a Mask-RCNN (ResNet50) [46] with the last classification layer replaced by class embeddings output by the text encoder. We train the detector for 12 epochs, with the learning rate decreased by a factor of 0.1 at 8 and 11 epochs with 8 V100 GPUs. The initial learning rate is set to 0.04 with batch size 32, and the weight decay is set to 1e-4. We extract the categories of large-scale object detection dataset (i.e., LVIS [3] and Objects365 [16]) as the combined categories set $C$, which contains about 1k categories. We generate non-base pseudo labels for training, while evaluate only on novel categories and base categories following [21].

COCO Dataset. We compare our VTP-OVD with existing open-vocabulary methods on the COCO dataset [2]. As shown in Table I, VTP-OVD achieves the state-of-the-art performance (i.e., 31.5% mAP) on the novel categories of the COCO dataset and 4.5% mAP improvement on overall categories compared to another self-training based method OVD-ALBEF [11]. Besides, by comparing with the VTP-OVD (w/o FT), e.g., 31.5% mAP vs. 29.8% mAP, we further demonstrate the necessity of the fine-grained adapting stage with learnable visual and text prompts for dense alignment tasks. Note that we utilize the basic vision-language model (VLM) CLIP [10] and do not use the caption data, guaranteeing the transfer ability to other VLMs or pre-training datasets without caption information. Clarification should be made that self-training-based methods (e.g., OVD-ALBEF [11] and ours) usually achieve slightly worse on base classes compared to the knowledge-distillation (KD) method (e.g., ViLD [22]). We attribute this to the reason that self-training methods make the model optimize more towards novel classes via generating massive pseudo labels on them, while KD-based methods try to keep the performance of the base class by only distilling the classification ability.

Generalization Abilities. To further demonstrate the open-vocabulary ability of the detector trained through the VTP-OVD pipeline, we directly transfer the final detection model trained on COCO datasets, to other detection datasets, including PASCAL VOC [1], Object365 [16] and LVIS [3]. Benefit from the training on non-base pseudo labels, experimental results in Table II show that our VTP-OVD achieves the best generalization ability even adopted to the datasets with much more categories than the pre-trained dataset, i.e., 81 classes in COCO vs. 1203 classes in LVIS. Note that we do not compare with ViLD [22] since it does not provide either this experiment result or the code.

However, we also find our VTP-OVD fails in detecting several novel categories, including skateboard, snowboard, etc, during the pseudo labeling stage and self-training stage. By observing the dense score map and the pseudo labels of skateboard class in Fig. 7, we attribute this phenomenon to the generation of wrong pseudo labels.

To explain the reason, we think that some objects are often related to the environment, such as skateboards, skateboarders, and skateboard playground often appear together, thus trained with the alignment based on global image information and text embeddings, current vision-language model usually cannot distinguish these objects from other objects in same scene.

We also visualize the distribution of the pixel embeddings generated by the vision-language model CLIP (with or without learnable prompt modules) via t-SNE [47] on novel classes of COCO dataset in Fig. 9. Observation can be made that adopting learnable prompt modules helps cluster the pixel embeddings in the same class and separate pixel embeddings from different categories, demonstrating their effects on dense alignments.