SIA-OVD: Shape-Invariant Adapter for Bridging the Image-Region Gap in Open-Vocabulary Detection

Knowledge-Distillation based OVD methods tend to distill the open-vocabulary knowledge of vision-language pre-trained models (VLM), e.g., CLIP, into traditional close-set object detectors. ViLD (gu2021open, ) designs a distillation loss between CLIP image embeddings and image features obtained by the detector’s backbone to transfer CLIP’s visual capabilities to the object detector and leverages the CLIP text encoder for efficient classification within the object detection framework. HierKD (ma2022open, ) proposes a global-level knowledge distillation method and combines it with instance-level knowledge distillation to simultaneously learn base and target categories. OADP (wang2023object, ) proposes an Object-Aware Distillation Pyramid to mitigate the information destruction and inefficient knowledge transfer in distillation-based OVD methods. MM-OVOD (kaul2023multi, ) leverages the powerful generative ability of LLMs to enrich the textual descriptions of unseen categories. Then, it constructs a multi-modal classifier by combining image exemplars and textual descriptions, which achieves higher classification performance for novel categories. VL-PLM (zhao2022exploiting, ) and SAS-Det (zhao2024taming, ) introduce a self-training strategy into OVD, using CLIP to generate pseudo labels for novel categories based on the region proposals predicted by a class-agnostic proposal generator. Then a close-set detector is trained on the pseudo-labels for novel categories generated by CLIP. SIC-CADS (fang2024simple, ) refines the object classification scores of an existing OVD model by leveraging global knowledge distilled from a frozen CLIP model. These knowledge-distillation-based OVD methods attempt to equip close-set detectors with the ability to detect open-vocabulary words. In contrast, we utilize the frozen CLIP image encoder as the backbone of our detector, which directly incorporates CLIP’s open-vocabulary knowledge into the OVD task.

Large-scale vision-language pre-trained model (VLM), e.g., CLIP, demonstrates impressive zero-shot performance across various downstream tasks (ye2022unsupervised, ; tao2023galip, ; liu2024visual, ; li2024flowgananomaly, ). Adapting VLMs to a specific task commonly involves two approaches: fine-tuning or prompt tuning. Recently, the focus of research has gradually shifted towards prompt tuning, which keeps VLM encoders frozen and learns additional prompts as input to VLMs. CoOp (zhou2022learning, ) replaces the manual template of textual prompts with learnable context tokens, which are trained with all categories. However, such fixed context tokens result in poor zero-shot capabilities and generalization to unseen categories, so CoOp is not compatible with the OVD task. CoCoOp (zhou2022conditional, ) integrates image features into the learning process by adding a lightweight meta-net to generate instance-conditional textual tokens to mitigate this issue. CoCoOp improves the open-vocabulary ability, but the memory consumption and computational cost are extremely high because the meta-net does not support batch processing. UPT (zang2022unified, ) developed a modality-agnostic prompt tuning method, leveraging a lightweight transformer layer to concurrently fine-tune both the visual and textual branches. CLIP-Adapter (gao2024clip, ) adapts the original clip model to downstream tasks by adding trainable linear layers after the vision encoder and text encoder. CLIP-Adapter achieves higher efficiency because the adapters are inserted after encoders, and the back-propagate paths are much shorter. FG-VPL (sun2023fine, ) explores the potential for fine-grained image recognition of CLIP by introducing fine-grained visual prompts. In this paper, we follow these prompt tuning methods with frozen backbones to maintain the generalization ability of CLIP.

Open-vocabulary detection (OVD) aims to train an object detector $\mathcal{D}$ to detect objects of novel categories $\mathcal{C}_{N}$ without the instance-level annotations for them. During the training phase, only instance-level annotations for objects of base categories $\mathcal{C}_{B}$ and the pre-trained CLIP model are available. During the inference phase, the detector is tested on a set of novel categories $\mathcal{C}_{N}$, where $\mathcal{C}_{B}\cap\mathcal{C}_{V}=\emptyset$. Extra training datasets of image-text pairs (e.g., COCO Captions (chen2015microsoft, ) and CC3M (sharma2018conceptual, )) are also available, but in this work, we do not utilize any extra datasets to train our model.

Specifically, our open-vocabulary object detection process is divided into two branches: object localization and region classification. We mainly focus on improving the accuracy of region classification in this work. For object localization, we use DAB-DETR (liu2022dab, ) as the localizer, which learns object queries with bounding boxes as input. DAB-DETR encodes the feature map output by the $3^{rd}$ layer of the CLIP image encoder and decodes the outputs of the bounding boxes of detected objects in both base and novel categories. For region classification, given a region proposal of 4 dimensions $(x_{i},y_{i},h_{i},w_{i})$ with its confidence score $conf_{i}$ decoded from DAB-DETR, we utilize RoIAlign (he2017mask, ) to obtain the region feature from CLIP’s image feature map. Then, we feed the region feature and the bounding box into the Shape-Invariant Adapter (SIA). SIA generates the weights for different adapters according to the shape of the bounding boxes and transforms the region features into shape-invariant ones that are better aligned with text embeddings. We perform region classification by calculating the cosine similarities between region embeddings and text embeddings of all categories. The overall framework is illustrated in Figure 2.

Structure of SIA. In this section, we introduce our Shape-Invariant Adapter illustrated in Figure 3. To mitigate the image-region gap in OVD, SIA aims to learn a mapping from the region feature space obtained by RoIAlign to the common space, which is aligned with CLIP visual and textual features. We are inspired by the fact that CoOp (zhou2022learning, ) learns fixed context tokens for all samples and performs poorly on unseen categories, while CoCoOp (zhou2022conditional, ) learns conditional context tokens based on each sample and achieves better performance on unseen categories. Therefore, we try to learn conditional feature transformation based on the shapes of region proposals.

Luckily, the shape information of a region is much simpler than its semantic information. We do not have to rely on the generative network of CoCoOp to generate conditional context, which results in a high computational cost. Specifically, SIA keeps a set of $N$ independent adapters $\{Adapter_{j},j=1,...,N\}$. Each adapter consists of two fully connected layers $\mathbf{W}_{1}^{j}$ and $\mathbf{W}_{2}^{j}$, a ReLU activation layer, and a residual factor $\lambda$. When the $i^{th}$ region feature $f_{r}^{i}\in\mathbb{R}^{D\times 1}$ is input to SIA, we obtain the adapted region embedding $\hat{f}_{r}^{i}$ through a residual connection:

Given the adapter allocation weights $Y_{i}$ for the $i^{th}$ region proposal and the region features adapted by all adapters in SIA $F_{r}^{i}$, we multiply the region features by the weights to select the optimal adapter:

Finally, to perform classification, the adapted region feature $\beta_{r}^{i}$ is multiplied by the CLIP text embeddings for each category name. The text embeddings are denoted as:

Training. We adopt a two-stage training process. In the first stage, we freeze all parameters except the adapters in SIA. For a training sample $\mathcal{R}_{i}$, the ground-truth bounding boxes are input to the classification branch, and we optimize SIA with Cross-Entropy Loss:

Then we keep the classification branch frozen and train the transformer encoder and decoder in the localization branch. Our localizer shares the same DAB-DETR structure as CORA (wu2023cora, ), so we utilize the same training object.

COCO-OVD Benchmark. COCO-OVD benchmark is proposed by OVR-CNN (zareian2021open, ), which splits the COCO categories into 48 base categories and 17 novel categories. Under the OVD setting, object detectors are trained on the bounding box annotations of 48 base categories in the training set. Detectors are evaluated on the validation set, which contains 28,538 instances of base categories and 4,614 instances of novel categories.

Model Architecture. The architecture of the object localizer is identical to CORA, which is a DAB-DETR (liu2022dab, ) consisting of a 3-layer transformer encoder, a 6-layer transformer decoder, and 1,000 learnable object queries as input. We utilize the 80 prompt templates provided by CLIP to calculate the text embeddings for each category.

Comparison with State-of-the-Art. Table 1 presents our main results on the COCO-OVD benchmark. The compared OVD methods include knowledge-distillation based methods (F-VLM (kuo2022f, ), SIC-CADS (fang2024simple, )), region-text pretraining based methods (RegionCLIP (zhong2022regionclip, ), BARON (wu2023aligning, ), CLIM (wu2024clim, )), pseudo-labeling methods (VL-PLM (zhao2022exploiting, )) and prompt-learning methods (PromptDet (feng2022promptdet, ), DetPro (du2022learning, ), CORA (wu2023cora, )). We do not compare SIA with SIC-CADS and CLIM, which are attached to an existing OVD model.

To conduct a fair comparison, we separately compare the aforementioned methods with different visual backbones: RN50 and RN50x4. For each method, we list its supervision from three aspects: Extra Dataset (data beyond the instance-level annotation for base categories), Pretrained Model (the CLIP image encoder and text encoder), and Require Novel Class (whether the names of novel categories is needed during training).

Among these baselines, SIA outperforms CORA (wu2023cora, ) by 0.4 AP50 on novel categories with RN50 backbone, and 0.3 AP50 with RN50x4 backbone. SIA only utilizes the pre-trained CLIP model and instance-level annotations for base categories, without the need for any extra dataset. Moreover, during the training phase, novel class names are not required either, which makes it more convenient to train the model and easier to expand to unseen categories.

To further demonstrate that SIA effectively improves the classification accuracy for regions with extreme aspect ratios, we report the classification performance of regions with different shapes in Figure 4. As shown in the figure, the original CLIP exhibits unsatisfactory classification accuracy for all shapes. CORA improves the overall performance of region classification, but there is still an obvious gap between regions with aspect ratios far smaller or larger than 1:1 and regions with aspect ratios close to 1:1. Our proposed approach, SIA, effectively enhances the classification performance of regions with extreme aspect ratios. We also report AP50 scores achieved by CLIP, CORA, and SIA (Ours) across each novel category, as illustrated in Table 3, where $\delta$ denotes the difference between AP50 scores of SIA and CORA. SIA achieves improvements over CORA across most categories, particularly for tie (+7.51%), skateboard (+7.69%), and keyboard (+13.26%). For cat, umbrella, and cake categories, SIA exhibits slight decreases compared to CORA.

Classification Results for Region Proposals. Figure 6 shows the region classification results in the COCO-OVD validation set. SIA predicts more reasonable classification results than CLIP and CORA for region proposals, especially for regions containing severely deformed objects, e.g., surfboards, ties, and knives. For example, in the first column, SIA successfully identifies the surfboard, while CLIP and CORA misclassify it as an airplane.