CP-DETR: Concept Prompt Guide DETR Toward Stronger
Universal Object Detection

The recent work can be divided into early fusion and late fusion, depending on the degree of exploitation of the prompt. The late fusion-based method only utilizes the prompt information in the classification. ViLD (Gu et al. 2022), RegionCLIP (Zhong et al. 2022) focuses on transferring knowledge from CLIP to detection. The OWL-ViT (Minderer et al. 2022, 2023) and DetCLIP series (Yao et al. 2022, 2023, 2024) tend to directly align language and image regions through pre-training, therefore scaling up the data to 10$B$ and 50$M$ levels by pseudo-labeling, respectively. The early fusion-based method considers the effect of the prompt on both classification and localization, using the prompt as a condition for image feature encoding. GLIP (Li et al. 2022b) fuses word-level text prompts with multi-scale image features through cross-attention and leverages grounding data to help learn aligned semantics. Grounding DINO (Liu et al. 2023) further proposes language-guided query selection and cross-modality decoder to achieve denser fusion. Then, APE (Shen et al. 2024) and GLEE (Wu et al. 2024) reduce the number of text prompts using sentence-level text encoding methods, significantly reducing the computational overhead of the fusion layer, and thus allowing more negative categories to be used during pre-training. However, previous work uses all visual features to interact with prompts, ignoring the semantic gap of features at different levels in the backbone. For this reason, we design a hybrid encoder to achieve efficient cross-modal interaction through progressive fusion from single to global scales.

Unlike text prompts, visual prompts use image information directly to refer to objects, avoiding misalignment due to incorrect descriptions. Since late fused detectors have a double-tower structure, work (Minderer et al. 2022; Zang et al. 2022) adopts raw image as a visual prompt and leverages image-text-aligned representation to transfer the concept to a visual prompt. MQ-Det (Xu et al. 2023) uses a mixed representation of visual prompts and text prompts. T-Rex2 (Jiang et al. 2024) uses visual instructions to achieve interactive detection, with input boxes and dots generating visual prompts to avoid context loss in cropped images.

The optimized prompt is generated by prompt tuning, which has proved effective for alignment in the classification (Zhou et al. 2022). PromptDet (Feng et al. 2022) uses this prompt as the context of the text prompt to guide the classification foundation model to achieve text and region alignment. GLIP (Li et al. 2022b) aligns concepts in downstream tasks by using optimized prompts as offsets to text prompts, noting that deep cross-modal fusion is critical to improving the effectiveness of prompt tuning. Recent work (Chen et al. 2024) directly learning prompts avoids the dependence on text prompts and further improves performance. The specificity of prompt tuning is that the optimization object is the activation value, which only reduces the alignment bias in the downstream task without changing the model. Therefore, we believe that the evaluation metrics of the optimized prompt can better reflect detector universality.

Previous early fusion-based work (Shen et al. 2024; Wu et al. 2024; Liu et al. 2023; Li et al. 2022b; Zhang et al. 2022) fused full-scale image feature maps and prompts simultaneously, which ignores the semantic gaps that exist between features at different scales. However, due to the lack of semantic concepts and feature duplication, it is inefficient to perform cross-modal interaction on low-level feature maps in the early stages of fusion. Therefore, we use a progressive single-scale fusion module that performs fusion scale-by-scale from high-level feature maps. In order to avoid multi-scale information loss during scale-by-scale fusion, we also designed multi-scale fusion gating to enhance the fusion of critical information.

In the DETR architecture of detector training, both classification and location losses are implemented on the object queries. However, due to the number of object query much smaller than the image features and using a one-to-one set matching scheme of label assignment, encoder output features get sparse supervision signals from the transformer decoder. We argue that these sparse supervision signals will reduce the learning efficiency of cross-scale and cross-modal interactions in the hybrid encoder, leading to sub-optimal results. Therefore, we introduce the auxiliary detection head and prompt multi-label loss to apply additional supervision to image features and conceptual prompts, respectively, which will facilitate fusion learning in the hybrid encoder.

Text prompts successfully unify the training of different datasets and achieve zero-shot detection through a unified semantic space. However, due to alignment bias, the detector is prone to associate with wrong objects when meeting long-tailed or inaccurately described text in downstream tasks. For ODinW (Li et al. 2022a), we observe that the performance of all existing universal models with zero-shot significantly lags behind the closed-set trained models. Therefore, in order to reduce the impact of alignment bias on models in downstream tasks, CP-DETR also introduces two prompt generation methods, namely visual prompts and optimized prompts, to fully stimulate the universal detection capability of pre-trained models.

We use multiple datasets with region-text annotations from different sources for joint training. For the object level, we use publicly available detection datasets, which contain Objects365 (Shao et al. 2019) (O365), OpenImages(Kuznetsova et al. 2020) (OI), V3Det (Wang et al. 2023), LVIS (Gupta, Dollar, and Girshick 2019) and COCO (Lin et al. 2014) datasets. For grounding or REC data, we used the GoldG (Kamath et al. 2021), RefCOCO/+/g (Yu et al. 2016; Mao et al. 2016), Visual Genome (Krishna et al. 2017) (VG) and PhraseCut (Wu et al. 2020) datasets, with a memory bank set length of 1000 in pre-training. where GoldG, RefCOCO/+/g, we used the cleaned labels from GLIP (Li et al. 2022b) and we combined RefCOCO/+/g into RefC by removing duplicate samples. For GoldG, PhraseCut, and VG, where object phrases are treated as categories. For RefC, we treat the entire description as a category. It is worth noting that the training labels we use all come from publicly available datasets and do not scale up the data by pseudo-labeling image-text pair data as most work (Yao et al. 2024; Wu et al. 2024) does.

In our experiments, we developed two model variants, CP-DETR-T and CP-DETR-L, by using Swin-Tiny and Swin-Large (Liu et al. 2021) as image backbone, respectively. We used CLIP-L (Fang et al. 2024) as the text encoder in all variants and only fine-tuned it during pre-training. For CP-DETR-T, we use O365, V3Det, and GoldG for pre-training with a total training epoch of 30. For CP-DETR-L, we train 1$M$ iterations using all training datasets. In all experiments, we use AdamW as the optimizer with weight decay set to 1e-4 and set a minibatch to 32 on 8 A100 40GB GPUs. In pre-training, the learning rate was set to 1e-5 for the text encoder and image backbone and 1e-4 for the rest of the modules, and a decay of 0.1 was applied at 80% and 90% of the total training steps. In visual prompt training, the O365, V3Det, GoldG, and OI datasets are used, the learning rate of the visual prompt encoder is set to 1e-4, and the training is performed for 0.5$M$ iterations. In the optimized prompt, the learning rate of the embedding layer is set to 5e-2, the total number of training epochs is 24, and a decay of 0.1 is applied at 80% of the total training steps.

We evaluated the universal detection ability on the COCO, LVIS, ODinW (Li et al. 2022a) and RefCOCO/+/g benchmarks. ODinW contains 35 real-world scenarios that can reflect the model’s universality in downstream tasks. For COCO, LVIS, and ODinW, the $AP$ is an evaluation metric. Following work (Liu et al. 2023), we also used RefCOCO/+/g to evaluate the ability of the model to understand complex textual descriptions with the [email protected] metric.

By switching among the three concept prompt generation methods, we demonstrate the universality and effectiveness of CP-DETR as an object detection model, both in the pre-training domain and downstream scenarios, while ensuring state-of-the-art performance. In all evaluations, CP-DETR only uses one weight.

In this section, we conducted ablation experiments. The different variants models all use the Swin-T backbone and are trained on O365, V3Det, and GoldG for 12 epochs. In order to show the impact of various components on the universality of the detector, we not only performed a zero-shot evaluation on LVIS but also employed a full-shot optimized prompt on ODinW13.

Table 4 demonstrates the effectiveness of the different designs, where row 0 is the CP-DETR base model without the inclusion of auxiliary supervision and super-class representation. The prompt visual hybrid encoder was ablated in rows 1 and 2; the results show that the hybrid fusion approach of PSF and MSG reduces the difficulty of alignment and contributes to the zero-shot generalization of concepts, and the metric is improved by 0.6$AP$ and 1.5$AP$ on LVIS, respectively. The hybrid encoder is the most important improvement, with the ODinW13 full-shot metric upgraded from 58.5$AP$ in row 1 to 64.0$AP$. This improvement reveals the importance of effective cross-modal fusion for universal location, encouraging the model to decode object boxes based on information in the prompt. Rows 3 and 4 experiments show that auxiliary supervision facilitates the hybrid encoder in learning cross-modal knowledge during the training phase and modestly improves the zero-shot and full-shot metrics performance. Row 5 further adds super-class in prompt fine-tuning, with the number set to 10 in the experiment, i.e., 10 prompt vectors represent a category. This design can effectively handle situations where different objects in downstream scenes are represented in the same category, thus further improving the ODinW13 metric to 67.0$AP$. See the Technical Appendix for more results.

For Hungarian matching, we following previous works (Liu et al. 2023; Shen et al. 2024), and set the weight of alignment costs, L1 costs, and GIoU costs as 2.0, 5.0, and 2.0, respectively. The corresponding loss weights in the transformer decoder are 1.0, 5.0 and 2.0, respectively. Since the transformer decoder computes losses at each layer, to balance the contribution of different losses, we refer to previous work (Zong, Song, and Liu 2023) and set the prompt multi-label loss weight to 6, and the class loss, centerness loss and IoU loss in the auxiliary detection header to 6, 6, and 12, respectively.

Due to GPUs resource limitation and in order to reduce memory spikes, we apply gradient checkpoints and automatically mixed precision (AMP) techniques in the images backbone and prompt visual hybrid encoder. Both our CP-DETR-T and CP-DETR-L use 4 scales image features, where 1/8 to 1/32 is from the image backbone and 1/64 is from channel mapping downsampling. For images augmentation, we use the default DETR (Zhang et al. 2023) augmentation in MMDetection (Chen et al. 2019) toolbox, which includes multi-scale training and random flip.

We compare the data usage of CP-DETR with other methods in table 5. It can be found that most of the methods construct private training annotations to better align different modalities by extending the richness of the training samples. In contrast, CP-DETR achieves excellent zero-shot performance using only publicly available annotations. We believe there are two reasons for this: firstly, the CLIP (Fang et al. 2024) text encoder has seen ample visual concepts and the proposed design is effective enough in exploiting concept information. Second, our using sentence-level representations, where a large number of negative categories can be used in a batch, reduces the illusion.

In addition, the sampling rates we configured for the different datasets are shown in table 6. It should be noted that GoldG (Kamath et al. 2021) data contains both GQA and Flickr30k components. However, we found that multiple samples in GQA shared a single image, so we merged these samples and reduced the data size from 0.62$M$ to 0.09$M$. O365 contains v1 and v2 versions, based on previous studies (Liu et al. 2023; Li et al. 2022b), we use v1 on CP-DETR-T and v2 on CP-DETR-L.

Since ablation experiments in the main manuscript reveal that the super-class representation has a large performance gain for optimized prompts, we experimented with the super-class representation length as well. As shown in figure 3, the performance on the downstream task gradually improves as the representation length increases, approaching saturation at 10, so we use 10 as the default length for optimized prompts.

In addition, table 7 compares the model size and inference efficiency of CP-DETR and Grounding DINO (Liu et al. 2023). For a fair comparison, we use the Grounding DINO implemented in MMDetection (Chen et al. 2019). Automatically mixed precision was kept off in all tests. The results show that our model is more inference efficient. There are three main reasons for this, firstly our cross-modal interactions are scale-by-scale, which has less computational overhead compared to works (Liu et al. 2023) which interact at all scales. Second, we rely on PAN (Liu et al. 2018) structure to fuse image features instead of dense deformable self-attention (Zhu et al. 2021) operation. Finally, Grounding DINO-L uses 1/4 to 1/64 of the image feature maps, while we only use 1/8 to 1/64 of the image feature maps on the largest scale model, requiring fewer image features to be processed.

Although our model exhibits strong universal detection performance, it still has some challenges. On the one hand, the pre-training of CP-DETR relies heavily on text quality, yet there are potential descriptive conflicts between different datasets, e.g., the noun ”mouse”, which denotes a computer device in most of the data, whereas it is used to describe an animal in some scenarios. We believe that such textual defects will reduce the model’s optimisation efficiency and affect the zero-shot capability. On the other hand, since we use average pooling to obtain sentence-level text prompts, this may lead to incorrect optimisation of objects in sentences. For example, if the training text ”person wearing helmet” exists, the zero-shot of ”helmet” will most likely frame out the person with a helmet after pre-training, assuming that there is a lack of category annotation of ”helmet” in the data. In addition, it can be observed in the main manuscript that the visual prompts in CP-DETR-L are significantly better than those in CP-DETR-T, so further scaling up of the model and training data is still necessary.

In this subsection, we demonstrate the generalisation capabilities of CP-DETR on various scenarios through qualitative visualisations. In figure 4, we visualise some zero-shot results through textual descriptions. Our model performs well in different scenarios and correctly processes descriptive text, such as the second row and second column in figure 4.

In figure 5, we visualise some visual prompt results. It can be observed that visual prompts perform well on dense objects and can be combined with text prompts, as shown in (c) of figure 5.

Recently, we tried to scale up the model parameters by updating the visual backbone network. After preliminary experiments, we found that the pre-training weights of the backbone network have a significant effect on the zero-shot performance. We tried EVA-02 (Fang et al. 2024) and Florence-2 (Xiao et al. 2024) and finally chose EVA-02 ViT-L as the visual backbone of CP-DETR-Pro. In the preliminary experiments, CP-DETR-Pro uses the same training data as CP-DETR-T and is trained for 16 epochs with batchsize 16. As shown in table 8, CP-DETR-Pro exhibits an amazing zero-shot generalization capability, which not only exceeds the best metrics of all open-source algorithms, but is also sufficient to compete with closed-source models trained with tens of times closed-source data.

The open source code needs to be permitted by China Mobile’s Ministry of Science and Innovation, and we are working on applying for it. If there are any changes, we will update the arXiv version to publish the link.