More Pictures Say More: Visual Intersection Network for Open Set Object Detection

With the emergence of large pre-trained vision-language models like CLIP(Radford et al. 2021) and ALIGN(Jia et al. 2021), methods based on vision and language(Kamath et al. 2021; Gu et al. 2021; Zhang et al. 2022, 2023; Yan et al. 2023) have gained significant popularity in the field of open-vocabulary object detection. These methods locate objects using language queries while effectively handling open set problems. OV-DETR(Zang et al. 2022) is the first end-to-end Transformer-based open-vocabulary detector, combining CLIP embeddings from both images and text as object queries for the DETR decoder. GLIP(Li et al. 2022) treats object detection as a grounding problem and achieves significant success by semantically aligning phrases with regions. To address the limitations of single-stage fusion in GLIP, Grounding DINO(Liu et al. 2023) enhances feature fusion at three stages: neck, query initialization, and head phases, thus tackling the issue of incomplete multimodal information fusion. Furthermore, APE(Shen et al. 2024) scales the model prompts to thousands of category vocabularies and region descriptions, significantly improving the model’s query efficiency for large-scale textual prompts. The language-based models aim to enhance the semantic description of language queries to adapt to various visual environments, achieving remarkable progress in zero-shot and few-shot settings. However, relying solely on text poses limitations due to language ambiguity and potential mismatches between textual descriptions and complex visual scenes. This underscores the ongoing need for improved integration of visual inputs to achieve more accurate and comprehensive results. Recent advancements suggest that incorporating richer visual prompts and enhancing multimodal fusion techniques are crucial for overcoming these challenges and pushing the boundaries of open-vocabulary object detection further.

Expanding on language-based object detectors, some methods(Zhou et al. 2022a, b)have introduced visual elements to enhance detection accuracy and semantic richness. MQ-Det(Xu et al. 2024) utilizes image examples as visual prompts to enhance textual semantics, thereby enabling more effective open-vocabulary object detection. However, it remains constrained by textual semantics. Additionally, some methods(Kang et al. 2019; Han et al. 2022b) explore the possibility of object detection using only visual prompts. It primarily address few-shot object detection(Fan et al. 2020; Han et al. 2021, 2022a) and typically employ a two-branch Siamese network. For example, FCT(Han et al. 2022b) uses a two-branch Siamese network to process target images and visual queries in parallel, computing the similarity between image regions and a few examples for few-shot object detection. OWL-ViT(Minderer et al. 2022) leverages CLIP’s parallel paradigm and uses detection datasets for fine-tuning to adopt image examples for one-shot image-conditioned object detection. Similarly, DINOv(Li et al. 2024) expands on this concept by employing visual instructions (such as boxes, points, masks, doodles, and specified regions referencing another image) to handle open set segmentation. These visual methods often adopt a Siamese network architecture, which has limitations in zero-shot learning capability. To address these limitations and improve semantic understanding, our goal is to learn the semantic intersection of multiple images. VINO enriches visual semantics by retaining semantic information in all time steps using a multi-image visual bank. Our approach not only improves the model’s ability to understand complex visual scenes but also enhances its robustness and generalization in open set scenarios.

Rethinking Features in the Multi-image Visual Bank: Our goal is to address the insufficiency of category semantics in a single time step by leveraging the semantic intersections across categories. These intersections capture common features and nuances, providing a more robust semantic representation. However, as the number of input images increases, retaining all the semantics of the label becomes impractical due to significant memory consumption. A straightforward FIFO (first-in, first-out) approach would result in the loss of valuable semantic information from previous time steps, which is unacceptable for maintaining accurate category descriptions over time.

As images of the same category continuously appear, they create a stream of visual prompts. While this can be seen as a continuous influx of information, the challenge lies in retaining essential semantics within limited memory constraints. We address this by constructing a multi-image visual bank and employing multi-image updating mechanism. Our approach allows us to compress and retain critical semantic information across all time steps without overwhelming memory resources. The visual bank dynamically processes ROI features and maintains a prompt feature library. It selectively retains the most representative semantic information by averaging and aligning new features with existing ones, thus preserving the core semantics of each category. Our method ensures that, even as visual prompts accumulate, the memory bank remains efficient and semantic integrity is maintained.

This step identifies the sub-feature that is most similar to the new feature, indicating redundancy or relevance in the semantic space.

Our model consists of several key components: the Image Backbone, a visual encoder that extracts features from the target image; the DETR Decoder, a visual decoder that identifies the location and semantic information of proposed regions; the Prompt Encoder, which extracts features from visual prompts; and the Multi-image Visual Bank, a memory bank that preserves visual prompt information for each category and outputs their semantic intersections. By aligning the semantic information of the proposed regions with the semantic intersections of the visual prompts, we can assign labels to each proposed region. Our goal is to identify objects of interest in a target image $I_{t}$.

Specifically, the model takes the target image $I_{t}\in\mathbb{R}^{3\times h\times w}$ and the set of labels $R=\{r_{1},r_{2},\ldots,r_{|R|}\}$ as input. Here, $r_{i}=(x_{1},y_{1},x_{2},y_{2},I_{i})\in\mathbb{R}^{5}$ represents the coordinates of the top-left and bottom-right corners, along with the corresponding category label.

To enable the model to flexibly accommodate new and unseen categories, we employ a DETR-like decoder to process the extracted features $F_{t}$. The decoder leverages object queries $F_{q}\in\mathbb{R}^{bs\times n_{q}\times D}$ as prompts, where $n_{q}$ represents the number of object queries used. These object queries serve as learnable parameters that guide the model in identifying potential object regions within the target image.

The DETR-like decoder operates by decoding the features $F_{t}$ into two outputs: the coordinates of the proposed regions $B\in\mathbb{R}^{bs\times n_{q}\times 4}$ and the corresponding feature representations of these proposed regions $F_{r}\in\mathbb{R}^{bs\times n_{q}\times D}$.

To further validate the broad applicability of visual intersections in visual tasks, we extend the model by incorporating a segmentation head. This addition allows the model to also output predicted masks $M\in\mathbb{R}^{bs\times n_{q}\times h\times w}$. By leveraging the semantic intersection mechanism, the model is capable of not only detecting objects but also segmenting them within complex visual scenes, thereby demonstrating the versatility of our approach.

Next, we perform feature fusion by updating the multi-image visual bank $\hat{F}_{i}$ with the features extracted from the regions, aligning them with the same category in the visual bank, as described in the previous section:

This fusion process integrates the new region features into the existing visual bank, ensuring that the updated bank retains and reflects the latest semantic information.

This step outputs $R\in\mathbb{R}^{bs\times n_{q}\times C}$, assigning the most probable semantic labels to each proposed region.

Dataset and Settings. To rigorously evaluate the performance of our open set object detector, we introduce VINO-D, which leverages pre-training on the Objects365v1 dataset(Shao et al. 2019). This dataset encompasses a comprehensive collection of 600K images spanning 365 object categories, marked with 30 million bounding boxes. For open set detection evaluation, VINO-D is tested on two benchmarks. Firstly, we employ the LVIS v1 validation set(Gupta, Dollar, and Girshick 2019), known for its long-tail distribution of 1,203 object categories. Secondly, on the ODinW35 dataset(Li et al. 2022), which consists of 35 diverse datasets designed to challenge model performance in varied real-world scenarios. This dataset includes many rare categories seldom represented in training datasets, providing a stringent test of our model’s transferability and effectiveness across common and rare object categories. In addition to VINO-D, we develop VINO-S by integrating a segmentation head to expand its capabilities to both detection and segmentation tasks. This model is pre-trained on the COCO2017 dataset(Lin et al. 2014), which comprises approximately 110K images used for both object detection and panoramic segmentation. VINO-S is meticulously evaluated on the LVIS v1 validation set, demonstrating the broad application prospects of semantic intersections of visual prompts in visual tasks.
Training Details. Both VINO-D and VINO-S utilize APE-D weights for processing target images, with ViT-L(Fang et al. 2024) serving as their mage backbone. The prompt encoder is CLIP-L, which remains frozen during training. The number of prompts is set to 5, and the number of object queries is set to 900. Training is conducted on 2 RTX4090 GPUs, with a batch size of 1, using the AdamW optimizer with a learning rate of 5e-5. VINO-D is pre-trained on the Objects365v1 dataset for 1 epoch, which takes approximately 7 RTX4090 GPU days. Similarly, VINO-S is pre-trained on the COCO2017 dataset for 1 epoch, requiring about 2 RTX4090 GPU days. To address the substantial domain shift caused by the prompt encoder taking cropped images as inputs(Li et al. 2024), we control the resolution of these cropped images to ensure high-quality vision prompts. Specifically, the resolution of the first prompt image is maintained at no less than 2000, while subsequent visual prompts have resolutions no lower than 1600.

1. Incremental Improvement and Diminishing Returns: As the number of prompts increases from 1 to 20, there is a clear improvement in detection AP, from 20.9 with a single prompt to 31.3 with 20 prompts. However, the rate of improvement decreases as the number of prompts increases.

2. Significance of Multiple Prompts: The substantial improvement from 1 to 5 prompts underscores the importance of using multiple prompts. It demonstrates that leveraging multiple vision prompts enables the model to better generalize and understand the various visual features within each category.

The visualizations in Figure 3 illustrate the superior capabilities of our model across a range of challenging scenarios. The first image demonstrates the model’s proficiency in single-prompt detection, where a single visual prompt is sufficient to accurately identify and detect all instances of the target objects within the image. In the second image, the model effectively handles the detection of multiple instances across several categories within a single target image. This scenario highlights the robustness of the model’s semantic intersection approach, which allows it to maintain high detection performance even in complex environments with diverse object categories. The third example provides a clear illustration of the model’s ability to distinguish between categories with similar semantic features. These examples collectively highlight the versatility and accuracy of our model in open set object detection tasks.

The visualization of VINO-S, as shown in Figure 4, demonstrates that using the semantic intersection of visual prompts enables more effective segmentation of target semantics.

Table 5 presents the zero-shot performance of VINO across 35 diverse datasets within the ODinW35 benchmark. These results demonstrate the model’s robustness and adaptability in handling a wide range of real-world scenarios by semantic intersection. VINO consistently performs well across varying contexts and object categories, indicating its strong generalization ability in open-set object detection tasks.