Learning to Detect and Segment for Open Vocabulary Object Detection

Dynamically Aggregated Head To leverage the large network capacity of complex heads, we propose to generate dynamic weights to aggregate a set of complex heads $\{\mathcal{B}_{1},{\mathcal{B}_{2}},...,\mathcal{B}_{H}\}$, with associated parameters $\{\theta_{1},{\theta_{2}},...,\theta_{H}\}$. These heads act as experts that are good at refining bounding box and predicting mask segmentation for objects categories with certain shapes or appearance. Specifically, the aggregation weights is generated by:

$\displaystyle\textbf{w}=\mathcal{A}_{\phi}(\textbf{s}_{c})$

(6)

where $\textbf{w}\in\mathbb{R}^{H}$, and $\mathcal{A}$ is a small neural network with parameter $\phi$. To regularize the weight aggregation space as a convex hull for optimization, the weight w is then normalized with Softmax function:

$\displaystyle w_{h}=\frac{e^{w_{h}}}{\sum_{h^{{}^{\prime}}=1}^{H}e^{w_{h^{{}^{\prime}}}}}$

(7)

which ensures the weights sums to $1$ and each element is larger than $0$. With the normalized aggregation weight, the expert heads are then combined to a single head $\hat{\mathcal{B}}$ with parameter $\hat{\theta}$:

$\displaystyle\hat{\theta}=\sum_{h=1}^{H}w_{h}*\theta_{h}$

(8)

Similarly, for a set of mask heads $\{\mathcal{M}_{1},{\mathcal{M}_{2}},...,\mathcal{M}_{H}\}$, with associated parameters $\{\vartheta_{1},{\vartheta_{2}},...,\vartheta_{H}\}$, the weight generation and aggregation can be conducted to obtain a single mask head $\hat{\mathcal{M}}$ with parameter $\hat{\vartheta}$. When the dynamic aggregation is applied to both box regression and mask segmentation, separate weight generation networks are used, as box regression and mask segmentation may not share the same attention across the expert models.

Dynamically Generated Head To introduce stronger conditioning on the semantic embedding, we propose to directly generate the network parameter. With bounding box regression head as an example, the parameter is first generated as:

$\displaystyle\dot{\theta}=\mathcal{D}_{\varphi}(\textbf{s}_{c})$

(9)

the parameter $\dot{\theta}$ is used to instantiate a new box head $\dot{\mathcal{B}}$. Since $\dot{\theta}$ is directly generated from the semantic embedding, it easily encodes general class-specific information like aspect ratio and color. Similarly, a dynamic mask segmentation head $\dot{\mathcal{M}}$ can be obtained with generated parameter $\dot{\vartheta}$.

Combining the Dynamic Predictions We then combine the above dynamically aggregated heads $\hat{\mathcal{B}}$, $\hat{\mathcal{M}}$ and dynamically generated heads $\dot{\mathcal{B}}$, $\dot{\mathcal{M}}$ to obtain the final result. We consider simple weighted averaging on their prediction:

Optimizing the Expert Heads with Temperature Annealing During the early optimization stage, the expert heads within the dynamically aggregated head all require training signals to initiate the learning of basic regression and segmentation capability. While in the later optimization stage, the experts are expected to provide specialized capability in dealing with different objects. We thus facilitate the optimization of the expert heads by a temperature annealing Softmax strategy. Concretely, the normalization of aggregation weights is performed with temperature $\tau$

$\displaystyle w_{h}=\frac{e^{w_{h}/\tau}}{\sum_{h^{{}^{\prime}}=1}^{H}e^{w_{h^{{}^{\prime}}}/\tau}}$

(12)

During the early training stages, the temperature is set as a large value and with the progress of training to provide nearly uniform gradients for all the expert heads, the temperature is gradually annealed to a smaller value to achieve the desired specialized learning.

Conditionally Parameterized Convolution CondHead is closely related to Dynamic Convolution [5] and CondConv [29]. They differ from CondHead in motivation and application. They aim to improve the representation capacity of convolution kernels through conditional parameterization, the conditioning is performed on the visual features of each network layer. Moreover, they are applied to the image recognition task. While CondHead aims to bridge the perception gap between the base training object categories and target novel object categories by conditioning on the strongly visual-aligned semantic embedding. CondHead is applied to the more challenging bounding box regression and mask segmentation task, which is not explored before.

Partially Supervised Instance Segmentation Recent works [13, 18] explore partially supervised instance segmentation, where the base categories have both box and mask annotation while the novel categories have only box annotation. Mask^X R-CNN [13] propose a parameterized transformation function to transfer the box regression weights to mask segmentation weights. Shapemask [18] explores strong shape priors to achieve better class-agnostic object segmentation, which generalizes better to novel categories without mask annotation. Our focused open vocabulary object detection and instance segmentation are much more challenging than the partially supervised instance segmentation, which is based on good detection quality and strong object region representation. Mask^X R-CNN [13] cannot be applied here as box annotation is not available. Shapemask [18] could be applied here, we compare to it for instance segmentation and evaluate an augmented version of CondHead with Shapemask.

Setup Since the prior works mainly focus on the classification of novel object categories, we evaluate CondHead by adopting the prior state-of-the-art open vocabulary methods and treating them as baselines. We employ the following methods with increasing quality of semantic-visual alignment:

• •

  OVR-CNN [31] proposes to pre-train the multi-modal encoder with image-caption pair data. The pre-trained rich visual-semantic representation is then transferred to recognize the objects with open vocabulary.

• •

  ViLD [11] achieves much higher performance than OVR-CNN by transferring the stronger visual-semantic representation of CLIP [22], which was pre-trained on a much larger scale of image-caption data.

• •

  RegionCLIP [32] further improves the alignment between the semantic and visual embeddings by conducting region-wise pre-training. The improved representation further improves the recognition of novel objects and thus brings significant performance gain.

Results on COCO We re-evaluate the baseline methods by adding class-agnostic mask segmentation heads. Then we replace the bounding box regression and mask segmentation heads with the proposed CondHead and re-run the experiments. As shown in Table 2, our method surpasses all baseline methods, especially on the novel object category set. We observe the improvements grow with the three baselines, e.g., the object detection AP improvements are 1.4 with OVR-CNN, 2.1 with ViLD, and 2.4 with RegionCLIP. A similar observation holds for the instance mask segmentation task. This phenomenon verifies that CondHead can leverage semantic embeddings that better align with visual representations to achieve stronger bounding box regression and mask segmentation ability.

Results on LVIS Following the evaluation on COCO, We then evaluate CondHead on LVIS dataset. As shown in Table 3, our method brings about 2.0-3.0 absolute improvement on the target novel object categories (i.e., AP_r), for both bounding box detection and instance segmentation. The improvement holds for stronger backbone networks. We also observe a similar growth of improvement as COCO, i.e., the RegionCLIP method gains more improvement with CondHead than that of ViLD. This is likely because RegionCLIP provides stronger semantic-visual alignment with its region-wise pre-training, and thus CondHead better generalizes to the target categories.

Cross Dataset Evaluation We further evaluate CondHead with the cross dataset scenarios, where the open vocabulary object detector is expected to generalize on datasets that are different from the one used for training. We simply replace the vocabulary of a trained model with target dataset vocabulary to evaluate the performance. As shown in Table 4, we observe the improvement of CondHead transfers well. For example, 3.5, 2.4, and 2.2 absolute improvements in AP${}_{5}0$ for COCO, PASCAL VOC, and Objects365 respectively.

Qualitative Results To demonstrate how $CondHead$ improves bounding box regression and mask segmentation, we visualize the example qualitative results and compare that with the baseline method. As shown in Figure 3, CondHead is better at refining the proposal box and segmenting the target objects. For example, the baseline ViLD wrongly regresses the target chessboard to the table, while CondHead correctly predicts the chessboard (Figure 3 first column). In addition, CondHead better segments the challenging sweat pants while ViLD predicts inferior segmentation mask (Figure 3 last column).

The Effects of Language Descriptions We examine how CondHead is affected by language descriptions. As shown in Figure 4, when the input text is for a wrong category label, the box regression deteriorates, e.g., the proposal for sunflower is misguided to predict the carrot. On the other hand, it is interesting that when supplied with manually tuned descriptions, the detection and segmentation quality is improved, e.g., when employing snowboard and motorcycle as descriptions, the snowmobile is predicted better. This intriguing observation suggests CondHead learns strong category-conditioned prediction and a better performance could be achieved by simply tuning the semantic descriptions during inference.

Naive Class-wise Transfer We also establish another intuitive experiment as discussed in Section 1 to demonstrate the effectiveness of the proposed method. Since our intuition (Figure 1) is that the class-specific knowledge could be shared across categories, we train the class-wise heads on the base categories and then perform inference on the target categories in a semantically guided way. This is achieved by gathering the class-specific predictions with semantic similarity to the base categories. We use cosine similarity on the semantic embedding and select the category with the highest similarity. As shown in Table 5, we train the ClassWise baseline on both ViLD and RegionCLIP. ClassWise method provides a small improvement to class-agnostic heads compared to CondHead, demonstrating that CondHead better generalizes to the target categories.

Augmenting CondHead with Shape Priors As discussed in Section 3.3, we compare to Shapemask [18] which introduces explicit class-agnostic shape priors. We then explore an augmented version of CondHead by integrating the Shapemask, the semantic embedding is utilized to estimate the initial shape prior, then the dynamic expert aggregation is conducted on the latter coarse mask estimation and refinement. Please refer to the supplementary for more implementation details. As shown in Table 6, Shapemask brings comparable improvement to CondHead, this is attributed to its strong prior and complex multi-stage mask refinement capability. Further integrating Shapemask into CondHead offers larger gain, e.g., 3.5 and 3.6 AP improvement for the two baselines. This implies CondHead can be improved with explicit shape priors.

Component Analysis As shown in Table 7, we perform an ablation study by applying CondHead on the box and mask heads independently, we find they improve box AP and mask AP independently. We also observe CondHead introduces a very small computation overhead. To understand how the number of expert heads and aggregation hyper-parameter affects CondHead, a detailed analysis is conducted as shown in Figure 5. We find the performance gain is diminishing beyond 40 expert heads and the hyper-parameters $\lambda$ and $\mu$ work best at between 0.5 and 0.7. More analysis experiments are presented in the supplementary.