Open-Vocabulary Object Detection via Scene Graph Discovery

The inputs of OV object detection are an image and a number of text, as shown in Fig. 1(b). During inference, the text are usually $C$ candidate object categories. OV detection networks output object proposals (bounding boxes) from the image, and determine the category of each object by matching the object embedding with $C$ candidate category embeddings. During training, the text can be any language description corresponding to objects in the image. By learning with these VL data, OV detection networks are able to recognize various objects.

Previous works (Lin et al., 2023; Li et al., 2022c) only use nouns (i.e., individual objects) from language descriptions, while ignoring other useful information such as object relations. Objects and relations can compose scene graphs, which provide important cues for OV object discovery, classification and localization. A scene graph triplet is formed as ‘subject-predicate(relation)-object’, where ‘subject’ and ‘object’ are two objects and ‘predicate’ is the relation between them. In this paper, we propose an SGDN that exploits scene graphs for OV object detection.

Our SGDN consists of three components as illustrated in Fig. 2. The first is Feature Encoding that extracts the embeddings of the input image and text. The second is SGDecoder to generate embeddings of objects and relations. In SGDecoder, we propose an SSGA module to enrich object embeddings by scene graph information to improve OV object discovery, classification and localization. The final component is SGPred that generates object bounding boxes and categories, as well as relation categories. In SGPred, an SGOR (Fig. 3) is used to mutually refine scene graph extraction and bounding boxes prediction. We also propose cross-modal learning, which takes scene graphs as bridges to enhance the consistency between cross-modal embeddings. Next, we introduce each module in detail.

Image encoder. We leverage the common transformer-based architecture (Zhu et al., 2021) for object detection, where the image encoder includes a backbone encoder (e.g., ResNet (He et al., 2016)) and a transformer encoder (Zhu et al., 2021). The output of our encoder is an image feature map $\mathbf{V}\in\mathbb{R}^{N_{v}\times D_{v}}$, where $N_{v}$ is the number of image patches, and $D_{v}$ is the dimension.

Text encoder. We extract textual embeddings by a pre-trained text encoder (e.g., BERT (Kenton and Toutanova, 2019) or RoBERTa (Liu et al., 2019)). Since we generate both object and relation predictions, our input text contains two parts: object categories and relation categories.

During inference, there are $C+1$ object categories, including $C$ candidate object categories in the target application and an additional ‘no object’ category to recognize false proposals. The text encoder generates a feature map $\mathbf{F}^{o}\in\mathbb{R}^{(C+1)\times D}$ for object categories, in which each feature vector encodes an object category, and $D$ is the feature dimension. Similarly, we have $M+1$ relation categories, i.e., $M$ candidate relation categories in the target application and an additional ‘no relation’ category. $\mathbf{F}^{p}\in\mathbb{R}^{(M+1)\times D}$ represents the output relation category feature map. Note that if an object or relation category contains multiple words, our text encoder can generate one feature vector of the entire category.

During training, we first use language parsing tools (Schuster et al., 2015) to extract nouns and relations of nouns from the language expression. Then, we take all nouns in this expression as the candidate object categories, and also add the ‘no object’ category. All relations in this expression as well as the ‘no relation’ category are treated as our relation categories.

Our method leverages SGDecoder to model scene graphs, and candidate relation categories are only used for relation prediction. If training data (e.g., fixed-set detection data) or target applications only require object detection results, our model can avoid these relation inputs and skip the relation prediction and cross-modal learning parts in SGPred.

After feature encoding, we build an SGDecoder to extract object and predicate embeddings. The inputs of our decoder are $N$ object tokens $\{\mathbf{o}_{n}\in\mathbb{R}^{D}\}_{n=1,...N}$ and a predicate token $\mathbf{p}\in\mathbb{R}^{D}$, where $D$ is the dimension of each token. Each object token $\mathbf{o}_{i}$ represents an object in the image. As investigated in previous scene graph generation works (Shit et al., 2022), the relation and scene graph between the $i$-th and $j$-th objects can be represented by the concatenation of the ‘subject’ embedding $\mathbf{o}_{i}$, the predicate embedding $\mathbf{p}$ as well as the ’object’ embedding $\mathbf{o}_{j}$; and the predicate embedding $\mathbf{p}$ can be shared for all object pairs. Therefore, we only use one relation token to capture object relations. Our decoder generates object and predicate embeddings from these $N+1$ object and relation tokens and the image feature $\mathbf{V}$.

Self-attention. The decoder contains $L$ blocks and each block includes a self-attention, an SSGA module and a cross-attention. The self-attention models long-range dependencies among object and predicate tokens, and updates these tokens as follows:

(1)

$$\begin{split}\{\mathbf{o}^{sl}_{1},...,\mathbf{o}^{sl}_{N},\mathbf{p}^{sl}\}=Attn(&query:\{\mathbf{o}_{1},...,\mathbf{o}_{N},\mathbf{p}\},\\ &key:\{\mathbf{o}_{1},...,\mathbf{o}_{N},\mathbf{p}\},\\ &value:\{\mathbf{o}_{1},...,\mathbf{o}_{N},\mathbf{p}\})\end{split}$$

where $Attn(\cdot)$ is a multi-head attention model (Vaswani et al., 2017). We take our object and predicate tokens $\{\mathbf{o}_{1},...,\mathbf{o}_{N},\mathbf{p}\}$ as queries, keys and values in this attention model. $\{\mathbf{o}^{sl}_{1},...,\mathbf{o}^{sl}_{N},\mathbf{p}^{sl}\}$ are the updated object and predicate tokens, where long-range dependencies are embedded, and every token is also $D$-dimension.

The SSGA module. We propose an SSGA module that further embeds scene graph information into object tokens to improve OV object discovery, classification and localization. Specifically, as shown in Fig. 3, we first generate scene graph embeddings:

(2)

$$\mathbf{g}_{i,j}=[\mathbf{o}^{sl}_{i},\mathbf{b}_{i},\mathbf{p}^{sl},\mathbf{o}^{sl}_{j},\mathbf{b}_{j}],\quad i,j=1,...,N\ and\ i\neq j$$

where $[\cdot]$ means token concatenation. $\mathbf{b}_{i},\mathbf{b}_{j}\in\mathbb{R}^{4}$ are bounding boxes of the $i$-th and $j$-th objects, respectively. We integrate bounding box information to generate more powerful scene graph embeddings. The details of bounding boxes will be introduced in Sec. 3.4. The scene graph embedding $\mathbf{g}_{i,j}\in\mathbb{R}^{3D+8}$ encodes the information of the $i$-th, $j$-th objects as well as their relation. All scene graph embeddings can compose a scene graph matrix $\mathbf{G}\in\mathbb{R}^{N(N-1)\times(3D+8)}$.

Our SSGA module then leverages an attention model to embed scene graphs into object tokens as

(3)

$$\begin{split}\{\mathbf{o}^{sg}_{1},...,\mathbf{o}^{sg}_{N}\}=SAttn(&query:\{\mathbf{o}^{sl}_{1},...,\mathbf{o}^{sl}_{N}\},\\ &key:\mathbf{G},\\ &value:\mathbf{G})\end{split}$$

where we take object tokens as queries, and treat scene graph embeddings as keys and values. $SAttn(\cdot)$ means a sparse attention model. To reduce computational costs, we only calculate attention between each object token and its related scene graph embeddings, i.e., this object acts as the ‘subject’ or ‘object’ in the scene graph embedding. For example, for the $n$-th object token, we only compute attention between $\mathbf{o}^{sl}_{n}$ and $\{\mathbf{g}_{n,j},\mathbf{g}_{j,n}\}_{j=1,...,N\ and\ j\neq n}$. The attention model incorporates scene graph guidance into object tokens and updates object tokes into $\{\mathbf{o}^{sg}_{n}\in\mathbb{R}^{D}\}_{n=1,...N}$. In this way, although our model has not seen some objects in training data, it can discover them from scene graph cues. Moreover, these scene graph cues are also helpful for object classification and localization.

Cross-attention. The cross-attention takes object and predicate tokens as queries, while using the image feature map $\mathbf{V}$ as keys and values to integrate visual information into these tokens:

(4)

$$\begin{split}\{\mathbf{o}^{cr}_{1},...,\mathbf{o}^{cr}_{N},\mathbf{p}^{cr}\}=Attn(&query:\{\mathbf{o}^{sg}_{1},...,\mathbf{o}^{sg}_{N},\mathbf{p}^{sl}\},\\ &key:\mathbf{V},\\ &value:\mathbf{V}).\end{split}$$

The output embeddings $\{\mathbf{o}^{cr}_{1},...,\mathbf{o}^{cr}_{N},\mathbf{p}^{cr}\}$ of each decoder block embed image, object and scene graph information. All embeddings are $D$-dimension vectors and can be further refined by the next decoder block.

Based on our object and predicate embeddings $\{\mathbf{o}^{cr}_{1},...,\mathbf{o}^{cr}_{N},\mathbf{p}^{cr}\}$, three types of results can be predicted, i.e., object bounding boxes and categories, as well as relation categories).

Object bounding box prediction. Let $\mathbf{b}_{n}=[x_{n},y_{n},w_{n},h_{n}]\ (n=1,...,N)$ represent the bounding box for the $n$-th object. $x_{n}$ and $y_{n}$ are the coordinates of the center point in the box, and $w_{n}$ and $h_{n}$ are the width and height of the bounding box. We can use MLPs (multi-layer perceptrons) to predict object bounding boxes $\{\mathbf{b}_{1},...,\mathbf{b}_{N}\}$ from object embeddings $\{\mathbf{o}^{cr}_{1},...,\mathbf{o}^{cr}_{N}\}$.

SGOR. Inspired by prior fixed-set object detection works (Zhu et al., 2021), we build an SGOR mechanism. On the one hand, iterative offset regression generates more accurate bounding boxes than one-step prediction (Zhu et al., 2021). More importantly, we leverage SGOR to allow mutual enhancement between scene graph and object localization.

Concretely, we first initialize $N$ object bounding boxes before the decoding in Sec. 3.3. Here, we use the same initialization as in (Zhu et al., 2021), which is based on deformable attention predictions. Then, in each block in our decoder, we leverage object boxes to enhance scene graphs by Eqn. (2). After each block, new object embeddings $\{\mathbf{o}^{cr}_{l,1},...,\mathbf{o}^{cr}_{l,N}\}$ are generated, where $l=1,..,L$ means the $l$-th decoder block. Our object embeddings include scene graph cues, and we leverage them to refine bounding boxes as follows:

(5)

$$\mathbf{\Delta b}_{l,n}=MLP([\mathbf{o}_{l,n}^{cr},\mathbf{b}_{l,n}])$$

where $\mathbf{b}_{l,n}\in\mathbb{R}^{4}$ is the bounding box of the $n$-th object in the $l$-th decoder block. We concatenate the embedding $\mathbf{o}_{l,n}^{cr}$ and the bounding box $\mathbf{b}_{l,n}$ of this object. After that, an MLP with two linear layers and Sigmoid activation functions is used to predict the box offset $\mathbf{\Delta b}_{l,n}\in\mathbb{R}^{4}$. The bounding box of this object is refined as

(6)

$$\mathbf{b}_{l+1,n}=\mathbf{b}_{l,n}+\mathbf{\Delta b}_{l,n}$$

where $\mathbf{b}_{l+1,n}$ is the refined box and can be used as the bounding box in the next decoder block. For each object, the final bounding box prediction is the refined box $\mathbf{b}_{L+1,n}=\mathbf{b}_{L,n}+\mathbf{\Delta b}_{L,n}$ after the last decoder block.

Object category prediction. We predict object and relation categories based on the object and predicate embeddings $\{\mathbf{o}^{cr}_{1},...,\mathbf{o}^{cr}_{N},\mathbf{p}^{cr}\}$ in the final decoder block. Let $\mathbf{O}\in\mathbb{R}^{N\times D}$ be a matirx composed by object embeddings, i.e., $\mathbf{O}=\{\mathbf{o}^{cr}_{1},...,\mathbf{o}^{cr}_{N}\}$. To predict object categories, we first use two-layer MLPs to refine object embeddings $\mathbf{O}$ into $\mathbf{\widetilde{O}}\in\mathbb{R}^{N\times D}$. Then, we generate a similarity matrix between object and category embeddings:

(7)

$$\mathbf{S}^{o}=\mathbf{\widetilde{O}}(\mathbf{F}^{o})^{T}$$

where $\mathbf{S}^{o}\in\mathbb{R}^{N\times(C+1)}$ is the object category matrix. In $\mathbf{S}^{o}$, each element $s^{o}_{n,c}$ means the similarity between the $n$-th object and $c$-th object category. For each object, we can find the category with the highest similarity as its classification result, and it is a false proposal if its category is ‘no object’.

Relation category prediction and joint learning. For relation prediction, we first leverage Eqn. (2) to generate scene graph embeddings $\mathbf{g}^{p}_{i,j}$ from our final object embeddings $\{\mathbf{o}^{cr}_{1},...,\mathbf{o}^{cr}_{N}\}$, predicate embedding $\mathbf{p}^{cr}$ and bounding boxes $\{\mathbf{b}_{L+1,1},...,\mathbf{b}_{L+1,N}\}$. These scene graph embeddings compose a scene graph matrix $\mathbf{G}^{p}\in\mathbb{R}^{N(N-1)\times(3D+8)}$. Two-layer MLPs are used to transform $\mathbf{G}^{p}$ into $\widetilde{\mathbf{G}^{p}}\in\mathbb{R}^{N(N-1)\times D}$. A relation similarity matrix $\mathbf{S}^{p}\in\mathbb{R}^{N(N-1)\times(M+1)}$ is calculated as

(8)

$$\mathbf{S}^{p}=\widetilde{\mathbf{G}^{p}}(\mathbf{F}^{p})^{T}.$$

Similar to object category prediction, we can determine the relation between every object pair by finding out the most similar relation category. There is no relation between two objects, if the predicted relation category is ‘no relation’. In this way, we can generate OV relation classification results. Moreover, since relation categories are predicted from object embeddings and bounding boxes, we can obtain better object embeddings and object localization results by joint learning with relation training data.

Cross-modal learning. We propose cross-modal learning to further exploit scene graph cues to classify OV objects. As OV detection models formulate object classification as a VL matching problem, the key to accurate classification is to learn consistent object and category embeddings. Therefore, we leverage relation supervision to enhance the consistency between object and category embeddings. Specifically, during training, we replace object embeddings in the scene graph matrix $\mathbf{G}^{p}$ with ground truth object category embeddings. Let $\mathbf{G}^{t}\in\mathbb{R}^{N(N-1)\times(3D+8)}$ represent the replaced matrix. We then use the same two-layer MLPs as in relation category prediction to transform the matrix into $\widetilde{\mathbf{G}^{t}}\in\mathbb{R}^{N(N-1)\times D}$, and generate the relation similarity matrix $\mathbf{S}^{t}\in\mathbb{R}^{N(N-1)\times(M+1)}$ as

(9)

$$\mathbf{S}^{t}=\widetilde{\mathbf{G}^{t}}(\mathbf{F}^{p})^{T}.$$

By learning $\mathbf{S}^{t}$ with relation supervisions, $\mathbf{S}^{t}$ and $\mathbf{S}^{p}$ will be closed, and thus our object and category embeddings will be more consistent. Note that we only use cross-modal learning during the training stage, because we do not have ground truth object categories during inference.

Our training objective contains four parts as follows:

(10)

$$Loss=\lambda_{1}L_{bb}+\lambda_{2}L_{ocls}+\lambda_{3}L_{pcls}+\lambda_{4}L_{cml},$$

where $L_{bb}$, $L_{ocls}$, $L_{pcls}$ and $L_{cml}$ are loss functions for object bounding box prediction, object category prediction, relation category prediction as well as cross-modal learning, respectively. $\lambda_{1},\lambda_{2},\lambda_{3}$ and $\lambda_{4}$ are hyperparameters to weight different losses.

Our bounding box loss $L_{bb}$ is smooth l1 loss. Since SGOR predicts offsets and refines bounding boxes in every decoder block, we calculate a smooth l1 loss in each block, and the final bounding box loss $L_{bb}$ is their sum. Bipartite Hungarian matching is used to align predicted boxes with ground truths.

Similar to previous OV works (Li et al., 2022c), our object category loss $L_{ocls}$ is the sum of binary cross-entropy losses for every element $s^{o}_{n,c}$ in our object similarity matrix $\mathbf{S}^{o}$. In particular, for each element $s^{o}_{n,c}$, we calculate a binary cross-entropy loss between it and the ground truth. The ground truth is 1 when the $n$-th object belongs to the $c$-th category; otherwise, it is 0. Similarly, relation category loss $L_{pcls}$ and cross-modal leanring loss $L_{cml}$ are also binary cross-entropy losses for similarity matrices $\mathbf{S}^{p}$ and $\mathbf{S}^{t}$, respectively.

We leverage referring grounding data for training. A referring grounding sample contains an image, a language and bounding box annotations for every noun. The relation between each object pair can be extracted by language parsing tools, as described in Sec. 3.2. We take them as relation classification ground truths. Fixed-set object detection data can also be used during training. We only use $L_{bb}$ and $L_{ocls}$ for these data, while fixing the predicate token and relation prediction parts.

Following prior works (Lin et al., 2023; Li et al., 2022c), we evaluate the OV ability by zero-shot experiments on COCO (Lin et al., 2014) and LVIS (Gupta et al., 2019), and leverage VL data during training to recognize OV objects.

VL data. Any extra VL can be used in the OV scenario. Here, we employ referring grounding data like previous methods (Li et al., 2022c). Flickr30K Entities (Plummer et al., 2015) includes 31K images with referring expressions and annotations. Visual Genome (Krishna et al., 2016) labels 108K image for referring grounding. We use these 140K training data. Our goal is to verify the effectiveness of our network rather than train a large pre-training model. Thus, we do not use millions of training data. Also, Visual Genome (Krishna et al., 2016) provides scene graph annotations, but we do not use them.

COCO. The COCO 2017 dataset (Lin et al., 2014) contains 120K training and 5K validation images. We use the generalized zero-shot setting (Bansal et al., 2018), which splits COCO into 48 base classes for training and 17 novel classes for validation. We combine 120K COCO training images and 140K grounding data to train our model. There are overlapped images between COCO (Lin et al., 2014) and Visual Genome (Krishna et al., 2016). We remove them and training samples that contain 17 novel classes.

LVIS. There are 100K training and 20K validation images on LVIS (Gupta et al., 2019). Categories in LVIS are divided into 405 frequent, 461 common and 337 rare classes. We combine 886 frequent and common classes for training, while using 337 rare categories for validation. 140K grounding and 100K LVIS training data are mixed during training, where rare classes are removed. Since LVIS (Gupta et al., 2019) also requires mask predictions, we use the external fully convolutional head in (Zang et al., 2022) to generate masks based on decoder embeddings. We also test this LVIS-trained model on COCO to verify the cross-dataset ability.

Metrics. We adopt $AP50$ for COCO (Lin et al., 2014) zero-shot detection, and $mAP$ for LVIS (Gupta et al., 2019) as well as the cross-dataset validation.

We choose RoBERTa (Liu et al., 2019) as our text encoder and add a linear layer to transform the textual feature dimension to $D$. $D$ is set to 512 in our experiments. We fix RoBERTa during training and only update the parameters of the linear layer. We do not use any prompt during training, only the prompt ‘A photo of a [query]’ is used during inference. Our image encoder is Deformable DETR (Zhu et al., 2021) with the ResNet50 (He et al., 2016) backbone pre-trained on ImageNet. Deformable attention is also used in our decoder. The numbers $N$ and $L$ of object tokens and decoder blocks are set to 100 and 6, respectively. We train our model on two stages. VL data are used during the first-stage training, while fixed-set detection data is used in the second stage. $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ and $\lambda_{4}$ are simply set to 1.5, 1.5, 1.0 and 1.0, respectively, and fixed for all datasets. Other network and training settings are the same as Deformable DETR (Zhu et al., 2021). All experiments are conducted on the Pytorch platform (Paszke et al., 2019) with 8 V100 GPUs.

We report the zero-shot results of our model and other state-of-the-art methods on COCO (Lin et al., 2014) in Table 1. Our SGDN outperforms Rasheed et al. (Rasheed et al., 2022), which achieve the best performance for novel classes in previous methods. Rasheed et al. (Rasheed et al., 2022) use the large-scale VL pretraining CLIP (Radford et al., 2021) and add COCO Caption data for training. They also leverage novel-class information during training, which is not practical for OV detection. In prior works without novel-class information, RegionCLIP (Zhong et al., 2022) shows the highest accuracy, which also uses CLIP (Radford et al., 2021) and three million extra data. Compared with it, our SGDN yields improvements of 6.1% for novel classes. GLIP (Li et al., 2022c) also uses referring grounding training data, but it is trained with 27 million data as a VL pre-training. Rather than designing a pre-training, our work aims to provide an architecture for OV detection. Therefore, we reproduce GLIP (Li et al., 2022c) with our 260K training data for a fair comparison. Our SGDN outperforms it by 6.8% for novel classes. We also achieve the best accuracy for all classes.

Table 2 shows the zero-shot results on LVIS (Gupta et al., 2019). We outperform the previous state-of-the-art method VLDet (Lin et al., 2023) by 1.9% for rare classes. Note that, VLDet (Lin et al., 2023) also uses novel-class information during training. Compared with GLIP (Li et al., 2022c), which uses the same VL training data, we achieve improvements of 3.9% for rare classes and 2.8% for all classes.

As several works (Gu et al., 2022; Du et al., 2022; Zang et al., 2022; Cai et al., 2022; Li et al., 2022c) show cross-dataset results to further verify the OV ability, we also report these results in Table 3. Our method outperforms all these methods except the original GLIP (Li et al., 2022c), which uses 27 million training data and a large Swin-L (Liu et al., 2021). Compared with GLIP (Li et al., 2022c) with the same backbone and training data, our SGDN obtains gains of 3.6%. Meanwhile, X-DETR (Cai et al., 2022) also employs grounding data for training, and we outperform it by 14%. All these superior results demonstrate the effectiveness of our scene-graph-based framework, as well as our proposed SGDecoder and SGPred modules.

To further verify the effectiveness of our SGDN, we conduct ablation studies on zero-shot COCO. All models are trained with 260K VL and COCO base data, and use the ResNet-50 backbone.

Scene graph for OV detection. We report the effects of our main components in Table 4. Since we use deformable attention (Zhu et al., 2021), we first build a Deformable-DETR-based OV detection model, ‘Model A’, as our baseline. In ‘Model A’, we add a text encoder to Deformable DETR (Zhu et al., 2021) and replace its classification head with our object category prediction. Then, we design a scene-graph-based ‘Model B’, where we incorporate the predicate token $\mathbf{p}$ as well as the relation category prediction module to ‘Model A’, and extract scene graphs for training. For novel classes, ‘Model B’ outperforms ‘Model A’ and GLIP (Li et al., 2022c) by 2.4% and 1.6%, respectively. These results show the effectiveness of scene graphs for OV object detection.

Main component. Our SGDecoder (‘Model C’) outperforms ‘Model B’ by 2.3% for novel classes, because our SGDecoder with SSGA leverages scene graphs to better embed objects. Our SGPred (‘Model D’) achieves gains of 3.5% and 3.2% for novel and all classes, which demonstrates the effectiveness of our SGOR and cross-modal learning. Compared with ‘Model B’, our final SGDN yields improvements of 5.2% and 4.6% for novel and all classes, respectively.

Dissecting SGDecoder. We then dissect our SGDecoder in Table 6. If we remove SSGA from ‘Model C’, the model is equal to ‘Model B’ and the performance significantly decreases. In ‘Model C w/o box’, we use SSGA but remove bounding boxes from scene graph embeddings. Compare to ‘Model B’, this model achieves improvements of 1.5% for novel classes. The reason is that SSGA better exploits scene graph information for OV detection. Bounding boxes generate gains of 0.8% for novel classes. In ‘Model C w/o sparse’, we employ vanilla deformable attention instead of the sparse one. The performance only slightly increases. However, vanilla attention requires much more computational costs than sparse attention.

Dissecting SGPred. In Table 5, we show the effects of main parts in our SGPred. Our SGOR mutually improves object localization and scene graph embedding, and thus yields gains of 1.5% and 2.3% for novel and all classes. Cross-modal learning increases the performance for novel classes by 1.1%, benefiting from the scene-graph-based cross-modal consistency enhancement.

We show our OV SGDet ability in Table 7. We conduct this experiment on Visual Genome (Krishna et al., 2016), and use the dataset split provided by (He et al., 2022). The training set includes 70% seen object and relation classes in Visual Genome grounding and scene graph data, while the 30% unseen object and relation classes are used for validation. Metrics are $Recall@50$ and $Recall@100$. It can be seen that our SGDN significantly outperforms previous OV scene graph methods on the OV SGCls task and can also predict OV SGDet results.

Qualitative results. Fig. 4 shows qualitative results on COCO. It can be observed that GLIP (Li et al., 2022c) misclassifies some unseen objects, such as the ‘couch’ in the upper right image in Fig. 4. We exploit scene graph information to better recognize OV objects. Normally, the object ‘near’ a ‘TV’ and a ‘chair’ is more like a ‘couch’ than a ‘bed’. Therefore, our approach reduces this classification error. GLIP [18] also misses a number of unseen objects. For example, in the bottom left and right images in Fig. 4, the ‘horse’, ‘umbrella’ and ‘handbag’ are missed by GLIP (Li et al., 2022c). Our SGDN successfully discovers them by exploiting scene graph cues ‘zebra near’ and ‘person holding’. Moreover, SGDN can better localize unseen objects (e.g., the ‘snowboard’ in the upper left image in Fig. 4) based on scene graphs. These results demonstrate the effectiveness of our SGDN for OV object discovery, classification and localization.

OV scene graph detection. Scene graph generation contains three tasks. The simplest is predicate classification (PredCls), where object bounding boxes and classes are provided, and only object relations (predicates) need to be classified. The second is scene graph classification (SGCls). By given object bounding boxes, SGCls expects to classify objects and relations. The hardest is scene graph detection (SGDet), which requires to predict all bounding boxes, object and relation classes. Previous OV scene graph generation methods cannot deal with the OV SGDet task, because they are not able to detect OV objects (He et al., 2022). Different from them, our SGDN can simultaneously detect OV objects and relations, and thus generates OV SGDet predictions.

We visualize OV scene graph detection results in Fig. 5. Since SVPR (He et al., 2022) does not release its source code, we only show the results from our SGDN in Fig. 5. We successfully localize and classify unseen objects, e.g., ‘man’, ‘cat’ and ‘window’. Meanwhile, unseen relations such as ‘sitting on’ are also predicted by our SGDN.