Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

The inputs of referring grounding are an image $I$ and a language query $Q$. Referring grounding expects to output a bounding box $P$ of the object described by the query. In this paper, we propose a bidirectional cross-modal matching (BiCM) framework to predict the bounding box.

Our BiCM framework contains four components as shown in Fig. 2: (a) a query-aware attention map (QAM) module that generates attention maps and predicts a candidate bounding box in a top-down manner; (b) a cross-modal object matching (COM) module that leverages CLIP feature space to select bounding boxes from object proposals in a bottom-up manner; (c) a similarity fusion (SF) module that integrates the top-down and bottom-up results; (d) and a knowledge adaptation matching (KAM) module that adapts CLIP knowledge to our target grounding data. We introduce the details of each component below.

The top-down QAM module aims to generate the grounding result from the global image and query. Inspired by Grad-CAM [46] in CNN, we design a back-propagation-based mechanism to extract query-aware visual attention maps in Transformers to achieve this goal.

As shown in Fig. 2 (a), we first input the image $I$ into the pre-trained CLIP visual encoder (ViT [47]) and output the feature vector $\mathbf{fi}$. In ViT, the input image is divided into $U$ patches (tokens) and a special [CLS] token is added to generate $\mathbf{fi}$. Visual attention is extracted from the last block in ViT as follows:

where $h=1,...,H$ and $H$ is the number of heads in the last block. $\mathbf{q}_{h}$ is the attention “query” of the [CLS] token in the $h$-th head, and $\mathbf{k}_{h,u}$ represents the attention “key” of the $u$-th image patch. $d$ is a scaling factor. $att_{h,u}$ is the attention score of the $u$-th image patch in the $h$-th head, which means the importance of this image patch for the final output. It can be used to highlight visually salient regions in the image. However, this attention score cannot find out the regions corresponding to the language query $Q$.

To generate query-aware attention scores, we leverage the pre-trained CLIP textual encoder (GPT-2 [48]) to encode the feature vector $\mathbf{fq}$ of the input query $Q$. Since the image feature vector $\mathbf{fi}$ and the query feature vector $\mathbf{fq}$ are embedded into the same feature space, we calculate their cosine similarity $S^{IQ}$ as

where $<,>$ represents inner product.

We then take the image-query similarity $S^{IQ}$ as a loss and propagate it back into the visual encoder:

where $\alpha_{h,u}$ is the gradient on $att_{h,u}$, which can be regarded as the importance of this attention score for the image-query similarity $S^{IQ}$. Thus, we use it to weight the attention score to find out the most important image patches for the language query as

where $\widetilde{att}_{h,u}$ is the weighted attention score, and ReLU function is used to filter out the negative importance scores. We average scores in all heads as our final query-aware visual attention score $qa_{u}$:

Scores $a_{u}$ for all image patches construct a query-aware visual attention map $A$. We finally upsample $A$ into the original image size by bilinear interpolation and use min-max normalization to normalize it. By empirically setting a threshold $Thr_{a}$, we can select out the desired object and generate the bounding box $P_{t}$.

Our QAM module is able to localize objects from the top-down perspective. Nevertheless, it cannot always capture compact object boundaries. Thus, we further introduce a bottom-up method to identify objects from pre-extracted object proposals.

In particular, an object detection model such as Faster RCNN [49] is first used to generate object proposals $\{P_{r}\}_{r=1}^{N_{r}}$ from the input image $I$, where $N_{r}$ denotes the number of proposals in the image. Then, we leverage the CLIP visual encoder to extract visual feature vector $\mathbf{fp}_{r}$ of each proposal. After that, we compute the similarity $S^{PQ}_{r}$ between the query and each proposal as

This similarity directly compares the visual features of objects with the language query, and thus avoids information loss caused by modality conversion.

In addition, previous works [25, 26] indicate that class names of object proposals are also important information. Therefore, we further calculate the similarity between the query $Q$ and each proposal class name $C_{r}$. We use the CLIP textual encoder to encode $C_{r}$ into the feature vector $\mathbf{fc}_{r}$ and then compute the cosine similarity:

The final bottom-up similarity $S^{BU}_{r}$ between each proposal and the query is defined as the sum of the two similarities:

The proposal $P_{r}$ with the highest similarity could be selected as the desired object.

Benefiting from the pretrained object detection model, our bottom-up COM module can extract compact object boundaries. However, it cannot predict the desired object if the pretrained detection model misses the object. To avoid this, we add our top-down prediction $P_{t}$ to the set of object proposals $\{P_{r}\}_{r=1}^{N_{r}}$ and select the desired object from the combined set $\{P_{1},...,P_{N_{r}},P_{t}\}$. We take the proposal class name $C_{r}$ which has the highest similarity $S^{CQ}_{r}$ with the query as the class name of $P_{t}$.

The SF module is to integrate the similarity scores from the top-down QAM module and the bottom-up COM module, as shown in Fig. 2 (c). For each object proposal in $\{P_{1},...,P_{R},P_{t}\}$, we generate its top-down similarity based on the query-aware visual attention map $A$ as

where $N_{v}$ is the number of pixels in this proposal and $a_{v}$ is the attention score of each pixel $v$ in $A$.

We then fuse the top-down and bottom-up scores as follows:

where $\lambda_{t}$ is the weight to trade off the two scores. The final result is the proposal whose fused score is the highest.

The purpose of this KAM module is to generate pseudo labels from unpaired training data and train a lightweight network to adapt CLIP knowledge to the target dataset and the grounding task. Specifically, given the unpaired image and query sets, we leverage CLIP visual and textual encoders to obtain the image and query features, respectively. For each image, we find out the query with the highest similarity, and treat the image and the query as a pseudo image-query pair. Then, for each image-query pair, we use the above three components to predict a bounding box. If its fused similarity score $S_{r}$ is higher than a threshold $Thr_{k}$, we choose the bounding box as a pseudo label or pseudo ground-truth.

After constructing pseudo labels, we then train a simple MLP (Multi-layer Perception) network, which takes the image features $\mathbf{fi}$, visual object proposal features $\mathbf{fp}_{r}$, object class name features $\mathbf{fc}_{r}$ and query features ${\mathbf{fq}}$ as inputs, and outputs another similarity score $S_{r}^{KAM}$. As shown in Fig. 3, our MLP in KAM consists of three fully-connected layers with batch normalization and ReLU activation. The first layer is used to fuse input features, the second layer is to transform the fused features and the final layer outputs the score $S_{r}^{KAM}$ which is normalized by a Sigmoid function. We adopt the loss in fully-supervised grounding [10] to train the MLP.

During inference, the score $S_{r}^{KAM}$ can be added to $S_{r}$ to select the target object:

where $\lambda_{k}$ is the weight of $S_{r}^{KAM}$.

We evaluate our method on three referring grounding datasets, including the Flickr30K Entities [30], ReferItGame [31], Google-Ref [11] datasets.

The Flickr30K Entities dataset [30] is a phrase grounding dataset. There are 31,783 images and 158,915 descriptions (five descriptions per image) in this dataset. 513,644 phrases in these descriptions describe 275,775 bounding boxes in images. Many phrases (e.g., several people) describe multiple bounding boxes in an image. In this case, following previous methods [25, 26], we merge these bounding boxes and use the union region as the ground-truth. The entire dataset is split into training, validation and testing sets, containing 29783, 1000 and 1000 images, respectively. We use unpaired data in training and validation sets to train our model, and evaluate it on the testing set. The testing set comprises 14,481 phrases for 1,000 images.

The ReferItGame dataset [31], also known as RefCLEF, contains 20,000 images, 9,000 for training, 1,000 for validation and 10,000 for testing. 130,525 phrase expressions describe 96,654 objects in these images. It is a more challenging dataset, because phrase lengths in this dataset are often longer than that in the Flickr30K Entities dataset. Different from the Flickr30K Entities dataset, every phrase in the ReferItGame dataset only describe one object bounding box. We also leverage unpaired data in training and validation sets to train, and use the testing set to estimate the accuracy of our model.

The Google-Ref dataset [11] collects 26,711 images from the MS COCO dataset [50]. There are 54,822 objects and 104,560 referring expressions, which are divided into training and validation, including 44,822 and 5,000 objects, respectively. We use the training set for training, and verify our model on the validation set.

Metrics. We adapt the grounding accuracy to estimate our grounding framework, which is percentage of predictions whose IoU with ground truth is higher than 0.5.

Implementation Details. We use Faster RCNN [49] pretrained on Visual Genome [51] to extract 100 object proposals for each image, and encode 512-dimension visual and textual features. Thresholds $Thr_{a}$ and $Thr_{k}$ are set to 0.7 and 0.9, respectively. We set $\lambda_{t}$ and $\lambda_{k}$ to 1000 and 1, to make the three scores have similar orders of magnitude. Our MLP is trained on one Nvidia RTX 3090 GPU for 50 epochs. Adam optimizer is used for training and the base learning rate is set to 0.0001. On the Flickr30K Entities dataset, because some queries refer to multiple bounding boxes in an image, all methods including ours merge multiple high-score bounding boxes as final results. We merge bounding boxes whose similarities are higher than the average similarity. On other datasets, a query only describes one object in an image. To fairly compared with previous work [25], we select the largest bounding box from above-average-similarity bounding boxes as our prediction.

Table 1 shows results of our and other state-of-the-art methods on the Flickr30K Entities dataset. For a fair comparison, all unpaired-data methods use Faster RCNN pretrained on Visual Genome [51] to extract proposals. Moreover, in [26], results of [25] and [26] on a non-standard testing set with 16,576 phrases are reported. We reproduce these methods on the standard testing set with 14,481 queries. Compared with Wang et al.’s method [25] which only uses class name information, our method achieves improvements by 8.21%. Parcalabescu et al. [26] employ scene graphs to improve referring grounding in their method. Our method outperforms their method by 6.55%.

Results on the ReferItGame dataset are shown in Table 2. It can be observed that our method outperforms [25] and [26] by 15.40% and 12.88%, respectively. Results on the Google-Ref dataset are shown in Table 3, where our method also significantly outperforms previous methods on all sets. These results demonstrate the effectiveness of our BiCM framework.

In Table 1, Table 2 and Table 3, we report results from some fully- and weakly-supervised methods as references. Our method and previous unpaired methods [25, 26] outperform many weakly-supervised methods. A reason is that we introduce external knowledge. Our method also shows competitive accuracy against some fully-supervised methods, such as [52] and [53].

We visualize some grounding results in Fig. 4. It is seen that while previous methods do well at finding objects when the query is a noun (the first row in Fig. 4), they mis-localize many objects for relatively complex queries (e.g., the second and third rows in Fig. 4. The reason is that previous technologies only use class names and scene graphs to compare with queries, while some key information described in these queries are not contained in object class names, such as color and position. Different from previous methods, our method leverages bidirectional matching and knowledge adaptation, and thus avoids these grounding errors. Fig. 5 shows more qualitative grounding results on the Flickr30K Entities and ReferItGame datasets. Compared with previous methods [25, 26], our method finds out the desired objects more accurately.

Effects of main components. We first study the effects of each main component in our framework (see Table 4). Compared with the baseline method, our COM module yields 4.35% improvements, because COM directly analyzes multi-modal data and thus avoids information loss during modality conversion. The performance of our QAM module is lower than the baseline. The reason is that our QAM can localize objects from a top-down perspective but cannot always capture accurate object boundaries. However, when we add the bounding box generated by QAM to COM (i.e., QAM+COM), we achieve higher accuracy than both the baseline and original COM, indicating that our top-down QAM and bottom-up COM are complementary. QAM can find objects which are missed in pre-extracted proposals in COM, while COM is able to select bounding boxes with better boundaries. Our SF module further fuses the top-down query-aware visual attention maps with bottom-up similarity scores, and thus improves the accuracy by 1.72%. The learnable KAM module achieves improvements of 1.31%, thanks to the knowledge adaptation.

BiCM with or without training. The QAM, COM and SF modules in our BiCM do not need any training. Compared with the baseline, our method without training yields 6.90% improvements. Our KAM module extracts pseudo labels from unpaired training data to train an MLP. Our method with KAM outperforms the baseline by 8.21%.

Different attention maps in the top-down module. We show the effects of our query-aware visual attention maps in Table 5. Vanilla visual attention maps are composed by attention scores $\{att_{h,u}\}_{u=1}^{U}$, which is able to highlight visually salient regions but cannot highlight regions corresponding to the language query. Therefore, it only slightly improves the performance, compared with only using COM. Our query-aware visual attention maps find out not only visual salient regions but also query-specific regions, as shown in Fig. 4. Thus, our query-aware visual attention maps achieve significant improvements.

Visualization of query-aware visual attention map. We visualize our query-aware visual attention map in Fig. 6. It can be seen that vanilla visual attention maps only find out visually salient regions (such as the tower in the first image in Fig. 6), while our query-aware maps highlight objects corresponding to different language queries.

Different information in the bottom-up module. Table 6 shows the effects of different information in our COM module. When COM only uses object class names, the performance is higher than the baseline. This is because our textual encoder models the information of the entire query while the baseline method only separately models information of every words. Compared with using object class names, using object visual information shows a lower accuracy. A reason is that class names and queries are in the same modality while visual information and queries are in different modalities. Even though our COM embeds information in different modalities into a same space, comparing single-modal information is still easier than comparing multi-modal information. In addition, many queries only contain one word or several words, which are very similar to class names. Therefore, using class names shows better performance. However, leveraging both class names and visual information obtains the best performance and gains significant improvements (3.51%) compared with using only a single modality. It is because class names lack some important information such as color, posture and so on, which can be provided by vision.

Precision of pseudo labels. To train the MLP in KAM, we extract pseudo labels from unpaired training data. Table 7 shows the precision of our pseudo labels, which is the percentage of correct ones in all pseudo labels we extracted. It can be observed that our precision is 74.7% when the threshold $Thr_{k}$ is 0.9. Under this threshold, we can generate 3,986 pseudo annotations.

Failure cases. We depict some failure cases in Fig. 7. There are two main types of failures. The first is caused by complex reasoning. For instance, in the top two images in Fig. 7, the queries require counting and analyzing numbers, which is hard to be learned by unpaired training data. The second type is inaccurate object boundary, such as the bottom two images in Fig. 7. Our BiCM finds out the desired objects but sometimes fails to capture their boundaries. Using better object detection models can reduce this type of errors.