Toward Fine-Grained 3D Visual Grounding through Referring Textual Phrases

3D visual grounding (3DVG) is an emerging research field. ScanRefer [4] and ReferIt3D [1] are pioneers for the 3DVG, in which both datasets and baseline methods are concurrently proposed. Previous studies [4, 1, 13, 44] formulate 3DVG as a matching problem. They first generate multiple 3D object proposals through either ground truths [1] or a 3D object detector [4], and then fuse 3D proposals with the language embeddings to predict proposals’ similarity scores. The proposal with the highest score will be selected as the target object. Recent works [37, 43, 47, 15] use Transformer [40] architecture to encode the relationship between language and objects’ proposals, where [43] and [37] exploit additional 2D images and orientation annotation during the training stage to boost the performance, respectively. Recently, [48, 3] combine the task of 3DVG with the task of 3D dense captioning to mutually enhance the performance of the two tasks. MVT [14] explores multi-view information in the 3D scene to make a more reliable grounding. Nevertheless, all of these methods exploit the entire sentence to identify a single target object. Such a scheme overlooks the fine-grained relationships between the language and all objects, especially non-target objects, making the grounding process unexplainable as well.

Different from visual grounding which adopts the whole sentence to localize the target object on images [41], phrase grounding is defined as spatial localization with a textual phrase. Existing approaches exploit different techniques to realize phrase grounding on 2D images, where some of them map the visual and textual modalities onto the same feature space [32, 41], and the others aim to reconstruct the corresponding phrase [38]. For the development of architecture, Hinami & Satoh [12] train the object detectors with an open vocabulary, Plummer et al. [31] condition the textual representation on the phrase category. Recent approaches integrate object proposals with Transformer models [22, 28]. Besides, some previous works focus on utilizing unsupervised [38], weakly supervised [46, 42, 6] manners to achieve phrase grounding.

Different from image-based phrase grounding, our 3D phrase aware grounding (3DPAG) has the following properties: 1) In the dataset aspect, sentences used in 2D phrase grounding are generally captions for the entire scene. In contrast, since our datasets are initiated for visual grounding, the majority of sentences are descriptions of the target object. 2) In the methodology aspect, approaches in 2D phrase grounding only take a textual phrase and the corresponding image as inputs. By contrast, we utilize the entire sentence and the 3D scene as inputs to localize the target object, while identifying all the phrase-related objects mentioned in the context is an auxiliary task.

Vision-language pre-training has become a hotspot in cross-modal learning [39, 23, 16, 36]. CLIP [35] uses independent encoders to extract features of language and vision separately and then aligns textual and visual information into a joint semantic space with a contrastive criterion. Existing methods usually pre-train transformer models on large-scale image-text paired datasets [25, 20] and fine-tune on downstream tasks. These approaches have achieved state-of-the-art results across various multi-modal tasks such as visual question answering [21], image captioning [24], phrase grounding [18] and image-text retrieval [45]. Nevertheless, due to the limited number of available 3D data and the difficulty of 3D representation learning, there is no pre-training study for 3DVG till now.

Figure 2 (a) illustrates how we can make a typical architecture for 3D visual grounding accessible to 3D phrase aware grounding (3DPAG). Generally, the network consists of two encoders to independently deal with the object proposals and language descriptions. On the one hand, for the visual encoder, the global features for objects are extracted by a feature extractor such as PointNet++ [34]. Then the fine-grained classes of the global features are predicted. Afterward, the position and size information, i.e., center and bounding box, are embedded into a positional encoding vector through linear transformation, which is subsequently concatenated with global features. On the other hand, the language encoder directly embeds the text input with a pre-trained BERT model [9], where an extra [CLS] token is set to predict the target category through the input sentence. After that, vision and language tokens are fed into a 3DVG model, obtaining updated object and language features, respectively. In practice, previous approaches design different 3DVG models through diverse architectures, such as Transformer [43] and language-guided GCN [44]. All of these methods can be directly adopted in our architecture. Moreover, we add an extra cross-attention layer [40] to fuse the language and object predictions. Thus the obtained attention maps can be used to predict the subsequently introduced phrase-object alignment (POA) map. Finally, the output features are fed into two fully connected layers, producing a set of confidence scores. The object proposal with the highest grounding score is selected as the final grounding prediction.

During the training phase, the phrase-object alignment (POA) map is not predicted directly as depicted in Eqn. (3). Intuitively, we find that the attention map in the last cross-attention layer is a nature POA map, which can be directly optimized. As shown in Figure 2 (b), we extract the multi-head attention maps from the last cross-attention layer and average them along the head dimension. It shares the same property as the POA map described in Eqn. (3). The ground truth (GT) map in Figure 2 (b) can be generated by phrase annotations. Assume there are $M$ object proposals and $L$ language tokens, the GT map $\hat{C}$ should have the shape of $M\times(L+1)$, where the additional dimension in the column represents [NoObj] correspondence.

For each object proposal (i.e., the row of the map), if it is mentioned by any token, the column of the corresponding token should be 1 (0 for other columns). If there is no correspondence, the column of [NoObj] is set to 1. For example, for the last two tokens, “the monitor” in Figure 2, which correspond to the third object, the last two columns in the third row should be set to 1. After that, we optimize the POA map using a multi-class cross entropy (CE) loss. During the inference, the POA map conducts a fine-grained 3DVG. If one wants to know which object corresponds to “the monitor” in Figure 2, the score for each object can be gained by averaging the last two columns and the object that has the highest score is the desired output.

Except for utilizing phrase information in the training stage, we also propose a phrase-specific pre-training strategy, which is demonstrated in Figure 3. The pre-training strategy is trained only on the 3DVG task but can be applied to both 3DVG and 3DPAG models. Specifically, we manually set several phrase-specific masks according to the number of objects mentioned in the sentence for pre-training. Suppose there are $K$ phrases, represented as $P=\{p_{i}\}_{i=1}^{K}$, in a query sentence, we define their phrase-specific masks $G=\{g_{i}\}_{i=1}^{K}\in\mathbbm{1}^{K\times L}$ as follows:

$\displaystyle g_{i,j}=1\ {\text{if}}~{}\ j\text{-th token}\in p_{i}\ {\text{else}}\ 0,$

(4)

where each $g_{i}\in G$ is a phrase-specific mask corresponding to the phrase $p_{i}$. Taking Figure 3 (a) as an example, since there are three objects mentioned in the sentence, i.e., “the office chair”, “the one” (another chair) and “the monitor”, we set three masks ($K=3$) whose length are equal to the number of tokens in the sentence. For each mask, only positions that are related to the specific object are equal to 1. After that, we use these masks as extra positional encoding and concatenate them with language features $E$. In each iteration, we randomly choose one mask $g_{i}\in G$ and object features $O$ to identify the object referred by $p_{i}$ via

$\displaystyle E\oplus g_{i}\mapsto E^{\prime},\ \text{3DVG}(O$

$\displaystyle,E^{\prime})\mapsto S(p_{i}),$

(5)

where $S(p_{i})$ means the model generates confidence scores for the object related to the phrase $p_{i}$. Through such a manner, fine-grained information can be exploited, making the model easier to capture non-target objects.

After pre-training, we fine-tune our model on either 3DVG or 3DPAG tasks, as shown in Figure 3 (b). Specifically, we add a linear layer to predict the mask that is only related to the target object, supervised by the ground truth.

There are two existing datasets for 3DVG, i.e., ScanRefer [4] and ReferIt3D [1].

• -

  ScanRefer [4] consists of 51.5K sentence descriptions for 800 ScanNet scenes [8]. The dataset is split into 36,655 samples for training, 9,508 samples for validation, and 5,410 samples for testing, respectively.

• -

  ReferIt3D [1] uses the same train/valid split with ScanRefer on ScanNet and contains two sub-datasets, where Sr3D (Spatial Reference in 3D) has 83.5K synthetic expressions generated by templates and Nr3D (Natural Reference in 3D) consists of 41.5K human annotations collected similarly as ReferItGame [19].

The 3D web-based UI allows users to select arbitrary texts, and it will automatically save the phrase by recording the start and end positions. When the annotators search for an object, the selected object is highlighted while others are faded out. Meanwhile, a set of captured image frames are shown to compensate for incomplete details of the 3D reconstruction. The position and viewpoint of the virtual camera are allowed for adjusting flexibility to examine the target object better. To ensure the annotation quality, we allow users to select a “not make sure” button during the annotation, and these suspicious sentences will go through a double-check by us. Each sentence will be labeled by two annotators. After labeling, the annotators will verify the results according to each other’s annotations. When labeling ScanRefer dataset, it takes an average of 40 minutes for an annotator to complete a scene. There are 3664 man-hours to complete the whole labeling.

In this paper, we also design a novel metric $Acc_{{\text{PAG}}}$ for the 3DPAG task. Specifically, for each sentence, we simultaneously evaluate the accuracy of a target object and mentioned non-target objects, and the final result is gained by multiplication of two accuracies:

$\displaystyle Acc_{{\text{PAG}}}=Acc_{\text{VG}}\times Acc_{{\text{non-target}}}.$

(6)

For each sentence, $Acc_{{\text{VG}}}$ is 1 if the target object is identified correctly, otherwise 0. $Acc_{{\text{non-target}}}$ is the grounding accuracy of non-target but mentioned objects (excluding the target one). If multiple objects are related by a phrase, we allow the model to choose an arbitrary one (less than 0.1%). The $Acc_{{\text{PAG}}}$ on the whole dataset is the averaged accuracy of all sentences, and thus it can be treated as the weighted version of $Acc_{{\text{VG}}}$. By using this weighted accuracy, we can prevent the model from only predicting the right result through learning data distribution, i.e., $Acc_{{\text{PAG}}}$ equals to 1 only if the model correctly identifies the target object and all non-target objects.

As shown in Table 4, MVT equipping our pre-training strategy (i.e., PSP) achieves the highest results in all metrics, especially on ScanRefer++ dataset which contains more ambiguous phrases as distractors. Specifically, it respectively improves $Acc_{\text{PAG}}$ and $Acc_{\text{PG}}$ by 7.4% and 6.0% compared with naive extension of MVT. Moreover, after phrase-specific pre-training (PSP) with phrase-level annotations, SAT also significantly improves the performance for 3DPAG, increasing 8.6% and 6.2% for PAG and PG.

In this section, we investigate the effect of different components proposed in our paper, as shown in Table 5. The baseline indicates the origin SAT and MVT models, which achieve 49.2% and 55.1% $Acc_{\text{VG}}$ on Nr3D, respectively. Purely exploiting an additional transformer layer leads to a slight performance decrease in both baselines (model A). With POA optimization, model B increases the accuracy to 50.4% and 56.0%, which outperforms model A by 2.1% and 1.2%, respectively. Exploiting phrase-specific pre-training (model C) also improves $Acc_{\text{VG}}$ from 48.3/54.8% to 53.0/56.4%. Both model B and model C demonstrate the effectiveness of phrase-level annotations. Finally, by merging all proposed components, the full architecture obtains the highest score, i.e., 54.4% and 59.0%, beyond our baseline by 5.2% and 3.9%, respectively.