Phrase Decoupling Cross-Modal Hierarchical Matching and Progressive Position Correction for Visual Grounding

Two-stage visual grounding first uses a pre-trained object detection model to generate region proposals and then selects the proposals that best match the textual description. In the first stage, detectors such as Fast R-CNN [31] or Mask R-CNN [32] are commonly used. In the second stage, proposals are matched with the provided textual description, and the proposal with the highest matching score is selected as the prediction output. To ensure accurate matching, a typical approach is to model the relationships between proposals and language descriptions by introducing a graph structure and transforming the visual grounding task into an optimal matching problem of graph nodes [33, 34, 35, 36]. However, such methods do not consider the impact of syntactic structures on text features, which limits the expressive power of text features. Parsing and understanding the semantics of sentences using syntactic tree structures and then matching them with proposals can alleviate this issue [20, 21]. Moreover, MattNet [19] employs a modular design strategy to describe the object in the text, and its position and relationship with the subject. To enhance the cross-modal feature matching capability, CM-Att-Erase [37] generates challenging samples by erasing words from the text or image content, emphasizing the remaining information’s role in cross-modal matching. Two-stage methods leverage object detection models to locate described objects, mitigating the challenges of blind matching between text and target regions. However, those methods require high precision of the object detection results. When the object detector fails or generates inaccurate region proposals, it may directly impact the subsequent matching of text and targe features.

The one-stage visual grounding typically involves 1) fusing visual and textual features, 2) partitioning the fused features into grids, and 3) performing position regression for each grid. This approach eliminates the reliance on object detectors in cross-modal matching. For example, FAOSA-VG [38] concatenates visual and textual features and feeds them into YOLOv3 [39] to regress the bounding boxes of the target objects. ReSC [40] splits the textual description using attention mechanism and performs multiple rounds of inference between the image and text to gradually reduce referring ambiguity, addressing the limitations of previous one-stage methods when understanding complex queries. RCCF [41] treats visual grounding as a correlation filtering process, mapping text features from the language domain to the visual domain and using the mapping results as templates (kernels) to filter image features to generate the center positions and sizes of target boxes.

Recently, the Transformer architecture has been extensively studied. It was initially introduced in [42] for neural machine translation. Influenced by the successful application of the self-attention mechanism, researchers have further explored applying the Transformer framework to the vision tasks, such as image classification [43, 44],action recognition [45, 46, 47, 48], and visual grounding [27, 49, 50, 51, 23, 52], among other tasks. In visual grounding tasks, transformers are used to enhance the interaction between vision and text, thereby improving visual grounding performance. Specifically, TransVG [27] uses a series of simple transformer encoders for multimodal fusion and inference in visual grounding, avoiding the need for complex handcrafted fusion modules. Dynamic MDETR [51] decouples the whole grounding process into encoding and decoding stages and uses sparse priors on object locations to develop a dynamic multimodal transformer decoder to improve the efficiency of the visual grounding process. CLIP-VG [52] employs adaptive curriculum learning and a simple end-to-end network to achieve visual grounding based on CLIP [53]. Word2Pix [23] utilizes a one-stage visual grounding network with an encoder-decoder transformer structure to achieve attention learning from words to pixels. Although these methods are effective, they do not match the text with the described objects at different hierarchies, failing to highlight the association between text and image. This limits further improvement in matching performance. The proposed method establishes a connection between text and image features across hierarchies within a one-stage framework, in which a progressive refinement of the target regions can be achieved.

The visual grounding model proposed in this paper comprises four components: Global Feature Cross-Modal Alignment (GFCMA), Hierarchical Mask Generation (HMG), Cross-Modal Hierarchical Matching (CMHM), and Progressive Position Correction (PPC), as shown in Fig. 2. GFCMA primarily focuses on establishing global relationship between text and image, providing global information for the subsequent creation of hierarchical association. HMG parses phrases from the input text and uses them to create masks for CMHM. Driven by the hierarchical mask, CMHM achieves cross-modal feature hierarchical matching, allowing features to be associated across different hierarchies. PPC uses the hierarchical matching results to progressively correct the target object position. In the following sections, we will elaborate on each component.

GFCMA mainly comprises a visual encoding branch and a textual encoding branch (Fig. 2). Assuming that the source image fed into the visual encoding branch is $v$, its corresponding sentence is $s$. $v$ is first input into ResNet101 to extract shallow features $\hat{\mathbf{F}}_{v}\in\mathbb{R}^{C\times H\times W}$. $C$, $H$, and $W$ denote the number of channels, height, and width of $\hat{\mathbf{F}}_{v}$, respectively. For each pixel position of $\hat{\mathbf{F}}_{v}$, the feature map is vectorized to obtain the feature $\mathbf{F}_{v}=\left[\mathbf{f}_{v}^{1},\mathbf{f}_{v}^{2},\cdots,\mathbf{f}_{v}^{N_{v}}\right]$, composed of tokens $\mathbf{f}_{v}^{i}\in\mathbb{R}^{C\times 1}(i=1,2\cdots,N_{v})$, where $N_{v}=H\times W$. $\mathbf{F}_{v}$ is input into the Transformer layer, resulting in $\mathbf{F}_{v,t}=\left[\mathbf{f}_{v,t}^{1},\mathbf{f}_{v,t}^{2},\cdots,\mathbf{f}_{v,t}^{N_{v}}\right]$, where $\mathbf{f}_{v,t}^{i}\in\mathbb{R}^{C\times 1}(i=1,2\cdots,{N_{v}})$. For textual branches, assume that the features extracted from text via the Transformer layer are $\mathbf{F}_{s,t}=\left[\mathbf{f}_{s,t}^{1},\mathbf{f}_{s,t}^{2},\cdots,\mathbf{f}_{s,t}^{N_{s}}\right]$, composed of tokens $\mathbf{f}_{s,t}^{j}\in\mathbb{R}^{C\times 1}(j=1,2\cdots,N_{s})$, where $N_{s}$ is the number of textual tokens.

As shown in Fig. 3, $\mathbf{F}_{v,t}$ and $\mathbf{F}_{s,t}$ pass through a cross-modal attention layer and a cross-modal global alignment layer to obtain aligned coarse-grained features. $\mathbf{F}_{v,t}$ is utilized to highlight the corresponding features between $\mathbf{F}_{s,t}$ and $\mathbf{F}_{v,t}$. Specifically, in the cross-modal attention layer, $\mathbf{F}_{v,t}$ is transformed into the query vector $\mathbf{Q}_{v,t}$ via linear mapping, while $\mathbf{F}_{s,t}$ is linearly mapped to both the key vector $\mathbf{K}_{s,t}$ and the value vector $\mathbf{V}_{s,t}$. The semantic feature $\mathbf{F}_{s,t}^{e}$ is then obtained through multi-head attention (MH Attn), with each token in $\mathbf{F}_{s,t}^{e}$ incorporating global information from the text.

Furthermore, in the cross-modal global alignment layer, the visual feature $\mathbf{F}_{v,t}$ and the semantic feature $\mathbf{F}_{s,t}^{e}$ are linearly mapped into the same semantic space. The resulting mapped features are denoted as $\mathbf{F}_{v,t}^{\prime}$ and $\mathbf{F}_{s,t}^{\prime e}$, respectively. Then, we can establish a global relationship between the tokens in $\mathbf{F}_{s,t}^{\prime e}$ and the corresponding tokens in $\mathbf{F}_{v,t}^{\prime}$, highlighting the feature tokens of the targeted object described in the text, without the need for pairwise feature metrics. Specifically, the similarity between the corresponding tokens of $\mathbf{F}_{v,t}^{\prime}$ and $\mathbf{F}_{s,t}^{\prime e}$ can be measured as:

By using the following formula, $s_{v,s}(i)$ is normalized to $[0,1]$, resulting in $\hat{s}_{v,s}$:

where $\lambda^{*}$ is the inverse temperature of the softmax function. Each value $\hat{s}_{v,s}$ indicates the degree of relevance between the visual token and the corresponding textual token. Leveraging $\hat{s}_{v,s}$, we obtain the globally aligned cross-modal features:

The cross-modal global alignment layer emphasizes the related features between images and text, suppressing the non-related ones. This provides support for subsequent hierarchical feature matching.

There is a hierarchical correspondence between the text phrase and the potential target area. That is, the more phrases describing the target object, the more accurate the target object position in the image. With this hierarchical association, we devise a feature cross-modal hierarchical matching scheme. In detail, we propose a hierarchical mask generation mechanism to achieve hierarchical matching of features. For texts that require phrase decoupling, we employ the phrase decoupling model from [54] to extract related noun phrases, verb phrases, and adjective phrases, etc. Take the text “white and black cat laying on orange cat” as an example. After model decoupling of this text, and we can obtain four phrases: “white and black cat” (noun phrase), “laying” (verb phrase), “on” (prepositional phrase), and “orange cat” (noun phrase). We assign a specific binary positional encoding to each phrase $\mathbf{P}=\left\{\mathbf{p}_{1},\mathbf{p}_{2},\ldots,\mathbf{p}_{L}\right\}$, where $\mathbf{p}_{i}=[\mathbf{0},\ldots,\underset{i}{\mathbf{1}},\ldots,\mathbf{0}]\quad(i=1,2,\ldots,L)$ denotes the position encoding of the $i$-th phrase, $\mathbf{0}\in\mathbb{R}^{1\times N_{i}}$ is a vector entirely constituted of 0, $\mathbf{1}\in\mathbb{R}^{1\times N_{i}}$ is a vector fully made of 1 and $N_{i}$ is the number of words that make up the current phrase. The mask at the $j$-th hierarchy is:

In the paper, we present CMHM for cross-modal matching, which relies on the generated hierarchical mask, as shown in Fig. 4. CMHM consists of multiple CMHM layers, each including a Hierarchical Mask Attention (HM Attn) and a Multi-head Attention (MH Attn). HM Attn is used to extract features consistent with hierarchical masks, while MH Attn highlights cross-modal consistent features. The number of hierarchies formed by the textual phrases corresponds to the number of layers in CMHM. For the $l$-th CMHM layer, the hierarchical mask can be represented as $\bm{M}_{l}$, which consists of $[\mathbf{1},\cdots,\underset{l}{\mathbf{1}},\mathbf{0},\cdots,\mathbf{0}]$. Assuming the input features to this layer are $\tilde{\mathbf{F}}_{g,v}^{l-1}$, $\mathbf{F}_{v,t}$ and $\mathbf{F}_{s,t}$, where $\tilde{\mathbf{F}}_{g,v}^{l-1}$ is the output of the $(l-1)$-th CMHM. When $l=1$, $\tilde{\mathbf{F}}_{g,v}^{0}=\mathbf{F}_{g,v}$. By performing linear mapping, $\tilde{\mathbf{F}}_{g,v}^{l-1}$ is transformed into the query vector $\mathbf{Q}_{h,v}^{l}$, and $\mathbf{F}_{s,t}$ is mapped to both the key vector $\mathbf{K}_{h,t}^{l}$ and the value vector $\mathbf{V}_{h,t}^{l}$. Thus, the resulting output by HM Attn can be represented as:

where $\mathbf{W}_{l}=\frac{(\mathbf{Q}_{h,v}^{l})^{T}\mathbf{K}_{h,t}^{l}}{\sqrt{d_{2}}}\odot\mathbf{M}_{l}$, $d_{2}$ indicates the dimensionality of $\mathbf{K}_{h,t}^{l}$, $\odot$ denotes the element-wise multiplication. $\lambda$ is a modulating factor greater than $1$. It is used to retain the visual information corresponding to the current hierarchical text query while appropriately weakening the visual information corresponding to the previous hierarchical text query. This ensures that the model does not overly rely on the previous hierarchical information during hierarchical cross-modal matching. Instead, it can fully utilize the new information of the current hierarchy in addition to referencing the previous hierarchical information.

To enhance the consistent information between $\mathbf{F}_{m}^{l}$ and $\tilde{\mathbf{F}}_{g,v}^{l-1}$, they are summed to produce:

After different linear mapping, the results of $\tilde{\mathbf{F}}_{mg,v}^{l}$ serve as Query and Key, and the visual feature $\mathbf{F}_{v,t}$ serves as Value in MH Attn, yielding an enhanced feature $\tilde{\mathbf{F}}_{g,v}^{l}$. Guided by the hierarchical text masks, we achieve a fine-grained interaction between textual and visual features. This interaction allows $\tilde{\mathbf{F}}_{g,v}^{l}$ to incorporate hierarchical textual information consistently. This addresses the limitations of coarse-grained matching and leverages the association across different hierarchies to boost cross-modal matching. It is worth noting that the context between words is retained by segmenting the text at the phrase level, allowing the model to understand these relationships and further encourage its fine-grained matching more comprehensively.

With the guidance of hierarchical matching, we develop a novel PPC module for target localization, which achieves precise localization by continuously refining the target’s position. As illustrated in Fig. 2, the PPC module consists of $L$ PPC layers. The PPC layer comprises Hierarchical Semantic Aggregation (HSA) and Hierarchical Position Correction (HPC), as depicted in Fig. 5. HSA is used to aggregate both textual and visual feature at the current hierarchy, consisting of HM Attn and MH Attn. HM Attn is used to extract features consistent with hierarchical masks, while MH Attn is used to highlight cross-modal consistent features. HPC is used to refine the position information of the target object from the previous layer by utilizing the target’s location data at the current layer, thereby strengthening the connections across layers.

Assume that the inputs of the $l$-th PPC layer are the target query $\tilde{\mathbf{Q}}^{l-1}$, textual features $\mathbf{F}_{s,t}$, hierarchical masks $\mathbf{M}_{l}$, visual features $\mathbf{F}_{v,t}$, and $\tilde{\mathbf{F}}_{g,v}^{L}$, and the output is $\tilde{\mathbf{Q}}^{l}$. In the first PPC layer, a random vector is used as $\tilde{\mathbf{Q}}^{0}$. In the $l$-th PPC layer, the HSA’s output is denoted as $\mathbf{Q}^{l}$. Since $\mathbf{Q}^{l}$ incorporates both textual and visual feature from the current layer, it carries more information about the target object compared to $\tilde{\mathbf{Q}}^{l-1}$. These information are directly related to the positional information of the target object. Consequently, $\mathbf{Q}^{l}$ can be used to refine the predicted position information from the previous hierarchy. Let $\mathbf{b}^{l-1}$ be the bounding box position predicted in the $(l-1)$-th hierarchy, $\mathbf{b}^{l}$ (when $l=0$, $\mathbf{b}^{0}$ is a position vector consisting of 0) be the corrected bounding box position, then the change $\mathbf{b}_{\Delta l}$ from $\mathbf{b}^{l-1}$ to $\mathbf{b}_{l}$ can be predicted by:

where $\sigma$ represents the sigmoid activation function, and MLP represents a multi-layer perceptron. In this case, the bounding box position predicted in the current stage can be expressed as:

In HPC, the current position information plays a crucial role in predicting the position at the next hierarchy. Therefore, we embed the position information predicted in the current stage into $\mathbf{Q}^{l}$ to assist the position predicting of the next hierarchy. Suppose the coordinate of $\mathbf{b}^{l}$ is $\left[x_{l},y_{l},w_{l},h_{l}\right]$, where $\left(x_{l},y_{l}\right)$ denotes the center point coordinate of the bounding box, $w_{l}$ and $h_{l}$ respectively denote the height and width of the box. Since the dimensions of $\mathbf{b}^{l}\in\mathbb{R}^{1\times 4}$ and $\mathbf{Q}^{l}\in\mathbb{R}^{1\times C}$ are not consistent, we adopt the method of [55] to encode the position of $\mathbf{b}^{l}$, then project it to the same dimension with $\mathbf{Q}^{l}$, resulting in $\mathbf{Q}_{b}^{l}\in\mathbb{R}^{1\times C}$:

where PE is the sine positional encoding function, $[.]$ represents concatenation along the channel dimension, and $\theta$ represents a fully connected layer. By adding $\mathbf{Q}_{b}^{l}$ to $\mathbf{Q}^{l}$, we obtain the target query for the $l+1$ hierarchy:

We perform $N$ iterations on the PPC module. The output $\tilde{\mathbf{Q}}_{n}^{L}$ of the $L$-th PPC-layer at $n$-th iteration is used to predict the bounding box $B_{n}$ corresponding to the textual reference. $B_{n}$ can be generated as:

where MLP represents a multi-layer perceptron. We use the Generalized IoU loss ($l_{g}$) [67] and $l_{1}$-loss to constrain $B_{n}$ in each iteration. The details are as follows:

where $B_{gt}$ represents the ground truth, $\lambda_{1}$ and $\lambda_{2}$ are balancing weights and $N$ is the total number of iterations. Let $\mathbf{b}_{n}^{L}$ be the position predicted by Eq.(8) at the $L$-th hierarchy and $n$-th iterations. Additionally, we use the $\ell_{1}$ to make $B_{gt}$ and $\mathbf{b}_{n}^{L}$ consistent:

The entire network is trained in an end-to-end manner, and model parameters are optimized using the total loss $\mathcal{L}_{\text{total }}$:

Datasets. The proposed method is evaluated on visual grounding datasets, including RefCOCO [68], RefCOCO+ [68], RefCOCOg [69] and ReferItGame [70]. Those datasets consists of images and their corresponding referring expressions.

RefCOCO/RefCOCO+/RefCOCOg The RefCOCO [68], RefCOCO+ [68] and RefCOCOg [69] datasets are created by selecting images and target objects from MSCOCO [71]. The RefCOCO dataset consists of $19,994$ images, which contains $142,210$ referring expressions for $50,000$ target objects. The RefCOCO+ dataset contains $19,992$ images with $141,564$ referring expressions for 49,856 objects. RefCOCOg includes $25,799$ images and $95,010$ expressions for $49,822$ objects. For the RefCOCO and RefCOCO+ datasets, they are divided into four sections: train, val, testA, and testB. Among them, testA focuses on images containing multiple persons, while testB involves images with a variety of non-human objects. The images in the RefCOCO and RefCOCO+ datasets contain multiple objects of the same category. Moreover, in RefCOCO+, the referring expressions exclude the words that denote absolute position (such as ’left’). The RefCOCOg dataset is divided into train, val, and test sections, with its images typically containing $2$ to $4$ objects of the same category. In addition, the average text length of RefCOCOg is $8.43$ words, which is significantly longer than that of the RefCOCO and RefCOCO+ datasets ($3.61$ and $3.53$ words, respectively)

ReferItGame The ReferItGame dataset [70] is created by selecting images and target objects from SAIAPR12 [72]. It contains $120,072$ expressions for $19,987$ objects within $20,000$ natural scene images. The ReferItGame is divided into train, val, and test sets.

Evaluation Protocol. We follow the method used in the work of [27] to evaluate the detection performance. A prediction is deemed correct and denoted as [email protected] when the Intersection over Union (IoU) between the predicted bounding box and its Ground Truth surpasses the threshold of $0.5$.

In this paper, the maximum text length is set to $40$ words and the input images are resized to $640\times 640$. For visual encoding branch, we use ResNet-50/101 [73] and $6$ transformer layers to extract visual features. For textual encoding branch, the basic BERT-base-uncased model [55] is used for initialization. We employ the phrase decoupling model from [54] for phrase decoupling. During training, we use the AdamW [74] optimizer with an initial learning rate of $10^{-4}$ and batch size of $64$ for model training. We set $\lambda_{1}=5$ and $\lambda_{2}=1$ in the loss function described in Eq.(13). To ensure a fair comparison with more methods, we added ViT-Base[44] and Swin-Base[75] as visual backbones, while keeping the text encoder unchanged. Inspired by the large-scale pre-training visual language models, we use the visual-language model CLIP [53] as the backbone of the proposed method. To improve training stability, we fix the CLIP encoder and introduce an MLP layer for feature modulation. All other settings are consistent with the original configurations. The training is conducted on four RTX4090 GPUs for $90$ epochs.

The experimental results of different methods on the ReferItGame, RefCOCO, RefCOCO+ and RefCOCOg datasets are compared in Table I. Those comparative methods are divided into two-stage and one-stage methods. The proposed method belongs to the one-stage methods. Compared to the one-stage methods using ResNet-50 or ResNet-101 as the backbone, the proposed method outperforms all the comparison methods. Specifically, when we use ResNet-50 as the visual backbone, the proposed method achieves a $3.07\%$ performance improvement on the RefCOCOg validation dataset compared to the D-MDETR[51]. Additionally, when using ResNet-101, the proposed method achieves a $3.52\%$ increase on the RefCOCO+ testB dataset compared to the VLTVG[3]. When we replace the ResNet-101 with ViT-base, the proposed method achieves a $8.88\%$ improvement compared to JMRI[66] on the RefCOCO+ testB dataset. Additionally, when we use Swin-Base as the visual backbone, the proposed method achieves a $3.25\%$ improvement compared to LUNA[65] on the RefCOCOg validation dataset. Furthermore, we compare the proposed method with CLIP-VG which is developed based on a large-scale visual language model. For a fair comparison, we replace the backbone in the proposed method with CLIP. As shown in Table I, the proposed method outperforms the CLIP-VG, demonstrating a significant enhancement of $4.73\%$ on the RefCOCO+ validation dataset. Additionally, it exhibits advancements of $2.97\%$ on the ReferItGame test dataset and $5.4\%$ on the RefCOCO+ testB dataset, respectively. It can be attributed to the effectiveness of the proposed hierarchical relationship mining and matching. Moreover, the proposed model using ResNet-50 as its backbone achieves the fastest speed, with an inference time of $29ms$ when evaluated under equivalent conditions.

In Fig. 6, we compare the results predicted by the proposed method, CLIP-VG [52] and Baseline. The red box represents the ground truth. The green and yellow boxes denote the results predicted by the proposed method and Baseline, respectively. The blue dashed box denotes the prediction results of CLIP-VG. Fig. 6(a) shows the prediction results when the query images and simple referring expressions are fed to the models. It can be seen that all the methods can focus well on the target objects when the hierarchy of the referring expressions is less. However, as the hierarchy of the referring expressions increases, a significant discrepancy emerges between the labels and the prediction results of both the Baseline and CLIP-VG, as shown in Fig. 6(b). In contrast, the proposed method consistently sustains high localization precision. It is attributed to our method’s ability to mine the hierarchical relations of text, thereby facilitating precise localization of the target objects. Additionally, Fig. 6(c) shows some failure cases for our method, CLIP-VG and the Baseline. In the first two test cases of Fig. 6(c), we observe that our model and other methods struggle to accurately locate text queries containing numbers. This difficulty arises from the models’ challenges in recognizing numerical information. In the last two failed test cases of Fig. 6(c), the ambiguous text queries lead to our method, CLIP-VG and the Baseline predicting boxes that diverge from the ground truth annotations.

The proposed network mainly includes GFCMA, CMHM and PPC modules. We conduct the ablation study on ReferItGame dataset to evaluation their contribution. We remove the cross-modal global alignment layer from the GFCMA module, and the residual part of the GFCMA module is used as the Baseline model. This network is trained with $l_{1}$ loss and generalized IoU loss. The results of ablation experiments are presented in Table II.

Effectiveness of GFCMA. As shown in Table II, when the GFCMA is added into the Baseline, the accuracy on [email protected] is improved by $2.96\%$ and $2.59\%$ on validation and test sets, respectively. The accuracy increases due to that the GFCMA focuses on global relationships between text and image, providing global information for subsequent fine-grained interactions.

Effectiveness of CMHM. When CMHM is added into Baseline+GFCMA, the performance of Baseline+GFCMA+CMHM is further improved on validation and test sets. It implies that the CMHM can mine correlation relationships between different hierarchies in cross-modal matching, and highlight the role of the shared features at different hierarchies.

Effectiveness of PCC. Compared to Baseline+GFCMA+CMHM, the inclusion of PCC results in an increase of $2.82\%$ ($2.86\%$) on the validation (test) set. It is because that the position of the detected bounding box can be gradually corrected by the PCC.

Effectiveness of HPC. To validate the contribution of the HPC in PPC module to the proposed model, the position correction in Eqs.(8)-(11) is removed and $\mathbf{Q}^{l}$ ($l=1,2,\cdots,L$) aggregates textual and visual information layer by layer. The $\mathbf{Q}^{L}$ is used to regress and determine the final bounding box location. The model without hierarchical position correction is denoted as ‘PPC w/o HPC’. As observed in Table II, the addition of HPC results in an increase of $1.21\%$ and $1.38\%$ in [email protected] on the validation and test sets of ReferItGame, respectively. It is proved that the position correction information $\mathbf{Q}_{b}^{l}$ is useful for precise localization of target object.

Analysis of iteration times $N$. In Fig. 7(a), we evaluate influence of the iteration times of PPC. As shown in Fig. 7 (a), the proposed mehtod achieves the best performance when the iteration times $N$ reaches $6$. Therefore, we set $N=6$ throughout the experiment.

Analysis of hyperparameter $\lambda$. In the proposed method, the hyperparameters $\lambda$ used in Eq. (6) is a constant greater than $1$. To determine the value of $\lambda$, Fig. 7 (b) shows the changes of model performance when $\lambda$ takes different values. From those results, we can see that the index [email protected] reaches its maximum when $\lambda=2$.

Analysis of hyperparameter $\lambda_{1}$ and $\lambda_{2}$. In this section, we analyze the impact of the hyperparameters $\lambda_{1}$ and $\lambda_{2}$ in Eq.(13) on detection performance on the test set of the RefCOCOg. Fig. 8 shows the changes of model performance when $\lambda_{1}$ and $\lambda_{2}$ take different values, respectively. It is observed from Fig. 8(a) that when $\lambda_{1}=2$, the [email protected] reaches its peak. Therefore, we set $\lambda_{1}$ to $2$. Moreover, the proposed mehtod achieves the best performance when $\lambda_{2}=5$, as shown in Fig. 8(b). Consequently, we set $\lambda_{2}=5$.

In the proposed method, the number of decoupling phrases in the text corresponds to the number of hierarchies. On the test and validation sets of the ReferItGame and RefCOCOg datasets, we analyze the influence of the number of hierarchies on the prediction accuracy of the proposed method. Fig. 9 describes the effect of the number of hierarchies on model performance. As shown in Fig. 9, as the number of hierarchies increases, the prediction accuracy of the model gradually improves. It indicates that the information extracted from each hierarchical phrase has been successfully leveraged, thereby improving the model performance.

To further understand this phenomenon, we analyze the changes in the attention maps of the CMHM at each hierarchy, as shown in Fig. 10. We find that as the hierarchy of text description increases, the attention received by the target area increases, and the attention received by the non-target area gradually decreases. It is mainly because the features at different hierarchies involve the description of the target object. Therefore, the features referring to the target object between levels are highlighted by exploiting the correlation between different hierarchies, while those related to non-target objects are suppressed. Taking “person in middle above rock” as an example, at the first hierarchy “person”, the model pays attention to the areas where people is located. The text “person” in this hierarchy contains multiple target locations including the target that the complete text refers to. On the second hierarchy ”person in middle”, since both this hierarchy and the previous hierarchy include the target person, in exploiting hierarchical relationships, the “person” related to the target object is highlighted, and target-irrelevant “person” is suppressed. The third hierarchy is ”person in middle above rock”. Since three hierarchies involve the person and two levels involve “person in middle”, the features related to “person” and “middle” are highlighted by the proposed method, and the person corresponding to the middle and rock receives more attention from the network. Therefore, the proposed hierarchical cross-modal matching method can highlight the features of the target object and suppress the features of non-target objects.

To further validate the effectiveness of the hierarchical cross-modal matching method, we conduct visualizations of the coarse-grained and fine-grained tokens, as shown in Fig. 11. It is evident from these visualizations that coarse-grained tokens mainly focus on the rough visual regions corresponding to the text query. After processing through the hierarchical cross-modal matching, the fine-grained tokens are able to focus more precisely on the specific regions specified by text query. This further demonstrates the effectiveness of the proposed hierarchical cross-modal matching method in highlighting the features of the target object and suppressing the features of non-target objects.