Two-stage Visual Cues Enhancement Network
for Referring Image Segmentation

Retrieval. For a given pair of input image $I$ and language expression $T$, we retrieve an image $I^{sim}$ semantically relevant to the referred object from data pool for enriching visual information of the input image. For an accurate retrieval result, both textual and visual similarities between each candidate sample in the data pool and input are measured. Specifically, an image encoder and a text encoder are first used to generate visual and textual features for input and each candidate sample. For description convenience, we regard input as query, and candidate samples in data pool as keys. Accordingly, the extracted visual feature and text feature of input/candidate are called visual query/key and textual query/key. The visual and textual query pair is denoted as Q = {$\mathbf{q}^{I}$, $\mathbf{q}^{T}$} and i-th visual and textual key pair is denoted as $K_{i}$ = {$\mathbf{k}_{i}^{I}$, $\mathbf{k}_{i}^{T}$}, $i$ $\in$ {1,2,…,N}, where N is the number of samples in data pool. Next, we match the query Q and all N keys $K_{i}$ in the data pool to select the most relevant image $I^{sim}$ for the following referring image segmentation process.

Enrichment. For the input image $I$ together with the retrieved similar image $I^{sim}$, a siamese CNN is adopted to simultaneously extract their visual feature $V^{in}$ and $V^{sim}$. Here, we define the low-resolution visual feature sets as LR = {$V_{3}$,$V_{4}$,$V_{5}$} containing feature maps from res3, res4 and res5 layers, and high-resolution feature sets as HR = {$V_{1}$,$V_{2}$} including feature maps from res1 as well as res2 layers of the siamese CNN. Semantic information exists in the low-resolution feature maps from deeper layers of CNN while the high-resolution feature maps mainly contain region details, such as edge or texture as implied in (Zeiler and Fergus, 2014). In order to incorporate visual information semantically relevant to the referred objects, the enrichment process is performed between the low-resolution visual feature sets of input image $\textit{LR}^{in}$ = {$V_{3}^{in}$,$V_{4}^{in}$,$V_{5}^{in}$} and the retrieved similar image $\textit{LR}^{sim}$ = {$V_{3}^{sim}$,$V_{4}^{sim}$,$V_{5}^{sim}$}.

The details of Enrichment process is depicted in Figure 3. Given visual feature $V_{l}^{in}$ $\in$ $\mathbb{R}^{h_{l}\times w_{l}\times d_{l}}$ and $V_{l}^{sim}$ $\in$ $\mathbb{R}^{h_{l}\times w_{l}\times d_{l}}$, where $h_{l}$, $w_{l}$ and $d_{l}$ are the height, width and channel dimensions respectively, we reorganize the feature representations at every spatial location in $V_{l}^{sim}$ according to their correlation with referent in $V_{l}^{in}$ via a cross-image attention:

$\mathbf{v}_{l}^{in}$ and $\mathbf{v}_{l}^{sim}$ $\in$ $\mathbb{R}^{h_{l}w_{l}\times d_{l}}$ are the reshaped $V_{l}^{in}$ as well as $V_{l}^{sim}$. $W_{1}$, $W_{2}$ and $W_{3}$ are learnable parameters. The cross-image attention is calculated in the embedded Gaussian version (Wang et al., 2018) to get an attention map $A_{l}$ which measures the importance of every region in $\mathbf{v}_{l}^{sim}$ to $\mathbf{v}_{l}^{in}$. Then $A_{l}$ is used for reorganizing the linear transformed $\mathbf{v}_{l}^{sim}$ to generate visual feature $\mathbf{v}_{l}^{sim^{\prime}}$ $\in$ $\mathbb{R}^{h_{l}w_{l}\times{d_{l}}}$, which will be served for boosting $\mathbf{v}_{l}^{in}$. Next, to enhance $\mathbf{v}_{l}$ with $\mathbf{v}_{l}^{sim^{\prime}}$, directly adding or concatenating them might introduce noise, especially in the case of inaccurate retrieved results. Thus, we devise a gate conditioning on the original input visual feature $V_{l}^{in}$ to selectivly incorporate $V_{l}^{sim^{\prime}}$ into $V_{l}$, which is formulated as bellow, where we omit the reshape transformation from $\mathbf{v}_{l}^{in}$/$\mathbf{v}_{l}^{sim^{\prime}}$ to $V_{l}^{in}$/$V_{l}^{sim^{\prime}}$:

$\sigma(\cdot)$ represents sigmoid function. Conv($\cdot$) denotes 1$\times$1 convolution operation, and $\odot$ is the notation of element-wise multiplication.

RIS model takes visual and textual features as input, and outputs a predicted segmentation mask. By now the enriched visual features $V^{\prime}_{l}$ ($l$ $\in$ $\{3,4,5\}$) achieved by RES have been well-prepared. As for language feature, we encode each word in sentence T with a LSTM to generate language feature $\mathbf{s}$ $\in$ $\mathbb{R}^{d_{s}}$, where $d_{s}$ is the dimension of language embedding space. Following prior works (Ye et al., 2019; Huang et al., 2020; Hu et al., 2020), an 8-D spatial coordinate feature $O\in\mathbb{R}^{h_{l}\times w_{l}\times 8}$ is integrated into the enriched visual features $V^{\prime}_{l}$ before cross-modal fusion.

As it is commonly assumed that the language refers to high-level visual concepts while leaving low-level visual processing unaffected (Vaswani et al., 2017), it is natural to aggregate language feature with high-level visual features (i.e., the low-resolution feature maps). Moreover, exploring cross-modal interaction is often formulated as a multi-step progressive process (Nam et al., 2017; Fan and Zhou, 2018; Mun et al., 2020; Huang et al., 2020). Based on the consideration above, we first adopt Bilinear fusion strategy (Ben-younes et al., 2017) for preliminary multimodal fusion, then refine the raw multimodal feature via integrating features of multiple levels. Such process is formulated as below:

where $W_{4}$ $\in$ $\mathbb{R}^{d_{s}\times d_{m}}$ and $W_{5}$ $\in$ $\mathbb{R}^{d_{m}\times d_{m}}$ are weight matrix for linear transformation, $\otimes$ and [;] represents the matrix multiplication and concatenation respectively. Gate($\cdot$) operation has many choices, such as LSTM-type gate in (Ye et al., 2019), vision-guided gate in (Hu et al., 2020) and language-guided gate in (Huang et al., 2020). We use the language-guided gate as implemented in  (Huang et al., 2020). Finally, we apply ConvLSTM (SHI et al., 2015) to aggregate multimodal features ($M_{3}^{raw^{\prime}}$, $M_{4}^{raw^{\prime}}$ and $M_{5}^{raw^{\prime}}$) of different levels to obtain a refined multimodal feature $M^{fine}$.

Different from low-resolution visual features, high-resolution visual feature accommodates more region details but are lack of semantics as implied in (Zeiler and Fergus, 2014). In order to adaptively incorporate region detail cues related to referents in high-resolution visual feature for further enhancement, we investigate a novel multi-resolution feature fusion strategy as illustrated in Figure 4. Instead of directly fusing the multimodal feature $M^{fine}$ with the high-resolution visual feature $V_{2}^{in}$, we suppress the regions irrelevant to referents in $V_{2}^{in}$ under the guidance of $M^{fine}$, and then dynamically reconcile their relative intensities via a gate during aggregation process.

Suppose $HR^{in}=\{V_{2}^{in},V_{1}^{in}\}$ represents the high-resolution input image I. To reduce computational complexity, we only use $V_{2}^{in}$ for promoting the visual details. Specifically, we first encode more visual contents into $C^{LR}$ with convolution and ReLU activation operations, then perform Pixel Shuffle (PS) (Shi et al., 2016), a kind of upsampling operation, on $C^{LR}$ to obtain $C^{HR}$ with the same resolution as $V_{2}$. Afterwards, $C^{HR}$ learns to suppress the regions irrelevant to referents in $V_{2}^{in}$ during training with concatenation and convolution operations:

For fusing the background suppressed feature $V^{\prime}_{2}$ with upsampled multimodal feature $M^{fine}$, simply adding or concatenating them fails to dynamically and adaptively balance their relative importance at different spatial location. Therefore, we reconcile their intensities using a gate operation, which consists of a sequence of convolution and sigmoid activation operations (i.e., black dotted rectangle parts at the midlle of Figure 4 ):

As shown in Equation (5), an intensity map $A_{2}$ $\in$ $\mathbb{R}^{h_{2}\times w_{2}\times 1}$ for $V^{\prime}_{2}$ is first generated with a gate. Each value in $A_{2}$ measures the relative intensity of region detail cues in $V^{\prime}_{2}$ at current position. Afterwards, the intensity map $A_{2}$ element-wisely multiplies on $V^{\prime}_{2}$ and concatenate with upsampled $M^{fine}$ to obtain the final enhanced feature $M^{fine^{\prime}}$, which will be used for the segmentation mask prediction.

Datasets. We conduct extensive experiments on four benchmark datasets for RIS: UNC (Yu et al., 2016), UNC+ (Yu et al., 2016), G-Ref (Mao et al., 2016) and ReferIt (Kazemzadeh et al., 2014). Images in UNC, UNC+ and G-Ref are all collected from MS COCO (Lin et al., 2014), while ReferIt selects images from IAPR TC-12 (Escalante et al., 2010). UNC contains 19,994 images with 142,209 language expressions for 50,000 segmentation regions. Objects of the same category may appear more than once in each image. UNC+ consists of 141,564 descriptions for 49,856 objects in 19,992 images. No location words appear in UNC+ language descriptions, which means the category and attributes are only hints for segmenting referred objects. G-Ref comprises of 104,560 expressions referring to 54,822 objects in 26,711 images. The length of description sentence is longer as well as diversiform. ReferIt includes 19,894 images with 96,654 objects referred by 130,525 language expressions, and Referents in it can be objects or stuff (e.g., ground and sky).

Implementation details. Our network consists of two execution stages. The first stage aims to retrieve a similar image offline and the second stage is an end-to-end training process. Details of experiment settings are listed as follows: In retrieval process, we adopt VGG16 (Simonyan and Zisserman, 2015) and DistilBERT (Sanh et al., 2019) with weights pretrained on ImageNet (Deng et al., 2009) and BookCorpus (Zhu et al., 2015) to preprocess the image and text. We simply use cosine similarity as measurement. Note that the matching measurement is not restricted to cosine similarity, any other strategies or well-designed models for calculating the similarity can serve as substitution to obtain more precise retrieval results in the future. Specifically, we first reduce the searching space by matching all text keys $\mathbf{k}_{i}^{T}$ in data pool with text query $\mathbf{q}^{T}$ to select 20 most relevant candidate samples. Further, the most resemble image $I^{sim}$ among these 20 samples is picked according to visual similarity. The whole retrieval process is finished offline, which means that the retrieved similar image for every paired input image and referring expression is prepared for the subsequent end-to-end training.

During the end-to-end training, every input image is resized and zero-padded to 320$\times$320, and fed into DeepLab ResNet-101v2 (Chen et al., 2018) pretrained on PASCAL-VOC dataset (Everingham et al., 2010) following the previous work (Ye et al., 2019; Huang et al., 2020; Chen et al., 2019). We extract feature maps from DeepLab ResNet-101v2 layers res3, res4, res5 as low-resolution feature set and res1, res2 as high-resolution feature set. The spatial scale of each feature map is 40$\times$40 in low-resolution feature set, and 80$\times$80 in high-resolution feature set. As for language processing, GloVe (Pennington et al., 2014) word embeddings pretrained on Common Crawl 840B tokens are firstly used for constructing word embedding look-up table. Then a LSTM takes GloVe word embeddings as input for capturing context information. Following previous work (Ye et al., 2019; Hu et al., 2016), We set the dimension of hidden state in each LSTM cell as 1000. Adam (Kingma and Ba, 2015) optimizer is initialized with learning rate of 0.00025 and equipped with polynomial decay with power of 0.9. And the weight decay is set as 0.0005. Considering the GPU memory limits, we set batch size as 1. Ordinary cross-entropy loss is adopted to supervise training process. It is worth noting that we don’t use DenseCRF (Krähenbühl and Koltun, 2011) for further refinement during inference like previous work (Ye et al., 2019; Hu et al., 2020; Huang et al., 2020), but still outperforms their models with DenseCRF for prediction mask refinement.

Evaluation metrics.  Following previous work (Ye et al., 2019; Chen et al., 2019; Hu et al., 2020; Huang et al., 2020), we also use two typical metrics for evaluation: Overall Intersection-over-Union (Overall IoU) and Prec@X, where X $\in$ {0.5,0.6,0.7,0.8,0.9}. However, previous works don’t discuss the intrinsic characteristics of these 2 metrics. Here we first state the formulas for Overall IoU and Prec@X, then analyze the specific model capability that each of the metric measures.

The formulas of Overall IoU and Prec@X are shown as below:

where $\mathrm{I}_{i}$, $\mathrm{U}_{i}$ and $\mathrm{IoU}_{i}$ represent the intersection region, union region and intersection region over union region of i-th data in test set. $M$ is the size of test set. As implied in Equation (6), Overall IoU calculates the total intersection region over union region of all samples. It mainly reflects model performance of pixel-level, which does harm to objects of small size, because small objects with limited pixels are easily dominated by the large objects with amounts of pixels. On the contrary, Prec@X mainly measures the model performance of object level, for it clearly calculates the percentage of objects in test set whose IoU higher than the threshold X.

The proposed network achieves visual boost by RES and AMF. To comprehend the effectiveness of each of them, we conduct ablation experiments on UNC+ and G-Ref dataset via quantitative as well as qualitative comparison. Due to page limitation, only ablation experiment results on UNC+ val set are shown in this paper with Table 1, and the remaining results can refer to the supplementary material. Moreover, in order to especially investigate the effectiveness on the referents weak in visual cues, we also test on small objects, a special case of objects with weak visual cues, inside each dataset. Here, we roughly define the concept of ”small” as: objects with mask accounting for less than 5% area of the whole image. For the Prec@X measure model performance of object level as stated in Section 5, we make analysis mainly based on Prec@X in Table 1.

RES.  We first explore the effectiveness of proposed RES on UNC+ val set, and the results are shown in rows 2, 3 of Table 1. Our baseline adopts DeepLab ResNet-101v2 and LSTM as backbone, and only language feature and 3 low-resolution visual features from res3, res4, res5 layers of ResNet-101v2 are used for cross-modal fusion and final prediction. For fair of comparison, we only add RES while keep other settings the same as baseline, which is denoted as Baseline+RES in Table 1. By comparing row 2 and row 3, we found that RES brings improvement on [email protected], [email protected] and [email protected], but decreases [email protected] and [email protected]. Such results is reasonable for RES introduces external relevant visual information that makes referents more indicative, thus achieves better performance when threshold of IoU is relatively low. Nevertheless, these introduced visual information may not be distributed to referents in original image consistently at spatial domain, which hinders the model from generating a precise mask. The [email protected] and [email protected], who set a high threshold, only accept generated mask highly consistent with ground-truth as correct prediction, therefore these 2 metrics drop after adding RES.

We further test the effectiveness of RES on small referents in rows 7, 8 of Table 1. By comparing the relative gain on all sizes of referents (row 2 and row 3) and small referents (row 7 and row 8), we notice that improvements brought by RES on small objects (1.12% for [email protected], 0.87% for [email protected], 0.60% for [email protected]) are more obvious than all sizes of objects (0.51% for [email protected], 0.49% for [email protected], 0.58% for [email protected]), which indicates the external knowledge is more helpful for referents deficient in visual cues (i.e. small objects in our experiment).

AMF.  In rows 3, 4, 5 of Table 1, we carefully control variable to verify the effectiveness of AMF. As described in Sec4.2, our baseline equipped with RES (Baseline+RES) only takes visual features from res3, res4 and res5 as input, therefore, it is unfair to directly compare our final model (Baseline+RES+AMF) that also uses res2 feature with Baseline+RES model. Based on the above consideration, we add another experiment that directly concatenate the high-resolution visual feature map from res2 with learned low-resolution multimodal feature, and use concatenation followed by a 1$\times$1 convolution layer for fusion, which is denoted as ”Baseline+RES+Cct” in Table 1. As illustrated in row 4 of Table 1, although directly concatenate high-resolution feature can achieve considerable improvement, our proposed AMF module even outperforms the Baseline+RES+Cct with a large margin. Moreover, the improvements on [email protected] and [email protected] are particularly evident, which implies that AMF more focus on generating a precise mask for referents.

Likewise, we conduct experiments on small objects (row 8, 9, 10 in Table 1). As shown in Table 1, our AMF improves all metrics for small referents more obviously than all sizes of objects, and the [email protected] for small objects even 2 times higher than the Baseline+RES+Cct.

Qualitative Results.  To intuitively understand the behavior of RES and MRE, we visualize the results predicted by different version of our model (Baseline, Baseline+RES, Full model) in Figure 5. As shown in the left part of the second row in Figure5, our baseline model pays more attention to the mineral water bottles near the milk bottle. We consider the reason for such results might be that large part of milk bottle is occluded by the hand, thus model unable to recognize it and choose the mineral water bottle in the middle as the referred object. After RES enriches visual information of referent, model can locate the correct referred object as shown in the ”milk bottle” case of Figure 5(c). Our AMF further refines the prediction results and generates a much better prediction mask than our baseline. The other examples in Figure 5 show a similar phenomenon, the difference is that RES helps model to select the correct object among multiple objects of the same category.

We compare the proposed approach with several State-Of-The-Art(SOTA) method (Ye et al., 2019; Huang et al., 2020; Hu et al., 2020), including 2 strongest models (i.e. CMPC (Huang et al., 2020) and LSCM (Hui et al., 2020)). The comparison results are shown in Table 2. From the result, We observe that our pure model even achieve better results than SOTA methods using DenseCRF (Krähenbühl and Koltun, 2011) for further refinement on most benchmark datasets. Remarkably, particularly obvious gain is obtained on G-Ref by our model, while the performance improvements are not as high on UNC and UNC+. One reason is that the UNC is easier as indicated by the metrics (Nearly 16% higher Overall IoU than G-Ref), and relatively early work (e.g. CSMA (Ye et al., 2019), STEP (Chen et al., 2019)) already perform well. Another reason is that small objects appear more frequently in G-Ref than in UNC+ as shown in 7, therefore, the G-Ref is better able to benefit from our model. In addition, two methods (STEP (Chen et al., 2019) and LSCM (Hui et al., 2020)) adopting high-resolution visual features should be especially noticed. STEP (Chen et al., 2019) iteratively fuses 5 levels of visual features for 25 times and LSCM (Hui et al., 2020) sequentially aggregates 4 levels of visual features through bottom-to-up and top-to-down style as in (Liu et al., 2018). Instead of repeatedly utilizing high-resolution visual feature, we directly feed high-resolution into proposed AMF block, but still yields 3.68% and 2.03% overall IoU boost against STEP and LSCM+DCRF on G-Ref val set, which indicates the effectiveness of our design.

To further investigate the effectiveness of proposed TV-Net on the cases weak in visual cues, we compare our model with two strong SOTA models (CMPC and LSCM) on small referents within each dataset in Table 3. Besides, in order to evaluate pure model ability, we don’t use DenseCRF (Krähenbühl and Koltun, 2011) as further refinement in LSCM as well as CMPC. As illustrated in Table 3, our method surpasses both CMPC and LSCM with a large margin on most of the benchmarks. In general, our model achieves significant superior results on cases weak in visual cues while also gets better performance on all objects than SOTA methods.

The qualitative results of LSCM, CMPC and our model are depicted in Figure6. From the first example in the left part of row one, we observe that CMPC unables to find the correct referred banana. Compared with CMPC, although LSCM can correctly locate target banana, it generates a prediction mask far from complete. Our prediction result is highly consistent with the ground-truth. In the example right part of row one, LSCM and CMPC ignore the lower part of the surfboard, while our model can accurately segment the whole surfboard, which proves that our model can well handle objects deficient in visual cues in RIS.

We also demonstrate some interesting failure cases in Figure 8. We argue that such failure cases are mainly due to following two aspects. First, the inaccurate language expressions mislead segmentation prediction. For example, the ground-truth mask for expression ”laptop bottom” only includes the screen part of referred laptop, while our model segments the complete laptop. Second, extreme ambiguity at boundaries (e.g., splash of water around bird wings in the second row of Figure 8) deteriorates segmentation results.