Position-Aware Contrastive Alignment Network for Referring Image Segmentation

Referring image segmentation is a fundamental vison language understanding task that aims to segment the target described by a natural language expression from the input image. This task has a wide range of potential application scenarios, such as language-based human-robot interaction and image editing. In contrast to traditional semantic and instance segmentation which require segmenting each visual region belonging to a predefined set of categories, referring image segmentation is not limited to specific categories, but needs to segment specific areas described by unrestricted linguistic expressions include representations words and phrases of concepts such as entity, action, attribute, location, etc.

The main challenge of this task is to jointly comprehend the content of vision and language, and accurately segment the referential target based on their association. Therefore, a core problem in referring image segmentation research is how to better model the interaction between vision and language information to align their feature representations. The most typical paradigm is to first extract visual and linguistic features based on convolution neural network and recurrent neural network respectively, then interact visual and linguistic features through concatenation or attention mechanism to obtain multi-modal features, and finally the multi-modal features are sent to the decoder module to obtain the mask of the reference target. Recently, Transformer has made rapid development in the field of vision. Recently, Transformer has achieved great success in many visual tasks. Compared with convolution neural network and recurrent neural network, transformer is better at modeling dependence on long distance, which plays an important role in promoting visual and language interaction. A lot of work has introduced transformer into referring image segmentation and achieved good performance.

Although significant improvements have been achieved, current methods are mainly based on an implicit fusion method for visual and linguistic information interaction, which still has limitations in multi-modal feature alignment. Specifically, the current effective fusion strategy is to implicitly obtain the potential target area by calculating the similarity between the two modal features, and the interaction between vision and language only depends on the supervision signal of the ground truth mask. Due to the randomness of unrestricted natural language expressions, it is difficult to learn aligned visual linguistic multi-modal feature representation based on implicit fusion. To enhance the effect of alignment, CRIS makes use of the explicitly aligned multi-modal response knowledge provided by CLIP, but still lacks explicit information guidance in the training stage.

How to explicitly improve multi-modal alignment in referring image segmentation is still an open problem. In this paper, we try to explicitly introduce additional prior information to promote the interaction between visual and linguistic information of the model, so as to improve multi-modal aligned feature representation ability of the model and achieve more precise location and segmentation. To this end, we propose the Position-Awared Contrastive Alignment Network(PCAN), which explicitly aligns visual cues with language by introducing position information related to language description. It consists of two modules:Position Aware Module (PAM) and Contrastive Language Understanding Module (CLUM). PAM first obtains object location information related to the natural language description in the image, and the location information will guide the model to focus explicitly on the referent target during the training process. CLUM then uses positional information as additional input to help better visual and linguistic feature interaction and alignment by comparing linguistic descriptions of the referent target and surrounding related object and finally based on highly aligned multimodal features, we accurately predict the masks of the referent targets.Notably, our proposed module is used only in training and discarded in the inference phase, without introducing additional overhead. Our method achieves SOTA results on three mainstream referring image segmentation datasets and proves its effectiveness.

Our main contributions can be summarized as follows:

• •

  We show that the introduction of a priori positional information can effectively improve the alignment of visual and linguistic features.

• •

  We propose the Contrastive Language Understanding module, which draws on the concept of contrastive learning to make full use of a priori knowledge.

• •

  We achieve new state-of-the-art results on three datasets for referring image segmentation, demonstrat- ing the effectiveness and generality of the proposed method.

The pipeline of our PVLU framework is illustrated in Fig. 2. Given an image $I\in\mathbb{R}^{H\times W\times 3}$ and a language expression $\mathcal{E}=\left\{e_{l}\right\}_{l=1}^{L}$ with L words, RIS aims to produce a binary segmentation masks of referred object $\mathcal{M}\in\mathbb{R}^{H\times W}$. Firstly, we extract multi-scale image features with visual backbones and extract text features with a Transformer. These features are further fused to obtain the initial multi-modal features. Secondly, we feed the multi-modal features and prior information into the CLUM module to get the explicitly aligned multi-modal features. Finally, we adopt Cross-modal Feature Pyramid network to obtain the final segmentation mask.

You must include your signed IEEE copyright release form when you submit your finished paper. We MUST have this form before your paper can be published in the proceedings.

Please direct any questions to the production editor in charge of these proceedings at the IEEE Computer Society Press: https://www.computer.org/about/contact.