Multi-Attention Network for Compressed Video Referring Object Segmentation

Text-based video object localization is a problem that connects computer vision and natural language processing. Xu et al. (Xu et al., 2015b; Xu and Corso, 2016; Yan et al., 2017) propose an actor-action semantic segmentation task, and introduces the challenging large-scale A2D dataset. To further study cross-modal video understanding, Gavrilyuk et al. (Gavrilyuk et al., 2018) augment A2D dataset by further adding human generated description of a specific object of each video. The new task can be modeled as spatio-temporal video object localization. Several works (Gavrilyuk et al., 2018; Wang et al., 2020; Hui et al., 2021; Botach et al., 2022) focus on dynamic convolution-based method; while the last two methods incorporate spatial context into filter generation. Several works (Wang et al., 2019; Liu et al., 2021a; Ye et al., 2021; Seo et al., 2020; Chen et al., 2021) focus on attention-based method. McIntosh et al. (McIntosh et al., 2020) further encode the visual and language features as capsules and integrate the visual and language information via a routing algorithm. The very recent method MTTR (Botach et al., 2022) leverages Transformer to do referring video object segmentation, and achieves remarkable performance. However, it requires well-prepared frame-level annotations to do instance segmentation first before segmenting referring object, and it needs a complex post-processing to select one instance sequential mask as referred one, which slows the inference down. Unlike previous works that focus on decompressed video frames, in this work, we aim to investigate compressed video referring object segmentation. It benefits us on several computing resource limited scenarios, such as online segmentation, real-time fast segmentation and run on embedded devices.

A lot of works have achieved great success in compressed video action recognition, they give us a great inspiration to extract powerful features from compressed video. In the pioneering works (Zhang et al., 2016, 2018), motion vectors are utilized to replace optical flow features. However, they leverage both compressed videos and raw videos simultaneously to do prediction. In order to improve the performance, some methods introduce optical flows into features from compressed video as extra information (Shou et al., 2019; Wu et al., 2018). Li et al. (Li et al., 2020b) propose a novel Slow-I-Fast-P (SIFP) neural network model for compressed video action recognition. It consists of the slow I pathway receiving a sparse sampling I-frame clip and the fast P pathway receiving a dense sampling pseudo optical flow clip. Hu et al. (Hu et al., 2020) train four independent classification networks and combine them by late fusion to make the final prediction. Unlike existing approaches that encode I-frames and the P-frames individually, Yu et al. (Yu et al., 2020) propose to jointly encode them by establishing bidirectional dynamic connections across streams. Inspired by previous works, we jointly encode three data modalities features to learn powerful video features for segmentation in this work.

Recently, attention mechanisms (Zhuo et al., 2017; Wang et al., 2018a) have been widely-used in visual-language task, such as VQA (Xu et al., 2015a; Yang et al., 2016; Lu et al., 2016; Gao et al., 2019; Kim et al., 2018; Lu et al., 2018; Han et al., 2020, 2021) and referring expression comprehension (Ye et al., 2019; Wang et al., 2018b). Fu et al. (Fu et al., 2019) develop a dual attention network, which contains two parallel attention modules: one is for spatial-attention and another for channel-wise attention. Tsai et al. (Tsai et al., 2019) introduce the cross-modal attention network to model word-based spatial attention, and further designs a novel transformer based on cross-modal attention. Ye et al. (Ye et al., 2019) propose a cross-modal self-attention to do referring segmentation in image-level. Wang et al. (Wang et al., 2019) design a asymmetric cross-modal attention module, which uses not only the gated self-attention to model the language’s impact on vision, but also co-attention to model the visual impact on language. Inspired by previous works that leverage dual attention to model cross-modalities correlations, we propose a dual-path dual-attention module in this paper. This module integrate the I-frame features into the Residuals and Motion Vectors by cascade dual-attention, including channel attention and spatial attention. By getting I-frame features attended the Residual features and Motion Vector features in a parallel manner and then fusing, the P-frame features are becoming powerful for segmentation.

The encoder-decoder architecture of the Transformer can be adopted to multi-modal tasks such as captioning, question-answering, reasoning, and visual grounding. Vision-and-language pre-trained model is becoming a trend in this field, such as (Li et al., 2019; Su et al., 2019; Lu et al., 2019). Sun et al. (Sun et al., 2019) propose an excellent work, VideoBERT, which learns joint video-text representations with Transformer in a self-supervised manner for downstream tasks. A multi-modal Transformer is proposed in (Yu et al., 2019) for image captioning. Object proposals generated from multiple detectors are fed into the Transformer encoder, and the Transformer decoder learns a language model conditioned on the encoder outputs. In referring expression comprehension task, Sun et al. (Suo et al., 2021) propose a set of Transformers to localize referred objects in a one-stage manner. Li et al. (Li and Sigal, 2021) jointly train their proposed novel Transformer for referring expression segmentation and comprehension tasks while enabling contextualized multi-expression references. Recently, MTTR (Botach et al., 2022) and MDETR (Kamath et al., 2021) leverage the successful encoder-decoder architecture of the Transformer to design their own methods for referring video/image segmentation. They develop their methods based on DETR (Carion et al., 2020) and VisTR (Wang et al., 2021). However, both two methods require a large number of object queries, and well-prepared annotations to train instance segmentation first.

Because of the lightweight the P-frame feature extractor ResNet-18, the feature map of the P-frame is not strong enough for segmentation. Meanwhile, since the Residuals in the P-frames maintain the RGB differences between I-frame and its reconstructed frames calculated by Motion Vectors in the P-frames after motion compensation, both Motion Vectors and Residuals are important for extracting powerful the P-frame features. Thus, we design a dual-path dual-attention module to encode powerful features from compressed video for segmentation.

Take one GoP and the last scale as an example. Firstly, for k-th the P-frame, we project the channel dimension of I-frame feature $X_{I_{s}}$ into the same channel dimension of Motion Vector features $X_{M_{s}}^{k}$. Meanwhile, to avoid losing part of Motion Vector information during feature extraction, we concatenate the spatial down-sampling Motion Vector $M^{k}$, Motion Vector features $X_{M_{s}}^{k}$ and I-frame features $X_{I_{s}}$ together first. Then, we leverage ${\rm Conv_{D}}$ to fully exchange and fuse two data modalities, I-frames features and Residuals. ${\rm Conv_{D}}$ is a dense-connected four convolutional layers, details are shown in 4.2. Thus, the formulation is:

where $Y_{IM_{s}}^{k}$ is the fused features of I-frame and Motion Vectors, $[;]$ is the concatenation operation.

Then, we leverage a fully connected layer to project the fused features to obtain the channel attention between I-frame and Motion Vectors:

where $W_{C}$ is the learnable coefficient, and $\sigma_{s}$ is the softmax activation. $\overline{Att}_{{\rm cha},IM_{s}}^{t}$ represents gated attention on the channel dimension with each element measuring the impact of I-frame channel on Motion Vector channel. We duplicate $\overline{Att}_{{\rm cha},IM_{s}}^{k}$ into the same size of I-frame features $X_{I_{s}}$, and element-wise multiple attention map with the $X_{I_{s}}$ to get the attend Motion Vector features $X_{{\rm cha},IM_{s}}^{k}$ in channel dimension:

where $\odot$ represents element-wise multiplication.

Meanwhile, in spatial attention, we firstly leverage a simple convolutional layer to get the attention map $\overline{Att}_{{\rm spa},IM}^{k}$ on spatial dimension between I-frame and Motion Vector. Each element in the attention map measuring the impact of I-frame spatial positions on Residual spatial position:

where $w_{spa}$ is the convolutional kernel, $\otimes$ means convolution operation. Then, duplicate the $\overline{Att}_{{\rm spa},IM}^{t}$ into the same dimension of $X_{I}$, and element-wise multiple with the channel attended features to get attended Motion Vector features of I-frame:

Thus, the total formulation is:

where $F_{M_{s}}^{k}$ is the final attended visual features in Motion Vector path.

Residuals maintain the RGB differences between I-frame and its reconstructed frame calculated by Motion Vectors after motion compensation. Therefore, we leverage the same dual-attention operation as Motion Vectors branch to integrate the Residual features into the P-frames features extraction. Similarly, the general formulation in Residual path is:

where $F_{R_{s}}^{k}$ represents the final attended visual features in Residual path.

After getting I-frame features attended Motion Vector features and the Residual features, both two are important for powerful the P-frame features extraction. We leverage a simple summation operation to fuse Motion Vector and Residual to get the P-frame features $F^{k}$:

where $F_{s}^{k}$ is the augmented features of the $k$-th the P-frame. Meanwhile, our proposed dual-path dual-attention calculation lies in shadow convolution layers and element-wise multiplication, which is computational efficient.

The above illustration is an example of one GoP. Thus, generally, $F_{s}^{k}$ is actually $F_{s}^{t,k}$. Thus, the output after passing dual-path dual-attention is $F_{s}=[F_{s}^{1,1},F_{s}^{1,2},...,F_{s}^{1,K},F_{s}^{2,1},F_{s}^{2,2},...,F_{s}^{T,K}]$. Then, we utilize a 4-layer 3D-CNN to downsample the temporal dimension to get clip features $V\in\mathbb{R}^{\frac{H}{16}\times\frac{W}{16}\times C_{V}}$, and then, we concatenate clip features in the channel dimension to each frame features to get the overall video features $F_{V}\in\mathbb{R}^{(T\times K)\times\frac{H}{16}\times\frac{W}{16}\times(2C_{V})}$. Meanwhile, there are another two scales dual-path dual-attention outputs $F_{m}$ and $F_{l}$, which are used for mask generation.

The proposed query-based cross-modal Transformer firstly leverages the Transformer encoder to fully exchange two modalities and reduce modalities’ gap, and generates language-guided visual features $F_{L\rightarrow V}$ and vision-guided linguistic features $F_{V\rightarrow L}$. Both focus on referring video objects. Then, the cross-modal features along with the object queries are fed into to the Transformer decoder, guiding the object queries focusing on learning the content of referring object from multi-modal perspectives.

The proposed query-based cross-modal Transformer employs the feature map of a video clip $F_{V}\in\mathbb{R}^{(T\times K)\times\frac{H}{16}\times\frac{W}{16}\times 2C_{V}}$ and language embedding vector $F_{L}\in\mathbb{R}^{N\times C_{L}}$ as inputs. These visual and linguistic features are linearly projected to a shared dimension $D$ at first. Then, we add a fixed 2D positional encoding to the feature map of each frame and the features of each word, following (Botach et al., 2022). The features of each frame are then flattened and concatenated with text embeddings respectively, resulting in multi-modal sequence whose size is $((T\times K)\times\frac{H}{16}\times\frac{W}{16}+N)\times D$.

The multi-modal sequence along with $N_{q}$ object queries are then fed into the Transformer decoder. The multi-modal sequence highlights the referring video objects in both visual and linguistic aspects, and the multi-modal sequence points out the learning keypoint of the object queries. Unlike previous work (Botach et al., 2022), our method sort and select the dynamic kernels before predicting the sequential masks, which greatly reduces the number of object queries, avoids post-processing to link each video object in each frame, so as to save a lot computational cost.

The mask generation module of the algorithm in this chapter adopts the structure of U-Net(Ronneberger et al., 2015) considering its successful performance. Our method predicts sequential masks after selecting kernels based on the matching scores, which greatly reduce the number of input object queries, and avoids well-prepared annotations to learn instance segmentation at first.

After the Transformer decoder, the query features $F_{q}$ is obtained, which is based on the observation of the overall content from both visual and linguistic aspects. We use two layers of MLP to map $F_{q}$ to get a set of dynamic convolution kernels $\Omega\in\mathbb{R}^{N_{q}\times(T\times K)\times\frac{H}{16}\times\frac{W}{16}}$. We convolve $\Omega$ with the language-guided visual features $F_{L\rightarrow V}$ obtained by the Transformer decoder to obtain coarse-grained masks ${\rm Mask}_{q,LR}$:

where $LR$ represent low-resolution, ${\rm Mask}_{q,LR}\in\mathbb{R}^{N_{q}\times(T\times K)\times\frac{H}{16}\times\frac{H}{16}}$. For each coarse-grained mask in ${\rm Mask}_{q,LR}$, we calculate the pixel-level IoU with the down-sampling ground truth, select the one with highest pixel-level IoU, denote it as ${\rm Mask}_{LR}$. Meanwhile, denote the indicators of positive/negative regions as $\Delta=[\delta_{1},\delta_{2},...,\delta_{N_{q}}]$, where $\delta_{j}=1$ when the mask ${\rm Mask}_{LR}^{j}$ has the highest pixel-level IoU with the down-sampling ground truth and otherwise 0. We utilize an additional fully-connected layer to obtain matching scores $S\in\mathbb{R}^{N_{q}}$, with each element indicate the matching scores of the object query with the referring object.

where $W$ is the learnable parameter, $\sigma_{s}$ is softmax activation, and [;] represents concatenate operation. For each resulting coarse-grained video mask, we perform down-sampling operation to the ground truths to supervise the generation of the coarse-grained mask by a cross-entropy loss. Meanwhile, we also incorporate the matching loss to get the total loss at low resolution:

where $i$ indicates $i$-th GoP, $j$ indicates $j$-th the P-frame, ${\rm GT}$ represents ground-truth, $CE$ represents cross-entropy loss, and $\beta$ is the hyper-parameter which aims to control the balance of two losses.

After getting the coarse-grained ${\rm Mask}_{LR}$, we combine it with the visual output of the Transformer encoder $F_{L\rightarrow V}\in\mathbb{R}^{(T\times K)\times\frac{H}{16}\times\frac{W}{16}}$ and the feature encoder corresponding scale features first. Then, we leverage the up-sampling and convolution operation to get up-scale mask.

Next, two additional up-sampling and convolution operation is applied, each time we concatenate the mask with corresponding scale feature of the dual-path dual-attention output. After two additional operation, we get the fine-grained ${\rm Mask}_{HR}$, and we leverage the annotations and two losses to supervised the fine-grained mask generation. The formulation is:

where $HR$ indicates the high-resolution, $Dice$ is the dice loss.

Thus, the total loss in the proposed method is:

During inference, we select the highest matching score and its corresponding coarse-grained mask. Then we leverage the coarse-grained mask to get the final refined mask with the output of transformer encoder and feature encoder.

We conduct extensive experiments on three referring video segmentation benchmarks including Actor-Action (A2D) Sentences, J-HMDB Sentences and Refer-YouTube-VOS.

A2D Sentences. The original A2D dataset contains 3,782 videos which is collected from YouTube. Following (Xu et al., 2015b), we split the dataset into 3,036 training videos and 746 testing videos. Gavrilyuk et al. (Gavrilyuk et al., 2018) augments the original A2D dataset to the A2D Sentences by providing a sentence description to each actor and its action in a video. There are totally 6,655 sentences. We train our network on the training split of A2D Sentences and test its performance on the testing split of A2D Sentences and J-HMDB Sentences for all experiments in this paper.

J-HMDB Sentences. The J-HMDB dataset contains 928 video clips of 21 different actions with mask annotations. Through Gavrilyuk et al. (Gavrilyuk et al., 2018) extending, J-HMDB Sentences has 928 corresponding sentence descriptions for 928 videos.

Refer-YouTube-VOS. To referring video object segmentation task, (Seo et al., 2020) constructs a large-scale video object segmentation dataset, which is called Refer-Youtube-VOS, with descriptive sentences. The dataset has 4,519 high-resolution videos containing 94 common object categories. There are pixel-level instance segmentation annotations every 5 frames in each 30fps video, and their duration is about 3 to 6 seconds.

Evaluation metrics. On A2D Sentences and J-HMDB Sentences datasets, following previous works (Gavrilyuk et al., 2018; Wang et al., 2019, 2020; McIntosh et al., 2020; Ning et al., 2020; Hui et al., 2021; Ye et al., 2021; Liu et al., 2021a; Chen et al., 2021), Overall IoU, Mean IoU, Precision@$K$ (P@$K$), and mAP are adopted as metrics to evaluate the performance. It is worth noting that all IoU in our task is the pixel-wise Intersection-over-Union. The precision@$K$ computes the percentage of samples whose IoU is higher than threshold $K$ ($K=\{0.5,0.6,0.7,0.8,0.9\}$). The mAP reports mean average precision (mAP) over 0.50:0.05:0.95. The overall IoU reports the ratio of the overall intersection area of all testing samples to the total union areas; while the mean IoU reports the average means of IoU over all the testing samples. On Refer-YouTube-VOS dataset, we leverage the standard evaluation metrics on this dataset, region similarity ($\mathcal{J}$), contour accuracy ($\mathcal{F}$) and their average value ($\mathcal{J}\&\mathcal{F}$). Since the annotations of validation set are not released publicly, we evaluate our method on the official challenge server.

The video clip input for this work consists of 36 frames, including 3 I-frames, and each I-frame is followed by 11 the P-frames, including motion vectors and residuals. In this work, all frame inputs are resized to 320*320. The dimension of visual features extracted by Video-Swin-T is 20*20*384, the dimension of visual features extracted by ResNet-18 are 20*20*512, and the dimension of language features extracted by BERT is 20*768 (each language description is padding to 20 words). All experiments were implemented using PyTorch. Some other settings: $D$ is 512, learning rate is set to 0.0001, learning momentum is 0.9, weight decay is 0.0005, optimizer is SGD-momentum, training steps and epoch are 1000 and 60, respectively. $\beta$ is set to 0.1 in this work.

Results on A2D Sentences. The results are shown in Table 1. From the resuls, we can see that our method is the only method that achieves the best result on both two major metrics, mean IoU and Overall IoU, and top-2 results on mAP, and also, we can observe that our method achieves the state-of-the-art results on almost all metrics, and achieves a substantial improvement. Thus, the effectiveness of our proposed method can be illustrated.

To further illustrate our method’s outperformance, we test the run time of our method and MTTR(Botach et al., 2022), which is method with the best performance so far and fast inference speed. On PC²²2Some details about our PC: 1) SSD: Samsung EVO 970 Plus; 2) CPU: AMD 3950X and 3) GPU: RTX 3090., our method achieves 77 fps³³3The inference speed is tested on the original codes without any quantification or accelerate operation.; while, MTTR can only run on 52 fps. The speed of our method is about 48% higher. Video decompression on our PC is very fast, running at over 1,000 fps, and it doesn’t make a big difference to MTTR’s inference speed. However, on the mobile terminal, video decompression requires a lot of time and computation. Here’s some statistics, for our mobile terminal (Jetson Xaiver NX), decompressing 360p A2D video runs at 50 fps, and decompressing 1080p A2D video runs at only 8 fps. Especially in online segmentation on mobile terminal, video decompression hinders the running time of existing methods, and since our method is directly applied to compressed video, there is little impact on our method.

Results on J-HMDB Sentences. We evaluated the generalization ability of the proposed method by directly testing the model, which is trained on A2D Sentences, on J-HMDB Sentences without any fine-tuning. The results are shown in Table 2, from the resuls, we can see that our method is the only method that achieves top-2 results in all three major metrics, mean IoU, Overall IoU and mAP.

Results on Refer-YouTube-VOS. Refer-YouTube-VOS is the most challenging dataset so far. The results on Refer-YouTube-VOS are shown in 3. From the results, it can be observed that our method achieves the best results on the overall $\mathcal{J}\&\mathcal{F}$ metric when compared to the existing works. On metric $\mathcal{J}$, our method is much better than the existing state-of-the-art methods, which we believe is because our method makes good use of multi-scale mask generation and can process more frames at a time when compared to the existing methods.

Qualitative Results. The qualitative results of our proposed method are shown in Fig. 3. The above two results are of A2D Sentences; whereas, the bottom results are of Refer-YouTube-VOS. It can be seen that our method segments right video objects in several challenging situations, i.e., complex video scenes, pose variation, occlusion and partially out of camera. Meanwhile, it can be seen that our segmentation results are of good quality and have smooth edges.

Discussion on the effectiveness of Dual-path Dual-attention module. We verify the effectiveness of our dual-path dual-attention module in this section. The details of every compared method are listed as follows: 1) Baseline+CoViAR (Wu et al., 2018) exchanges our the encoder of features from compressed video and dual-path dual-attention into CoViAR method, which concat the I-frame feature and the P-frame feature. 2) Baseline+DMC-Net (Shou et al., 2019) is similar to CoViAR. The difference is that we use another resnet-18 to extract DMC features. From the results, we can see that our method achieves large advances to the existing encoder of features from compressed video. A possible reason is that our dual-path dual-attention focuses on enhancing the visual features in the spatial dimension, and at the same time, the encoder of the reference frames (I-frames) in this work adopts the latest and most powerful Video-Swin-T, which makes our video features good for segmentation. Meanwhile, the lightweight feature extractor makes our network segment 36 frames at a time, which helps maintain segmenting results coherent across the video to further improve the segmentation results.

Discussion on the effectiveness of query-based cross-modal Transformer. In this section, we evaluate the effectiveness of the query-based cross-modal Transformer by replacing it with the existing method’s module. We leverage the very recent method MTTR(Botach et al., 2022) to replace our Transformer and mask generation with their query-based segmentation and instance sequence matching module. The only difference between two comparing methods is that the number of video frames training at a time. The results are shown in Table 5. As illustrated, our method achieves better segmenting results than both two comparing methods. Since MTTR adopted Video-Swin-T to extract all video frames features, their features are stronger than ours. A possible reason for achieving better performance is that our features from compressed video are good for learning segmenting one object at a time since instance segmentation requires more powerful visual features.

Effectiveness of the number of object queries. The results are shown in Table 6. We can see that as the number of object queries increases from 1 to 5, the performance becomes better. All evaluation metrics reached the optimal value when $N_{q}=5$. When the number of queries continues to increase, the performance degrades and becomes to converge since the performance of $N_{q}=7$ and $N_{q}=9$ are quite the same. This is because our convolution kernel is generated and selected by a hard selection method. When the number of queries increases, the selection matching accuracy will inevitably decrease, thus affecting the segmentation effect. When the number of queries is small, the object query is not enough to cover the objects of the entire video scene, which affects the segmentation results.