Memory Matching is not Enough: Jointly Improving Memory Matching and Decoding for Video Object Segmentation

Cost Volume Generation. The first step of short-term memory matching requires the generation of initial information representing the variations between adjacent frames. Instead of the local correlation matching in AOT [27] and its variants [25, 29] with high computational cost, we build a global 4D cost volume $c_{m}\in\mathbb{R}^{H_{1}\times W_{1}\times H_{2}\times W_{2}}$ for $e_{q}$ and $e_{k}$, the volume is constructed through dot product operations. Here, $H_{1}$ and $W_{1}$ donate the $e_{k}$’s height and width, $H_{2}$ and $W_{2}$ donate the $e_{q}$’s height and width. As shown in the bottom right corner of 2, each 2D sub-map in cost volume $c_{m}$ can be regarded as the visual similarity between the source pixel and all target pixels, and $e_{k}$ is derived from the $e_{q}$ of the previous frame. We patch embed the cost volume, which divides the cost volume into multiple patches according to the patch size, and perform feature embedding on each patch, as shown in the upper right corner of 2. We define the number of the patches as $S$, embedding dimension as $C^{\prime}$, the patch embedding features $P_{s}\in\mathbb{R}^{(H_{1}*W_{1})\times S\times C^{\prime}}$ as:

Patch Features Tokenization. For complicated variations between adjacent frames, we need to pay more attention to the patches relating to the target objects. To achieve this goal, we introduce a set of learnable tokens $L_{q}\in\mathbb{R}^{N_{l}\times C_{l}}$ to extract the patch features $P_{s}^{\prime}\in\mathbb{R}^{(H_{1}*W_{1})\times N_{l}\times C_{l}}$ that contains latent critical variations, where $N_{l}$ is the token number and $C_{l}$ is the tokens’ embedding dimension. We implement a cross-attention, which applies the learnable tokens $L_{q}$ as query, patch embedding features $P_{s}$ as the key and value:

Producing Readout Feature. We implement the SS-attention [3], which focus more on capturing spatial features to produce short-term readout result $r_{s}$:

Context Embedding. Context block (CB) applies the same residual network layer as the encoder and three context blocks form a cascaded structure. Prompt features $g_{i}$ and the residual encoder output $Res(f_{i-1})$ fuse as $f_{i}$ in each cascade:

Recursive Decoding Process. Inspired by [15], we introduce the "looking and thinking twice" idea into memory readout decoding, namely recursive decoding, which consists of pre-decoding and post-decoding. Recursive decoding process shares weights in upsample blocks (UP, as shown in 3), which is essential for improving readout decoding due to seeing both the pre-decoded features and the context compensated features. Let $D^{b}_{i}$ and $D^{p}_{i}$ denote pre-decoding and post-decoding outputs, $i$ is the cascading level. The implementation of UP can be formally defined as follows:

Training. We deploy the ResNet50 as the backbone for embedding encoder. Following popular matching-based VOS methods [13, 2, 26], we employ the two-stage training strategy. In the first training stage, the model is pre-trained on static datasets from AOT [27], where objects with masks are augmented (e.g., flip, shift, crop) and randomly synthesized onto the backgrounds. In the second training stage, we perform training on real videos. For the evaluation of DAVIS [14], we use the training sets of DAVIS and YouTube, while for the evaluation of YouTubeVOS [24], we only use the YouTube training set. The embedding encoder is frozen in the second training stage to avoid overfitting to the seen object categories. All modules apply the learning rates from initial 2e-4 then the learning rates gradually decay to 2e-5 in a polynomial manner. We employ the cross entropy loss and soft jaccard loss [11] to train the model. During training, we use an input size of $384\times 384$ and a batch size of 8, which is distributed on 4 RTX3090 GPUs. Note that our method do not adopt the BL30K for training in the following reports.

Inference. Our method uses a resolution of 480p by inference. Following common matching-based methods, we set the update frequency of memory to 5 (i.e., every five frames stack the short-term memory in long-term memory). For a fair comparison, we do not employ the multi-scale inference trick on val/test datasets.

Quantitative Comparison. As shown in Table 1, we compare the performance of our method with up-to-date methods on a series of DAVIS datasets. Compared with the state-of-the-art memory matching-based methods (i.e., XMem [1]), we achieve a 2.9% and 1.9% J&F improvement on DAVIS2017 Test and DAVIS2017 Val, respectively. Compared with the latest understanding-based method (i.e., ISVOS [22]), we improve the J&F by 1.1% and 1% on DAVIS2017 Test&Val, respectively. Our method achieves the top ranking of F performance on DAVIS 2016 single object performance, DAVIS2017 Test&Val multiple objects performance. On the Youtube dataset validation, for a fair comparison, we only adopt YouTubeVOS’s train-set for training and compare it with XMem [1] schemes using the same training data. As shown in Table 3, even without any fancy training and inference tricks, our method still achieves excellent performances in YouTube2018&2019.

Unseen Categories. Youtube dataset needs to evaluate the performance of unseen categories in training, as shown in Table 3; our method outperforms other methods under the metrics of unseen categories. Our proposed compensatory decoding stage generates guide information in pre-decoding that can more discriminatively help segment unseen category objects.

Qualitative Results. We visualize some video segmentation results in evaluation, including some common VOS challenge cases (i.e., tremendous motion, object reappearance, similar objects confusion), and compare them with two state-of-the-art matching-based methods AOT-L [27] and XMem [1], as shown in Fig. 4. Our method is superior to the other two methods in obtaining object details. We can see that AOT-L [27] and XMem [1] make obvious errors when the human body appears in drastic motions. XMem [1] is more prone to errors when dealing with objects of different scales (e.g., the rope and the human body in the Col 5 and 6 of Fig. 4). Furthermore, we compare the classical method STM [13] based on memory matching for a long span. Our method does not have object confusion after multiple similar objects are occluded, indicating that the proposed jointly improving method is helpful in overcoming the challenge of similar objects.

Impact of Memory Matching Mechanism. As shown from Row 2 to Row 4 in Table 3, both our cost-aware and cross-scale in the memory matching stage play positive roles when compensatory decoding is removed. The improvement of cross-scale matching is more conducive to improving regional similarity (i.e., $J$), and cost-aware matching improves both metrics and edge accuracy (i.e., $F$). Cross-scale matching has more powerful matching for objects of different scales in the video and thus is beneficial to improve region similarity. Cost-aware constructs the cost volume for learning, explores the changes between two frames, and therefore is more effective for preserving object details and edge features. Cost-aware matching also improves the inference speed (FPS) in Col 7 of Table 3, which abandons the calculation of neighbourhood cross-correlation in short-term memory and constructs the pixel relationship by dot product that is more efficient. We can observe that the combined improvement strategy of cost-aware and cross-scale improves J by 2.6% and F by 2% compared to the baseline.

Impact of Decoding Mechanism. Compensatory decoding provides an essential information supplement for matching readout results and suppresses false matching to a certain extent by implementing context embedding compensation in the decoding stage. Row 5 of Table 3 demonstrates the improvement of compensatory decoding over the baseline. To study the binding function effect in compensatory decoding, as shown in Table 5, we remove three implements or modules (i.e., context compensation (Context), adaptive weighting (AW), and spatial block (SP)) and evaluate the gains separately. CD-Baseline in Table 5 for the experimental setup in the 5th row of Table 3. On the context removal setting, we simultaneously remove the share-weighted recursive decoder and replace it with two decoders in the cascade that do not share weights. As shown in Table 5, after removing context comparison, we can observe the most significant drop in performance (i.e., 1.4% drop in $J$ and 0.9% drop in $F$). Therefore, context compensation is vital in improving the decoding stage. Furthermore, we migrate our decoding improved mechanism to other existing matching-based methods, as shown in Table 5, which indicates the feasibility and the plug-and-play potential of our compensatory decoding.

Impact of Matching and Decoding Improved Jointly. The contribution of this paper is to explore the role of joint improvement of memory matching and decoding. Rows 6 to 8 from Table 3 show the gain of the matching and decoding jointly improving mechanisms. We can see that the combination of either CD & CA or CD & CS has a more substantial positive effect than the memory matching improved alone. Therefore, we believe the joint improvement of memory matching and decoding proposed in this paper is crucial to the matching-based VOS approach.

Visualize the Readout Features. We visually compared the readout features with AOT [27] as shown in Fig. 5. After cost-aware and cross-scale matching, our initial readout results are significantly better than AOT [27]. We can see that our initial readout still has false matching area at the right wheel of the bike. However, after context embedding, the final readout results suppress false matches and increase some high-response features. This suggests that our approach of jointly improving the matching and decoding stages could facilitate producing more accurate and precise masks.