Contrastive Masked Autoencoders for Self-Supervised Video Hashing

ConMH is a simple yet effective masked autoencoders framework for self-supervised video hashing. To better understand the semantic information of the video, ConMH reconstructs the original video given its highly temporal-masked frames. Following MAE (He et al. 2022), the ConMH architecture is asymmetric. Specifically, the encoder of ConMH only takes a small randomly sampled subset of the original frames as input to extract latent representations and binary hash tokens. And the decoder of ConMH reconstructs the original video frames from generated hash tokens and masked tokens, illustrated in Figure 2.

Masking. To make the model better understand the semantic information of the video, we adopt a highly temporal-masking strategy. Specifically, given the frames $\{v_{i}^{m}\}_{m=1}^{M}$ of a video, we randomly sample a small subset of the frames for training and mask the remaining ones. Typically, a higher masking ratio means less available information the model can get, leading to a more challenging reconstruction task. Given these intuitions, the model can learn high-quality representations and generate effective hash codes by a reasonable self-supervised pretext task. Besides, with a high masking ratio, the encoder only operates on a small subset of an input, accelerating the training phase.

ConMH Encoder. Our encoder is composed of a standard Transformer (Vaswani et al. 2017) and a hash layer. The multi-head attention module in Transformer is able to capture the temporal information and correlations among video frames, which is critical for video understanding. Different from the setting in (Vaswani et al. 2017), we only use a small subset (e.g., 25%) of the full video frames as the transformer input during training. Through multiple transformer blocks, we can obtain the latent representations $\{h_{i}^{m}\}_{m=1}^{\tilde{M}}$ for each video, where $\tilde{M}$ is the frame number of the sampled subset. To obtain compact binary code, we further adopt a hash layer to generate binary hash tokens $\{t_{i}^{m}\}_{m=1}^{\tilde{M}}$. Finally, we generate a binary code for each video by operating the mean pooling and sign function on its binary hash tokens. Since the sign function is not differentiable, we employ the approach in (Li et al. 2021) to realize the training phase in an end-to-end manner.

ConMH Decoder. Our decoder is composed of another series of Transformer blocks. Different from the encoder, our decoder takes binary hash tokens and corresponding masked tokens as input. Positional encoding is also inserted to identify the masked token with their corresponding frames. The decoder is only used during the training phase, so we can flexibly design the decoder architecture. Following the asymmetric nature of MAE (He et al. 2022), we design a small Transformer decoder, with which the encoder has to learn high-quality feature representations and hash codes to complete the reconstruction task.

Reconstruction Task. After ConMH encoder and decoder module, we can obtain a full set of reconstruction frames $\{\tilde{v}_{i}^{m}\}_{m=1}^{M}$. Following MAE, we only reconstruct the masked frames using corresponding reconstruction frames $\{\tilde{v}_{i}^{m}\}_{m=1}^{M-\tilde{M}}$. We adopt Mean Square Error (MSE) to measure the reconstruction quality as following:

Although the reconstruction task above can help the model to extract the discriminative visual semantic information among videos, it neglects the similarity structure between videos, which is critical for the retrieval task. In this subsection, we will discuss how to combine instance-discriminative contrastive learning with the proposed masked autoencoder in a single stage.

Debiased Contrastive Loss. In ConMH, we consider random temporal masking as a data augmentation for contrastive learning and sample two non-overlapping small subsets from a single video, as shown in Figure 1. Then we feed these two subsets to the ConMH encoder to obtain two hash codes from different views of the same video, denoted as $b_{i}$ and $b_{j}$. For ConMH, we adopt a debiased contrastive loss (Chuang et al. 2020) $\ell^{de}(i,j)$ to correct the sampling bias of same-label data points:

where $\tau$ denotes a temperature parameter and $\rho$ is the class probability indicating that two randomly sampled videos belong to the same class with probability $\rho$. Because we cannot get the actual number of categories, we treat $\rho$ as a hyperparameter. Given a mini-batch of $N$ samples, we calculate the contrastive loss over all samples below:

Loss Objective. The overall loss of ConMH for network optimization is a combination of reconstruction loss and constrastive loss as following,

where $\alpha$ is a hyperparameter that balances the above-mentioned two losses.

Dataset: We evaluate our ConMH on three large-scale video datasets FCVID (Jiang et al. 2017), ActivityNet (Caba Heilbron et al. 2015) and YFCC (Thomee et al. 2015). FCVID contains 91,223 videos in 239 categories. Following (Song et al. 2018), we use 45,585 videos among FCVID for training and 45,600 videos for retrieval database and queries. ActivityNet covers 200 categories of various human activities. Due to the lack of original test labels, we use 9,722 videos as our training set and the validation set as our test set. Similar to (Li et al. 2021), we uniformly sample 1,000 videos in 200 categories as queries and the remaining 3,758 videos as retrieval database. YFCC contains 0.8M videos in 80 categories, which is one of the largest public video datasets available in the real world. We use 409,788 videos for training and 101,256 videos for testing. We randomly choose 1,000 videos with non-zero labels among the testing set as queries and the remaining ones as our retrieval database.

Evaluation Protocols: Following the same evaluation protocols in (Li et al. 2021), mean Average Precision at top-K retrieval results (mAP@K) is used to evaluate our ConMH performance. Videos are sorted according to their Hamming distance from the query. We evaluate the retrieval results with code lengths of 16, 32, and 64 bits.

Implementation Details: Similar to BTH (Li et al. 2021), on FCVID and YFCC datasets, we uniformly sample 25 frames for each video and use VGG-16 network (Simonyan and Zisserman 2014) pre-trained on ImageNet (Russakovsky et al. 2015) to extract 4096-D frame features. And on ActivityNet, we uniformly sample 30 frames for each video and use ResNet50 (He et al. 2016) pre-trained on ImageNet to extract 2048-D frame features.

We follow the design of ViT-small (Dosovitskiy et al. 2020) to build our model. Specifically, for the Transformer encoder, we set the depth as 12, multi-head number as 6, and hidden size as 256. For the Transformer decoder, we set the depth as 2, multi-head number as 3, and hidden size as 192.

During training, we set the batchsize as 512, masking ratio as 0.75, $\alpha$ as 1.0, $\tau$ as 0.5, $\rho$ as 0.1, and train our model for 800 epochs, 500 epochs, and 40 epochs for FCVID, ActivityNet and YFCC respectively.

We set the initial learning rate as 0.0001, and decay it to 90% every 20 epochs with a minimal learning rate of 0.00001. We optimize our model by using Adam optimizer algorithm (Kingma and Ba 2014). Our model is implemented in Pytorch with an Nvidia RTX2080Ti GPU.

Comparisons with State-of-the-arts: We compare our ConMH with several state-of-the-art self-supervised video hashing methods: ITQ (Gong et al. 2012), MFH (Song et al. 2011), DH (Erin Liong et al. 2015), SSTH (Zhang et al. 2016), JTAE (Li et al. 2017), SSVH (Song et al. 2018), NPH (Li et al. 2019), BTH (Li et al. 2021) and MCMSH (Hao et al. 2022). Following (Song et al. 2018), we extend the image hashing methods ITQ and DH to video hashing by applying them on CNN frame features.

As Figure 3 shows, our ConMH outperforms other methods on three large-scale video datasets with all code lengths, demonstrating its superiority in retrieval accuracy. Specifically, compared with the strongest competitor MCMSH, our ConMH surpasses it by 5.7%, 11.4%, and 4.7% with 16 bits in terms of mAP@5 on FCVID, ActivityNet, and YFCC, respectively. ConMH achieves impressive retrieval results by incorporating video semantic information and video similarity relationship understanding in a single stage. These two tasks are critical for the retrieval task and can complement each other to further improve retrieval performance when combined together.

Besides, from Figure 3, ConMH surpasses other methods by a considerable margin with low-bit hash codes, proving the superiority of ConMH in high real-time demand scenarios. In particular, as we decrease the bit length, more semantic information will be lost by the compressive hashing technique, leading to poor retrieval performance. However, instead of representation learning followed by quantization, we design hashing-oriented learning objectives to suppress the quantization loss. In this way, more semantic information will be retained by hash codes. On the contrary, when the bit-length is increasing, the quantization loss is reduced with a higher information capacity, and the hashing learning task also becomes simpler. Therefore, we can see different methods show approaching performance under 64 bits or larger bit-length settings. This phenomenon has widely occurred in previous methods, as shown in Figure 3 (f), (i).

Precision-Recall Curves: The precision-recall curves of BTH, MCMSH and ConMH on FCVID and ActivityNet are shown in Figure 4. As can be seen, compared with BTH and MCMSH, our ConMH achieves higher precision at the same recall rate with all code lengths.

Cross-dataset Evaluation Comparisons: To investigate the generalization of our ConMH, we evaluate cross-dataset retrieval performance among different methods in this subsection. In detail, we train various methods on FCVID and test them on YFCC. We compare the retrieval results with a single-dataset case (both training and testing on YFCC). Table 1 shows mAP@20 results of various methods with 64 bits in the cross-dataset setting. Although all methods degrade in performance, ConMH still performs the best. Besides, the performance drop of ConMH is negligible, showing the generalization of ConMH across different datasets.

Qualitative Results: We randomly sample 10 categories in FCVID, each with 80 videos, constituting FCVID-small. And we show the t-SNE visualizations of MCMSH and ConMH with 64 bits on FCVID-small. As shown in Figure 5, hash codes generated by ConMH better distinguish different categories of videos.

Effectiveness of $\mathcal{L}_{recon}$ and $\mathcal{L}_{contra}$: To analyse the effectiveness of two pretext tasks (i.e., $\mathcal{L}_{recon}$ and $\mathcal{L}_{contra}$) of ConMH, we construct several ConMH variants: (1) ConMH w/o $\mathcal{L}_{contra}$ removes the relational understanding task and only uses $\mathcal{L}_{recon}$ to train the model. (2) ConMH w/o $\mathcal{L}_{recon}$ removes the reconstruction task and only uses $\mathcal{L}_{contra}$ to train the model. (3) ConMH-norm removes the temporal-masking operation based on ConMH w/o $\mathcal{L}_{contra}$ and reconstructs the full video frames during training. As shown in the first two rows of Table 2, the results of ConMH w/o $\mathcal{L}_{contra}$ outperform ConMH-norm with all code lengths, showing that the model can better understand the semantic information of video from our highly temporal-masking reconstruction task. Besides, from Table 2, training with only $\mathcal{L}_{recon}$ or $\mathcal{L}_{contra}$ both lead to the reduction of model performance compared with ConMH. With only $\mathcal{L}_{recon}$, although the model can understand the semantic information of video, it ignores the similarity relationship between videos which is disadvantageous for the retrieval task. And with only $\mathcal{L}_{contra}$, the model cannot understand the video content well. Therefore, taking care of both the reconstruction and discrimination tasks is critical for good retrieval performance.

Furthermore, we explain the effectiveness of hashing-oriented learning objectives in ConMH on hash code learning. Most existing methods perform general feature learning (GFL) first, making the hidden representations discriminative. Specifically, they use the hidden representations to complete pretext tasks and binarize them directly to obtain the hash codes. Besides, they introduce a quantization loss to reduce the quantization error from the binarization. Such an approach is hard to balance feature learning and quantization error reduction. However, we directly use the generated hash codes to complete the reconstruction and discriminative tasks, which enables the hash codes to retain more information. We have also tried general feature learning approach to use hidden representations to complete the pretext tasks and introduce a quantization loss, denoted as ConMH w/ GFL. As shown in Table 2, the hashing-oriented approach is better.

Retrieval Performance with Swin Transformer: Recently, Swin Transformer (Liu et al. 2021) has proved its superiority in image feature extracting, compared with the CNN model. To investigate how ConMH performs using a stronger feature extraction backbone, we use the ImageNet-pretrained Swin Transformer backbone to replace the CNN backbone. Then we evaluate the performance of different methods under such a stronger setting. As shown in Table 3, with a stronger backbone, ConMH still performs better than BTH and MCMSH, proving the scalability of ConMH.

Effects of Different Masking Ratios: We evaluate model performance with different masking ratios on FCVID with 64 bits. As shown in Figure 6 (a), a high masking ratio (i.e., 50% to 75%) works well. Both low and extremely high masking ratios will degrade model performance. Furthermore, we study the effect of masking ratios in the case of only a single task. As shown in Figure 6 (b), when only the reconstruction task is employed, a high masking ratio (e.g., 90%) works better. As for the contrastive task, a moderately lower masking ratio (e.g., 30%) is preferred. The reasons are two-fold. On the one hand, a high masking ratio will benefit intra-sample semantic understanding via reconstruction task. On the other hand, the large information loss it incurs will increase the difficulty of learning semantically-invariant information between positive samples in the contrastive task and even harm the task convergence. Besides, we find that adopting a compromise masking ratio (e.g., 50%-75%) helps to balance and unify two objectives in one learning framework effectively. As we can learn from the above results, the complete version of ConMH outperforms both single-task models when choosing proper masking ratios.

Effects of Different Masking Strategies: In this subsection, we investigate the impact of different masking strategies. From Table 4, we can learn that the entire frame masking strategy achieves the best performance, while tube-based masking and token-based masking strategies show not very satisfactory results. The reason might be that our method (which follows the standard SSVH settings) intrinsically operates on frame-level granularity. Therefore, when combining our method with the tube-based or token-based masking strategies, the input video frames will suffer from a large proportion of information loss. It results in weak frame-level features and thus adversely reduces the effectiveness of the reconstruction task. Besides, we have to preserve all the frames for the reconstruction task and suffer from larger memory overhead.

Effects of Different Sampling Strategies: In this subsection, we investigate the impact of different sampling strategies as shown in Table 5. In general, both overlapped and non-overlapped sampling strategies behave similarly in terms of performance. It might be due to the information redundancy among successive frames. Besides, the long and short sampling strategy does not exhibit an advantage in our case. We guess the reason might be that the constrained pattern reduces the sampling’s diversity (or randomness), which might be less effective on untrimmed videos.

Effects of Different Model Sizes: In this subsection, we explore the impact of different model sizes on FCVID with 64 bits. As shown in the first ten rows of Table 6, using a small decoder to reconstruct the original videos is better. When the decoder is big (e.g., with 12 depth or 1024 width), the retrieval performance of the model, in turn, will degrade. That is because a small decoder needs more semantic information to complete the reconstruction task, forcing the encoder to generate more informative representations and hash tokens. However, the decoder cannot be too small (e.g., too small width or a simple FC layer), which will make the reconstruction task too difficult to accomplish, leading to poor performance. Although the decoder structure can be chosen arbitrarily because it will be discarded during testing, it still affects the training time. Therefore, we comprehensively consider the training time and model performance, then set the depth and width of the decoder as 2 and 192, respectively. Besides, as shown in the last three rows of Table 6, when the encoder size becomes larger, the performance of the model is correspondingly improved, demonstrating that ConMH is a scalable framework for self-supervised video hashing, which still works well even with a huge model structure. However, large model structures make inference time long, violating the real-time requirement of video hashing. So we choose ViT-small as our defaulting encoder structure.