Self-supervised Video Representation Learning by Context and Motion Decoupling

There is an increasing interest in learning representation from unlabeled videos. Many works explore the intrinsic structure of videos and design video-specific pretext tasks, such as estimating video playback rates [5, 49, 57], verifying temporal orders of clips [33, 27, 54], predicting video rotations [21], solving space-time cubic puzzles [22], and dense predictive coding [17, 18].

Contrastive learning has recently shown great success in image representation learning [35, 19, 10], where self-supervised pretraining is approaching the performance of the supervised counterpart [12, 11]. More recently, contrastive learning has been introduced to video domain [42, 50, 38], where clips from the same video are pulled together while clips from different videos are pushed away. Another type of contrastive learning methods employ adaptive cluster assignment [8, 3, 2], where the representation and embedding clusters are simultaneously learned. However, since most methods apply contrastive learning on raw RGB clips without separating the motion information, the learned representation may suffer from the context bias problem. This work also follows the contrastive learning paradigm, but we explicitly decouple the motion supervision from the context bias in the pretext task.

Considering the multi-modality of videos, many works explore mutual supervision across modalities to improve the learned representation. For example, they regard the temporal or semantic consistency between videos and the corresponding audios [23, 2, 53, 37], narrations [32], or a combination of different modalities [36] as a natural source of supervision for representation learning.

A recent work named DSM [48] shares some similarities with our framework. It also tries to enhance the learned video representation by decoupling the scene and the motion. However, it achieves this by simply changing the construction of positive and negative pairs in contrastive learning, and it still learns on raw video clips; while our approach explicitly decouples the context and motion information in the source of supervision. Besides, the significantly better performance of our approach than DSM on downstream tasks also verifies the superiority of our work.

Video compression techniques (\eg, H.264 and MPEG-4) usually store only a few key frames completely, and reconstruct other frames using motion vectors and residual errors from the key frames. Taking advantage of this, many works propose to build video models directly on the compressed data to achieve faster inference and better performance [60, 61, 52, 39]. Pioneering works [60, 61] replace the optical flow stream in two-stream action recognition models [40] with a motion vector stream, thereby avoiding slow optical flow extraction. CoViAR [52] further utilizes all modalities (\ie, I-frames, motion vectors, and residuals) in compressed video and bypasses the decoding of RGB frames for efficient video action recognition. DMC [39] improves the quality of the motion vector maps by jointly learning a generative adversarial network. More recently, Wang et al. [51] presents a method for fast object detection in compressed video, showing the potential of using compressed data for more fine-grained tasks. This work uses MPEG-2 Part2 [25] for video encoding, as practiced in CoViAR, where every I-frame is followed by 11 consecutive P-frames. We take the I-frames and motion vectors as proxies of the context and motion information for self-supervised video representation learning.

Motion prediction task usually refers to deduce the states (\eg, position, orientation, speed, or posture) of an object in a near future [14, 55]. Example applications include human pose prediction [14, 31, 30] and traffic prediction [55, 13]. Typical models for solving this task include RNNs [26, 31], Transformers [1, 58], and graph neural networks [30]. In this work, we leverage a simple Transformer encoder-decoder network [46] for predicting future motion features based on current visual observations.

This work presents a self-supervised video representation learning method with two decoupled pretext tasks: a context matching task for learning coarse-grained context representation; and a motion prediction task for learning fine-grained motion representation. For efficiency, we use the I-frames and motion vectors in compressed video as the sources of supervision for context and motion tasks, respectively. Figure 2 shows an overview of our framework, where the V-Network, I-Network, and M-Network are used to extract video, context, and motion features, respectively.

In the context matching task, we compare global features of video clips and I-frames. An InfoNCE loss [35] is cast, where (video clip, I-frame) pairs from the same videos are pulled together, while pairs from different videos are pushed away. By matching still images to video clips, the learned representation is supposed to capture the global and coarse-grained contextual information of the video.

Unlike context, motion information is relatively sparse, localized and fine-grained. Therefore, we prefer to use a pointwise task to guide the V-Network to capture motion representation. A simple choice is to estimate motion vectors correspond to a video clip. However, this would lead the model to learn low-level offsets (\eg, optical flow) instead of high-level trajectories or actions that are more preferred. In this work, we use motion prediction as the pretext task, where visual data of the current clip is used to predict motion information in a near future. The assumption is that, to predict motion dynamics in future frames, V-Network needs to extract low-level movements from the video and reorganize them into high-level trajectories. In this way, the learned representation can capture long-term motion information, which is beneficial for downstream tasks.

In our implementation, instead of directly predicting the motion vector values, we estimate the features of motion vector maps using an encoder-decoder network (\ie, Transformer [46]), and compare the predicted and “groundtruth” motion features in a pointwise way. We find that leads to more stable pretraining and better transferring performance. The context matching and motion prediction tasks are jointly optimized in an end-to-end fashion. Next we will introduce these pretext tasks respectively in details.

The context of a video is relatively static in a period of time. It depicts a global environment in which the action takes place. We present a context matching task for the video model to capture such information, where the source of supervision comes from the I-frames extracted from the compressed video. A brief overview of the context matching process is shown in the upper half of Figure 2.

Specifically, we extract features $\mathbf{x}_{i}\in\mathbb{R}^{C_{1}\times T_{1}\times H_{1}\times W_{1}}$ of a random clip in video $i$, and features $\mathbf{z}_{i}\in\mathbb{R}^{C_{2}\times H_{2}\times W_{2}}$ of a random I-frame surrounding the clip. Then we use global average pooling to obtain their global representation $\mathbf{x}_{i}^{\prime}\in\mathbb{R}^{C_{1}}$ and $\mathbf{z}_{i}^{\prime}\in\mathbb{R}^{C_{2}}$. Following the design improvements used in recent unsupervised frameworks [10, 12], we apply two-layer MLP heads $g^{V}$ and $g^{I}$ on the clip and I-frame features respectively to obtain $\mathbf{x}_{i}^{*}=g^{V}(\mathbf{x}_{i}^{\prime})\in\mathbb{R}^{C}$ and $\mathbf{z}_{i}^{*}=g^{I}(\mathbf{z}_{i}^{\prime})\in\mathbb{R}^{C}$. Then the InfoNCE loss is applied:

where $B$ denotes the number of samples in the minibatch, $\cos(\mathbf{z},\mathbf{x})=(\mathbf{z}^{\mathrm{T}}\mathbf{x})/(\|\mathbf{z}\|_{2}\cdot\|\mathbf{x}\|_{2})$ denotes the cosine similarity between $\mathbf{z}$ and $\mathbf{x}$, and $\tau$ is a temperature adjusting the scale of cosine similarities. The loss function pulls video clips and I-frames from the same videos together, and pushes those from different videos far apart.

Compared with contextual information, motion information is more fine-grained and position (in $x$, $y$, and $t$ dimensions) sensitive. To encourage the video network to capture high-level and long-term motion information, we design a motion prediction task where visual data of current clip is used to predict motion dynamics in the near future. We use motion vectors extracted from the compressed video as the source of supervision. The lower half of Figure 2 shows a brief overview of the motion prediction process.

Specifically, we extract features $\mathbf{x}_{i}\in\mathbb{R}^{C_{1}\times T_{1}\times H_{1}\times W_{1}}$ from a clip of video $i$, and features $\mathbf{v}_{i}\in\mathbb{R}^{C_{3}\times T_{3}\times H_{3}\times W_{3}}$ from the motion vector maps in a near future after the clip. Subsequently, we feed $\mathbf{x}_{i}$ to an encoder-decoder network $\mathcal{T}$ (\ie, Transformer [46] or ConvGRU [4, 18]) to predict the motion features $\hat{\mathbf{v}}_{i}=\mathcal{T}(\mathbf{x}_{i})\in\mathbb{R}^{C_{3}\times T_{3}\times H_{3}\times W_{3}}$. The “groundtruth” and predicted motion features are then flattened and fed into two-layer MLP heads $g_{1}^{M}$ and $g_{2}^{M}$ respectively to obtain projected embeddings $\mathbf{v}_{i}^{*}=g_{1}^{M}(\mathrm{flatten}(\mathbf{v}_{i}))\in\mathbb{R}^{C\times N}$ and $\hat{\mathbf{v}}_{i}^{*}=g_{2}^{M}(\mathrm{flatten}(\hat{\mathbf{v}}_{i}))\in\mathbb{R}^{C\times N}$, where $N=T_{3}\cdot H_{3}\cdot W_{3}$. The pointwise InfoNCE loss is then applied to $\mathbf{v}_{i}^{*}$ and $\hat{\mathbf{v}}_{i}^{*}$:

where $i,k=1\sim B,~{}j,l=1\sim N$, $\mathbf{v}_{ij}^{*}\in\mathbb{R}^{C}$ denotes the $j$th column of $\mathbf{v}_{i}^{*}$. In the loss function, only feature points corresponding to the same video $i$ and at the same spatial and temporal position $(x,y,t)$ are regarded as positive pairs, otherwise they are regarded as negative pairs. The pointwise InfoNCE loss aims to lead the video network to learn fine-grained motion representation.

We test several modeling options and configurations of the motion prediction task, \eg, whether to predict current or future motion information, whether to match features using the InfoNCE loss or to directly predict motion vector values, and the use of different encoder-decoder networks, \etcComparison of these settings is shown in Table 1. To summarize, we find that: 1) Predicting future motion information leads to significantly better video retrieval performance compared with estimating current motion information; 2) Matching predicted and “groundtruth” motion features using the pointwise InfoNCE loss brings better results than directly estimating motion vector values; 3) Different encoder-decoder networks lead to similar results, while using Transformer performs slightly better.

In this work, we follow the optimal setting, where we predict future motion features using a Transformer network, and we employ the pointwise InfoNCE loss for training the model. When applying the Transformer network, we simply consider the input $\mathbf{x}_{i}\in\mathbb{R}^{C_{1}\times T_{1}\times H_{1}\times W_{1}}$ as a $1$-D sequence of length $T_{1}\cdot H_{1}\cdot W_{1}$, and the output $\hat{\mathbf{v}}_{i}\in\mathbb{R}^{C_{3}\times T_{3}\times H_{3}\times W_{3}}$ as a a $1$-D sequence of length $T_{3}\cdot H_{3}\cdot W_{3}$.

We linearly combine the context matching loss and the motion prediction loss to obtain the final loss:

where the $\alpha$ is a scalar hyper-parameter within $[0,1]$. We simply set $\alpha=0.5$ in our experiments, where the context and motion losses are equally weighted. The V-Network, I-Network, M-Network, Transformer $\mathcal{T}$, and all MLP heads (\ie, $g^{V}$, $g^{I}$, $g^{M}_{1}$, and $g^{M}_{2}$) are jointly optimized with loss function (3) in an end-to-end fashion.

This section validates several modeling and configuration options in our framework. The finetuning or video retrieval results on UCF101 and HMDB51 datasets are used as the performance indicators.