Video 3D Sampling for Self-supervised Representation Learning

By generating pseudo labels from the raw data, self-supervised learning methods can learn rich representations without leveraging expensive human-annotated labels. Self-supervised image representation learning has witnessed rapid progress. Various proxy tasks have been proposed such as jigsaw puzzle (Noroozi and Favaro, 2016), rotation prediction (Gidaris et al., 2018), colorization (Zhang et al., 2016), inpainting (Pathak et al., 2016), and context prediction (Doersch et al., 2015), to name a few. Video representation learning can be categorized into temporal representation learning and spatiao-temporal representation learning.

Action recognition is one of the most important tasks for video understanding. It takes a video clip as the input and outputs the specific action category of the video. Since the dynamic information is complex to understand, action recognition is challenging.

Based on the 2D CNNs feature extractor, (Simonyan and Zisserman, 2014) proposed two-stream convolutional networks where the results of RGB stream and optical flow stream were fused. TSN (Wang et al., 2016) extracted multiple clips for a video and utilized the whole action video level supervision. (Zhou et al., 2018) built temporal dependencies among video frames for action recognition.

Recently, 3D CNNs feature extractors have attracted much attention due to their strong ability for temporal modeling. C3D (Tran et al., 2015) designed 3D convolutional kernels, which can model spatial and temporal features simultaneously. R3D (Tran et al., 2018) extended C3D with ResNet(He et al., 2016). S3D-G (Xie et al., 2018) replaced 3D CNNs at the bottom of the network with low-cost 2D convolutions.

To encourage the model to learn spatial representations, we design two transformations on the appearance of video clips. Let $I(x,y)$ be the original frame, $I(u,v)$ be the transformed frame. There is a conversion formula which maps ($x$,$y$) to ($u$,$v$):

, where $(x_{i},y_{i}),(u_{j},v_{j})(i,j=1,2,3,4)$ are the coordinates of the four vertices of the original and the transformed image. To explain it in detail, we firstly obtain the coordinates $(x,y)$ of four vertices in the original frame, and then calculate the coordinates $(u,v)$ of vertices after transformation. Then, $m_{i}(i=0,...,7)$ is calculated by the above formula. Thus, every pixel in the original image can find the corresponding location in the transformed frame.

To make it simple, we set $(x_{0},y_{0})=(0,0),(x_{1},y_{1})=(0,H),(x_{2},y_{2})$ $=(W,H),(x_{3},y_{3})=(W,0)$, where $W,H$ denotes the width and the height of the original image. We also set $(u_{0},v_{0})=(0,0)$. In the following, we are going to describe the detail of the transformations.

Scale: In order to change the size (aspect ratio) of the object, we modify the height or width of the video frame. It should be noted that we do not scale the height and width equally (with the same rate), because this only changes the resolution of the image, which is trivial to learn. Moreover, if we change the aspect ratio of the object, and the network can still learn its original proportion, it demonstrates that the network has learned the semantic information of the object.

In scale, we set $(u_{1},v_{1})=(0,b*H),(u_{2},v_{2})=(a*W,b*H),(u_{3},v_{3})=(a*W,0)$. It denotes that the width and height of the transformed video frame is $a$ and $b$ times of that of the original video frame. In our implementation, $(a,b)$ is the hyperparameter and the learning target. Fig: 2 shows an example of $(a,b)=(1,0.3)$.

Projection: Projection transforms a cuboid into a trapezoid. The head end of the trapezoid is smaller than the tail end, which has the effect of expanding objects close to the camera and shrinking distant objects. In order to change the sizes of objects in different regions with different rates, we apply projection on the video frame.

To transform the frame to a trapezoid, we randomly choose one side as the head end and shorten the length. For example, we take the right side as the head end and set $(u_{1},v_{1})=(0,H)$, $(u_{2},v_{2})$ =$(W,(H+c*H)/2),(u_{3},v_{3})=(W,(H-c*H)/2)$. It denotes that the transformed frame is a trapezoid which takes the left side as the bottom end and the right as the head end. The length of the head end is =$c*H$. For spatial projection, $c$ and the head end side is the learning target. Fig. 2 shows an example of $c=0.5$ and the head end is the right side.

Through spatial transformations, we can change the size (aspect ratio) of the objects uniformly (scale) or non-uniformly (projection) whilst maintaining the semantic information. Fig. 3 shows the example of a three-frame video with an object moving in a straight line. Through spatial transformations, we can modify the direction of motions.

To encourage the model to learn rich temporal representations, two temporal transformations on video clips are carried out. We design different frame sampling strategies for different temporal transformations.

Scale: The scale transformation in spatial modifies the size of the object along its width or height. Accordingly, we change the duration of an action in temporal scale. To achieve that, we speed up a video by interval sampling, and the interval is saved as the learning target. Specifically, let $n$ be the speed, the interval is $n-1$. That is, given a video $V=\{v_{i}\}$, where $v_{i}$ is the $i$-$th$ frame in $V$. We generated $V^{s}$, which is s times speed of $V$ as $V^{s}=[{v_{r},v_{r+s-1},v_{r+2(s-1)},...,v_{r+l(s-1)}}]$, where $r+l(s-1)\leq$ the length of $V$, $r$ is a random start frame in $V$ and $l$ is the length of $V^{s}$. In our implementation, $l=16$. For temporal scale, the speed $s$ is the learning target. Fig. 2 shows an example of $s=2$.

Projection: The spatial projection scales objects in different regions of the video frame at different rates. Accordingly, we propose temporal projection to progressively speed up a video so that it has different playback rates in different stages. In our implementation, we use a multi-stage speed-up mode, which means the video has multiple speeds in the same video. To generate the training samples, when a video $V$ is given, we first sample $l_{1}$ frames at $s_{1}-1$ intervals, then $l_{2}$ frames at $s_{2}-1$ intervals. The total length of $V^{p}$ $l=l_{1}+l_{2}$. In our implementation, $l_{1}=l_{2}=8$. Given a specific speed pattern $p=(s_{1},s_{2})$ and $V$, we generate $V^{p}$ as $V^{s_{1},s_{2}}=[{V^{S_{1}},V^{S_{2}}}]$. The generation of $V^{S_{1}},V^{S_{2}}$ is same as the operation in temporal scale. For temporal projection learning, $p$ is the learning target. Fig. 2 shows an example of $p=(1,2)$.

With temporal transformation, we speed up the video straightforwardly (scale) or progressively (projection), which encourages the model to capture rich temporal representations.

Given a video clip, we first apply a spatial transformation and then a temporal transformation on it. Then we feed it to a backbone to extract features and use a multi-task network to predict the specific transformation.

Feature Extraction: To extract video representations, we choose C3D (Tran et al., 2015), R3D, R(2+1)D (Tran et al., 2018), and S3D-G (Xie et al., 2018) as backbones. C3D stacks five 3D convolution blocks with 3$\times$3$\times$3 convolution kernels in all layers. Within the framework of residual learning, R3D block consists of two 3D convolution layers followed by batch normalization and ReLU layers. R(2+1)D are ResNets with (2+1)D convolutions, which decompose full 3D convolutions into a 2D convolution followed by a 1D convolution. Unlike many other 3D CNNs, S3D-G replaces many of the 3D convolutions, especially the 3D convolutions at the bottom of the network, by low-cost 2D convolutions. S3D-G exhibits distinguished feature extraction ability for action recognition.

Category Prediction: To complete the prediction, we take it as a classification task. For each transformation, we fix the parameters and take them as a specific category for classification. To be noted, to make most use of spatial and temporal transformation, we take V3S as a multi-task network.

In Fig. 4, we show the motion speediness curve (Benaim et al., 2020) of cliff diving action. As one can see, the speediness of the action is different in different stages, rather than a constant value. Take the cliff diving action as an example, there is a rapid rise of the motion speediness from stage 1 to stage 3. We argue that if the network can sense the variation of speed, it can better understand the characteristics of an action. V3S proposes to utilize temporal projection to capture the variation of speed, thereby completes the deficiencies in previous speed-based methods.

In this section, we evaluate the effectiveness and discuss the hyperparameters of the designed four transformations on the first split of UCF101. For simplicity, we choose R(2+1)D as the backbone for our ablation studies.

As shown in table 1, we conduct extensive experiments for the selection of each parameter. In table 1, we discuss the influence of hyperparameters for spatial transformations. To explain it further, we discuss $(a,b)$ for spatial scale $S_{S}$, and $c$ for spatial projection $S_{P}$. For $S_{S}$ , we randomly select $(a,b)$ $\in$ {(1,1.15), (1,1.3), (1,1.45), (1.15,1), (1.3,1), (1.45,1)} in the following experiments, because it demonstrates the best performance among the settings (73.2% to 71.1%$\backslash$72.8%). For $S_{P}$, we accordingly select projection magnitude $c$ $\in$ {0.8, 0.65, 0.5} in the following experiments.

In table 1 (bottom), we discuss the hyperparameters for temporal transformations, which are $s$ for $T_{S}$ and $p$ for $T_{P}$. For $T_{S}$ , we randomly select $s$ from $\{1,2\}$, $\{1,2,3\}$ or $\{1,2,3,4\}$. When sampling $s$ $\in$ $\{1,2,3\}$, it demonstrates the best performance (76.4% to 71.0%$\backslash$76.3%). we thus set a sampling speed $s\in\{1,2,3\}$ in the following experiments. For $T_{P}$, we accordingly select the speed pattern $p$ $\in$ {(1,2), (2,3), (3,4), (4,5), (2,1), (3,2), (4,3), (5,4)} in the following experiments.

To combine the designed transformations, we integrate $S_{S}$, $S_{P}$, $T_{S}$, $T_{P}$ in Table 3. After combining spatial scale $S_{S}$ and spatial projection $S_{P}$, V3S achieves 73.8%. V3S also show better performance (78.0%) when combining $T_{S}$ and $T_{P}$. After combining all these four spatial and temporal transformations, the accuracy of 79.1% is achieved, which surpasses the accuracy of the individual spatial or temporal transformations. In summary, the spatial and temporal transformations are complementary, thus with the combination as the final proxy task, more powerful representations can be learned.

Utilizing self-supervised pre-training to initialize action recognition models is an established and effective way for evaluating the representation learned via self-supervised tasks. To verify the effectiveness of our method, we conduct experiments on the action recognition task. We initialize the backbone with V3S pre-trained model, and initialize the fully connected layer randomly. Following the protocol of (Xu et al., 2019), we train backbones for 300 epochs during training and make the fine-tuning procedure stop after 160 epochs. For testing, we sample 10 clips for each video and average the possibility of predictions to obtain the final action category.

For C3D, R3D and R(2+1)D, we set the initial learning rate to be 0.001. The number of the clip frames is 16 and each frame is first resized to 128 $\times$ 171 and randomly cropped to 112 $\times$ 112. For S3D-G, we set the initial learning rate to be 0.01, the input clip length is 64 frames and each frame is first resized to 256 $\times$ 256 and randomly cropped to 224 $\times$ 224. For action recognition, the batch size is set to 8 and the momentum is set to 0.9.

Table 2 shows the split-1 accuracy on UCF101 and HMDB51 for action recognition task. With S3D-G pretrained on UCF101, V3S obtains 85.4% and 53.2% on UCF101 and HMDB51 respectively, outperforms CoCLR (Han et al., 2020b) by 4.0% and 1.1%. With R(2+1)D pretrained on Kinetics-400, V3S achieves 79.2% and 40.4% on UCF101 and HMDB51, which outperforms PacePred(Wang et al., 2020) by 2.1% and 5.4%. With C3D, R3D, V3S obtains 74.8%$\backslash$34.9% and 74.0%$\backslash$38.0%. To be noted, V3S is purely pretext task based, and does not incorporate contrastive learning. However, its performance is better than a series of methods with contrastive loss such as PacePrediction(Wang et al., 2020) and CoCLR(Han et al., 2020b). This can further verify its effectiveness.

To further validate the effectiveness of V3S, we adopt video retrieval as another downstream task. In the process of retrieval, we generate features at the last pooling layer of extractors. For each clip in the testing split, we query top-K nearest videos from the training set by computing the cosine similarity between two feature vectors. When the testing video and a retrieved video are from the same category, a correct retrieval is counted.

Video retrieval results on UCF101 and HMDB51 are listed in Table 4 and Table 5 respectively. Note that we outperform SOTA methods with different backbones from Top1 to Top50. These results indicate that in addition to providing a good weight initialization for the downstream model, V3S can also extract high-quality and discriminative spatiao-temporal features.

In this section, we exploit action similarity labeling task (Kliper-Gross et al., 2011) to verify the quality of the learned spatio-temporal representations from another perspective on the ASLAN dataset(Kliper-Gross et al., 2011). Unlike action recognition task, the action similarity labeling task focuses on action similarity (same/not-same). The model needs to determine whether the action categories of the two videos are the same. This task is quite challenging as the test set contains never-before-seen actions.

To evaluate on the action similarity labeling task, we use pre-trained model to extract features from video pairs, and use a linear SVM for the binary classification. Specifically, for each video pairs, each video is first split into 16-frame clips with stride length 8. The network takes these video clips as input to extract clip-level features from the pool3, pool4 and pool5 layers. Then, the clip-level features are averaged and L2 Normalized to get a spatiao-temporal video-level feature. In order to measure the similarity of these two video-level features, we calculate 12 different distances for each feature as described in (Kliper-Gross et al., 2011). A 36-dimensional feature is obtained by concatenating these three types of video features and this feature is normalized to ensure that the scale of each distance is the same, following (Tran et al., 2015). Finally, a linear SVM is used to determine whether the two videos are in the same category or not.

Action similarity results on ASLAN are listed in Table 6. The results show that the accuracy of V3S on R3D outperforms that of the previous methods, and it further shortens the gap between supervised and unsupervised methods. It demonstrates that the features extracted by the V3S network have excellent intra-class similarity and inter-class dissimilarity.

In order to gain a better understanding of what V3S learns, we adopt the Gradient-weighted Class Activation Mapping (Grad-CAM) (Selvaraju et al., 2017), an improved version of CAM(Zhou et al., 2016) to visualize the attention map.

Fig. 5 shows the samples (baseball pitching, archery, band marching, baseball pitch, walking with dog, wall pushups, typing) of such heat maps. One can see that the highly activated regions of these heat maps have a great correlation with the movement of actions. For example, in different stages of the archery action, the activation area varies along the motion. When the man takes out of the arrow, the activation area concentrates on the left hand that is drawing the arrow, and when the man is drawing the bow, the activation area moves to the right hand.