Self-Supervised Video Representation Learning by Video Incoherence Detection

To utilize VID, we first generate incoherent clips from raw videos. Given a raw video $V$, the incoherent clip $\mathcal{V}_{inc}$ is constructed by $k$ sub-clips $\mathcal{V}_{1},\mathcal{V}_{2},...,\mathcal{V}_{k}$ sampled from $V$ with a certain length of incoherence between each other. The location $L_{loc}$ and length $l_{inc}$ of incoherence are both randomly generated. The length of incoherence $l_{inc}$ between sub-clips is limited within the range of:

$$l_{inc}\in[l_{inc}^{min},l_{inc}^{min}+1,...,l_{inc}^{max}],$$

(1)

where $l_{inc}^{min},l_{inc}^{max}$ are both hyper-parameters indicating the upper and lower bounds of the incoherence length, respectively. The purposes of this constraint are two-fold. Firstly, the constraint on $l_{inc}$ is necessary for our Incoherence Length Detection (LeD). Secondly, the constraint on incoherence length prevents the incoherence between sub-clips from being either too vague or too obvious, which avoids learning trivial solutions. For simplicity, we take the case where $\mathcal{V}_{inc}$ is constructed by two sub-clips as an example to thoroughly illustrate the generation process of incoherent clips as shown in Figure 2.

Selection of incoherence location. The incoherence location $L_{loc}$ refers to the relative concatenation location between two sub-clips. Formally, given the desired length of incoherent clips $l_{0}$, the location of incoherence $L_{loc}$ is uniformly selected as:

	$\displaystyle l_{1}\in\{1,2,...,l_{0}-1\},\ \ l_{2}=l_{0}-l_{1},$		(2)
	$\displaystyle L_{loc}=l_{1}-1,$		(3)

where $l_{1}$, $l_{2}$ are the length of $\mathcal{V}_{1}$, $\mathcal{V}_{2}$ as illustrated in Figure 2(b), where squares in different colors refer to allocated frame positions for different sub-clips in $\mathcal{V}_{inc}$. $L_{loc}$ is the relative location of incoherence where sub-clips concatenate as well as the label for the following LoD task.

Hierarchical selection of sub-clips. Given the sub-clip lengths $l_{1},l_{2}$, sub-clips $\mathcal{V}_{1}$, $\mathcal{V}_{2}$ are hierarchically sampled from the raw video $V$ with incoherence between each other. The incoherent clip $\mathcal{V}_{inc}$ is generated as the temporal concatenation of $\mathcal{V}_{1}$, $\mathcal{V}_{2}$. While previous work (Wang, Jiao, and Liu 2020) proposed to sample frames by looping over the raw video, such strategy is not compatible with our proposed VID since it could introduce unexpected incoherence when looping from the end to the start of the video. Instead, to preserve the sequential relationship of the raw video, we propose a hierarchical sampling strategy that maximizes the sample range of each sub-clip while satisfying Equation 1.

Illustrated as the upper row in Figure 2(c), given the raw video $V$ of the length $T$, the sample range $T_{1}$ of the first sub-clip $\mathcal{V}_{1}$ is determined by reserving sufficient frames for subsequent sub-clips. In this way, the sub-clip $\mathcal{V}_{2}$ can be sampled from the rest of the raw frames wherever the $\mathcal{V}_{1}$ locates in $T_{1}$, which preserves the sequential relationship and satisfies the constraint in Equation 1. Formally, given $l_{2}$ and $l_{inc}^{min}$, the sample range $T_{1}$ is computed as:

	$\displaystyle t_{1}^{min}$	$\displaystyle=1,$		(4)
	$\displaystyle t_{1}^{max}$	$\displaystyle=T-l_{inc}^{min}-l_{2},$		(5)
	$\displaystyle T_{1}$	$\displaystyle=\{t_{1}^{min},t_{1}^{min}+1,...,t_{1}^{max}\},$		(6)

where $t_{1}^{min}$, $t_{1}^{max}$ are the lower and upper bound of the range $T_{1}$. Given the range $T_{1}$, $\mathcal{V}_{1}$ is uniformly sampled as $\mathcal{V}_{1}\in T_{1}$ illustrated as the lower row in Figure 2(c).

Hierarchically, the range of the second sub-clip $T_{2}$ is decided by the sampled sub-clip $\mathcal{V}_{1}$ and the range of $l_{inc}$ in Equation 1. As shown in the upper row of Figure 2(d), given the raw frame index of the last frame in $\mathcal{V}_{1}$ denoted as $m_{1}=\max(\mathcal{V}_{1})$, the sample range $T_{2}$ is computed as:

	$\displaystyle t_{2}^{min}$	$\displaystyle=m_{1}+l_{inc}^{min}+1,$		(7)
	$\displaystyle t_{2}^{max}$	$\displaystyle=\min(m_{1}+l_{inc}^{max}+l_{2},T),$		(8)
	$\displaystyle T_{2}$	$\displaystyle=\{t_{2}^{min},t_{2}^{min}+1,...,t_{2}^{max}\},$		(9)

where $t_{2}^{min}$, $t_{2}^{max}$ are the upper and lower bounds which ensure that $l_{inc}$, the length of incoherence between $\mathcal{V}_{2}$ and $\mathcal{V}_{1}$, always satisfies the constraints in Equation 1. Similar to $\mathcal{V}_{1}$, the second clip $\mathcal{V}_{2}$ is uniformly sampled as $\mathcal{V}_{2}\in T_{2}$, as in the lower row of Figure 2(d).

Given the sub-clips $\mathcal{V}_{1},\mathcal{V}_{2}$, the incoherent clip $\mathcal{V}_{inc}$ and its label $L_{len}$ for Incoherence Length Detection (LeD) task are generated as:

	$\displaystyle\mathcal{V}_{inc}$	$\displaystyle=\mathcal{V}_{1}\oplus\mathcal{V}_{2},$		(10)
	$\displaystyle l_{inc}$	$\displaystyle=\min(\mathcal{V}_{2})-\max(\mathcal{V}_{1}),$		(11)
	$\displaystyle L_{len}$	$\displaystyle=l_{inc}-l_{inc}^{min},$		(12)

where $\oplus$ indicates the concatenation of two sub-clips $\mathcal{V}_{1}$, $\mathcal{V}_{2}$ along the temporal dimension.

We propose two novel self-supervised tasks, including Incoherence Location Detection (LoD) and Incoherence Length Detection (LeD) to detect the incoherence in incoherence clips while maximizing the mutual information between different incoherent clips from the same raw video. Specifically, given an incoherent clip $\mathcal{V}_{inc}$, the high-level representation is first extracted as $h=f(\mathcal{V}_{inc})$ where $f(\cdot)$ denotes the encoder. Given the representation $h$, the optimization objectives of VID include three components: Incoherence Location Detection (LoD), Incoherence Length Detection (LeD) and Intra-Video Contrastive Learning (ICL).

Incoherence Location Detection (LoD). Given the high-level representation $h$ and its location label $L_{loc}$, the network is required to predict the location of incoherence in $\mathcal{V}_{inc}$. This is mainly inspired by humans’ sensitivity towards the loss frames within video clips. The network is driven to identify the abnormal motion caused by incoherence, which encourages the network to learn semantic representation about videos. The LoD task is formulated as a single-label classification problem. Given the representation $h$ and label $L_{loc}$, the network is optimized by the cross-entropy loss illustrated as:

$$l_{LoD}=-\sum_{i=0}^{l_{0}-1}y^{loc}_{i}\log(\frac{\exp{(z^{loc}_{i})}}{\sum_{j=0}^{l_{0}-1}\exp{(z^{loc}_{j}})}),$$

(13)

where $z^{loc}\in\mathbb{R}^{l_{0}-1}$ is the output of fully-connected layers $\phi^{loc}(\cdot)$ given the representation $h$ as input. $y^{loc}\in\mathbb{R}^{l_{0}-1}$ is the ground-truth label vector whose element at $L_{loc}$ equals 1 while the rest to 0. In practice, given a mini-batch of representation $Z^{loc}$, Equation 13 is applied to each representation in $Z^{loc}\in\mathbb{R}^{N\times(l_{0}-1)}$, where $N$ denotes the batch size. The $\mathcal{L}_{loc}$ loss is then calculated as the average of all losses of representation in $Z^{loc}$.

Incoherence Length Detection (LeD). In addition to LoD, given the high-level representation $h$ and its corresponding label $L_{len}$ of Equation 12, the network is required to predict the length of incoherence. The proposed LeD task is designed as a regularization measure to avoid trivial learning. In some cases, it is possible that the incoherence locates at the period when the distribution of low-level representation intensively changes (e.g. intensive movement of the camera or sudden changes of light conditions). This could cause a distinct difference in low-level representation between sub-clips of the incoherent clip $\mathcal{V}_{inc}$, leading to trivial learning during LoD. compared with LoD which extracts semantic representation, our proposed LeD can be regarded as a simple yet challenging task that extracts additional temporal information by enforcing the network to deduce the length of incoherence with respect to the raw video. In practice, the accuracy of LeD is relatively low (with Top1 about 25%), while our ablation experiment shows that it can bring noticeable improvement to the performance mainly because it can improve the robustness of VID towards intensive changes of low-level information.

Similar to LoD, the LeD task can also be formulated as a classification problem, where cross-entropy loss is utilized for optimization as:

$$l_{LeD}=-\sum_{i=0}^{\Delta l_{inc}}y^{len}_{i}\log(\frac{\exp{(z^{len}_{i})}}{\sum_{j=0}^{\Delta l_{inc}}\exp{(z^{len}_{j}})}),$$

(14)

where $z^{len}\in\mathbb{R}^{l_{0}-1}$ is the output of fully-connected layers $\phi^{len}(\cdot)$ given $h$ as input. $y^{len}\in\mathbb{R}^{\Delta l_{inc}}$ is the ground-truth label of incoherence length and $\Delta l_{inc}$ is the difference between the upper bound and lower bound of incoherence length. Similar to LoD, Equation 14 is also applied to each representation of the mini-batch $Z^{len}\in\mathbb{R}^{N\times\Delta l_{inc}}$, whose loss $\mathcal{L}_{LeD}$ is calculated as the average of all losses of $Z^{len}$.

Intra-Video Contrastive Learning (ICL). Contrastive learning can effectively extract the mutual information between variously augmented samples from the same source. Recent works (Yao et al. 2021; Wang, Jiao, and Liu 2020) demonstrate its great potential, exceeding other self-supervised or even supervised methods. In this work, we include Intra-Video Contrastive Learning (ICL) as an extra optimization objective to maximize the mutual information between different incoherent clips from the same video. This is inspired by the fact that our visual systems extract mutual information from incoherent clips, and thus can still recognize the correct motions even under incoherent circumstances.

Formally, given a mini-batch of $N$ raw videos $\mathbf{V}=\{V_{1},V_{2},...,V_{N}\}$, two incoherent clips are randomly generated as demonstrated in Section Generation of incoherent video clips for each raw video $V_{i}\in\mathbf{V}$. The incoherent clips from the same raw video $V_{i}$ are considered as positive pairs denoted as $\{\mathcal{V}_{inc}^{i},\widetilde{\mathcal{V}}_{inc}^{i}\}$. The incoherent clips from different raw videos are regarded as negative pairs denoted as $\{\mathcal{V}_{inc}^{i},\mathcal{V}_{inc}^{k}\},k\neq i$. Each incoherent clip $\mathcal{V}_{inc}^{i}$ is then fed to the network $f(\cdot)$, forming the high-level representation $h_{i}$. The representation $h_{i}$ is subsequently fed to a fully-connected layer $\phi^{cl}(\cdot)$ followed by a non-linear ReLU activation, generating features $z_{i}^{cl}$. Provided with features of positive pairs $\{z_{i}^{cl},\widetilde{z_{i}}^{cl}\}$ and features of negative pairs $\{z_{i}^{cl},z_{k}^{cl}\},k\neq i$, the contrastive loss is computed as:

$$\mathcal{L}_{ICL}=-\frac{1}{2N}\sum_{i=1}^{2N}\log(\frac{\exp{(s(z^{cl}_{i},\widetilde{z_{i}}^{cl})})}{\exp{(s(z^{cl}_{i},\widetilde{z_{i}}^{cl})})+\mathcal{D}(i))}),$$

(15)

$$\mathcal{D}(i)=\sum\nolimits_{k\neq i}\exp{(s(z^{cl}_{i},z_{k}^{cl}))},$$

(16)

where $s(u,v)=u^{\top}v/\left\|u\right\|\left\|v\right\|$ denotes the similarity between feature $u$ and $v$. $\mathcal{D}(i)$ is the summation of exponential similarity between features of negative pairs.

The overall network structure is illustrated as Figure 3. Given the unlabeled raw video, the incoherent clips are first generated as described in Section Generation of incoherent video clips, where each raw video randomly generates two incoherent clips as shown in Figure 3(a). Subsequently, the batch of incoherent clips is fed to the encoder $f(\cdot)$ implemented as the 3D CNN backbone. The ultimate optimization objective is formulated as:

$$\mathcal{L}=\alpha\mathcal{L}_{LoD}+\beta\mathcal{L}_{LeD}+\lambda\mathcal{L}_{ICL},$$

(17)

where $\alpha$, $\beta$, $\lambda$ are the coefficients of three loss terms from the sub-tasks, respectively.

Datasets. We evaluate our VID across three action recognition datasets, including UCF101 (Soomro, Zamir, and Shah 2012), HMDB51 (Kuehne et al. 2011) and Kinetics-400 (Kay et al. 2017). UCF101 is a widely used video dataset for action recognition, which contains 13,320 videos with 101 action categories. HMDB51 is a relatively smaller yet challenging dataset for action recognition. It includes about 7,000 videos with 51 action classes. Both UCF101 and HMDB51 are divided into three training and testing splits. Kinetics-400, denoted as K-400, is a large dataset for action recognition. It contains about 304,000 videos with 400 action classes collected from the online video platform YouTube. Same as the setting of prior work (Wang, Jiao, and Liu 2020; Jenni, Meishvili, and Favaro 2020), we utilize the training split of Kinetics-400 and the training split 1 of UCF101 for self-supervised pre-training. The training split 1 of UCF101 and HMDB51 are utilized during fine-tuning for action recognition.

Backbone networks. As for the 3D CNN backbones, to fairly compare our proposed methods with others (Wang, Jiao, and Liu 2020; Xu et al. 2019), we utilize three different 3D CNN networks in our experiments, including C3D (Tran et al. 2015), R3D (Hara, Kataoka, and Satoh 2018) and R(2+1)D (Tran et al. 2018). The mentioned backbones have been widely used to evaluate self-supervised methods in previous research. Specifically, C3D (Tran et al. 2015) is constructed by direct extending 2D kernels of 2D CNN to 3D ones. R3D (Hara, Kataoka, and Satoh 2018) introduces the residual connections from 2D CNNs to 3D CNNs. Following previous works (Yao et al. 2020; Jenni, Meishvili, and Favaro 2020; Kim, Cho, and Kweon 2019), we utilize R3D-18 which is the 18-layer variant of R3D. R(2+1)D (Tran et al. 2018) proposes to replace the traditional 3D kernel with the combination of a 2D kernel and a 1D kernel for spatial and temporal feature extraction, respectively. In this work, we mainly conduct our experiments with R(2+1)D thanks to its superior performance compared with others.

Augmentation and parameters. Following the setting of prior work (Jenni, Meishvili, and Favaro 2020; Wang, Jiao, and Liu 2020), each incoherent clip includes 16 frames. The sampling for each sub-clip is 1 and the range of incoherence length is set as $l_{inc}\in\{3,4,...,10\}$. While pre-training on UCF101, we follow the setting in (Wang, Jiao, and Liu 2020; Alwassel et al. 2019) which increases the epoch size from 9k to 90k with color jittering along the temporal dimension. Frames are resized to $128\times 171$ and then randomly cropped to $112\times 112$. The whole input clip is then flipped horizontally with a probability of $50\%$. The network is trained with a batch-size of 30. The stochastic gradient descent (Bottou 2010) is utilized for optimization with the weight decay set to 0.005 and the momentum set to 0.9. The coefficients of sub-tasks $\alpha$, $\beta$, $\lambda$ are empirically set to 1, 0.1, 0.1. The learning rate is initialized as 0.001 and divided by 10 every 6 epochs with a total training epoch of 18.

In this section, we justify the design of our proposed VID by ablation studies. We first illustrate the optimal range of the incoherence length, and then conduct experiments utilizing different sub-tasks. Our proposed incoherence detection is additionally evaluated with various backbones compared with previous coherence-based methods. Except as otherwise specified, our ablation studies are conducted with R(2+1)D (Tran et al. 2018) pre-trained on UCF101.

Range of the incoherence length. We first explore the best range of incoherence length $l_{inc}$. Illustrated as Table 1, the experiments are conducted by changing either the lower bound $l_{inc}^{min}$ or the upper bound $l_{inc}^{max}$ of the range. As $l_{inc}^{min}$ increases from 2, we can also observe an improvement of performance from $76.7\%$ peaking at $78.1\%$ with $l_{inc}^{min}=3$, while the performance begins to decrease when $l_{inc}^{min}$ further increases. When $l_{inc}^{min}$ is smaller than 3, the incoherence between sub-clips are too difficult for the network to identify. The further increase of lower bound decreases the variety of incoherence length, leading to a drop in performance. Similar to the lower bound, when the upper bound of incoherence length increases from $l_{inc}^{max}=6$, the performance of VID rises consistently from $76.8\%$ which reaches a climax when $l_{inc}^{max}=10$, whereas a decreasing performance is observed as upper bound further increases, dropping from $78.1\%$ with $l_{inc}^{max}=10$ to $76.7\%$ with $l_{inc}^{max}=14$. As $l_{inc}^{max}$ increases, the sample range of incoherence becomes more abundant, while the incoherence becomes too obvious when $l_{inc}^{max}>10$. This observation indicates that an inappropriate range of $l_{inc}$ can result in too vague or too obvious incoherence, which leads to inferior performance. We thus set the range of $l_{inc}$ as $[3,10]$ in the following experiments.

Different sub-tasks. We further evaluate the performance of different sub-tasks. As shown in Table 2, when utilizing a single sub-task, we observe that networks pre-trained with any sub-task significantly exceed random initialization with a relative improvement of more than $25.0\%$. The network with LoD obtains the highest performance of $75.4\%$ on UCF101, which justifies the effectiveness of LoD. The network with ICL also achieves a competitive performance of $72.1\%$. However, when optimizing with only LeD, the network is required to directly predict the incoherence length without locating it. Therefore, the network can not fully leverage incoherence detection in videos, leading to inferior performance of $70.9\%$.

As for arbitrary pairs of sub-tasks, we observe that LoD-based pairs (LoD+LeD and LoD+ICL) surpass the single LoD with noticeable margins of more than $2.0\%$. This justifies the effectiveness of LeD and ICL as additional objectives. We also observe an inferior performance of the LeD-based pair (LeD+ICL), which aligns with the performance of the network with a single LeD.

Comparison with coherence-based methods. To justify the effectiveness of incoherence detection, we evaluate our VID without additional ICL sub-task compared with previous coherence-based methods (Luo et al. 2020; Xu et al. 2019) utilizing order prediction. Illustrated in Figure 4, it is seen that our VID outperforms previous coherence-based methods across various backbones on different datasets. On UCF101, VID exceeds the previous VCOP (Xu et al. 2019) and VCP (Luo et al. 2020) by $4.3\%$-$7.9\%$ and $1.4\%$-$11.0\%$, respectively. On HMDB51, the proposed VID surpasses VCOP (Xu et al. 2019) and VCP (Luo et al. 2020) by over $10\%$ relatively. The improvement indicates that incoherence detection requires a more comprehensive understanding of videos compared with frame order reasoning.

Action recognition. To verify the effectiveness of our proposed VID, we evaluate our VID with different backbones on action recognition which is a primary downstream task adopted in prior works (Wang, Jiao, and Liu 2020; Yao et al. 2020; Jenni, Meishvili, and Favaro 2020). For action recognition, the network is initialized with the weights of the pre-trained model while the fully-connected layer is randomly initialized. The whole network is trained using the cross-entropy loss with an initial learning rate of 0.003. Other augmentation and parameter settings are the same as the pre-training stage. For testing, following the evaluation protocol of previous works (Wang, Jiao, and Liu 2020; Yao et al. 2020), we uniformly sample 10 clips from each video followed by a center crop. The final predictions for each video are the average result of all sampled clips.

As shown in Table 3, our proposed VID achieves state-of-the-art (SOTA) results compared with self-supervised spatio-temporal reasoning methods which utilize a single data transformation. For C3D, our proposed VID outperforms the SOTA PRP method (Yao et al. 2020) by $1.1\%$ on UCF101 and $3.2\%$ on HMDB51. For R3D, VID also exceeds PRP and ST-Puzzle (Kim, Cho, and Kweon 2019), which are the previous SOTA method on UCF101 and HMDB51, with noticeable margins of $7.1\%$ on UCF101 and $5.4\%$ on HMDB51. With R(2+1)D pre-trained on UCF101, VID further surpasses the previous SOTA method PP (Wang, Jiao, and Liu 2020) by $1.0\%$ on UCF101 and $3.1\%$ on HMDB51. When pre-trained on Kinetics-400, the margins of improvement further expand to $1.4\%$ and $4.5\%$, respectively. Provided by the illustrated results, VID can learn more abundant spatio-temporal representations compared with previous single-transformation methods.

In Table 3, we also include the results of RTT (Jenni, Meishvili, and Favaro 2020) which assembles multiple transformations, leading to superior performance compared with single-transformation methods. Nevertheless, for C3D, VID is the only single-transformation method that outperforms RTT by $0.3\%$ on UCF101 and provides competitive performance on HMDB51. It is possible to further improve the performance of ensemble-based methods by including our VID, while we mainly focus on leveraging video coherence by using a single temporal transformation in this work.

Video retrieval. We further evaluate our VID on another downstream task of nearest-neighbour video retrieval, which evaluates the quality of features extracted by the self-supervised pre-trained model. To make a fair comparison, our evaluation follows the protocol of previous state-of-the-art methods (Wang, Jiao, and Liu 2020; Yao et al. 2020). All models are pre-trained on UCF101. Given ten 16-frame clips sampled from each video, their features are extracted from the last pooling layer of the pre-trained backbone model. During inference, frames of each clip are first resized to 128 × 171 and then centrally cropped 112 × 112. Clips in the testing split are utilized to query the Top $k$ nearest samples based on their corresponding features. Here we consider $k$ equal to 1, 5, 10, 20, 50.

As shown in Table 4 and Table 5, VID outperforms state-of-the-art methods PP (Wang, Jiao, and Liu 2020) and PRP (Yao et al. 2020) on most evaluation metrics of UCF101 and HMDB51 across all backbones with significant marginals (e.g. $1.7\%$-$4.4\%$ for Top1 on UCF101). Specifically, VID surpasses all previous methods on HMDB51 across all evaluation metrics except for Top50, with improvement ranging from $0.3\%$ to $8.1\%$. The significant improvement further justifies that our VID extracts more effective spatio-temporal representation for downstream tasks.

In addition to experiment settings in our paper, we present the implementation details of our experiments. As mentioned in Sec. 4.1, color jittering is applied to each extracted incoherent clip along the temporal dimension. Specifically, we randomly modify the brightness, contrast, saturation and hue for each frame in the incoherent clip, whose augmentation ranges are $[0.2,1.8]$, $[0.2,1.8]$, $[0.2,1.8]$ and $[-0.2,0.2]$, respectively. Color jittering is enabled during the pre-training and fine-tuning stages and is disabled during evaluation. We conduct our experiments utilizing PyTorch (Paszke et al. 2017) with two NVIDIA Tesla P100. Our code is available in the appendix.

We visualize the heat map of extracted representation to justify the effectiveness of LeD task as shown in Figure 5. To validate the regularization effect of LeD, we additionally present corresponding heat maps from the network pre-trained without LeD. As shown in the last two rows, when there is subtle difference between scenes of sub-clips, networks

pre-trained with or without LeD both focus on the actors to detect abnormal motion caused by incoherence, which proves that incoherence detection requires motion understanding. When scenes change intensively as shown in the first row, the network pre-trained with LeD maintains its concentration on motion areas, while the network without LeD is distracted by the dynamic scene. This observation justifies that the utilization of LeD increases the robustness of VID towards the severe changes of low-level information, which avoids trivial learning.

In addition to Figure 5, we present more heat map visualization of our VID to justify its effectiveness. As shown in Figure 6, each row is augmented frames extracted from the test sample of UCF101 (Soomro, Zamir, and Shah 2012). The first column is the original frame representing the input sample. The following columns are heat maps visualization by Grad-CAM (Selvaraju et al. 2017) of different sub-clips.

The heat maps provided in Figure 6 justify our assumption that incoherence detection requires an understanding of motion in videos. For example, given the golf swing in the first row, the network pre-trained with our VID concentrates on the upper body of the actor to detect the movements for incoherence detection. Additionally, as illustrated in the lower three rows in Figure 6, when there are intensive changes between scenes of different sub-clip, our VID can maintain its concentration on the motion areas to detect incoherence. For example, given input indicating horse racing in the last row, the network pre-trained with our VID continuously focus on the horses and riders regardless of the intensive changes of backgrounds.

We present multiple examples of video retrieval results in comparison with the previous state-of-the-art method PRP (Yao et al. 2020). Following the evaluation protocols in previous works (Yao et al. 2020; Xu et al. 2019), we extract the features from the last pooling layer of the pre-trained network. For PRP (Yao et al. 2020), we evaluate its performance with its provided pre-trained model. Both our VID and PRP (Yao et al. 2020) are evaluated with ResNet18, which is the 18-layer variant of R3D (Tran et al. 2018). Illustrated as Figure 7, our proposed VID provide more reasonable results compared to previous PRP (Yao et al. 2020). For example, given the query of applying eye makeup as shown in the first row, our VID retrieves two samples of the same action classes as the query among Top 3, while PRP retrieves samples that belong to similar-yet-incorrect actions, such as haircut and blow-dry hair. The retrieval results indicate that the network pre-trained with our VID obtains a more comprehensive understanding of videos compared to previous methods.