Soft-Landing Strategy for Alleviating the Task Discrepancy Problem
in Temporal Action Localization Tasks

As a fundamental task for processing long-form videos, Temporal Action Localization (TAL) has drawn significant attention among computer vision researchers leading to a plethora of seminal works [34, 19, 37, 4, 17, 42, 31]. Beyond the standard fully-supervised and offline setting, various extensions of the task were suggested, including its online [14, 16], weakly-supervised [21, 35, 39] and unsupervised [9] variants.

Although numerous attempts have been made for designing better TAL heads, relatively less attention has been paid for devising a good snippet encoder, despite the fact that all TAL methods and its variants start from the pre-extracted snippet feature sequences. A major problem of the TAC-pretrained snippet encoder is its insensitivity to different snippets in the same video clip. Apparently, this insensitivity problem can be resolved by adopting a “temporally sensitive” pretraining method for the snippet encoder. In this perspective, [1] rigorously exploited temporal annotations for training the snippet encoder. However, as it requires a large scale and temporally annotated video dataset, general applicability of this approach is limited. On the other hand, [32] adopted data-generation approach which only exploited the action class labels of the Kinetics400 [15] dataset. To be specific, various boundaries were generated by simply stitching intra-/inter- class videos, and a pretext task of guessing the type of the generated boundaries was proposed. Going one step further, a recent work [41] introduced a completely unsupervised pretraining methodology for the snippet encoder, greatly expanding its possible applications. In addition, it is worth noting that [33] made an initial attempt on designing an end-to-end TAL with a low-fidelity snippet encoder, while [20] provided exhaustive empirical studies on these end-to-end TAL approaches. Nevertheless, all previous works involve trainable snippet encoder assumption, while our SoLa strategy only requires pre-extracted feature sequence for its adoption.

Temporal Self-similarity Matrix (TSM) is a square matrix, where each of its element corresponds to its self-similarity score. From a given video with $L$ frames, each value at position $(i,j)$ in TSM is calculated using a similarity metric (e.g. cosine similarity) between frame i and j, resulting in an $L\times L$ matrix. As an effective and interpretable intermediate representation of a given video, several recent works [13, 12] exploited TSM to tackle various video understanding tasks, including Generic Event Boundary Detection [23] and repetitive action counting [7]. In our work, we focus on the certain similarity patterns that arise in general TSM, motivating us to design a new Similarity Matching objective.

Let $V:=v_{\psi=1}^{l}$ be a video consisting of $l$ frames. We assume having a pretrained snippet encoder that takes a length $\alpha$ snippet $v_{\alpha\psi+1}^{\alpha\psi+\alpha}$ as its input and emits a vector $f\in\mathbb{R}^{m}$. With the snippet encoder, we convert $V$ into a snippet feature sequence $f_{\tau=1}^{L}$, where $L=\lceil l/\alpha\rceil$ if there is no overlap among the snippets. Here, we introduce the Soft-Landing (SoLa) module $SoLa(\cdot):\mathbb{R}^{L\times m}\to\mathbb{R}^{L\times m}$, where $F_{\tau=1}^{L}=SoLa(f_{\tau=1}^{L})$ and $F_{\tau=1}^{L}$ denotes the transformed feature sequence¹¹1In general, output dimension of the $SoLa(\cdot)$ can be different. But here we only consider the same dimension case for clarity.. For a wider and more general applicability of the transformed feature sequence $F_{\tau=1}^{L}$, we assume to have access to the unlabeled downstream dataset, meaning that we only know the target data but do not know the target task. Our ultimate goal is to devise a proper unsupervised training method to train the SoLa module that produces temporally sensitive transformed feature sequence $F_{\tau=1}^{L}$. We expect the transformed feature sequence $F_{\tau=1}^{L}$ to perform better than the original feature sequence $f_{\tau=1}^{L}$ in various downstream tasks, in which the temporal sensitivity is important.

The main idea of the Soft-Landing (SoLa) strategy is placing a light-weight neural network, or SoLa module, between the pretrained encoder and the downstream head. In our framework, there is no need for retraining or fine-tuning of the pretrained encoder since the SoLa module is solely responsible for narrowing the task discrepancy gap between the pretrained encoder and the downstream head. Besides, following the standard two-stage TAL convention, our SoLa strategy works on the snippet feature level and hence eliminates the need for an access to raw RGB frames. As most properties of the standard TAL framework are preserved, our SoLa strategy requires only a minimal modification of the existing framework for its integration.

For a clear demonstration, overall schematic diagram of the training and the inference stage of the SoLa strategy is illustrated in Figure 2 and Figure 1 (b) respectively. In the training stage, we first sample a fixed-size local subsequence from the pre-extracted snippet features $f_{\tau=1}^{L}$, which originate from the pretrained snippet encoder. Next, the sampled subsequence is put into the SoLa module, yielding a transformed feature sequence. After assembling each element with a step size $s$, the shortened feature sequence is projected through an MLP and pairwise similarities between the sequence before the projection and after the projection are computed, forming a TSM-like matrix. In Figure 2, we denote this asymmetric self similarity matrix as a predicted TSM. With a predefined target TSM which solely depends on the frame interval, Similarity Matching loss $\mathcal{L}^{\mathrm{SM}}(\cdot,\cdot)$ is posed in each position of the matrices. Note that the above training procedure does not demand any additional label as the frame interval (distance between the frames) is its only supervisory signal. The exact procedure for generating the target TSM and calculating the loss will be discussed in the next section. (Section 3.3)

During the inference stage, we put the snippet feature sequence $f_{\tau=1}^{L}$ to the trained SoLa module $SoLa(\cdot)$ and get an enhanced feature sequence $F_{\tau=1}^{L}$. Then, $F_{\tau=1}^{L}$ is directly used for the downstream task.

From its definition, the SoLa module can be any neural networks that take a tensor with shape $(L,m)$ as its input and output a tensor of the same shape. To demonstrate it as a proof of concept, we employed the simplest architecture: a shallow 1D CNN. While devising effective architecture for the SoLa module is an interesting future research direction, we found that this 1D CNN architecture works surprisingly well. More detailed architectural configurations and ablation studies on this topic are provided in the supplementary materials.

Due to the unlabeled target dataset assumption, it is obvious that the training of the SoLa module must be done in a self-supervised way. While recent studies [6, 10] have shown remarkable success of contrastive learning in the self-supervised representation learning domain, all of these methods rely on extensive data augmentation. Specifically, [5] pointed out that a strong data augmentation is essential for successful training of the self-supervised contrastive learning models. Nevertheless, existing data augmentation methods are rgb frame-level operation (e.g. random cropping, color distortion, etc.), whereas our SoLa module deals with feature sequences $f_{\tau=1}^{L}$. Since feature-level data augmentation is infeasible, straightforward application of previous contrastive learning approach is non-trivial.

Instead of designing a feature-level data augmentation technique, we pay attention to the temporal similarity structure that general videos naturally convey: “Adjacent frames are similar, while remote frames remain distinct.” This intuition is validated in Figure 3, where the similarities among the snippet features from the same video and their average Temporal Self-similarity Matrix (TSM) are plotted.

As expected, the frame similarity decays as the frame interval increases, regardless of the backbone snippet encoder’s architecture. Note that although specific similarity scores may vary between the backbone architectures, the overall tendency of the similarity decay and the average TSM is preserved, indicating that the temporal similarity structure is a common characteristic of general videos. With this empirical observation, we propose a novel pretext task for feature level self-supervised pretraining called Similarity Matching, which utilizes frame interval as its only learning signal.

One of the possible approaches for exploiting the frame distances to train the model is directly using their positional information  [36, 38]. However, instead of using the raw frame interval as the learning signal, we go one step further to produce a snippet feature sequence that is even more temporally sensitive. To this end, we introduce a deterministic function $\lambda(\cdot):\mathbb{R}\to\mathbb{R}$ that maps a frame interval to a desirable, i.e., target similarity score. With these scores, the SoLa module is trained with the Similarity Matching. Here, the Similarity Matching denotes a direct optimization of the feature similarity; if the target similarity score is given as 0.75, the objective function would penalize the feature encoder if the similarity score between the two snippet features deviates from 0.75. In this manner, the objective forces the similarity between the two snippet features to be close to the target similarity score, which depends on the frame interval between them.

We designed the $\lambda$ function to enhance the temporal similarity structure of the empirical similarity score distribution. To achieve the goal, the function should make the adjacent snippet features be more similar while remote snippet features remain distinct. To do so, the $\lambda$ function is defined as follows with a constant $K$:

$$\lambda(d)=\sigma\Big{(}\frac{K}{d^{2}}\Big{)},d\neq 0,$$

(1)

where $d$ stands for the frame interval between the two snippet representations and $\sigma(\cdot)$ denotes the sigmoid function. High $K$ value in Equation (1) leads to a $\lambda$ function that amplifies the similarities of neighboring snippet features (refer to the Target TSM and the Average TSM in Figure 3). We found that this target assignment is important for achieving a good downstream performance (see ablation study results in the supplementary materials). Furthermore, it is worth mentioning that the target similarity score never goes to zero with the above $\lambda$ function if both snippet features are from the same video, reflecting our another prior knowledge: “There is an invariant feature that is shared among frames in the same video”, which also corresponds to the basic concept of the slow feature analysis [28].

Remaining design choice for the Similarity Matching is choosing an appropriate way to produce a similarity prediction $\hat{p}$ from the given snippet feature pair $(f_{i},f_{j})$. As the frame interval between different videos cannot be defined and to avoid trivial dissimilarity, we presume the snippet feature pair $(f_{i},f_{j})$ to come from the same video.

Motivated by the Simsiam framework [6], we adopt asymmetric projector network $Proj(\cdot):\mathbb{R}^{m}\to\mathbb{R}^{m}$, which consists of two fully-connected layers. Thus, we first calculate $z_{i}=SoLa({f_{\tau=1}^{L}})[i],z_{j}=Proj(SoLa(f_{\tau=1}^{L})[j])$ and utilize the rescaled cosine similarity between them as the network prediction:

$$\hat{p}=\frac{1}{2}(\frac{z_{i}\cdot z_{j}}{\parallel z_{i}\parallel\parallel z_{j}\parallel}+1).$$

(2)

Finally, with those $\hat{p}$ and target similarity $\Lambda$ (output from the $\lambda$ function in Equation (1)), Similarity Matching loss $\mathcal{L}^{\mathrm{SM}}$ is computed as follows:

$$\mathcal{L}^{\mathrm{SM}}(\Lambda,\hat{p})=-\Lambda\log\hat{p}-(1-\Lambda)\log(1-\hat{p}).$$

(3)

Note that $\mathcal{L}^{\mathrm{SM}}(\cdot,\cdot)$ is merely a standard Binary Cross Entropy (BCE) loss with a soft target. In the supplementary materials, we provide the connection between the Similarity Matching loss and the contrastive learning.

Focusing on the fact that both $\Lambda$ and $\hat{p}$ values represent feature similarities, we can derive an interesting interpretation: view the set of $\Lambda$ and $\hat{p}$ values as TSMs. In this point of view, posing standard BCE loss between $\Lambda$ and $\hat{p}$ values becomes a TSM Matching. The target TSM, presented in Figure 3, visually illustrates $\Lambda$ assignments following Equation (1); it can be seen as a general and desirable TSM of an arbitrary video – maintaining high similarity between close frames while low similarities among distant frames. For each $\hat{p}$ and $\Lambda$ in the corresponding position, Equation (3) is computed, resulting in a group of BCE losses. This group of BCE losses can be succinctly described as an aforementioned “TSM matching” in that the losses force the predicted TSM to resemble the target TSM as training proceeds.

One may suspect that the TSM matching with a fixed target TSM would induce monotonous TSM prediction. However, we observed that only the average of the predicted TSMs goes to the target TSM as training proceeds while each sample shows diverse patterns (Figure 4).

As a relatively new task, many of the preceding works did not release their codes yet, making the exact fair comparison difficult. Thus, we will mainly focus on the gain, which reflects given method’s efficacy compared to their baselines.

We conducted three major ablation studies on the SoLa module with the G-TAD downstream head by experimenting with the followings: i) receptive fields, ii) unified training of the SoLa module, iii) different snippet encoder which only uses RGB-only frames.

As we discussed in Section 3.2, the SoLa module consists of simple 1D convolutional layers. However, finding the best-performing convolution kernel size remains as a design choice. To figure out the influence of the design choice, we tested kernel_size $\in\{1,3,5,7\}$.

Moreover, a recent work on MLP projector [27] claims that additional MLP between the classifier head and the feature encoder can reduce the transferability gap, inducing better downstream task performances. From this, it is natural to ask if the SoLa module itself (without Similarity Matching) can play a role as a buffer between the pretrained encoder and the TAL heads. To answer this question, we tried a unified training of the SoLa module where the SoLa module is directly attached to the very front part of the TAL head and trained simultaneously with the head. It can be seen as a TAL head with an additional local aggregation module on its front part, or a truncated version of [33].

The prevailing convention in TAL is using the twostream network. However, to ensure our SoLa strategy’s general applicability, we also conducted experiments on RGB only I3D [3] features⁴⁴4The feature is available in https://github.com/Alvin-Zeng/PGCN

Table 4 describes the result of ablation studies on our SoLa strategy. The result suggests that while the local feature aggregation plays a key role in the SoLa module, the wider receptive field does not necessarily entail a performance improvement; we attribute it to the need for an appropriate information bottleneck since the frame interval is exploited as the model’s learning signal. In addition, we find out that while the SoLa module itself can slightly boost the TAL performance, it does not match the performance of the SoLa module trained with Similarity Matching. It implies that training with Similarity Matching is essential for the successful training of the SoLa module.

Lastly, we observed that the SoLa module performs reasonably well in the RGB-only case (bottom part in Table 4). However, the gain is lower than the two-stream case, indicating that there is an inherent limitation that comes from the frozen snippet encoder assumption. For instance, if the base feature does not contain sufficient information, it serves as an information bottleneck for the SoLa module, imposing an upper bounds of the gain that SoLa module can achieve. For more ablation studies and results about our model, please refer to our supplementary materials.