Learning Action Completeness from Points
for Weakly-supervised Temporal Action Localization

Our baseline is shown in the upper part of Fig. 2. We first divide the input video into 16-frame segments, which are then fed to the pre-trained feature extractor. Following [21, 48], we exploit both of RGB and flow streams with early-fusion. The two-stream features are fused by concatenation, resulting in $X\in\mathbb{R}^{D\times T}$, where $D$ and $T$ denote the feature dimension and the number of segments, respectively.

The extracted features then go through a single 1D convolutional layer followed by ReLU activation, which produces the embedded features $F$. In practice, we set the dimension of the embedded features to the same as that of the extracted features $X$, i.e., $F\in\mathbb{R}^{D\times T}$. Afterward, the embedded features are fed into a 1D convolutional layer with the sigmoid function, to predict the segment-level class scores $P\in\mathbb{R}^{C\times T}$, where $C$ indicates the number of action classes. Meanwhile, we derive the class-agnostic background scores $Q\in\mathbb{R}^{T}$, to model background frames which do not belong to any action classes. Thereafter, we fuse the action scores with the complement of background probability to get the final scores $\hat{P}$, i.e., $\hat{p}_{t}[c]=p_{t}[c](1-q_{t})$. This fusion strategy is similar to that of [22], although the out-of-distribution modeling is not incorporated in our model.

The segment-level action scores are then aggregated to build a single video-level class score. We use the temporal top-$k$ pooling for aggregation as in [21, 48]. Formally, the video-level probability is calculated as follows.

where $k=\lfloor\frac{T}{8}\rfloor$ and $S$ denotes all possible subsets of $\hat{P}[c,:]$ containing $k$ segments, i.e., $|S|=k$.

Our baseline model includes two loss functions using video- and point-level labels respectively. As aforementioned, the video-level class label $y^{\text{vid}}[c]$ can be derived by accumulating the point-level labels. The video-level classification loss is then calculated with binary cross-entropy.

The point-level classification loss is also computed by binary cross-entropy but involving the background term for effectively training $Q$. In addition, we adopt the focal loss [28] to facilitate the training process. Formally, the classification loss for action points is defined as follows.

where $M^{\text{act}}$ indicates the number of action instances in the video and $\beta$ is the focusing parameter, which is set to $2$ following the original paper [28].

Training only with action points would lead the network to always produce low background scores rather than learn to separate action and background. Therefore, we gather some pseudo background points to supplement action ones. Our principle for selection is that at least one background frame must be placed between two adjacent action instances to separate them. By the problem definition, two different action points are sampled from different instances, so we use the action points as surrogates for the corresponding instances. Concretely, between two adjacent action points, we find the segments whose background scores $q_{t}$ are larger than the threshold $\gamma$. If no segment satisfies the condition in a section, we select one with the largest background score. Meanwhile, for the case where multiple background points are selected in a section, we mark all points between them as background, since it is trivial that no action exists there. In practice, this strategy is shown to be more effective than global mining [35] by collecting more hard points. Given the pseudo background point set, $\mathcal{B}^{\text{bkg}}=\{t_{j}\}_{j=1}^{M^{\text{bkg}}}$, the classification loss for background points is computed by:

where $M^{\text{bkg}}$ denotes the number of the selected background points and $\beta$ is the focusing factor, the same with (3). For pseudo background points, we penalize the final scores for all action classes, while encouraging the background scores.

The total point-level loss function is defined as the sum of the losses for action and pseudo background points.

As discussed in Sec. 1, the point-level classification loss is insufficient to learn action completeness, as point labels cover only a small portion of action instances. Therefore, we propose to generate dense pseudo-labels that can offer some hints about action completeness for the model. In detail, we consider all possible sequence candidates consistent with the action and pseudo background points. Among them, we find the optimal sequence that can provide good completeness guidance to the model. However, it is non-trivial without full supervision to measure how well a candidate sequence covers complete action instances. To enable it, we re-purpose the outer-inner-contrast concept [52] as a proxy for judging the completeness score of a sequence. Intuitively, the contrast between inner and outer scores is likely to be large for a complete action instance but small for a fragmentary one. Note that our purpose is different from the original paper [52]. It was originally designed for parametric boundary regression. In contrast, we utilize it as a scoring function to search for the optimal sequence, from which the model could learn action completeness.

Before detailing the scoring function, we present the formulation of candidate sequences. Due to the multi-label nature of temporal action localization, we consider class-specific sequences for each action class. Note that all segments belonging to other action classes are considered background for sequences of class $c$. Then, a sequence is defined as multiple action and background (including other actions) instances that alternate consecutively. Formally, a sequence of class $c$ can be expressed as $\pi_{c}=\{(s_{n}^{c},e_{n}^{c},z_{n}^{c})\}_{n=1}^{N_{c}}$, where $s_{n}^{c}$ and $e_{n}^{c}$ denote the start and end points of the $n$-th instance, respectively, while $N_{c}$ is the total number of instances for class $c$. In addition, $z_{n}^{c}\in\{0,1\}$ indicates the type of the instance, i.e., $z_{n}^{c}=1$ if $n$-th instance is of the $c$-th action class, otherwise $0$ (background).

Given an input sequence, we compute its completeness score by averaging the contrast scores of individual action and background instances contained in the sequence. It would be noted that the contrast scores of background instances are included in the calculation, which proves to be effective for finding more accurate optimal sequences, as will be shown in Sec. 4.3. Formally, the completeness score of a sequence $\pi_{c}$ for the $c$-th action class is computed by:

$l_{n}^{c}=e_{n}^{c}-s_{n}^{c}+1$ is the temporal length of the $n$-th instance of $\pi_{c}$, $\delta$ is a hyper-parameter adjusting the outer range (set to 0.25), and $N_{c}$ is the total number of action and background instances for class $c$. Then, the optimal sequence for class $c$ can be obtained by finding the sequence that maximizes the score, i.e., $\pi_{c}^{*}=\operatorname*{arg\,max}_{\pi_{c}}\mathcal{R}(\pi_{c})$ using (6). The optimal sequence search process is illustrated in Fig. 3. By evaluating the completeness score, our method can reject underestimation (Fig. 3a) and overestimation (Fig. 3b) cases. Consequently, we obtain the optimal sequence that is most likely to contain complete action instances.

However, the search space grows exponentially as $T$ increases, leading to the exorbitant cost for optimal sequence search. To relieve the issue, we implement the search process with a greedy algorithm under a limited budget, which results in greatly saving the computational cost. Detailed algorithm and cost analysis are presented in Sec. B of the appendix. Note that optimal sequence search is performed only for the action classes contained in the video.

Given the class-specific optimal sequences $\{\pi_{c}^{*}\}_{c=1}^{C}$, our goal is to let the model learn action completeness. To this end, we design two losses that enable completeness learning by contrasting action instances from background ones. This helps in complete action predictions, as validated in Sec. 4.

Firstly, we propose score contrastive loss that encourages the model to separate action (background) instances from their surroundings in terms of final scores. It can be also interpreted as fitting the model outputs to the optimal sequences (Fig. 2a). Formally, the loss is computed by:

where we use $\beta$-squared term to focus on the instances that are largely inconsistent with the optimal sequence ($\beta=2$).

Secondly, inspired by the recent success of contrastive learning [6, 9, 16], we design feature contrastive loss. Our intuition is that features from different instances but with the same action class should be closer to each other than any other background instances in the same video (Fig. 2b). We note that our loss differs from [6, 9, 16] in that they pull different views of an input image, whereas ours attracts different action instances in a given video. In addition, ours does not need negative sampling from different images, as background instances are obtained from the same video.

To extract the representative feature for each action (or background) instance, we modify the segment of interest (SOI) pooling [5] by replacing max-pooling with random sampling. In detail, we evenly divide each input instance into three intervals, from each of which a single segment is randomly sampled. Then, the embedded features of the sampled segments are averaged, producing the representative feature $f_{n}^{c}$ for the $n$-th instance of the sequence $\pi_{c}^{*}$.

Taking the normalized instance features $\bar{f}_{n}^{c}$ as inputs, we derive feature contrastive loss. The loss is computed only for the classes whose action counts are larger than 1, i.e., at least two action instances exist in the video. Note that background instances do not attract each other. Given the optimal sequences $\big{\{}\pi_{c}^{*}=\{(s_{n}^{c},e_{n}^{c},z_{n}^{c})\}_{n=1}^{N_{c}}\big{\}}_{c=1}^{C}$, the proposed feature contrastive loss is formulated as:

where $\ell_{\text{feat}}^{c}$ is the partial loss for class $c$, $\tau$ denotes the temperature parameter, and $\mathbbm{1}\left[\cdot\right]$ denotes the indicator function.

The overall training objective of our model is as follows.

where $\lambda_{*}$ are weighting parameters for balancing the losses, which are determined empirically.

During the test time, we first threshold on the video score $\hat{\mathbf{p}}^{\text{vid}}$ with $\theta^{\text{vid}}$ to determine which action categories are to be localized. Then, only for the remaining classes, we threshold on the segment-level final scores $\hat{\mathbf{p}}_{t}$ with $\theta^{\text{seg}}$ to select candidate segments. Afterward, consecutive candidates are merged into a single proposal, which becomes a localization result. We set the confidence of each proposal to its outer-inner-contrast score, as in [21, 29]. To augment the proposal pool, we use multiple thresholds for $\theta^{\text{seg}}$ and perform non-maximum suppression (NMS) to remove overlapping proposals. Note that the optimal sequence search is not performed at test time, so does not affect the inference time.

In Table 1, we compare our method with state-of-the-art models under different levels of supervision on THUMOS’14. We note that fully-supervised models require far more expensive annotation costs compared to weakly-supervised counterparts. In the comparison, our model significantly outperforms the state-of-the-art point-supervised approaches. We also notice the large performance margins at high IoU thresholds, e.g., $\sim$11% in [email protected] and $\sim$9% in [email protected]. This confirms that the proposed method aids in locating the complete action instances. At the same time, our model largely surpasses the video-supervised methods with the comparable labeling cost. Further, our model even performs favorably against the fully-supervised methods in terms of average mAPs at the much lower annotation cost. It is, however, also shown that ours lags behind them at high IoU thresholds, due to the lack of boundary information.

We provide the experimental results on GTEA and BEOID benchmarks in Table 2. On the both datasets, our method beats the existing state-of-the-art methods with a large gap. Notably, our method shows significant performance boosts under the high thresholds of 0.5 and 0.7, verifying the efficacy of the proposed completeness learning.

Table 3 and Table 4 summarize the results on ActivityNet. Our model shows the superior performances over all the existing weakly-supervised approaches on both versions. It can be also observed that the performance gains upon video-level labels are relatively small compared to THUMOS’14, which we conjecture is due to the far less frequent action instances (1.5 vs. 15 instances per video).

We present qualitative comparisons with SF-Net [35] in Fig. 4. It can be clearly noticed that our method locates the action instances more precisely. Specifically, in the left example, SF-Net produces fragmentary predictions with false negatives, whereas our method detects the complete action instances without splitting them. In the right sample, while SF-Net overestimates the action instances with false positives, our method produces precise detection results by contrasting action frames from background ones well. The red boxes highlight the false negatives and false positives of SF-Net in the left and right examples, respectively. We note that all the predictions of our model in both examples have high IoUs larger than 0.6 with the corresponding ground-truth instances, validating the effectiveness of our completeness learning. Comparisons on other benchmarks and more visualization results can be found in Sec. C of the appendix.

To analyze the correlation between score contrast and action completeness, we draw the scatter plot of score contrast vs. IoUs with ground-truth action instances, using the randomly sampled 2,000 temporal intervals in the THUMOS’14 training videos. For reference, we also present the scatter plot of inner action scores vs. IoUs with the same intervals. In the experiments, we use the baseline model for fair comparison. Fig. 5a demonstrates that there is a moderate correlation between inner action scores and IoUs, but there are many cases with large inner scores but low IoUs (see bottom right). On the contrary, as shown in Fig. 5b, score contrast correlates much stronger with IoUs, demonstrating its efficacy as a proxy for measuring the action completeness without any supervision.

We compare different variants of pseudo background mining on THUMOS’14. Specifically, we consider three variants: (1) “Global mining” selects the top $\eta M^{\text{act}}$ points throughout the whole video without considering their locations as in SF-Net [35], where $M^{\text{act}}$ is the number of action instances and $\eta$ is set to 5, (2) “Ours w/o filling” follows the principle described in Sec. 3.1 except the filling stage, i.e., we select at least one background point for each section between two action points, and (3) “Ours” mines all points between the background points for each section if multiple points are found in the second variant. Note that we use the baseline model without completeness learning for clear comparison.

The results are demonstrated in Table 9. It can be observed that both of our methods significantly outperform the “Global mining” approach, which verifies the effectiveness of our selection principle that at least one background point should be placed for each section. Moreover, by ensuring at least one background point for each section, the search space of optimal sequence selection can be significantly reduced, although we do not include the cost analysis for this experiment. Meanwhile, we notice that filling between two background points slightly boosts the localization performance. This is presumably because hard background points with low background scores can be collected in the filling step.

In Fig. 6, we visualize the obtained optimal sequences for the examples from the three benchmarks. In the first example from THUMOS’14 (a), the optimal sequence covers the ground-truth action instances well so that the model could learn action completeness from it. Moreover, although the examples from GTEA (b) and BEOID (c) contain a variety of action classes in a single video, our method successfully finds the optimal sequence that shows large overlaps with the ground-truth ones. Overall, it is shown from all the examples that the optimal sequences are quite accurate even though they are selected based on point-level labels without full supervision. They in turn provide completeness guidance to our model, which proves to improve localization performances at high IoU thresholds in Sec. 4.3 of the main paper.

We qualitatively compare our method with SF-Net [35] on the three benchmarks. The comparison on THUMOS’14 [13] is demonstrated in Fig. 7. As shown, SF-Net produces fragmentary predictions by splitting action instances, whereas our method outputs complete ones with high IoUs even for the extremely long action instance (b). The comparison result on GTEA [23] is presented in Fig. 8. It would be noted that action localization on GTEA is challenging as the frames with different action categories are visually similar, leading to false positives. We see that SF-Net has difficulty in distinguishing action instances from background ones, resulting in inaccurate localization. On the other hand, our method successfully finds the action instances by learning completeness, showing fewer false positives. Lastly, the comparison on BEOID [7] is shown in Fig. 9. It can be clearly noticed that SF-Net fails to predict the ending times of action instances, leading to the overestimation problem. On the contrary, with the help of the completeness guidance, our method better separates actions from their surroundings and locates the action instances more precisely.