Turning to A Teacher
for Timestamp Supervised Temporal Action Segmentation

Given a video $X=\{x_{1},\dots,x_{T}\}$ with $T$ frames, the goal of the temporal action segmentation task is to predict a sequence of the frame-wise labels $[\tilde{a}_{1},\dots,\tilde{a}_{T}]$. Unlike full supervision, timestamp supervision provides a set of annotated timestamps $A=\{a_{t_{1}},\dots,a_{t_{n}},\dots,a_{t_{N}}\}$ with each video, where $N$ is the number of the annotated timestamps. Especially, for each action instance within the video $X$, we have access to one frame labeled with timestamp $t_{n}\in[1,T]$ and action class $a_{t_{n}}$, where $n$ denotes the $n$-th action instance and $n\in[1,N]$. Although a video may contain multiple action instances, the number of labeled timestamps $N$ is much smaller than the number of frames $T$, i.e. , $N\ll T$. This reveals the sparse nature of timestamp supervision.

We propose a framework for timestamp supervised temporal action segmentation including a warm-up phase and a mean teacher phase. As shown in Figure 2, we use the feature $F\in\mathbb{R}^{T\times D}$ extracted by I3D [17] as input to our framework. We first train a segmentation model on the timestamp annotations, which is denominated as the warm-up phase. Then we adopt a label generator to generate pseudo frame-wise labels to alleviate the problem of the sparse nature of timestamp supervision. However, this will introduce oscillation that results in performance degeneration. To alleviate this problem, inspired by Mean Teacher [16], we propose a teacher model parallel to the segmentation model, which is updated by the EMA of the segmentation model. In turn, the teacher model predictions are used to update the segmentation model. This forms a circular dependency that helps the framework maintain stability and obtain better performance, which is denominated as the mean teacher phase. The combination of the segmentation and the teacher models is similar to a real-life scenario. A student, who encounters problems, will turn to a teacher. The process of solving problems helps the student enrich knowledge and the teacher gather experience.

In the warm-up phase, we only have access to a sparse set of annotated frames. To train a segmentation model, we apply a cross entropy loss as the partial classification loss

$$L_{cls}=\frac{1}{N}\sum_{n=1}^{N}{-log(y_{t_{n},c})},$$

(1)

where $y_{t_{n},c}$ denotes the predicted probability for the target action class $c$ at annotated timestamp $t_{n}$. It ignores all frames that are not annotated. Note that the predicted probability is the Softmax output of the action scores $S\in\mathbb{R}^{T\times N_{c}}$, where $N_{c}$ indicates the number of action classes.

Due to the sparse nature of timestamp supervision, it is observed that the segmentation model only learns from each action instance the annotated frame and other informative frames which have similar features to the annotated frame. Intuitively, the frames between the annotated frames and the informative frames should have similar predicted probability, and these frames should be predicted to have high confidence. Based on this, we propose a segmentally smoothing loss

$$\begin{split}&L_{smo}=\sum_{\begin{subarray}{c}a_{t_{n}}\in A\end{subarray}}{\left(\frac{1}{T^{\prime}}\sum_{t=t_{n-1}+1}^{t_{n+1}}{\delta_{a_{t_{n}},t}}\right)}\\ s.t.&\\ &\delta_{a_{t_{n}},t}=\min(\tau,|\log{y_{t,a_{t_{n}}}-\log{y_{t-1,a_{t_{n}}}}}|),\end{split}$$

(2)

where $y_{t,a_{t_{n}}}$ refers to the predicted probability for the annotated action class $a_{t_{n}}$ at time $t$, $T^{\prime}$ denotes the number of frames between the annotated frames $t_{n-1}$ and $t_{n+1}$, and $\tau$ is a hyperparameter. The segmentally smoothing loss encourages smooth transition of the frames between high confidence frames and therefore suppresses the low confidence regions surrounded by high confidence frames (see Figure 3). It helps the segmentation model to learn from not only the annotated and the informative frames but also those frames in low confidence regions. Unlike TMSE [4], which encourages smooth transition of the predicted probabilities for all frames in an input video and all action classes, the proposed loss accumulates loss segmentally to encourage smooth transition for the labeled action categories within action instances in the input video. So it is more focused and cohesive.

The overall loss function for the segmentation model in the warm-up phase is a weighted sum of the above two losses

$$\mathcal{L}=L_{cls}+\alpha L_{smo},$$

(3)

where $\alpha$ is a hyperparameter to balance the contribution of each loss.

Although the smoothing loss is able to help pay attention to the low confidence parts of an action instance, it still ignores many frames without annotations. Following [15], we apply a label generator to generate pseudo frame-wise labels by detecting action changes. However, we find that the generated pseudo frame-wise labels are not stable and cause oscillation during the training process. To alleviate this problem, we introduce a teacher model parallel to the segmentation model to help stabilize the system, which has the same architecture as the segmentation model. We update the teacher model by the EMA of the segmentation model. The EMA is formed by an ensemble of the current version and those earlier versions of the segmentation model. At $i$-th iteration, it is defined as

$$\overline{\theta}_{i}=\left\{\begin{array}[]{ll}(1-\frac{1}{i})\overline{\theta}_{i-1}+\frac{1}{i}\theta_{i}&1-\frac{1}{i}<\lambda,\\ \lambda\overline{\theta}_{i-1}+(1-\lambda)\theta_{i}&otherwise,\\ \end{array}\right.$$

(4)

where $\theta$ and $\overline{\theta}$ are the weights of the segmentation model and the teacher model respectively, and $\lambda$ is a hyperparameter. At the initial training iterations, the teacher model is a bare model without training. To help the teacher model learn quickly, we use absolute average [18] instead of EMA when $1-\frac{1}{i}<\lambda$.

To make the consistency constraint between the segmentation and the teacher models, we adopt a frame consistency loss $L_{cons}$ between their predictions, in the form of mean square error (MSE)[16].

$$L_{cons}=-\frac{1}{T}\sum_{t\in T}\|f(F,\theta)-f(F,\overline{\theta})\|^{2},$$

(5)

where $f(\cdot)$ denotes the predicted probabilities of a model. With this loss, the segmentation model learns from the teacher model, thereby forming a circular dependence as shown in Figure 2. It helps the two models and the generated pseudo labels stay steady.

The overall loss function for the segmentation model in the mean teacher phase is a weighted sum of three losses

$$\mathcal{L}=L_{cls}+\alpha L_{smo}+\beta L_{cons},$$

(6)

where $\beta$, akin to $\alpha$, is a hyperparameter. For simplicity, we use $L_{cls}$ to denote both pseudo classification and partial classification losses. While they are in the same form of cross-entropy loss, the pseudo classification loss

$$L_{cls}=\frac{1}{T}\sum_{t=1}^{T}{-log(y_{t,c})}$$

(7)

is supervised by the pseudo frame-wise labels. To generate the pseudo frame-wise labels, we regard the timestamp that minimizes the energy function in [15] as the action change between any two annotated frames. And then, we assign the same label to each frame located between an annotated frame and an estimated action change. Different from [15], we utilize action scores as the input feature to the energy function, rather than the output of the penultimate layer in the segmentation model. It is worth mentioning that we find that there is no benefit to perform forward-backward estimating as in [15] experimentally, so we only adopt the stamp-to-stamp version of it. Note that the above loss function is only computed for the segmentation model, and no gradient is computed on the teacher model.

We evaluate our method on three public datasets, namely Georgia Tech Egocentric Activities (GTEA)[19], 50Salads[20] and Breakfast[14].

GTEA contains 28 videos, 11 action classes and 31,222($\approx$ 31K) frames. It covers 7 daily activities which were performed by 4 actors in a kitchen. For evaluation, we use fourfold cross-validation and report the average as in [4].

50Salads contains 50 videos with 17 action classes and 518,411($\approx$ 520K) frames. On average, each video contains 20 action instances and is 6.4 minutes long. These action instances were performed by 25 actors and recorded from the top view, where each actor prepared a mixed salad twice. We use fivefold cross-validation for evaluation and report the average following [4].

Breakfast is the largest among the three datasets. It contains 1712 third-person view videos related to breakfast preparation activities. These videos were recorded in 18 different kitchens. Overall, there are 48 different action classes and roughly 3.6M frames. Each video contains 6 action instances on average. To evaluate, we use the standard 4 splits proposed in [14] and report the average.

Following [4], for all the above datasets, we use the temporal resolution of 15 fps. We use the timestamp annotations that Li et al.[15] provide.

Evaluation Metrics. Following the standard protocol of temporal action segmentation[15, 2], we report frame-wise accuracy (Acc), segmental edit distance (Edit) and segmental F1 scores at intersection over union (IoU) thresholds $10\%$, $25\%$ and $50\%$.

We take the I3D [17] features extracted by [4] as the input to our framework and the MS-TCN [4] as our segmentation model. Following [15], we use two parallel stages for the first stage with kernel size 3 and 5, change the number of layers in the first stage from 10 to 12, and pass the sum of both outputs to the next stages. We train our model for 50 epochs with Adam optimizer. Out of the 50 epochs, the warm-up phase takes 30 epochs, while the mean teacher phase takes the remaining 20 epochs. We use the learning rate 0.0001 for Breakfast and 0.0005 for GTEA and 50Salads, then multiply them by 0.1 every 40 epochs. For the hyperparameters, they are determined empirically. We set $\lambda$ to 0.9, 0.99 and 0.999 for GTEA, 50Salads and Breakfast respectively. Both $\tau$ and $\alpha$ are set to 1. To help the teacher model learn progressively, we increase $\beta$ linearly from 0 to 0.5 during the 20 epochs.

During the inference time, we take the output of the teacher model as the final predictions [16], and classify each frame as the label with maximum probability.

We compare our method with recent approaches under different levels of supervision. The results are presented in Tables 1, 2 and 3. As shown in these tables, our method outperforms the state-of-the-art timestamp supervised approach with a large margin on all datasets (up to 9.4% for the F1 score at 50% IoU threshold on GTEA). At the same time, our method exceeds the approaches under weaker levels of supervision in the form of transcript or set at a comparable annotation cost. Furthermore, our method even performs comparably against the fully-supervised methods in terms of F1 scores and Edit at a much lower annotation cost. Despite the lack of the information of the action boundary, our method achieves about $97\%$ frame-wise accuracy of the full supervision method MS-TCN [4].

Study on Teacher Model. To validate the efficacy of the proposed teacher model, we compare the results of our framework with and without the teacher model. We adopt the conventional two-step training scheme for the latter one. As shown in Table 4, the framework with the teacher model outperforms the one without it. We also present the evolution processes of frame-wise accuracy during the mean teacher phase for the two settings in Figure 1(a), and it demonstrates that the teacher model is able to maintain stability during training and achieves favorable performance. On the contrary, the conventional two-step training scheme (without the teacher model) introduces a severe oscillation which results in performance degeneration.

We also verify the effectiveness of the proposed segmentally smoothing loss. We list the results of three runs on GTEA and 50Salads in Table 4, including the one without any smoothing loss, the one with TMSE [4] and the one with the proposed segmentally smoothing loss. As analyzed earlier, the method without smoothing loss suffers from a severe over-segmentation error, which is indicated by the low F1 scores and Edit. The proposed segmentally smoothing loss outperforms the TMSE by $5\%\sim 9\%$ and $3\%\sim 4\%$ on F1 scores on the two datasets respectively. Furthermore, to present the impact of the proposed segmentally smoothing loss visually, we show two representative examples from 50Salads in Figure 4.