OnlineTAS: An Online Baseline for Temporal
Action Segmentation

Consider an untrimmed video $v=\{x_{t}\}_{t=1}^{T}$ of $T$ frames, where $x_{t}\in\mathbb{R}^{D}$ is the per-extracted frame feature at time $t$ and $D$ is the feature dimension. An action segmentation model partitions $v$ into $N$ contiguous and non-overlapping temporal segments $\{s_{i}=(y_{i},\ell_{i})\}_{i=1}^{N}$ corresponding to actions $y\in\mathcal{Y}$ present in the video, where $\ell$ indicates the segment length and $\mathcal{Y}$ defines the action space survey . A widely adopted strategy for TAS is to design a segmentation model $\mathcal{M}$ that predicts the action label $y_{t}$ for each frame $x_{t}$, akin to a frame-wise classification. In the offline setting ms-tcn ; yi2021asformer ; liu2023diffusion , the per-frame prediction $\hat{y}_{t}$ is based on the entire video sequence. The online setting uses only frames up to the current prediction time $t$, without access to future frames. Comparatively:

where $p_{t}\in\mathbb{R}^{|\mathcal{Y}|}$ is the estimated action probability for frame $x_{t}$. Most existing offline TAS works ms-tcn ; yi2021asformer ; li2020ms train the segmentation model $\mathcal{M}$ using a cross-entropy loss ($\mathcal{L}_{\text{cls}}$) for frame-wise classification which is balanced by a smoothing loss ($\mathcal{L}_{\text{sm}}$) that encourages smooth transitions between consecutive frames:

In this paper, we opt for the widely recognized convolution-based architecture MS-TCN ms-tcn as our foundational framework. This choice is driven by its relatively lower computational requirements than the attention- or diffusion-based models yi2021asformer ; liu2023diffusion . A straightforward transition from offline mode to an online mode of the segmentation model $\mathcal{M}$ is to substitute standard convolutions with causal convolutions. Causal and standard convolutions differ in their receptive field in that causal convolutions consider only past and present inputs while standard convolutions may incorporate both past and future inputs within a kernel. Mathematical details and illustrations of the two are shown in the Appendix.

The context-aware feature augmentation (CFA) module generates enhanced clip-wise features through interactions with temporal context captured by an adaptive memory bank. The module operates on a clip-wise basis. During training, video $v$ is split into $K$ non-overlapping clips $\{c_{k}\}_{k=1}^{K}$. Each clip has a window size $w$ and is sampled from $v$ with a stride of $\delta=w$, where the final clip $c_{K}$ is padded if $|c_{K}|<w$. The CFA module integrates the original pre-extracted frame features $c_{k}=\{x_{t}\}_{t=(k-1)w+1}^{kw}$ with temporal context to produce a context-enhanced version of representations $\tilde{c}_{k}=\{\tilde{x}_{t}\}_{t=(k-1)w+1}^{kw}$. Like lstr ; unders1 , our CFA module is also equipped with a simultaneously updated memory bank $M_{k}$ as a context resource for feature augmentation. The memory bank is further described in Sec. 3.3.

At each step $k$, context is accumulated by feeding $c_{k}$ through a lightweight GRU gru to obtain $c_{k}^{\text{GRU}}$. The GRU is reliable in capturing information over long video sequences robust . The clip is then passed through a context aggregation block to be augmented. The context aggregation block incorporates the GRU features $c_{k}^{\text{GRU}}$ with the memory state $M_{k-1}$ from the previous step for $I$ iterations. Concretely, we pass $c_{k}^{\text{GRU}}$ through a self-attention (SA) block to encourage information exchange with the local clip window. Additionally, we leverage a Transformer decoder vaswani2017attention to achieve a more effective memory encoding $\tilde{M}_{k-1}^{\text{TD}}$, i.e.,

The outputs from the self-attention module ($c_{k}^{\text{SA}}$) and the transformer decoder ($c_{k}^{\text{TD}}$) are then combined with cross attention (CA) and merged with $c_{k}^{\text{GRU}}$ to produce the context-augmented features:

The detailed formulas of SelfAttn(), TransDecoder(), CrossAttn() are given in the Appendix. An illustration of our CFA module is provided in Fig. 1.

In a similar spirit with lstr , our memory is designed to account for both short- and long-term context, i.e., $M=[M^{\text{long}},M^{\text{short}}]$. Short-term memory helps capture the local action dynamics while long-term memory retains information across extended durations important for TAS ltcontext ; sener2020temporal .

Short Memory $M^{\text{short}}$. Given that our enhancement module works on a per-clip basis with temporal stride $w$, we directly regard the enhanced feature $\tilde{c}_{k-1}$ from the last clip as the short-term memory, i.e., $M_{k}^{\text{short}}=\tilde{c}_{k-1}\in\mathbb{R}^{Dw}$.

Long Memory $M^{\text{long}}$. We update our long-term memory with information from processed clips. Specifically, we apply a convolutional layer on top of our context-enhanced representation $\tilde{c}_{k}$, where $\tilde{c}_{k}\in\mathbb{R}^{Dw}$, to collapse the temporal dimension and yield a memory token $m_{k}=\text{Conv1D}(\tilde{c}_{k})\in\mathbb{R}^{D}$. This memory token is then appended to the current long-term memory.

Adaptive Memory Update. The memory is updated whenever a new clip is processed. In practice, we constrain the total footprint of both short- and long-term memory to match the size of the processing clip, i.e., $M\in\mathbb{R}^{Dw}$. At the beginning of each sequence, the memory bank is initialized with short-term information only, i.e., $M\!=\!M_{0}^{\text{short}}\!=\!c_{1}$ and $M_{0}^{\text{long}}\!=\!\varnothing$. As more clips are processed, we gradually increase the budget to accommodate longer-term information. However, in anticipation of $M^{\text{long}}$ draining the entire budget in prolonged sequences, we cap its utilization at a maximum of two-thirds of the total budget. In instances where this threshold is exceeded, the earliest token is discarded, i.e.,:

The remaining budget is allocated for the short-term information accordingly:

As video progresses, our feature augmentation module (Sec. 3.2) receives more longer-term context while emphasizing only shorter and more relevant short-term information. This adaptive approach enables the context memory to flexibly shift its attention between short and long-term information as the video progresses. Algorithm 1 summarizes the update mechanism.

Discussion. Our module integrates context memory on top of a GRU layer. While the GRU captures temporal dependencies, it may struggle, especially in thousands-frame long sequences common in TAS. The context memory supplements the GRU by allowing selective access and updates, thereby enabling the retrieval and manipulation of long-term information. Such a design is supported by our empirical study that the explicit memory can extend the capacity of the GRU’s internal state. Supporting ablations are found in Sec. 4.1.

Training. Our final online segmentation model is constructed by combining our CFA module with a single-stage TCN ms-tcn with causal convolutions. Specifically, TCN takes as input the enhanced representations $\tilde{c}_{k}$ and maps them to the same labels for $c_{k}$. We train the framework end-to-end with the loss function formulated in Eq. 2, but on a clip basis with $T$ replaced by $w$:

we set $\lambda=0.15$ following ms-tcn . Like lstr , training on a clip basis provides better efficiency.

Inference. We present two distinct inference approaches by manipulating the clip sampling stride parameter $\delta$. The first mode of inference, referred to as online, is characterized by setting $\delta=1$. In such a setting, a video clip of $w$ frames is processed, with emphasis placed solely on the prediction derived from the final frame and rest are discarded. This facilitates the scenario when frame-by-frame prediction is preferred. The alternate mode of inference, termed semi-online, adheres to the training regime by setting $\delta=w$. In this mode, video clips are processed, and the dense predictions generated across all $w$ frames are preserved as final output. An illustration of two modes of inference is provided in Fig. 2.

Our intuition is that a valid action segment should not fall below a minimum length threshold unless there is high confidence in the prediction to justify a change in the action class. Specifically, we consider the maximum softmax probability of a prediction as its confidence measure, denoted as $q_{t}=\max(p_{t})$, for frame $x_{t}$, as $q_{t}$ to some extent indicates its reliability. A prediction is considered “unreliable” if its confidence measure scores below a certain threshold $\theta$, i.e., $q_{t}<\theta$. For the frame with “unreliable” prediction, we disregard its current prediction and assign the action label of its proceeding frame $\hat{y}_{t-1}^{*}$, when the previous action segment is shorter than the minimum length. Otherwise, we retain the original prediction. We set the length threshold $\ell_{\text{min}}=\sigma\times T_{\text{max}}$ with $\sigma\in(0,1)$, in proportion to to the longest video duration $T_{\text{max}}$ in training set.

Our post-processing mitigates the over-segmentation by adjusting action boundaries according to network predictions and action length, which is very efficient compared to unsup2 ; unsup3 that calculates frame similarities. The procedure for post-processing is illustrated in Algorithm 2.

Effectiveness. We report the overall performance for online and semi-online inference (see Sec. 3.4) in in Table 1. Our baseline is a single-stage causal TCN and we build our framework on top of it. Across all three datasets, the integration of our CFA module leads to a consistent boost in the segmentation performance. Specifically, our approach gained 5.7% (75.2% vs. 80.9%) in Acc and 9.2% (19.6% vs. 28.8%) in Edit on 50Salads. While the improvements on other datasets are not as significant, they still show effectiveness, with a margin of about 2%. Generally, semi-online inference achieves better performance over online across all metrics. Such improvement is likely because clip-wise prediction better preserves the local temporal continuity of labels compared to step-by-step single frame prediction.

Comparing across the metrics, segmental scores appear to be significantly low. On breakfast breakfast with our online inference, a frame-wise accuracy of 56.7% only corresponds to a 9.3% F1 score with 50% IoU. Such score indicates a severe over-segmentation issue and necessitates an effective post-processing. However, a significant performance increase is observed on Edit and F1 scores after our proposed pose-processing. For example, the same F1 score increases to 30.5%, tripling its original value. Although post-processing could lead to a slight decrease in accuracy, it demonstrates great effectiveness in mitigating the over-segmentation problem.

Ablation study. Table 3 evaluates the components in our CFA module. The first row is our single-layer causal TCN baseline with strong frame-wise accuracy but poor segmental metrics. The GRU boosts segment metrics (7-11%) over the baseline, showing its ability to accumulate context information. While CFA using the current clip as pseudo memory predictably leads to a performance drop (5%) compared to GRU due to lack of any context information. Combining either GRU or our adaptive memory with our CFA achieves very close performance (rows 4 and 5), highlighting the importance of the context information for TAS. The complete model yields the best performance and boosts Acc by 7% and average segmental scores by 15.3%. This validates the complementary effect of GRU’s internal state and our explicit memory design.

Number of layers in CFA. Table 3 explores the interaction iterations $I$ in CFA. The results indicate that the performance is not significantly affected by the number of iterations. In practice, we set the number of iterations to 2, as it achieves a good balance between performance and efficiency.

Memory composition ($M^{\text{short}}$/$M^{\text{long}}$). We assess the impact of memory types and present results in Table 5. It shows comparable performances for each memory type when considered individually. However, the combination of both yields a 2% improvement in Acc and 1.5% for averaged segmental metric, suggesting the significance of incorporating diverse memory for TAS.

Clip size $w$ / Memory size $\text{len}(M)$. In our implementation, we set clip size and memory size to be equal and we report its influence on performance in Table 5. It shows a larger clip size leads to better segmental results; this is because temporal continuity can be better modeled with longer clips for learning. However, the information of short actions could be diluted when compressed to form the memory token if the window size is too large. For memory sizes of 16, 32, and 64, the earliest memory are discarded as the average length of 50Salads is $\approx$5.8k frames. With the size reducing, the performance gradually drops and reaches the lowest of 79.8% in Acc compared to the peak of 82.4%. Note that with the memory size set to 16, our approach only retains long-term information from up to 192 frames, 30$\times$ less than the average video length.

Post-processing hyperparameters. Two hyperparameters are defined in our post-processing: confidence threshold $\theta$ and the minimum segment length $\ell_{\text{min}}$. We vary its scaling factor $\sigma$ to assess $\ell_{\text{min}}$. In Table 7, increasing $\theta$ greatly enhances the segment results, with a 18.1% increase observed when $\theta=0.7$. Although the accuracy tends to decrease as $\theta$ becomes larger, the drop is not as substantial (3.1%) compared to the improvements in segmental results. While Table 7 shows the segmental performance stops increasing and stays stable when $\sigma>\frac{1}{16}$ with a fixed confidence score $\theta=0.9$. In conclusion, employing a higher confidence threshold can help better mitigate the over-segmentation because it makes more sense to prioritize preserving the continuity of a segment that includes frames with highly confident predictions given a fixed length budget.

Tables 8 and 9 compare our approach against the state-of-the-art TAS approaches on all three benchmarks. Due to the absence of dedicated online TAS methods, we benchmark against the online TAD approach LSTR lstr . We train LSTR on TAS datasets using the official code implementation¹¹1https://github.com/amazon-science/long-short-term-transformer. To ensure a fair comparison, we configure their working (short-term) memory to be the same as ours ($w$). Additionally, we adjust its long memory accordingly to provide access to the entire past sequence. As evident from Tables 8 and 9, LSTR lstr consistently achieves relatively low performance, particularly with Edit scores of 5.0% and 4.9% on 50Salads and Breakfast datasets, respectively. This suggests severe over-segmentation in their predictions. Moreover, these performances are inferior even to those of our baseline model (casual TCN), indicating that a direct adoption of online detection models for the segmentation task is not ideal.

Amongst all datasets, Breakfast is the most challenging, with a significant performance gap between offline and online models, particularly on segmental metrics. Notably, the F1@50 score on Breakfast experiences a drastic drop of 4/5, from 47.5% to 8.3%, highlighting the difficulty of the online segmentation task with videos that are more complex. Nonetheless, we still achieve a modest absolute performance improvement of 2%. Furthermore, our post-processing technique, significantly boosts segmental performance, nearly tripling the original performance, albeit with a slight decrease in Acc. This underscores the effectiveness of our post-processing technique in mitigating the over-segmentation. MV-TAS weakonline tackles online segmentation but under a multi-view setting. It leverages multi-view information and an offline model as a reference for online segmentation. Despite this, even our baseline model, depicted in the third-to-last row of Table 9, showcases a notable performance improvement (55.3% vs. 41.6%) over MV-TAS weakonline . This considerable margin emphasizes the competitiveness of our baseline model.

When compared to offline models, our semi-online inference with post-processing manages to surpass the offline model MS-TCN ms-tcn on 50Salads dataset across the segmental metrics and reaches around 90% of the accuracy of the best-performing DiffAct liu2023diffusion . On Breakfast, our approach lags behind the offline model in both frame-wise accuracy and segmental metrics.

Qualitative Result. Fig. 3 qualitatively compares the segmentation results from different approaches. It is clear to see that LSTR lstr suffers from the most prominent over-segmentation issue, which remains significant after the post-processing. Under the same configuration, our semi-online achieves slightly better results compared to the frame-by-frame online inference. Our post-processing, when applied, successfully removes the short fragments (blue boxes) in the raw prediction and refines the segmentation output. However, it may reduce accuracy, particularly at action boundaries (red boxes).

For failure cases, we have the following two observations: 1) Detection of action start often delays due to the need for more frame information to predict new actions, especially when facing semantic ambiguities at action boundaries; 2) Persistent over-segmentation happens when the network makes incorrect but confident predictions, which could be improved with a stronger backbone or better temporal context modeling.

We evaluate the runtime performance of our approach using an Nvidia A40 GPU with both pre-computed I3D features and raw streaming RGB inputs, and present the inference times in Table 10. As shown, our approach can achieve up to 238.1 FPS when using pre-computed I3D features. To calculate the runtime for the entire segmentation pipeline, we take into consideration of the computational overheads of optical flow calculation and the I3D feature extraction. By leveraging a GPU backend for optical flow calculation, our full framework is able to achieve a runtime of 33.8 FPS.

Inference Latency. The inference speed presented above is identical for both online and semi-online inference modes since their input sizes are the same. However, the latency can differ. In the online mode, inference is performed on a per-frame basis, meaning its latency is only dependent on the inference speed. In contrast, the semi-online mode incurs additional latency as it requires gathering frames up to the clip lengths before forming inputs.

Online inference offers better real-time responsiveness compared to semi-online inference, but the latter achieves superior performance as we discussed in Sec. 4.1. The choice between these two modes depends on the application’s priorities: if the real-time inference is critical, online inference is preferable; however, if accuracy is more important and the task is less time-sensitive, semi-online inference is recommended.

Limitation. In this work, we only evaluate our approach on cooking videos, however, handling diverse and real-world videos may present several additional challenges. One common scenario involves interrupted actions, where a subject abruptly switches to a different action, leaving the ongoing action unfinished. These interruptions can be challenging for the model to handle effectively. Additionally, the extended length of the video poses another challenge. Streaming videos can be infinitely long, so effectively managing and preserving long-form history within a fixed memory budget becomes a critical issue.