Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videos

Problem Definition. The training set $\mathbb{V}=\{\mathbf{v}_{i}\}_{i=1}^{N}$ consists of N videos $\mathbf{v}_{i}$. From each $\mathbf{v}_{i}$ we extract a feature vector $\mathbf{x}_{i}\in\mathbb{R}^{F\times T_{i}}$, that consists of $T_{i}$ frames of $F$-dimensional features. We denote by $\mathbb{C}=\{c_{i}\}_{i=1}^{|\mathbb{C}|}$ the set of all top-level task labels, and by $\mathbb{A}=\{a_{j}\}_{j=1}^{|\mathbb{A}|}$ the set of all lower-level attribute labels. As an implementation choice, these attributes can be the set of actions in the dataset, or the set of verb/object components of those actions. Each video $\mathbf{v}_{i}$ is labeled by a task $c_{i}\in\mathbb{C}$, and also by a set $\mathbb{A}_{i}\subseteq\mathbb{A}$ of $M_{i}$ attributes, so that $\mathbb{A}_{i}=\{a_{i,j}\}_{j=1}^{M_{i}}$. At test time, given an input video, the system estimates the top-level task.

Semantic Hierarchy Stream (SHS). To recognize the task, one approach is to directly estimate $p(c_{i}|\mathbf{x}_{i})$. However, this approach is prone to overfitting when the number of video samples per task is limited. As attributes can be shared among tasks, the average number of training videos per attribute is typically greater than the average number of videos per task. Using attribute information also helps the model learn similarities and differences of spatio-temporal patterns in different tasks.

Thus, we model task recognition as $p(c_{i}|\psi_{i}^{a}).p(\psi_{i}^{a}|\mathbf{x}_{i})$, where $\mathbf{\psi}_{i}^{a}$ is an intermediate vector of attribute scores that is computed for each $\mathbf{x}_{i}$. The system learns a mapping function $\mathcal{M}_{x}^{a}:\mathbb{R}^{F\times T_{i}}\rightarrow\mathbb{R}^{|\mathbb{A}|}$, that maps each vector $\mathbf{x}_{i}$ to attribute score vector $\mathbf{\psi}_{i}^{a}$. It also learns a function $\mathcal{M}_{a}^{c}:\mathbb{R}^{|\mathbb{A}|}\rightarrow\mathbb{R}^{|\mathbb{C}|}$, that maps each attribute score vector $\psi_{i}^{a}$ to a task score vector $\psi_{i}^{c}$ (Fig.2).

Temporal Hierarchy Stream (THS). Tasks in instructional videos are usually performed in a relative order of steps. Understanding the task-discriminative stages of a video is essential in distinguishing tasks that share similar-looking actions. Thus, we divide each video into $K$ stages of equal duration, and train a classifier $\mathcal{S}_{\kappa}:\mathbb{R}^{F\times\frac{T_{i}}{K}}\rightarrow\mathbb{R}^{|\mathbb{C}|}$ for each stage $\kappa$. The system also learns an aggregation function $\mathcal{T}:\mathbb{R}^{K|\mathbb{C}|}\rightarrow\mathbb{R}^{|\mathbb{C}|}$, that maps stage-wise predictions to classification scores $\vartheta_{i,total}$ of the entire video.

Stream Fusion Module. In the end, we fuse the predictions of the SHS and THS streams to output the final task prediction scores $f_{i}^{c}$ of the entire model. A high-level diagram of the overall network is shown on Fig.2. The network is optimized using a loss function for the fusion module, as well as separate loss functions for the SHS and THS streams.

In this section we explain in detail the architecture of our two-stream hierarchical model (Fig.3), and we derive the three proposed loss functions.

We used the standard dish labels in both datasets as task labels. All experiments on the BD dataset for all models, except the unsupervised CTE [26], were done for 9 tasks after we combined the two dishes of frying and scrambling eggs as the top-level task of making eggs, because both share almost the same set of actions. For CTE, we report the results on the original 10 classes, as given by the authors. We note that CTE is unsupervised and not a direct competitor.

Task Classification. Table 1 shows quantitative results on task recognition, for our method as well as other methods that use different types of supervision. Particularly, [29, 44] are the state-of-the-art open-source weakly-supervised segmentation methods. They implicitly identify the task corresponding to the inferred sequence of actions during inference. Our explicit task modeling significantly outperforms them in accuracy by around 5 to 9 percent on the BD dataset, and by 16 to 21 percent in the 58 tasks of the C2 dataset.

Originally, [35, 39] are the state-of-the-art open-source weakly-supervised methods with specific loss functions to classify and localize sparse action instances in videos. To compare with them, we trained both to classify tasks. Also, [18] and [19] classify tasks in long videos by training multi-scale temporal convolutions and graph based representations respectively. Both networks make heavy use of memory and suffer from overfitting specifically in the C2 dataset, where using low-level attributes is key. While such direct task modelings under weak supervision prove to be more effective than the implicit classification using fine-grained action segments [29, 44], our hierarchical approach outperforms all competitors considerably in all metrics and datasets. Table 1 shows our t-mAP on the C2 dataset comes close to the fully-supervised baseline [46], which is trained on frame-level action ground-truth. Comparison results on 10 classes of the BD dataset are in the supp. material.

Action Segmentation. Table 2 shows results for action segmentation. In our experiments, we combined our two-stream task prediction framework on top of the state-of-the-art weakly-supervised segmentation methods [29, 44] and achieved new state-of-the art results on both datasets, manifested more vividly in acc-bg, because background frames are independent of the task. Therefore, excluding background frames highlights the contribution of the correct task label in segmentation. This consistent improvement in all metrics, while decreasing the inference time by 80-90% (Table 3), demonstrates the potential of the proposed top-down approach for weakly-supervised segmentation. Moreover, CDFL+GT in Table 2 represents CDFL segmentation results constrained by ground-truth task labels, which serves as an upper bound for our proposed top-down model.

Stream-Specific Results. We evaluated the contribution of the SHS and THS streams separately in Table 4. The SHS stream is more effective on the C2 dataset because of two main reasons: First, the average number of videos per task (3.4) is low compared to that of videos per attribute (28.4), so any direct way of task modeling is prone to overfitting. Second, the large number of attributes per task allows the learning of a discriminative attribute-to-task mapping. Meanwhile in the THS stream, despite the weak classification power of the stage-specific classifiers, our hierarchical modeling is able to aggregate stage-wise predictions effectively and produce significantly superior results. This shows that different stages provide complimentary information. Note that the THS stream alone achieves state-of-the-art on the BD dataset with only task label supervision.

Semantic Hierarchy Ablation. As shown in Table 5a, removing the attributes from the semantic hierarchy loss (Eq.7), and directly classifying tasks from the feature encoding $\mathbb{\phi}(\mathbf{x})$, leads to around $12\%$ drop in t-acc on the C2 dataset. Simply sharing low-level attributes among tasks is beneficial, and using the TF-IDF weights led to an additional $7\%$ difference in t-acc (Table 5b).

Temporal Hierarchy Analysis. Modeling tasks as a temporal hierarchy of multiple stages improves the performance compared to the single-stage approach. Furthermore, as indicated by Table 7, such an approach is not sensitive to the number of stages ($K>1$) in the hierarchy. This concludes that these stages provide complimentary information for the stage-aggregation function regardless of their exact positioning or duration in the video.

captionskip=2pt

capposition=bottom

Comparison of the Stream Fusion Mechanisms. Table 7 compares different mechanisms to fuse the predictions of SHS and THS streams. Specifically, the Task-wise Switching Gates are trained to identify the stronger stream per task and perform best, while the vanilla and weighted averaging compromise between both streams and produce sub-optimal results. In the BD dataset these gates propagate the results of the THS stream, whereas in the C2 dataset they switch between both for different tasks and combine predictions (see row 1, 5 and 6 of Table 4).

Fig.4 compares results of the THS stream with the single-stage baseline on four challenging videos from the BD dataset. Due to similar-looking actions shared between tasks, the single stage baseline misclassifies the video task, whereas our THS stream outputs the correct task under different stage-wise settings. Specifically, in the first and third videos, our model classifies the task correctly although only one of the stage predictions is correct. For example, according to the stage-wise accuracy of the task making chocolate milk in Fig.4, the last stage of this task is the most discriminative one. Thus, our model learns to put more weight on the predictions of this stage, which compensates for the first two stages outputting the wrong class of making cereal.

The two tasks of making coffee and making tea share similar-looking actions in the first and second videos, so analyzing the entire video in one step produces wrong predictions for both cases. The second video, in particular, provides an interesting case where similar visuals of the action taking cup between tasks of making chocolate milk and making tea led to confusion of the first stage. Also, the later two stages mistakenly predicted the task of making coffee, because the two actions of pouring and stirring are shared between both tasks of making coffee and making tea. Although all three stage-wise predictions are wrong, the aggregated result of those stages is correct. This shows that the proposed hierarchical model not only considers the predicted class of each stage, but also learns the relationship between stages and their fine-grained prediction scores.

In a given stage, the short discriminative part may be dominated by the longer ambiguous section. For example, the final stage of the last video depicts how stirring while occluding the bowl dominates the shorter and more discriminative action of pouring milk. This effectively resembles the appearance and motion of flipping a pancake by spatula, but the complementary information of the first two stages eventually results in the correct aggregated recognition.

In this section, we compare I3D and iDT features for the purpose of task recognition in weakly-labeled instructional videos. Specifically, we present results of existing models using I3D (Table 8) and iDT (Table 9) features on the MPII Cooking 2 dataset [46] as well as the first split of the Breakfast dataset [23].

We used the Fisher vectors of iDT features as in [23, 43]. The Fisher vectors for each frame are extracted over a sliding window of 20 frames. They are first projected to a 64-dimensional space by PCA, and then normalized along each dimension. Also, we extracted the I3D features of the Cooking 2 dataset using TV-L1 optical flow [55] on a moving window of 32 frames with a stride of 2, and the pre-computed I3D features of the Breakfast dataset were obtained from [12]. Furthermore, we applied PCA to the extracted I3D features to reduce the dimensionality of RGB and optical flow channels from 1024 to 128. We fed the same features to all competitors except [54] in Table 10 whose code is not publicly available, so we compare with their reported result on ResNet101 [16] features.

In the Cooking 2 dataset, we train the models on the training split and test on the test split. However as [29] and [44] take a long time to train and infer the segments, in Tables 8 and 9, we only use the first split of the Breakfast dataset to evaluate the difference in performance of all models when using I3D and iDT features as input. Note that the reported task recognition results on the Breakfast dataset in the paper are the average of all four splits using the best case for each method.

Explicit task classification methods, e.g., ours, W-TALC [39] and 3C-Net [35], consistently perform better with I3D features on both datasets, whereas the bottom-up inference of tasks in NNViterbi [44] and CDFL [29] produces mixed result. In particular, the performance of [29] and [44] on the Breakfast dataset considerably improves upon using iDT features. Overall, the more significant presence of object information in I3D features helps to classify top-level tasks more accurately, while detecting fine-grained actions seems to be less affected by such appearance information.

Timception [18], VideoGraph [19] and RhyRNN [54] are the latest state-of-the-art methods to classify tasks in minutes-long videos and are the closest competitors to our work. We compared the standard four fold cross validated results of Timeception and VideoGraph over 9 classes of the Breakfast dataset in Table 1 of the paper, however, we could not compare our method to RhyRNN because the source code of RhyRNN is not publicly available to adjust that model to our evaluation settings. Hence, in Table 10, we present comparison results of our method with the reported accuracy of this method and different versions of other models over the original 10 classes of the Breakfast dataset. For a direct comparison with RhyRNN , we show results on the first split as reported in RhyRNN.

Furthermore, Table 10 shows the original reported results of Timeception and VideoGraph, which are lower than our re-implemented versions in both cases. Contrary to the standard splitting rule of the Breakfast dataset, both works have used the last 0.15% of subjects in the dataset (8 subjects) to test their performance. Our result on this split significantly outperforms previous methods (Table 10). [18] and [19] also use the output before the last average pooling layer (pre pooling) in the I3D network as features, unlike us, where we use the features after the pooling layer (post pooling). The results in Table 10 suggest the superiority of the latter, because the lower dimension after pooling allows each network to be given more features as input, which increases their input temporal range.

Interestingly, the task accuracy for most models, including ours, hardly drops upon evaluation on 10 classes and our method is still superior than different versions of state-of-the-art.

As there are similar terms and many symbols used in the paper, here, we provide specific definitions of terms (Table 11) and symbols (Table 12) for readers to refer to.