Self-Regulated Learning for
Egocentric Video Activity Anticipation

Egocentric video recognition has been undergoing speedy development in recent years, evidenced by the emergence of many new large-scale egocentric video datasets  [1], [2], [3], [4], and a huge body of research work found in literature  [14, 15, 2, 16, 17, 18, 19, 20, 21, 22, 23].

As an early endeavor, Spriggs et al. [19] utilize a wearable camera and Inertial Measurement Units (IMUs) to explore first-person sensing and perform temporal segmentation and classification of human activity. Considering the strong contextual relation shown in first-person video, Fathi et al. [14] exploit the consistency of appearance representation of actions, hands, and objects, and propose a hierarchical activity modeling framework. Later, it was found that gaze location is a very important clue for egocentric video recognition, so the gaze location is first used for identifying salient visual information in [22]. A probabilistic generative model is proposed to learn the spatio-temporal relationship between gaze point, scene objects, and activity label in first-person video for daily activity recognition in [15].

With the development of deep learning, CNN is employed in [17, 23] for video feature representation in egocentric video recognition. Ma et al. [16] design a CNN-based two-stream network to integrate appearance and motion for egocentric activity recognition. Li et al. [2] consider the gaze as a probabilistic variable and use a deep network to model its distribution for joint egocentric video recognition and gaze prediction. Sudhakaran et al. [20] propose Long Short-Term Attention (LSTA), a new recurrent neural unit to pay attention to features from relevant spatial parts for egocentric activity recognition.

The rapid development of activity recognition deepens our understanding of egocentric video, and also benefits research on other related topics, such as egocentric video summarization, egocentric object detection and egocentric video anticipation.

Activity anticipation for egocentric vision has been extensively studied in [4, 9, 10, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36]. Some works focus on predicting the next activity. Qi et al. [24] propose a spatial-temporal And-Or graph (AOG) to represent events, and an early parsing method using temporal grammar is established to anticipate the next activity. Others focus on predicting what will happen after a long time interval. Ke et al. [9] propose to explicitly condition the anticipation on time, which is shown to be efficient and effective for long-term activity anticipation. Furnari et al. [30] propose to explore the dynamics of the scene and introduce a model to analyze fixed-length trajectory segments to forecast the next active objects.

Recently, a new egocentric activity anticipation challenge is proposed in [4]. On this challenging dataset, Furnari et al. [25] propose new loss functions for activity forecasting, which incorporate the uncertainty of the prediction of future activities. A learning architecture, i.e., RU-LSTM [10], is proposed, which processes RGB, optical flow and object-based features using two LSTMs and a modality attention mechanism to anticipate future activities.

Most of the above activity anticipation techniques have not taken full advantage of the rich context in the video content in their models. Besides, the semantic context information existing among the target activity is left to be uninvestigated. In comparison, we develop a self-regulated learning process which uses the contextual relation along the temporal direction in video sequences on both feature and semantic level to adaptively refine the video features for activity anticipation. Our model can be applied to different activity anticipation settings, like long time anticipation as in [7] and the challenge in [4] on large-scale datasets.

Numerous works based on deep CNN have been proposed for video action recognition. For typical 2D-CNN-based methods [37], 2D convolution is simply applied on single video frame and the frame features are fused. To model temporal information, two-stream-based methods [38, 39, 40] are introduced to model appearance and dynamics separately with two networks and they are fused in the middle stage or at the end. In another branch, many 3D-CNN-based methods (C3D [41], I3D [42], ECO [43] and SlowFast [44]) have been proposed to learn spatio-temporal features from RGB frames directly. Besides, the concept representations are utilized to perform video recognition [45, 46], which is believed to provide better interpretability.

Action recognition models, which can efficiently extract video feature representations, provide backbone models for the development of egocentric video understanding. For example, the TSN model [40] is utilized to extract observed video clip representations for egocentric activity anticipation in RU-LSTM  [10]. In our model, the TSN and I3D model are used as video feature extractor.

Third-person video anticipation predicts future activity categories from the third-person perspective [47, 7, 48, 49, 11, 50, 51]. Mahmud et al. [11] propose a hybrid Siamese network for jointly predicting the label and the starting time of future unobserved activity. Farha et al. [7] propose an RNN-based model and a CNN-based model to obtain accurate predictions, which scale well on different datasets and videos with varying lengths and huge variations in the possible future actions.

Third-person action prediction, also referred to as early stage action recognition, aims at predicting the label of an action as early as possible from partial observations [6]. Much effort has been devoted to action prediction [52, 6, 51, 53, 54, 55, 56]. As one of the earliest works on this problem, Ryoo et al. [52] use dynamic visual bag-of-words to model changes in feature distribution over time. Shi et al. [55] first try to obtain future visual feature through regression, and then an action recognition model is used for action prediction. Kong et al. [54] propose a deep sequential context network to reconstruct missing information of the partial observed video for action prediction. Compared to first-person videos, the semantic relation between temporally adjacent video segments appears to be more diversified.

For activity anticipation, all the available information we can get for prediction is obtained from the observed video clip. Therefore, how to effectively encode the observed video clip is the foundation for subsequent recursive sequence prediction process. As shown in Figure 2, given the observed video clip $I=\left\{{I_{1},I_{2},...,I_{o}}\right\}$, first a feature extractor $\phi$ is utilized to obtain the feature representation $F_{j}\in{\mathbb{R}^{d}}$ of the observed video frame at each time-step $j$. We can use many base models as $\phi$, ${\it e.g.}$, the TSN model [40] and the I3D model [42]. Then we use an aggregation function $\Phi$ (such as RNN model) to encode the observed video representations, and obtain the hidden representation $h_{o}\in{\mathbb{R}^{d}}$ at the last observed time-step. The process is shown as

The obtained representation $\boldsymbol{F}=\left\{F_{1},F_{2},...,F_{o}\right\}$ and $h_{o}$ will be utilized in the following recursive sequence prediction process. The choice of $\phi$ and $\Phi$ will be given in Section 4.2.1 and 4.3.7.

For recursive sequence prediction, given the observed video representation $\boldsymbol{F}$ and the hidden representation $h_{o}$ at the last observed time-step, we predict what will happen at the next anticipation time-step repeatedly until the target anticipation time-step is reached.

As mentioned in Section 1, due to error accumulation of recursive sequence prediction, the anticipated intermediate feature representation may be inaccurate. How to utilize the diversified content and rich context contained in the video to regulate the predicted representation is the core issue at the anticipation process for obtaining an accurate and complete intermediate representation.

Specifically, in Figure 2, at each anticipation time-step, given the feature representation $h_{o}$ at the last observed time-step and the predicted feature representation $h^{2}_{t-1}$ at the last anticipation time-step, a $GRU$ layer ($GRU_{1}$) is first employed to obtain the initial prediction feature representation $h^{1}_{t}$ at this anticipation time-step as

Applying the initial feature representation directly for subsequent predictions will lead to error accumulation and inaccurate final anticipation results. To get a more accurate feature representation, we utilize the contrastive loss function to rectify the predicted representation. The detail analysis will be shown in Section 3.2.

After the representation revision process, a representation on the video content that is expected to be more accurate at this anticipation time-step is obtained. The next step is to obtain useful information from the observed video clip that is related to the current video content. As shown in Figure 2, the representation $h^{1}_{t}$ will be used to dynamically attend to the observed video representation and acquire useful information $f^{1}_{t}$ related to $h^{1}_{t}$. Then $f^{1}_{t}$ and $h^{1}_{t}$ are fused to get the final intermediate representation $h^{2}_{t}$, which will be used to anticipate what will happen. We will give detail analysis in Section 3.3.

Finally, we perform the above procedures iteratively until the target anticipation time-step is reached.

After the recursive sequence prediction, we can obtain the representation $h^{2}_{t_{s}}$ at the final anticipation time-step $t_{s}$. Next, we describe in detail how to model semantic context information related to the target activity for obtaining the final accurate activity prediction results in Section 3.4.

The ultimate goal of our model is to get the activity categories at the target anticipation time-step. The probability distribution of the target activity $p^{a}_{t_{s}}\in{\mathbb{R}^{N_{a}}}$ at the final time-step can be calculated by a linear layer with softmax activation function,

where $\boldsymbol{W}_{a}\in{\mathbb{R}^{d\times{N_{a}}}}$ is the learnable parameters, and $N_{a}$ is the number of activity categories. $\hat{h}^{2}_{t_{s}}$ is the concatenation of $h^{2}_{t_{s}}$ and $h^{1}_{t_{s}}$, which increases the representation capability. In our work, we optimize the cross entropy loss $L_{a}$ to train the activity anticipation model.

Once the SRL has been trained, we predict what will happen at multiple future moments following a recursive sequence anticipation style. Given an observed video clip, our model acquires the feature representation of the clip. At each anticipation time-step $t$, we can get the final feature representation $h_{t}^{2}$ through the Recursive Sequence Prediction module. $h_{t}^{2}$ will be used to obtain the activity label that is taking place at this time-step through the Target Activity Anticipation module. On the other hand, $h_{t}^{2}$ is again forwarded through the Recursive Sequence Prediction module and Target Activity Anticipation module to produce the next prediction. The anticipation results at multiple future time-steps are obtained by repeatedly forwarding the previously generated prediction through the Recursive Sequence Prediction module and Target Activity Anticipation module until the desired final moment is reached.

EPIC-Kitchens Dataset [4] is a large scale cooking video dataset from a first person view captured by 32 subjects in 32 different kitchens. Each video is composed of multiple activity segments, annotated with 125 action and 352 object classes. There are 272 video sequences with 28561 activity segments for training/validation and 160 video sequences with 11003 activity segments for testing. Since the annotations of the test videos are not available, following [10], we split the training set into training and validation sets by randomly choosing 232 videos for training and 40 videos for validation. We consider all unique (action, object) class pairs in the public training set, and obtain 2513 unique activity classes. We also report results on the test set with seen (S1) and unseen (S2) kitchens. S1 indicates the test set includes scenes appearing in the training set, and S2 means the test set includes scenes not appearing in the training set.

EGTEA Gaze+ Dataset [2] contains 28 hours of first person cooking activity videos from 86 unique sessions of 32 subjects performing 7 meal preparation tasks. Each video contains audios, gaze tracking, human annotations of activities and hand masks. This dataset includes 10325 instances of activities, 19 action classes, 51 object classes and 106 unique activity classes. Three different train/test splits are provided by the authors, and we report the average performance of our model across all three splits.

50 Salads Dataset [12] contains 50 videos of salads preparation activities which are performed by 25 actors. The dataset is composed of 17 fine-grained activity classes. As no action and object classes are provided, we decouple 7 unique action classes and 14 object classes from all the activity categories. Following [12], we utilize a five-fold cross-validation for evaluation.

Breakfast Dataset [13] is composed of 1712 videos of people preparing breakfast meals. It contains 48 fine-grained activity categories. Similar to 50 Salads dataset, we also decouple 15 unique action classes and 36 object classes from all the activity categories. The videos are recorded in 18 different kitchens containing 52 different actors from third person view. The dataset is split into four different train/test splits: S1, S2, S3 and S4. We use these four splits for evaluation.

For EPIC-Kitchens, following [4], we use the Top-5 accuracy as a class-agnostic measure and Mean Top-5 Recall as a class-aware metric. Specifically, Mean Top-5 Recall is averaged over the provided list of many-shot actions, objects and activities. For the EPIC-Kitchens test set, we use the official evaluation metrics, i.e., Top-1 accuracy, Top-5 accuracy, Average Class Precision and Average Class Recall. For EGTEA Gaze+, Top-5 accuracy is used as the evaluation criterion. For 50 Salads and Breakfast, we use mean accuracy over classes for performance comparison.

For EPIC-Kitchens and EGTEA Gaze+, all video clips are processed every 0.25s. The input of our model is a fixed-length video clip (i.e., 1.5s in our experiments), and the goal is to anticipate what will happen at multiple time-steps (i.e., 0.25s, 0.5s, 0.75s, 1s, 1.25s, 1.5s, 1.75s and 2s). In other words, the observed time-step $o$ is 6 and the anticipation time-step $a$ is 8. For 50 Salads and Breakfast, we follow the dense anticipation protocol in [7] for the convenience of comparison with other methods. In this setting, the input is a particular percentage (i.e., 20% and 30%) of each video, and the goal is to anticipate the activity labels of the following sub-sequence with a percentage (i.e., 10%, 20%, 30% and 50%) of the video.

For aggregation function $\Phi$, we use a simple GRU layer. Other aggregation function like average pooling can also be utilized in our model. We will give detail analysis of different aggregation functions in Section 4.3.7.

For EPIC-Kitchens and EGTEA Gaze+, we use the feature provided by [10] directly. For 50 Salads, we simply use the feature provided by [63]. For Breakfast, we use I3D [42] to extract the feature representation which will be released along with our source code.

For EPIC-Kitchens and EGTEA Gaze+, when we train our model, we first randomly sample a training instance with 14 ($o$ + $a$) frames before the target activity. Then, we split it into 8 training instances with different length of anticipation time-steps (from 1 to 8). Finally, all instances are used to train our model jointly. We use SGD optimizer to train our model. The momentum and weight decay are set to 0.9 and 0.00005, respectively. The mini-batch size is 128. We also utilize dropout layer with dropout ratio 0.5. The provided action and object classes and the synthetic activity classes are utilized as labels to form $L_{v}$, $L_{n}$ and $L_{a}$. Specially, for EPIC-Kitchens, we set the initial learning rate as 0.1. The training procedure stops after 100 epochs. For the weight of each loss function, we set $\alpha$ as $0.01$ and $\beta$ as $0.8$, and the setting is determined via a cross-validation. For EGTEA Gaze+, the model is trained with an initial learning rate of 0.1 and 100 epochs. Through cross-validation, we choose $\alpha$ as $0.5$ and $\beta$ as $0.5$.

For 50 Salads and Breakfast, considering the dense anticipation protocol, we design a new training instance generation method. Specifically, we enlarge the values of $o$ and $a$ to $16$ and $16$. Besides, we use a temporal sliding window of 32 ($o+a$) to generate training instances from the beginning to the end of each video. We use Adam optimizer to train our model. $\beta_{1}$, $\beta_{2}$ and weight decay are set to 0.9, 0.999 and 0.00005, respectively. We set the mini-batch size as 128. Dropout layer with dropout ratio 0.5 is also used. We utilize the activity labels to train $L_{a}$ and the mined action and object labels to train $L_{v}$ and $L_{n}$. For 50 Salads, the learning rate starts from 0.001. The training procedure stops after 100 epochs. We set $\alpha$ as $0.9$ and $\beta$ as $0.1$ via cross-validation for the weights of each loss function. For Breakfast, the model is trained with an initial learning rate of 0.01. The training procedure stops after 80 epochs. We choose $\alpha$ as $0.5$ and $\beta$ as $0.5$ through cross-validation.

All experiments are implemented under the pytorch framework. For datasets EGTEA Gaze+, Breakfast and 50 Salads, several activity categories contain multiple objects. The annotation template for most of these activity categories is ‘put (or place) one object to (or into) another object’ (e.g., ‘put egg to plate’). For these activity categories, the first object indicates the major object in this activity. Hence, we simply utilize the first object as the object label.

For the baseline model (‘Baseline’ in Table I), the observed information encoding step is the same as SRL. After that, only a single $GRU$ layer is utilized in the recursive sequence prediction process to predict the feature representation recursively at each anticipation time. Finally, a fully connected layer with softmax activation function is used to predict the target activity. As shown in Table I, the top-5 accuracy of the baseline is 29.18% at anticipation time 1s.

As shown in Table I, compared to baseline model, the representation revision operation (‘+Rev’) boosts the activity anticipation performance from 29.18% to 30.27% at anticipation time 1s, which proves the effectiveness of the representation revision. Actually, at all anticipation times (i.e., 0.25s to 2s), the representation revision operation can lead to performance improvements. By comparing the row of ‘+Rea & SecCon’ and row of ‘SRL’ in Table I, we can see that after removing this operation, the performance is degraded to varying degrees at all anticipation times, which further proves the validity of the representation revision operation. The representation revision operation demonstrates even larger improvement with shorter time windows. For example, the performance gap between ‘Baseline’ and ‘Baseline+Rev’ at anticipation time 0.5s is 1.26% that is larger than 0.76% at 1.75s. This is mainly because that the feature representation becomes more difficult to be revised as anticipation time increases. Although our model can alleviate the error accumulation, it will inevitably cause a certain degree of error accumulation. Thus, for longer anticipation time, the representation revision operation will face greater challenge compared to situation of shorter anticipation time.

The core of the representation revision operation is the contrastive loss. We sample negative samples randomly from video clips with different activity labels as the selected negative set. We set the value of $N$ as $128$. Moreover, we conduct experiments to verify the impact of the samping methods and the number of samples $N$, as shown in Table II. ‘same video’ means we sample negative samples randomly from video clips that have the same video id as the positive sample. ‘other video’ means we sample negative samples randomly from video clips that have different video id from the positive sample. ‘all video’ means we sample negative samples randomly from all video clips of the training set.

From the Table II, we can get several important observations. First, the ‘all video’ sampling method achieves better results at most anticipation time-steps. However, compared with other sampling methods, the performance advantage is not obvious. On EPIC-Kitchens, videos are captured by different actors in the kitchen scenes. These videos contain diversified and similar semantic information, which leads to more hard negatives in the sampled batch. Hence, sampling from all video clips can better guarantee the diversity and the similarity of the negative samples compared to positive sample, which helps to get more accurate anticipation results. Accordingly, we choose this sampling method in our experiments. Second, the number of samples has a slight influence on our model. Different number of samples has their own prediction performance advantages at some anticipation times. Hence, we choose the appropriate number of samples according to the convenience of the implementation.

From Table I, with the dynamically reattending operation (‘+Rea’ in the table), the performance is 1.53% higher than baseline at anticipation time 1s. The performance improvements are also achieved at other anticipation times, showing that this operation is useful for activity anticipation. Its effectiveness can be further demonstrated by comparing the results in the line ‘+Rev & SecCon’ and line ‘SRL’ of the Table I. Besides, by seeing the performance gap between ‘Baseline’ and ‘Baseline+Rea’ at anticipation time 0.5s (1.99%) and 1.75s (1.93%), the dynamically reattending operation gives similar improvement for different anticipation times. This phenomenon is different from the representation revision operation. This is mainly because that the dynamically reattending operation can obtain useful information at different anticipation times. At each anticipation time, even though the generated feature representation contains noise, it can still coarsely represent the current video content. Accordingly, the dynamically reattending operation can use the predicted representation to capture useful observed information to some extent. Therefore, the dynamically attending operation is less sensitive to the anticipation time.

In Table I, by exploring semantic context information related to the target activity (‘+SecCon’), the performance is 1.21% higher than baseline. Moreover, by comparing the results of the line ‘+Rev & Rea’ and the line ‘SRL’, we can find the performance is degraded at each anticipation time by removing the semantic context exploration operation. These phenomenons verify the effectiveness of the semantic context exploration operation.

To show the validity of this operation more clearly, we visualize the semantic context information predicted by SRL in Figure 3. Take the first one as an example, at the target anticipation time, our model obtains the action (‘roll’) and object (‘dough’). Indeed, the ‘roll’ reveals the action of the target activity and the ‘dough’ reveals the object involved in the target activity. The obtained action and object do closely relate to the target activity. Hence, the semantic context helps us get more accurate anticipation results. Similar conclusions can also be drawn from other examples.

The validity of each component of SRL has been demonstrated in the above experiments. We also explore the effectiveness of any two combination of these components, the results are shown in the 5th~7th row in Table I. We can find that the performance of the combination of two components is higher than that of a single component. For example, the 5th row of Table I shows the result of the combination of the representation revision operation and dynamically attending operation, the top-5 accuracy at all anticipation times is higher than any single operation.

So far, all experiments are based on RGB features. To further verify the validity of SRL, we conduct extensive experiments on other types of feature representations (i.e., optical-flow and object features). The results are shown in Table III. ‘RU(Late)’ and ‘SRL(Late)’ indicate the models using late fusion strategy to merge the prediction results of the three feature modalities. ‘RU(Atten)’ and ‘SRL(Atten)’ indicate the models using an attention module to combine the results of the three feature modalities.

For a fair comparison, we use the pre-computed optical-flow and object features provided by [10]. The optical-flow features are extracted using a Batch Normalized Inception CNN. The object features are extracted using Faster R-CNN [64]. See [10] for more details about the utilized features. For models that use optical-flow or object features, the observed time-step $o$ and the anticipation time-step $a$ are also set to 6 and 8, respectively. A simple GRU layer is used as the aggregation function $\Phi$. We use SGD optimizer with the mini-batch size of 128. For model that uses optical-flow features, the initial learning rate is set as 0.05. The momentum and weight decay are set to 0.9 and 0.00005, respectively. The training procedure stops after 100 epochs. For the weights of each loss function, we set $\alpha$ as $0.5$ and $\beta$ as $0.5$, which is determined via cross-validation. For model that uses object features, we train the model with an initial learning rate of 0.1. The momentum and weight decay are set to 0.9 and 0.0001, respectively. The training procedure stops after 100 epochs. Through cross-validation, we set $\alpha$ as $0.8$ and $\beta$ as $0.8$ for the loss function.

It can be seen from Table III that our model achieves better anticipation performance under different feature modalities (i.e., RGB, OBJ or FLOW) at all anticipation times compared to [10]. Compared to the results using single feature, our model can also achieve higher anticipation accuracy at all anticipation times under the setting of any two feature modalities, i.e., RGB+OBJ, RGB+FLOW, and OBJ+FLOW. This phenomenon suggests that the OBJ and FLOW features are helpful for getting more accurate anticipation results. By comparing the results of RU(Late) and SRL(Late), we can find that our SRL(Late) achieves better performance at all anticipation times using the same features and fusion method. Specifically, at anticipation time 0.25s, our model can improve the top-5 accuracy from 37.61% to 40.21%, resulting in a 2.6% increase. Actually, the abundant visual information and semantic context information contained in the video content is not fully utilized by RU(Late). The better performance of SRL demonstrates the effectiveness and necessity of each component in our model.

Moreover, the performance of SRL(Late) is comparable to that of RU(Atten) at all anticipation time stamps. Note that RU(Atten) designs a Modality ATTention (MATT) module that calculates attention scores to indicate the relative importance of each feature modality for the final anticipation. Therefore, we design a similar attention fusion method to fuse the results of different feature modalities. At the target anticipation time-step, we first concatenate the observed video clip representations of each feature modality. Then, we use an MLP network with three layers to produce the modality-wise attention score. Finally, the attention score is used to fuse the anticipation results of each feature modality. From the experimental results, we can see that SRL(Atten) can obtain higher performance at all anticipation time stamps compared to RU(Atten) and SRL(Late). In summary, the above experiments verify the validity of single feature and feature combination. For simplicity, in the following experiments, we only use the RGB features in our model.

In our model, we consider three crucial factors when we choose GRU as our aggregation function $\Phi$. First, as a sequence model, GRU can encode the observed video information more effectively compared to pooling methods. The aggregated representation $h_{o}$ at the last observed time-step contains complex history information about the observed video clip. Second, adjacent video frames are more likely to have strong correlation. Using GRU, the aggregated representation $h_{o}$ can be pushed to the feature representation at the last observation time-step. It assists us to get more accurate prediction results at the first anticipation time-step, and benefit consequent anticipations. Third, compared to LSTM, GRU has fewer parameters. We also try other aggregation functions to compare the experimental results. They are average pooling (Avg), max pooling (Max) and LSTM (LSTM). The results are shown in Table V. We can find that GRU can get better results compared to other aggregation functions.

We compare SRL with the following anticipation models: DMR [65], ATSN [4], MCE [25], VN-CE [4], SVM-TOP3 [66], SVM-TOP5 [66], VNMCE+T3 [25], VNMCE+T5 [25], ED [53], FN [67], RL [68], EL [49] and RU-RGB [10]. Note that we only compare RU using RGB features (RU-RGB), and the results using other features are shown in Table III. We use Top-5 accuracy for activity prediction at different anticipation times (i.e., 0.25s~2s), Top-5 accuracy and mean Top-5 recall for action, object and activity prediction at anticipation time 1s to evaluate performance. The experimental results are shown in Table IV.

Table IV clearly shows that SRL achieves the start-of-the-art anticipation performance at most anticipation times. ATSN model, which simply uses the recognition model TSN [40] for activity prediction, only achieves 16.86% Top-5 accuracy at anticipation time 1s. The low performance suggests that we need to design special models to adapt to the video data property of the activity anticipation task. Methods like MCE, FN, RL and EL anticipate the target activity from the observed video clip directly. Instead, methods like RU-RGB and SRL are developed upon the recursive anticipation framework. By comparing the performance differences between these two types of methods, we can find that the recursive anticipation pattern is more suitable for activity anticipation task.

RU-RGB employs a sequence completion pre-training in their model to improve the anticipation performance. Without this pre-training setup, we can still achieve better performance. In fact, there are no enhancements to the predicted feature representation in RU-RGB. The better performance of SRL verifies the necessary of exploiting the informative visual cues contained in the video in the anticipation stage. When inspecting the Top-5 accuracy and the mean Top-5 recall for action, object and activity prediction at anticipation time 1s, we can find a relatively large improvement compared to previous methods. The improvement of action and object prediction performance shows that our semantic context exploration operation is effective, and this indeed assists us to obtain better activity prediction results.

We also conduct experiments on the EPIC-Kitchens test set with seen (S1) and unseen (S2) kitchens. The results are shown in Table VI. We can find that SRL obtains better anticipation performance than existing methods except TAR at most evaluation metrics, especially on S2. Essentially, TAR creates ensembles of multi-scale feature representations from the observed video clip. This operation is beneficial to predict the next activity. Instead, our SRL addresses the error accumulation issue over long periods of anticipation time. The Top-1 accuracy and Top-5 accuracy metrics are micro-averaged while the Average Class Precision and Average Class Recall metrics are macro-averaged. In a multi-class classification task, the micro-average is preferable if there exists class imbalance. For EPIC-Kitchens, the distribution of categories is imbalance. Accordingly, the higher performance on Top-1 and Top-5 accuracy further verifies the effectiveness of our model.

We compare SRL with other anticipation models on EGTEA Gaze+, including DMR [65], ATSN [4], MCE [25], ED [53], FN [67], RL [68]. EL [49] and RU [10]. We evaluate the performance using Top-5 accuracy at different anticipation times (i.e., 0.25s~2s). The comparison results are shown in Table VII. We can find that the performance of SRL is better than all competitors at all anticipation times.

In order to verify the generality of SRL, we also conduct experiments on third-person video datasets 50 Salads and Breakfast. We compare SRL with six third-person activity anticipation methods: Nearest-Neighbor, CNN model [7], Grammar-based [70], Uncertainty-based [8], RNN model [7] and Time-cond. [9]. We also compare SRL with the state-of-the-art egocentric video activity anticipaiton method RU-RGB [10]. The experimental results are shown in Table VIII.

We can clearly see from Table VIII that SRL outperforms most existing methods on Breakfast. On 50 Salads, in addition to individual prediction moments, SRL also achieves better activity anticipation results than existing methods. We can also find that the performance of Time-cond. model is better than SRL at some longer anticipation times. This is most likely because the Time-cond. introduces a time parameter $t$, which denotes the anticipation times. Specifically, the time parameter $t$ is fed to an MLP network to produce a time representation. Then, the time representation and representations of each observed time-step are combined for further processing. This explicit modeling of anticipation time improves the performance of their models for long-term prediction.

We can find poor anticipation performance for 50% anticipation on 50 Salads from Table VIII. We think this is mainly due to the unique characteristics of videos in the 50 Salads. First, the length of the video in the 50 Salads varies from more than 4 minutes to more than 10 minutes. Second, there are some background frames that do not contain activity information. These two factors pose great challenges to the representation revision and dynamically reattending operations in our model. Hence, given a long observed video clip, the performance of SRL for predicting activities over an exceedingly long period may degrade.

Besides, when we focus on the performance differences between RU-RGB and SRL in Table IV and Table VIII, we can find that SRL obtains good anticipation performance on both EPIC-Kitchens and longer-term anticipation on 50 Salads and Breakfast, while RU-RGB is less successful on 50 Salads and Breakfast. This observation indicates the strong predictive performance of SRL on both egocentric and third-person videos.

Since the egocentric and third-person activity ancipitation tasks are very different due to the time window they cover (the egocentric anticipation tasks are only focused on the immediate next few seconds rather than the entire rest of the video), we also conduct ablation studies on 50 Salads to see the efficiency of each component of SRL. The results are shown in Table IX.

From the table we can obtain several important conclusions. First, since the third-person datasets do not provide the action and object annotations, we derive the action and object labels from the provided activity annotations. The performance degradation from ‘SRL’ to ‘+Rev & Rea’ (or from ‘+SecCon’ to ‘Baseline’) indicates that the prediction of activity-related actions and objects is also necessary for third-person anticipation task. Second, the anticipation performance has different degrees of improvement by adding different single module or any two modules of our SRL to baseline model. These experimental results demonstrate the necessity and the validity of each component of SRL. Therefore, our specific designed future activity anticipation framework for egocentric videos is also effective for third-person videos. Third, by comparing the ablation study results on egocentric and third-person video datasets in Table I and Table IX, we can find that the most effective component of our SRL, which is ‘Rev’ on 50 Salads and ‘Rea’ (or ‘SecCon’) on EPIC-Kitchens, is different for egocentric and third-person ancipitation tasks. Actually, for third-person anticipation task, it has longer anticipation time window, which may introduce more error accumulation for recursive sequence prediction paradigm. Hence, the ‘Rev’ module of our SRL will be more significant compared with other components, which can be used to improve the representational ability of the predicted intermediate feature and bring greater performance improvements.