Predicting the Next Action by Modeling the Abstract Goal

The activity label of the entire action sequence is used to anticipate the next action in [33]. In  [31], observed features are used to obtain a fixed latent goal from visual features. However, humans often pursue multiple goals simultaneously [9]. Hence, we propose an abstract goal distribution that serves as a representation for one or more underlying intention(s). The final visual representation is considered as the goal in [6]. Our abstract goal allows us to anticipate actions without any knowledge of the overall activity label [33] or final visual representation [6].

Multiple representations of such as spatial, motion, and bag of objects are used to predict future actions in [13]. The authors showed that unrolling an LSTM for multiple time steps in the future is beneficial for the next action prediction. In [30], human-object interactions are encoded as features and fed to a transformer encoder-decoder to predict the features of future frames and the corresponding future actions. In [23], spatial attention maps of future human-object interactions are estimated to predict the next action.

In [36], multiple future visual representations are generated from an input frame using a CNN and the future action is predicted. Similarly in [39], an RNN is used to generate the intermediate frames between the observed frames and the anticipated action. In [20], temporal features are computed using time-conditioned skip connections to anticipate the next action. In [2, 24], RNNs are used to predict future actions conditioned on observed action labels. Similarly, in [17], a transformer is used to encode past actions and duration while another transformer decoder is used to predict both future actions and their duration. In [16], every frame is divided into patches and combined using the Visual Transformer (ViT) [10] to obtain a frame representation. These frame representations are combined using a temporal transformer to predict future features and action labels. In [38], long-range sequences are summarized by processing smaller temporal sequences using multi-scale ViTs and caching them in memory as context. Each context is attended hierarchically in time using multiple temporal transformers for action anticipation. We use a Gated Recurrent Unit (GRU) to summarize frame features and learn feature-based abstract goal distribution parameters to generate observed and next action representations.

An action anticipation model that correlates past observed features with the future using Jaccard vector similarity is presented in [11]. In [15] proposes a neural memory network to compare an input (spatial representation or labels) with the existing memory content to predict future action labels. Similarly, [29] proposes an action anticipation framework with a self-regulated learning process. They correlate similarities and differences between past and current frames. In [26], a predictive model directly anticipates the future action and a low-rank linear transitional model predicts the current action and correlates with the predicted future actions. Similarly, counterfactual reasoning is used to improve action anticipation in [41]. Our approach correlates the past and future by enforcing goal consistency between the two abstract goal distributions computed using observed features and the next action. Although our method can be also used for long term action forecasting [28] and early action recognition [34, 32], in this work we primarily focus on action anticipation.

In this section we describe how to generate feature-based abstract goal representation using variational RNN (VRNN) framework [7, 12]. Let us denote the observed feature sequence by $\mathbf{x}_{1},\mathbf{x_{2}},\cdots,\mathbf{x}_{T}$ where $\mathbf{x}_{t}\in\mathbb{R}^{d_{f}}$. Following standard VRNN, a Gaussian distribution $q_{t}=\mathcal{N}(\boldsymbol{\boldsymbol{\mu}}_{t,prior},\boldsymbol{\boldsymbol{\sigma}}_{t,prior})$ is used to model the prior distribution of the abstract goal ($\mathbf{z_{t}}$) given the observed feature sequence $t\in\{1,\cdots,T\}$ where $\boldsymbol{\mu}_{t,prior},\boldsymbol{\boldsymbol{\sigma}}_{t,prior}\in\mathbb{R}^{d_{z}}$. The parameters of abstract goal distribution $q_{t}$ are estimated using an MLP denoted by $\phi_{prior}:\mathbb{R}^{d_{h}}\to\mathbb{R}^{d_{z}}$) using the hidden state of a RNN ($\mathbf{h}_{t-1}\in\mathbb{R}^{d_{h}}$) learned from the previous $t-1$ features as follows:

	$\displaystyle q(\mathbf{z}_{t}|\mathbf{x}_{1:t-1})\sim\mathcal{N}(\boldsymbol{\boldsymbol{\mu}}_{t,prior},\boldsymbol{\boldsymbol{\sigma}}_{t,prior}),$		(1)
	$\displaystyle\text{ where }(\boldsymbol{\boldsymbol{\mu}}_{t,prior},\boldsymbol{\boldsymbol{\sigma}}_{t,prior})=\phi_{prior}(\mathbf{h}_{t-1}).$		(2)

Note that there are two separate MLPs, one to obtain $\boldsymbol{\boldsymbol{\mu}}_{t,prior}$ and another to obtain $\boldsymbol{\boldsymbol{\sigma}}_{t,prior}$. We apply $softplus$ activation to the $\phi_{prior}$ network that estimates the standard deviation ($\boldsymbol{\boldsymbol{\sigma}}_{t,prior}$). Unless otherwise specified, all $\phi_{<>}(\cdot)$ used in our model are two layered neural networks with ReLU activation.

The posterior distribution of the abstract goal ($r$) computes the effect of observing the incoming new feature $\mathbf{x}_{t}$ as follows:

$\displaystyle r(\mathbf{z}_{t}|\mathbf{x}_{1:t})\sim\mathcal{N}(\boldsymbol{\boldsymbol{\mu}}_{t,pos},\boldsymbol{\boldsymbol{\sigma}}_{t,pos}).$

(3)

As before, the parameters of $r$ are computed by a two layered MLP $\phi_{pos}:\mathbb{R}^{2\times d_{z}}\to\mathbb{R}^{d_{z}}$ using both the last hidden state of RNN ($\mathbf{h}_{t-1}$) and the incoming feature ($\mathbf{x}_{t}$) as follows:

$\displaystyle\boldsymbol{\boldsymbol{\mu}}_{t,pos},\boldsymbol{\boldsymbol{\sigma}}_{t,pos}=\phi_{pos}([\phi_{x}(\mathbf{x}_{t}),\phi_{h}(\mathbf{h}_{t-1})]),$

(4)

where $\phi_{x}:\mathbb{R}^{d_{f}}\to\mathbb{R}^{d_{z}}$, $\phi_{h}:\mathbb{R}^{d_{h}}\to\mathbb{R}^{d_{z}}$ are linear layers and $[\cdot,\cdot]$ represents vector concatenation. We use the reparameterization trick [21] to sample an abstract goal ($\mathbf{z}_{t}\in\mathbb{R}^{d_{z}}$) from the prior distribution $q(\mathbf{z}_{t}|\mathbf{x}_{1:t-1})$ as follows:

$$\mathbf{z}_{t}=\boldsymbol{\boldsymbol{\mu}}_{t,prior}+\boldsymbol{\boldsymbol{\sigma}}_{t,prior}\odot\boldsymbol{\epsilon},$$

(5)

where $\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{1})\in\mathbb{R}^{d_{z}}$ is a standard Gaussian distribution. Then sampled $\mathbf{z}_{t}$ is used to obtain the next hidden state of the RNN¹¹1Our RNN is a standard GRU cell. as follows:

$$\mathbf{h}_{t}=RNN(\mathbf{h}_{t-1},[\phi_{x}(\mathbf{x}_{t}),\phi_{z}(\mathbf{z}_{t})]),\forall t\in{1,\cdots,T}$$

(6)

where $\phi_{z}:\mathbb{R}^{d_{z}}\to\mathbb{R}^{d_{z}}$ acts as a feature extractor over $\mathbf{z}_{t}$. The sampled abstract goal ($\mathbf{z}_{t}$) can be used to reconstruct (or generate) the feature sequence as done in VRNN framework [7, 12]. However, we use it to represent feature-based abstract goal. Our intuition comes from the fact that humans derive action plans from goals, and videos are a realization of this action plan. Therefore, by construction, goal determines the video (feature evolution in our case). Interestingly, as the abstract goal latent variable encapsulates the video feature generation process, by analogical similarity, we make the proposition that latent variable ($\mathbf{z}_{t}$) represents the notion of feature-based abstract goal.

Therefore, we denote the \sayfeature-based abstract goal distribution as follows:

$$p(\mathbf{z}_{T})=q(\mathbf{z}_{T}|\mathbf{x}_{1:T-1}).$$

(7)

The abstract goal distribution represents all abstract goals with respect to a particular observed features sequence. Any observed action may lead to more than one goal. Our abstract goal representation captures these variations.

Human actions are causal in nature and the next action in a sequence depends on the earlier actions. For example, washing vegetables is succeeded by cutting vegetables when the goal is \saymaking a salad. We capture the causality between observed and next actions using “the observed action representation” and the “next action representation”. We obtain the observed action representation ($\mathbf{a}_{O}$) using feature-based abstract goal and the hidden state of RNN as follows:

$$\mathbf{a}_{O}=\phi_{O}([\phi_{z}(\mathbf{z}_{T}),\phi_{h}(\mathbf{h}_{T})]).$$

(8)

Here $\phi_{O}:\mathbb{R}^{2\times d_{z}}\to\mathbb{R}^{d_{h}}$ and $\mathbf{z}_{T}$ is sampled from the abstract goal distribution $p(\mathbf{z}_{T})$ using Equation 5.

Then we obtain the distribution of next action representation ($\mathbf{a}_{N}$) conditioned on the hidden state of the RNN and the observed action representation denoted by $p(\mathbf{a}_{N}|\mathbf{h}_{T},\mathbf{a}_{O})$. The reason for modeling next action representation as a distribution conditioned on hidden state and the observed action representation is twofold. First, a particular observed action may lead to different next actions depending on the context and goal. Note that in our model, both observed action representation $\mathbf{a}_{O}$ and the RNN hidden state $\mathbf{h}_{T}$ depend on the feature-based abstract goal representation. Second, there can be variations in human behavior when executing the same task. The next action representations are generated using a Gaussian distribution $\mathcal{N}(\boldsymbol{\mu}_{\mathbf{a}_{N}},\boldsymbol{\sigma}^{2}_{\mathbf{a}_{N}})$ where $\boldsymbol{\mu}_{\mathbf{a}_{N}},\boldsymbol{\sigma}^{2}_{\mathbf{a}_{N}}\in\mathbb{R}^{d_{z}}$ whose parameters are estimated as follows:

	$\displaystyle p(\mathbf{a}_{N}|\mathbf{h}_{T},\mathbf{a}_{O})\sim\mathcal{N}(\boldsymbol{\mu}_{\mathbf{a}_{N}},\boldsymbol{\sigma}^{2}_{\mathbf{a}_{N}})$
	$\displaystyle\text{ where }(\boldsymbol{\mu}_{\mathbf{a}_{N}},(\boldsymbol{\sigma}_{\mathbf{a}_{N}}))=\phi_{N}([\phi_{h}(\mathbf{h}_{T}),\phi_{a}(\mathbf{a}_{O})]),$		(9)

where $\phi_{a}:\mathbb{R}^{d_{h}}\to\mathbb{R}^{d_{z}}$ and $\phi_{N}:\mathbb{R}^{2\times d_{z}}\to\mathbb{R}^{d_{z}}$ are two separate MLPs. Now we sample multiple next action representations from the next action representation distribution using the reparameterization trick as in Equation 10,

$$\mathbf{a}_{N}=\boldsymbol{\mu}_{\mathbf{a}_{N}}+\boldsymbol{\sigma}_{\mathbf{a}_{N}}\odot\boldsymbol{\epsilon},$$

(10)

where $\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{1})\in\mathbb{R}^{d_{z}}$ is a standard Gaussian distribution.

Now, we obtain action-based abstract goal from observed and next action representations using generative variational framework [21]. The distribution for action-based abstract goal is modeled with a Gaussian distribution conditioned on the next action representation denoted by $q(\mathbf{z}_{N}|\mathbf{a}_{N})$ whose parameters are computed as:

	$\displaystyle q(\mathbf{z}_{N}|\mathbf{a}_{N})\sim\mathcal{N}(\boldsymbol{\mu}_{Nq},\boldsymbol{\sigma}_{Nq}),$
	$\displaystyle\text{ where }(\boldsymbol{\mu}_{Nq},\boldsymbol{\sigma}_{Nq})=\phi_{Nq}(\phi_{a}(\mathbf{a}_{N})),$		(11)

where $\boldsymbol{\mu}_{Nq},\boldsymbol{\sigma}_{Nq}\in\mathbb{R}^{d_{z}}$ and $\phi_{Nq}:\mathbb{R}^{d_{h}}\to\mathbb{R}^{d_{z}}$ is implemented with two MLPs. On the other hand, parameters of the action-based abstract goal distribution conditioned on both observed and next action representation are given as follows:

	$\displaystyle r(\mathbf{z}_{N}|\mathbf{a}_{N},\mathbf{a}_{O})\sim\mathcal{N}(\boldsymbol{\mu}_{Nr},\boldsymbol{\sigma}_{Nr})$
	$\displaystyle\text{ where }(\boldsymbol{\mu}_{Nr},\boldsymbol{\sigma}_{Nr})=\phi_{Nr}([\phi_{a}(\mathbf{a}_{N}),\phi_{a}(\mathbf{a}_{O})])$		(12)

where $\boldsymbol{\mu}_{Nr},\boldsymbol{\sigma}_{Nr}\in\mathbb{R}^{d_{z}}$ and $\phi_{Nr}:\mathbb{R}^{d_{h}}\to\mathbb{R}^{d_{z}}$ is a dual headed MLP. Finally, the action-based abstract goal distribution for the next action $p(\mathbf{z}_{N})$ is given by the distribution

$$p(\mathbf{z}_{N})=q(\mathbf{z}_{N}|\mathbf{a}_{N}).$$

(13)

We use both feature-based and action-based abstract goal representation to find the best candidate for next action as explained in next section. It should be noted that while the $q(\mathbf{z}_{N}|\mathbf{a}_{N})$ only depends on $\mathbf{a}_{N}$, the $r(\mathbf{z}_{N}|\mathbf{a}_{N},\mathbf{a}_{O})$ depends on both $\mathbf{a}_{N}$ and $\mathbf{a}_{O}$.

Given a sampled feature-based abstract goal $\mathbf{z}_{T}$, we select the best next action representation $\mathbf{a}_{N}^{*}$ using the divergence between $p(\mathbf{z}_{T})$ distribution (eq. 7) and $p(\mathbf{z}_{N})$ distribution (eq. 13). We call this divergence as the goal consistency criterion. Given $\mathbf{z}_{T}$, observed action $\mathbf{a}_{O}$ and the next sampled action $\mathbf{a}_{N}$, the goal consistency criterion is derived from the symmetric KL-divergence between $p(\mathbf{z}_{T})$ and $p(\mathbf{z}_{N})$ as follows:

$$D(\mathbf{a}_{N})=\frac{D_{KL}(p(\mathbf{z}_{T})||p(\mathbf{z}_{N}))+D_{KL}(p(\mathbf{z}_{N})||p(\mathbf{z}_{T}))}{2}.$$

(14)

We choose the best next action candidate (i.e. the anticipated action candidate representation) $\mathbf{a}_{N}^{*}$ that minimizes the goal consistency criterion. The rationale is that the best anticipated action should have an action-based abstract goal representation $p(\mathbf{z}_{N})$ that aligns with the feature-based abstract goal distribution $p(\mathbf{z}_{T})$. We use the following algorithm to find the best next action candidate $\mathbf{a}_{N}^{*}$.

Finally, we predict the anticipated action from the selected next action representation as follows:

$$\hat{\mathbf{y}}=\phi_{c}(\mathbf{a}_{N}^{*})$$

(15)

where $\phi_{c}:\mathbb{R}^{d_{z}}\to\mathbb{R}^{d_{c}}$ is the MLP classifier and $\hat{\mathbf{y}}$ is the class score vector. It should be noted that in Algorithm 1, we sample only one feature-based abstraction goal in line 1 of the algorithm. However, during training we sample $Q$ number of feature-based abstraction goals and for each of them we sample $K$ number of next action representations. In this case, we select the best candidate from all $K\times Q$ next action representation candidates using Equation 14. So, the next best action is consistent and does not rely too much on sampling as long as we sample sufficient candidate next actions.

Even if the feature-based abstract goal $P(\mathbf{z}_{T})$ is obtained from VRNN framework [7, 12], the formulation of action representations $\mathbf{a_{O}}$ and $\mathbf{a_{N}}$, action-based abstract goal $P(\mathbf{z}_{N})$ and goal consistency criterion is drastically different from  [1, 25]. In [31], goal consistency is defined between latent goals before and after the action using a hard threshold. Instead, our goal consistency is a symmetric KL divergence between $p(\mathbf{z}_{T})$ and $p(\mathbf{z}_{N})$ distributions which aims to align the two abstract goal distributions. This also results in a massive improvement in next action anticipation performance as shown in the experiments.

Our anticipation network is trained using a number of losses. In contrast to prior stochastic methods [25] [1], we introduce three KL divergence losses, based on a) feature-based abstract goal, b). action-based abstract goal ($L_{NG}$), 3. goal-consistency loss ($L_{GC}$). The first loss function is used to learn the parameters of the feature-based abstract goal distribution. We compute the KL-divergence between the conditional prior and posterior distributions for every feature in the observed feature sequence and minimize the sum given as follows:

$$\mathcal{L}_{OG}=\sum_{t=1}^{T}D_{KL}(r(\mathbf{z}_{t}|\mathbf{x}_{1:t})||q(\mathbf{z}_{t}|\mathbf{x}_{1:t-1})).$$

(16)

This is based on the intuition that the abstract goal should not change due to a new observed feature. Our second loss arises when we learn the action-based abstract goal distribution. As in equation 16 above, we compute the KL-divergence between $r(\mathbf{z}_{N}|\mathbf{a}_{N}^{*},\mathbf{a}_{O})$ and $q(\mathbf{z}_{N}|\mathbf{a}_{N}^{*})$ distributions of action-based abstract goal distributions as follows:

$$\mathcal{L}_{NG}=D_{KL}(r(\mathbf{z}_{N}|\mathbf{a}_{N}^{*},\mathbf{a}_{O})||q(\mathbf{z}_{N}|\mathbf{a}_{N}^{*})).$$

(17)

We denote the corresponding best action-based abstract goal distribution by $p(\mathbf{z}^{*}_{N})=q(\mathbf{z}_{N}|\mathbf{a}_{N}^{*})$. The intuition is same as before, the goal should not change because of the next best action $\mathbf{a}_{N}^{*}$. Furthermore, the feature-based and action-based abstract goal distributions should be aligned with respect to the selected next best action $\mathbf{a}_{N}^{*}$. Therefore, we minimize the symmetric KL-Divergence between the feature-based and best-action-based abstract goal distribution as follows:

$\displaystyle\mathcal{L}_{GC}=$

$\displaystyle\frac{D_{KL}(p(\mathbf{z}_{T})||p(\mathbf{z}^{*}_{N})+D_{KL}(p(\mathbf{z}^{*}_{N})||p(\mathbf{z}_{T})}{2}.$

(18)

We coin this loss as goal consistency loss. This loss is based on $D({\mathbf{a}_{N}})$ in Equation 14 with the only difference being that $p(\mathbf{z}^{*}_{N})=q(\mathbf{z}_{N}|\mathbf{a}_{N}^{*})$ is computed with respect to the selected best next action representation $\mathbf{a}_{N}^{*}$. Finally, we have the cross-entropy loss for comparing the model’s prediction $\mathbf{\hat{y}}$ with the ground truth one-hot label $\mathbf{y}$ as follows

$\displaystyle\mathcal{L}_{NA}=-\sum\mathbf{y}\odot\log(\mathbf{\hat{y}}).$

(19)

The loss function to train the model is a combination of all losses given as follows:

$$\mathcal{L}_{total}=\mathcal{L}_{OG}+\mathcal{L}_{NG}+\mathcal{L}_{GC}+\mathcal{L}_{NA}.$$

(20)

We experimented with adding different weights to each loss but there is no significant difference in learning. Therefore, we weight them equally.

We use three well known action anticipation datasets, EPIC-KITCHENS55[8] (EK55), EPIC-KITCHENS100[9] (EK100) and EGTEA Gaze+[22] to evaluate. We validate our models using the TSN features obtained from RGB and optical flow videos, and bag of object features provided by [13] for a fair comparison with existing approaches. Our base model has the following parameters: observed duration - 2 seconds, frame rate - 3 fps, RNN (GRU) hidden dimension $d_{h}=$ 256, abstract goal dimension $d_{z}=$ 128, number of sampled feature-based abstract goals ($Q=3$), number of next-action-representation candidates ($K=10$), and fixed anticipation time - 1s (following EK55 and EK100 evaluation server criteria), unless specified otherwise. We use a batch size of 128 videos and train for 15 epochs with a learning rate of 0.001 using Adam optimizer with weight decay (AdamW) in Pytorch. All our MLPs have 256 hidden dimensions.

We compare the performance of Abstract Goal (our method) with current state-of-the-art approaches on both the seen and unseen test sets of EK55 datasets in Table 1 using a late fusion of TSN-RGB, TSN-Flow, and Object features like most of the prior work. We train separate models for verb and noun anticipation and combine their predictions to obtain action anticipation accuracy. Our method outperforms all other prior state-of-the-art methods by a significant margin for both seen kitchens (S1) and unseen kitchens (S2). Notably, we outperform Transformer-based AVT [16] and Temporal-Aggregation [33] in all measures in both seen and unseen kitchens except for Top-5 accuracy on unseen kitchens. A similar trend can be seen for EGTEA Gaze+ dataset in Table 2 where our method outperforms all compared methods. We believe this improvement is due to two factors, (i) stochastic modeling is massively important for action anticipation, and (ii) the effective use of goal information is paramount for better action anticipation.

Despite, these excellent results in both EK55 and EGTEA Gaze+ datasets, our overall results on EK100 are not state-of-the-art–see Table 3. The overall performance is affected by tail classes where our method performs not as well as recent transformer methods (see Supplementary Table 1) that are heavily trained on external data [16, 38]. EK100 dataset is dominated by long-tailed distribution where 228 noun classes out of 300 are in the tail classes. Similarly, 86 verb classes out of 97 are in the tail classes. In our model, the next-action-representation is modeled with a Gaussian distribution (Equation 11), and therefore, it is not able to cater to exceptionally long tail class distributions as in EK100. This is a limitation of our method. We do not witness the tail-class issue in EK55 as the performance measure used is accuracy compared to mean-recall in EK100. Accuracy is influenced heavily by frequent classes but mean-recall treats all classes equally. However, our model shows excellent generalization results on unseen kitchens of EK100 dataset outperforming the best Transformer model [38].

In this section, we evaluate the impact of Goal Consistency (GC) criterion and loss ($\mathcal{L}_{GC}$) using the validation set of EK55 and EK100 datasets. We train separate models for verb and noun anticipation using TSN-RGB (RGB) and Object (OBJ) features, respectively. As Mean and Median sampling are used in prior variational prediction models [1], here we use mean and median sampling as two baselines to show the effect of GC. After sampling $Q\times K$ number of next-action representations ($\mathbf{a_{N}}$), instead of selecting the best next-action candidate using GC (Algorithm 1), we obtain the mean/median vector of all sampled candidates and then make the prediction using the classifier (e.g. mean vector$=\frac{\sum\mathbf{a_{N}}}{Q\times K}$). We also experimented with a majority/median class prediction baseline. In this case, we take all $Q\times K$ predictions from the classifier (from the next action-representation candidates) and pick the majority/median class as the final prediction. Everything else stays the same for all these mean/majority/median baseline models, except we do not use GC loss ($\mathcal{L}_{GC}$) or the GC criterion (Equation 14). Results are reported in Table 4.

As can be seen from the results, there is a significant impact of GC. Especially, there is an improvement of 3.39 and 2.37 for top-1 verb and noun accuracy respectively using our GC model in the EK55 dataset for $Q=1,K=10$ over Mean sampling baseline. Similar trend can be seen for EK100 and $Q=3,K=10$ as well. Our model also outperforms majority and median class sampling baselines for both [$Q=1,K=10$] and [$Q=3,K=10$] configurations indicating the effectiveness of goal consistency. Overall, our method with GC loss and criterion performs better than all other variants. Perhaps this is because the GC criterion allows the model to regularize the candidate selection while GC loss allows the model to enforce this during the training. This clearly shows the impact of goal consistency formulation of our model for action anticipation. To understand goal consistency qualitatively, we measured KLD between abstract goals of different videos. We found that videos with similar goals have lower KLD ($6.41\times 10^{-5}$) than dissimilar goals ($2\times 10^{-4}$). Details in Supplementary Table 2.

We perform a more controlled experiment to further evaluate the impact of GC loss where we set $Q=1$ and $K=1$ and train our model with and without GC loss ($\mathcal{L}_{GC}$). It should be noted that when $Q=1$ and $K=1$, there is no impact of GC criterion. To obtain a statistically meaningful result, we repeat this experiment 10 times and report the mean performance. As it can be seen from the results in Table 4 (last two rows), clearly the GC loss has a positive impact even when we just sample a single action candidate from our stochastic model. Adding GC loss improves the top1-verb prediction by 2.57 and top1-noun prediction by 2.35 on EK55 dataset. On EK100 dataset, the improvement is 2.98 and 4.55 for top-1 verb and noun prediction. This clearly proves that not only does the GC criterion help, but also the GC loss improves the performance of the model. We see that compared to our model variant [$Q=1,K=1$ with $\mathcal{L}_{GC}$] the [$Q=1,K=10$ with $\mathcal{L}_{GC}$] model performs significantly better (last row vs row 5 of Table 4). This indicates the impact of next-action-representation sampling (Equation 11) even for a single sampled feature-based abstract goal ($Q=1$). We conclude that the goal consistency loss, the goal consistency criterion, and next-action-representation distribution modeling (all novel concepts introduced in this paper) are effective for action anticipation.

Next, we evaluate how sensitive our model is to the number of sampled feature-based abstract goals $Q$ and sampled next action-representation candidates $K$. Results are shown in Table 5. Our model is not that sensitive to the sampled number of feature-based abstract goals $Q$, especially when $K>1$. However, having $Q=3$ is better than $Q=1$ for the majority of the performance measures except for top5-noun accuracy. As the model is not that sensitive to $Q$, we select $Q=3$ as the default. The performance of the model increases with increasing $K$. We select $K=10$ as our default based on results in Table 5. Our model uses both GC loss and criterion during training. GC criterion helps the model to pick the most suitable next action candidate while GC loss encourages the model to generate good candidates. Therefore, our model is regularized to predict the most likely candidates accurately and may not need to rely too much on the sampling process. We also notice that the model is not sensitive to different starting seeds in the sampling process–see Supplementary Table 4.

We also study the impact of each loss function described in Section 3.5 and report the results in Table 6. If we use only the supervised cross-entropy loss (i.e., $\mathcal{L}_{NA}$), then the performance is the worst, especially for verbs. Both $\mathcal{L}_{OG}$ and $\mathcal{L}_{NG}$ help in regularizing the abstract goal representations ($\mathbf{z_{t}}$ and $\mathbf{a_{N}}$), and therefore results improve significantly. Especially, the $\mathcal{L}_{NA}+\mathcal{L}_{NG}$ is the best loss combination for a pair of losses. When we combine all four losses, we get the best results. While $\mathcal{L}_{NA}+\mathcal{L}_{NG}$ regularizes the learning of abstract goal representations, $\mathcal{L}_{GC}$ which minimizes the divergence between feature-based and action-based goal distributions improves the choice of next verb or noun among the plausible candidates. We conclude that all four losses are important for our model.

We also compare our model with standard Variational RNN classification. We obtain the latent variable $\mathbf{z_{T}}$ and the observed action representation $\mathbf{a_{O}}$ from Equation 8, we classify $\mathbf{a_{O}}$ using a classifier to obtain the future action. In this case, we train the VRNN model with cross-entropy loss and KL-divergence (Equation 16). We call this baseline as VRNN, where we sample 30 candidates for $\mathbf{a_{O}}$ to match our baseline architecture with 30 next action candidates ($Q=3,K=10$). As shown in Table 7, our method performs much better than the VRNN method (either considering mean or median prediction).

In Table 8, we see that our model produces consistent anticipation performance for various anticipation time horizons on EK55 dataset. One model is trained for each anticipation duration. On all anticipation times, abstract goal outperforms existing approaches substantially (5-12%). Even when the anticipation time is increased to 2 seconds, abstract goal performance does not drop significantly compared to RU-LSTM [14] or Temp. Agg. [33]. We conclude that our model can produce better predictions over longer anticipation times, due to stochastic modeling of action anticipation with goal consistency. Additional ablation studies in Supplementary Tables 3-9.