StillFast: An End-to-End Approach for
Short-Term Object Interaction Anticipation

Different works aimed to predict future actions before they happen from a third person point of view[22, 37, 10, 14]. The authors of [22] proposed a new hierarchical representation called hierarchical movemes which describes human movements considering multiple levels of granularity to infer future actions from still images or short video clips. The authors of [37] explored the task of anticipating future actions and objects learning to predict future visual representations from unlabeled videos. The authors of [14] proposed a reinforced encoder-decoder architecture (RED) for action anticipation which contains a reinforcement module to encourage the system to predict the correct action as early as possible. Also long-term action anticipation task has been explored in previous works [18] with the aim to predict minutes-long sequences of future actions. Even in the sport domain the problem of forecasting human moves has been explored [10]. Indeed, the authors of [10] proposed a generic framework to anticipate future events in team sports videos directly from visual inputs. The anticipation problem has been explored even predicting the future location of users which allows to build an advanced surveillance systems able to predict people’s activities[29, 23] or for autonomous vehicles to understand pedestrian intents to avoid accidents[27, 28]. In particular, the authors of [27] tackled the problem to jointly predicting the future spatial position and the body keypoints of pedestrians to have a deeper understanding of pedestrians behavior. The authors of [28] proposed a factorized multimodal approach to predict long-term trajectories of the user focusing on RGB observations and past motion history. Recently, the authors of [29] studied the problem of using few input observations to predict accurate pedestrians position proposing a new teacher-student technique.

While prediction’s tasks addressed from the third person point of view are useful in different scenarios, we focused on user-object interactions anticipation for which the egocentric point of view offers several advantages, therefore, we considered this anticipation problem relying on the large-scale egocentric dataset EGO4D[19].

Different past works have explored problems related to anticipation from the first person point of view. Furnari et al. [12, 13] proposed a model based on LSTM networks to encode past features and predict future actions from egocentric videos. The authors of [30] extended RULSTM [13] with a novel attention-based technique to consider simultaneously “slow” and “fast” features. The authors of [40] proposed a novel transform-based fusion approach which combines multi-modal features (i.e., audio and visual) to predict future actions. To better understand future human’s behavior, past works also focused on predicting future hands and objects locations. The authors of [26] presented a two-stream fully convolutional network to forecast the presence and location of hands and objects in egocentric videos. Liu et al. [26] focused on hands movements as visual representations to predict future interaction hotspots and future actions.

While different tasks related to the anticipation problem have been deeply studied, the considered task have not been addressed systematically.

The problem of Short-Term Object Interaction Anticipation has been addressed in different forms. Some works studied the problem of predicting the objects which will be involved in a future human interaction (next-active objects). The authors of [11] have been the first to explore explicitly this task proposing to analyze objects trajectories. The authors of [5] focused on the prediction of hand-objects contact representations to anticipate future actions. The authors of [21] investigated the prediction of a visual attention probability map from images considering hands and objects features. The authors of [32] presented a new egocentric dataset captured in a procedural scenario specifically annotated to address the next-active object detection task, which is tackled using simple object detectors. All these works addressed different versions of the considered task, which made the proposed approaches difficult to compare and extend over. The first attempt to systematically study the problem of Short-Term Object Interaction Anticipation has been done by the authors of [19], provided a formal definition of the task which includes the prediction of next-active objects and associated verbs and time to contact (see Figure 1). Despite this effort, few approaches have been proposed so far to tackle the task in the form defined in [19]. Among them, the baseline proposed in [19] consists of two components trained independently: A Faster R-CNN [34] object detector to detect next-active objects in the last frame of the input video, and a SlowFast [9] action recognition model to attach each bounding box a verb and a time to contact prediction. The model works as a two-branch network. The 2D Faster R-CNN model processes a high resolution image (the most recent frame of the input video) to predict next-active objects and their related classes. The high resolution is needed to have enough detail in order to detect objects at different scales. The 3D SlowFast model processes a low-resolution video clip ending at the current timestamp to predict verb labels and time-to-contact values. The approach is trained in two stages. First, the Faster RCNN model is trained using all training next-active object bounding box and class labels. This object detector is hence run over the training, validation and test data examples to complement them with a set of next-active object bounding box proposals with associated noun labels and detection scores. In a separate stage, a SlowFast model is trained to predict a verb label and a positive time-to-action real number for each bounding box proposal. The authors of [2] extended this approach by employing a DINO[39] object detector to detect and recognize next-active objects in keyframes and a VideoMAE-pretrained transformer network [36] to extract features to predict verbs and time to contact for each detected bounding box.

In this work, we propose an end-to-end approach specifically designed to anticipate the location of next-active objects, the verb which describes the future interaction and how soon the interaction will take place (time to contact). The proposed method can be trained in a single stage, reducing training times nd simplifying the research cycle.

Figure 2 reports a diagram of the StillFast backbone. Given an input video $V$ and a timestamp $t$, the proposed model takes as input a high resolution frame $V_{t}$ sampled from video $V$ at time $t$ and a low resolution video $\mathcal{S}(V_{(t-\tau_{o}):t})$ which is obtained by spatially subsampling with function $\mathcal{S}$ the video $v_{(t-\tau_{o}):t}$ of length $\tau_{o}$ (observation time) starting at time $t-\tau_{o}$ and ending at time $t$.

To process the input image and video simultaneously, the proposed model comprises a two-branch backbone.

A 2D CNN $\Phi_{2D}$ (the “still” branch) processes the high resolution frame $V_{t}$ and produces a stack of 2D features at different spatial resolutions $\Phi_{2D}(V_{t})$. The stack of features is obtained by collecting activations at the inner layers of the 2D network. As detailed later, we follow [24] to obtain the stack of features to produce a feature pyramid in order to enable multi-scale object detection as done in the standard [34] architecture.

A 3D CNN $\Phi_{3D}$ (the “fast” branch) processes the video $V_{(t-\tau_{o}):t}$ and outputs a stack of 3D features $\Phi_{3D}(V_{(t-\tau_{o}):t})$. We perform mid-level feature fusion by combining the 2D and the 3D feature stacks with a Combined Feature Pyramid Layer illustrated in the center part of Figure 2. This layer first up-samples the 3D feature map with nearest neighbor interpolation to match the spatial resolution of the 2D features and averages over the temporal dimension to obtain the $\Phi^{2D}_{3D}(V_{(t-\tau_{o}):t})$ features, which now have the same shape as the 2D features $\Phi_{2D}(V_{t})$. These features are hence passed through a 3x3 convolutional layer, summed to the 2D features $\Phi_{2D}(V_{t})$ and then passed through another 3x3 convolutional layer. The rationale behind these 2D convolutional layers is to cope with artifacts introduced with the up-sampling and sum operations. The resulting feature map is fed to a standard Feature Pyramid Layer [24] to obtain the final feature pyramid $P_{t}$.

In our experiment we use a ResNet-50 as 2D CNN and an X3D-M as 3D CNN. See the supplementary material for more details.

Figure 3 shows a diagram of the proposed prediction head, which is based on the Faster R-CNN prediction head [34] as implemented in Detectron2 [38]. A Region Proposal Network (RPN) predicts region proposals from the feature pyramid $P_{t}$. A RoiAlign layer is used to extract local features from the region proposals. We found it useful to enrich these local features with global representations of the scene. To do so, we apply global average pooling to the upper layer of the combined feature pyramid and obtain a global image representation, which we concatenate to each local feature extracted from the region proposals. The concatenated features are passed through a fusion network comprising two fully connected layers. The resulting representations are summed to the original local features through a residual connection. This allows to use global features to modulate the content of local features rather than to replace them. These fused local-global representations are used to predict object (noun) probability distributions and class-specific bounding box regression offsets using linear layers as in [34]. The predicted noun probability distributions $p(n)$ include a background class to reduce the magnitude of positive prediction probabilities in the case of uncertain predictions or proposals falling in background areas. The same features are used to predict a verb probability distribution $p(v)$ and time-to-contact TTC using linear layers. We include a background class in the verb prediction layer as well to further discard false positives when the verb cannot be reliably predicted. A softplus activation is used to predict positive TTC values. For each object proposal, we multiply the noun probability $p(n)$ by the predicted probability of the Top-1 verb, excluding the background class. This is done to make sure that the uncertainty in verb prediction influences the final prediction score $s=p(n)\cdot max_{v}\{p(v)\}$. The final predictions are obtained by considering objects with a prediction score larger than or equal to $0.05$. The model is trained end-to-end adding a cross entropy verb loss $\mathcal{L}_{v}$ and a smooth L1 [34] time-to-contact loss $\mathcal{L}_{ttc}$ to the standard Faster R-CNN losses. We weigh the $\mathcal{L}_{v}$ with $0.1$ and the $\mathcal{L}_{ttc}$ with $0.5$ in our experiments. See the supplementary material for more details.

We performed experiments on the large-scale egocentric dataset EGO4D[19]. We consider both the initial version of the dataset described in [19] (denoted as “v1” in this paper), and the recently released update of the dataset (denoted as “v2”), described at this page: https://ego4d-data.org/docs/updates/#v20-update, following the official training/evaliuation/testing splits. We focus on the subset of the EGO4D dataset which has been explicitly labeled for the Short-Term Object Interaction Anticipation task. Version v1 of this dataset consists in $120$ hours of annotated clips, including $27,801$ training, $17,217$ validation, and $19,780$ test examples, annotated with a taxonomy of $87$ noun and $74$ verb classes. Version v2 consists in $243$ hours of annotated clips, including $98,276$ training, $47,395$ validation, and $19,780$ test examples, annotated with a taxonomy of $128$ noun and $81$ verb classes. It should be noted that v1 and v2 have different training and validation sets, but they share the same test set, which makes test results obtained on the two dataset comparable. Annotations for this shared test set are not publicly available and results can be obtained by sending predictions to an evaluation server¹¹1https://eval.ai/web/challenges/challenge-page/1623/.

The evaluation has been performed using Top-K mean Average Precision, which does not penalize methods predicting up to K - 1 next-active objects which are not annotated as defined in [19]. In particular, we evaluated methods with different Top-5 mAP measures, i.e., Top-5 mAP Noun, Top-5 mAP Noun+Verb, Top-5 mAP Noun+TTC and Top-5 mAP Noun+Verb+TTC which we named Top-5 mAP Overall, to assess the ability of the model to anticipate next-active object interactions considering different levels of granularity (i.e., nouns, verbs, time-to-contact). We used $K=5$ as defined in [19].

We compare our method with respect to different approaches which addressed the considered Short-Term Object Interaction Anticipation problem.

• •

  Faster R-CNN + Random[19]: uses Faster R-CNN to detect and recognize next active objects, then predicts verbs and time to contact randomly following the distributions of training labels;

• •

  Faster R-CNN + SlowFast[19]: the baseline defined in [19] composed of two components trained independently. A Faster R-CNN [34] object detector which detects and recognizes next-active objects and a SlowFast [8] 3D network to predict for each bounding box a verb and a time to contact;

• •

  InternVideo[2]: follows the same two-stage recipe of Faster R-CNN + SlowFast. A DINO[39] object detector is used to predict next-active objects and a VideoMAE-pretrained trasnformer video backbone [36] to predict verb and time to contact for each detected next-active object.

• •

  Faster R-CNN + Features: a Faster R-CNN object detector is used to detect and recognize next active objects, then pre-extracted Ominvore features [15] are used to predict the associated verb and time to contact. This baseline has been proposed as a quickstart approach with the EGO4D dataset. In our experiments, we consider the results reported in this document: https://colab.research.google.com/drive/1Ok_6F1O6K8kX1S4sEnU62HoOBw_CPngR.

Table 1 reports the results of the compared methods the validation and test sets of both v1 and v2 of the EGO4D dataset [19] using the aforementioned Top-5 mAP evaluation measure. As can be noted from Table 1, the proposed method improves over the FRCNN+Rnd. and FRCNN+SF baselines by significant margins in the v1 validation set on verb-related metrics: $7.47\%$ vs $5.19\%$ Noun+Verb Top-5 mAP ($+2.28\%$), and $2.48\%$ vs $2.07\%$ Overall Top-5 mAP ($+0.41\%$). These results suggest that the ability of the proposed approach to process images and video with a unified backbone and accounting for the uncertainty in verb prediction as described in Section 3.2 is beneficial to performance. Indeed, it should be noted that both are based on components with similar performance: a ResNet-50 2D backbone in both cases, a SlowFast 3D backbone for FRCNN+SF and an X3D-M backbone for StillFast. On the downside, the proposed StillFast approach achieves worse performance on Noun Top-5 mAP ($16.20\%$ vs $17.55\%$, hence $-1.35\%$) and on Noun + TTC Top-5 mAP ($4.94\%$ vs $5.37\%$, hence $-0.43\%$). We speculate that this is due to training instabilities due to the multi-task nature of our training procedure, as compared with FRCNN+SF, which is trained in two separate stages (Section 5.2 further analyses performance when multi-tasking is reduced within our architecture). This seems to be mitigated when more training data is available. Indeed, when training on v2 (fourth and fifth row of Table 1), StillFast consistently outperforms FRCNN+SF on Noun + Verb, Noun + TTC and Overall Top-5 mAP, while the loss of performance on Noun Top-5 mAP is much smaller ($20.26\%$ vs $21.00\%$, hence only $-0.74\%$).

The advantages of StillFast are more evident in the test set, which suggests better generalization of the proposed approach. In v1, the proposed method outperforms FRCNN+SF by $+3.17\%$ ($9.95\%$ vs $6.78\%$) according to Noun+Verb Top-5 mAP, $+0.28\%$ ($6.45\%$ vs $6.17\%$) according to Noun+TTC Top-5 mAP, and $+1.04\%$ ($3.49\%$ vs $2.45\%$) according to Overall Top-5 mAP. Noun Top-5 mAP results are still lower, but closer to the ones of FRCNN+SF ($19.51\%$ vs $20.45\%$, hence $-0.95\%$). Also in the test set, using more training data brings larger performance margins to StillFast. In v2 test set, the proposed approach surpasses FRCNN+SF by $+3.84\%$ ($13.29\%$ vs $9.45\%$) according to Noun + Verb Top-5 mAP, $+0.45\%$ ($9.14\%$ vs $8.69\%$) according to Noun + TTC Top-5 mAP, and $+1.51\%$ ($5.12\%$ vs $3.61\%$) according to Overall Top-5 mAP. If we consider that v1 and v2 share the same test set, StillFast jumps from a $9.95\%$ of v1 to $13.29\%$ of v2 Noun + Verb Top-5 mAP and from $3.49\%$ of v1 to $5.12\%$ of v2 Overall mAP, by merely adding more training data, which suggests that the proposed approach can scale in the presence of larger datasets.

In the v1 test set, we are able to also compare StillFast with other methods for which results are publicly available, including InternVideo [2] and FRCNN+Feat. StillFast outperforms FRCNN+Feat. on all evaluation measures except Noun Top-5 mAP, where it achieves slightly worse performance ($19.51\%$ vs $20.45\%$). Note that FRCNN+Feat. uses the same Faster RCNN object detector component as FRCNN+SF and is trained in two stages, hence similar considerations as the ones made while comparing the proposed approach with FRCNN+SF apply here as well. StillFast still outperforms InternVideo with respect to Noun+Verb Top-5 mAP ($9.95\%$ vs $9.18\%$) and Overall Top-5 mAP ($3.49\%$ vs $3.40\%$) despite being based on less advanced components (StillFast relies on a Faster R-CNN object detector, while InternVideo relies on DINO [39], StillFast relies on an X3D video backbone, while InternVideo relies on Video-MAE [36] pre-trained transform-based video backbones) and being trained end-to-end (InternVideo is trained in two stages as FRCNN+SF). We leave the integration of higher performing components in StillFast to future works.

Figure 4 reports two success (left) and two failure examples (right). The model struggles with uncertain future actions (“put cement” vs “mold cement”) and unusual actions (“clean book” vs “put book”).

In this section, we further investigate other aspects of the proposed StillFast architecture including the verb/noun prediction trade-off introduced by the prediction head described in Section 3.2, and the contribution of different architectural choices to the overall performance. The tables in this section report validation results of experiments performed on EGO4D v1 data. All results are in Top-5 mAP%.

Prediction Head Table 2 compares the results obtained by the proposed model when different prediction heads are used. The results of FRCNN+SF are also reported for reference. “Nouns Only” predicts only next-active objects and does not predict any verb or time to contact, “Standard Head” refers to the standard prediction head proposed in [19] which uses a SlowFast model to attach a verb and a time to contact prediction to each detected bounding box. “Proposed Head” refers to the head described in 3.2, including global features and accounting for verb prediction uncertainty. As can be noted, predicting only nouns outperforms the FRCNN+SF baseline by $+2.05\%$ ($19.69\%$ vs $17.55\%$) with respect to Noun Top-5 mAP. Adding verb and time to contact prediction with a standard head decreases Noun Top-5 mAP performance to $18.42\%$, which still outperforms FRCNN+SF ($17.55\%$), while obtaining better or comparable performance on the other metrics. This suggests that the proposed approach is penalized by the training instabilities caused by multi-tasking (i.e., predicting simultaneously nouns, verbs and time to contact). The proposed head achieves the best results in terms of Noun+Verb Top-5 mAP and Overall performance thanks to the use of global features and verb-noun score product, which effectively accounts for the uncertainty in the prediction of verbs. Table 3 assesses the impact of the main components of the proposed head, comparing the performance of the overall module with versions in which we remove global features, residual connections, and the verb-noun score product module. We also compare the head with a version which replaces the proposed global-local fusion mechanism by a simple sum (last row). As can be observed, the proposed architecture obtains overall the best results. Sum fusion achieves slightly better overall Top-5 mAP ($+0.01\%$), but lower results according to the other metrics, for which we prefer the proposed concatenation + residual connection design.

Backbone Table 4 shows the impact of the different components in the backbone when the proposed head is used. We observe that without a 3D branch (second row) the performance of the method drops according to all mAP measures by significant margins (e.g., $1.82\%$ vs $2.48\%$ Overall Top-5 mAP). This suggests that the proposed backbone design succesfully combines 3D and 2D features to anticipate future interactions. The third row shows that removing 2D convolutional blocks after the sum operation in the combined feature pyramid layer negatively affect performance (e.g., $2.25\%$ vs $2.48\%$ Top-5 Overall mAP). We speculate that the use of these layers allows to deal with artifacts potentially introduced in the upsampling and sum operations. The last row reports the performance of an alternative design of the network which attaches a 2D feature pyramid to the 2D network and a 3D feature pyramid to the 3D network and then fuses the resulting feature pyramids. As can be noted, this design is less effective than the proposed one, with a Top-5 Overall mAP score equal to $2.34\%$ (vs $2.48\%$ of the proposed design).