Bidirectional Progressive Transformer for Interaction Intention Anticipation

Overview. As shown in Fig. 3, given the original input video $\mathcal{I}$, we aim to predict the future hand trajectory $\mathcal{H}$ and interaction hotspots $\mathcal{O}_{\mathrm{I}}$. Initially, to alleviate information conflicts arising from changes in the field of view, the original input video undergoes a Spatial-Temporal Reconstruction Module that integrates spatial information from the last observation frame and temporal information from the entire video, producing spatial features for each future time step. Subsequently, these features are separated, with the last one serving as the contact feature while the preceding ones are for trajectory features. These features are then fed into the interaction hotspots and trajectory prediction branches, respectively. The former employs an overlap patch embedding layer to enhance attention at the center of the field of view, while the latter introduces temporal order to trajectory features through Cross-Transformers [11]. Simultaneously, we employ a Bi-Progressive Enhancement Module consisting of cross-attention blocks between dual branches, introducing a Bi-Progressive mechanism to the joint prediction. This alternately and iteratively corrects the features of trajectories and interaction hotspots in chronological order, thereby mitigating the accumulation of prediction errors over time. Finally, considering the inherent uncertainty in natural human behaviors [70, 56, 49], we introduce a Stochastic Unit for trajectory forecasting while employing a C-VAE [60] for interaction hotspots, adding appropriate uncertainty during the completion of position decoding to achieve comprehensive predictions.

Interaction Hotspots Prediction Branch. Due to the spatial distribution of contact points closer to the central field of view, the contact feature $\mathbf{F}_{\mathrm{C}}$ undergoes processing through an overlap patch embedding layer:

Where $\mathbf{F}^{\mathrm{o}}_{\mathrm{C}}$ represents the processed contact feature.

Hand Trajectory Prediction Branch. We employ original trajectory features as primary features and utilize features from the previously predicted frame as conditional features, subsequently merging them through the cross-attention block $\mathit{f}$, ensuring the coherence of the trajectory over time. Assuming the trajectory features are $\left\{{\mathbf{F}_{\mathrm{N+1}},\mathbf{F}_{\mathrm{N}+2},\cdots,\mathbf{F}_{\mathrm{N+T-1}}}\right\}$ and the feature map of the last observation frame is $\mathbf{F}_{\mathrm{N}}$, they are updated in chronological order:

Where ${\mathbf{{F}}^{\mathrm{o}}_{{t}}}$ stands for the updated trajectory feature, $t$ is the prediction time step.

H-O Bi-Progressive Enhancement. Expanding on the groundwork laid by two distinct prediction branches, and to mitigate the accumulation of errors as predictions unfold over time, we introduce a Bi-Progressive Enhancement Module (Fig. 4), which alternately corrects the predictions of trajectories $\mathcal{H}$ and interaction hotspots $\mathcal{C}_{\mathrm{I}}$ at each time step $t$. Specifically, owing to observation frames and the continuity of hand trajectories, the trajectory feature $\mathbf{F}^{\mathrm{o}}_{\mathrm{N+1}}$ at the initial predicted time step $\mathrm{N+1}$ demonstrates a high level of accuracy. Exploiting this, we enhance the precision of the contact feature $\mathbf{F}^{\mathrm{o}}_{\mathrm{C}}$ through the cross-attention block f, thereby minimizing the error with the ground truth. Moving to the subsequent time step $\mathrm{N+2}$, we employ the first refined contact feature $\mathbf{F}^{\mathrm{r_{1}}}_{\mathrm{C}}$ from the previous time step $\mathrm{N+1}$ as a condition to correct the trajectory feature $\mathbf{F}^{\mathrm{o}}_{\mathrm{N+2}}$. Following this, the corrected trajectory feature is utilized to further refine the contact feature. This iterative correction process persists until all trajectory features have undergone a complete round of updates. It can be represented as a recursive procedure:

Where $\mathbf{F}^{\mathrm{r}}_{t}$ represents the refined trajectory feature map at time step $t\in[\mathrm{N+1,N+T-1}]$, $\mathbf{F}^{\mathrm{r}_{i}}_{\mathrm{C}}$ denotes the contact feature after the $i$-th refinement. Via this module, we attain refined features for both trajectories and interaction hotspots, significantly reducing error accumulation in predictions.

Interaction Hotspots Decoder. Given the presence of inherent uncertainty associated with interaction hotspots [39, 41, 42], we employ a C-VAE [60] as the decoder. Due to the excessive randomness associated with directly decoding the coordinates of contact points, we instruct the C-VAE to output a feature map and select the location of the maximum value from this map as the output, thereby confining the uncertainty within a reasonable range. The C-VAE follows an encoder-decoder structure. For the encoder block, we concatenate the input $\Gamma$ with the condition $\mathbf{C}$ through an encoding function $\mathbf{P}_{\mathrm{e}}$, obtaining features in latent space $\mathbf{Z}$ which parameterized by mean $\mu$ and co-variance $\sigma$. In this context, the input $\Gamma$ is the ground truth heatmap, while the condition $\mathbf{C}$ is the refined contact feature $\mathbf{F}^{\mathrm{r_{3}}}_{\mathrm{C}}$. In the decoder block, the first step involves sampling $\mathbf{Z}$ from the latent space and concatenating it with condition $\mathbf{C}$. Following that, the output heatmap $\hat{\Gamma}$ is reconstructed by a decoding function $\mathbf{P}_{d}$. In summary, the entire process can be formulated as follows:

Where $\mathcal{N}$ represents a normal distribution, $\hat{\Gamma}$ is the reconstructed heatmap. Both encoding and decoding blocks consist of two MLP layers. We introduce KL-Divergence and the reconstruction error as the overall loss:

In this paper, we set $\lambda=0.01$.

Hand Trajectory Decoder. Similar to interaction hotspots, hand trajectories also exhibit uncertainty [45, 29, 33]. Given that trajectory feature maps with distinct distribution characteristics have been obtained through the Bi-Progressive Enhancement Module, in conjunction with sampling efficiency, we don’t utilize C-VAE, rather, we introduce a Trajectory Stochastic Unit to incorporate uncertainty into trajectory forecasting by calculating uncertain regions for each prediction time step instead. Since optimizing directly using point coordinates for loss calculation can pose challenges, we initially introduce a Gaussian distribution for each ground truth hand position to generate the ground truth heatmap. Subsequently, we calculate the loss between the corrected feature map and the generated ground truth heatmap, thereby enabling the network to learn the distribution of potential hand positions. The associated loss function $\mathcal{L}_{h}$ is based on MSE Loss:

Where $\mathbf{G}_{t}$ represents the ground truth heatmap at time step $t$. We designate the coordinates of the maximum value within each feature map as the predicted trajectory points, which are then fed into the stochastic unit to generate uncertain regions. The size of the region is determined by the time step as well as the length of trajectory segments in preceding and subsequent stages. The shape is constrained by the original image dimensions and the orientation of the trajectory segments.

Formally, the uncertain region (Fig. 5) can be shown as:

Where $\mathcal{R}_{t}$ is the uncertain region in future time step $t$, $\mathrm{W}$ and $\mathrm{L}$ are the width and length of the input video, respectively. $\mathcal{R}_{t}$ satisfies the following constraints:

Where $\mathcal{R}_{t1}$ and $\mathcal{R}_{t2}$ are elliptical regions constrained by $h_{t-1}$ and $h_{t+1}$, which can be represented as:

Training. During training, we freeze parameters in TSN [67]. Training loss combines both trajectory loss and interaction hotspots loss:

In this paper, we set $\varphi=1$.

Inference. We use a Trajectory Stochastic Unit and a C-VAE to introduce uncertainty during inference. Following previous work [45, 39], we report the minimum among 20 samples for trajectory evaluation.

We conducted experiments on three first-person video datasets Epic-Kitchen s-100 [14], EGO4D [28], and EGTEA Gaze+ [36]. Epic-Kitchens-100 [14] and EGTEA Gaze+ [36] predominantly include kitchen scenes, while EGO4D [28] encompasses a broader range of activities such as handicrafts, painting, and outdoor tasks. For each dataset, the observation time is set to 2 seconds. We configure the prediction time horizon $\mathrm{T}$ to 5, corresponding to 4 trajectory points and one contact point for each hand. Specifically, we predict 1-second trajectories on Epic-Kitchens-100 [14] and EGO4D [28], while on EGTEA Gaze+ [36], 0.5-second trajectories are forecast. Additional dataset information is available in the supplementary materials.

In EGO4D [28], We use the method [39] to automatically generate hand trajectories, while contact points are manually annotated based on the last observation frame and the contact frame. We employ GMFLOW [69] to generate optical flow from EGO4D [28] and EGTEA Gaze+ [36]. For each dataset, we resize the original videos to 456 $\times$ 256 size, extracting both trajectory and contact features of size 114 $\times$ 64. In the Spatial-Temporal Reconstruction Module, we set the dimension $D$ in Cross-Transformer to 1024, with depth $=6$ and attention head number $=8$. For the trajectory forecasting branch and H-O Bi-Progressive Enhancement Module, we configure the dimension $D$ to 256, with depth $=2$ and head number $=8$. Additionally, for the introduced C-VAE, we set the dimension of both input and condition to 114 $\times$ 64, while the hidden and latent sizes are set to $1024$ and $2048$, respectively. Our method is implemented by PyTorch and trained with the AdamW optimizer. We train the model for 100 epochs on 4 NVIDIA 3090 GPUs with an initial learning rate of $3e$-$5$ and a batch size of 40.

For comparison, We first select methods for individual tasks including Kalman Filter (KF) [8], Seq2Seq [63] and Hotspots [51]. Additionally, the methods for image/video understanding are chosen as the compared methods, which lack a joint mechanism. The former includes VIT [16], HRNet [62], UNet [58], Segformer [68], while the latter contains I3D [10], Slowfast [19], Timesformer [7], Uniformer-V2 [35]. Furthermore, we select methods with high task relevance and joint design, including FHOI [37] and OCT [39]. We conduct experiments using these approaches on Epic-kithens-100 [14], EGO4D [28], and EGTEA Gaze+ [36] datasets.

Hand Trajectory Evaluation. The predicted hand trajectories are referenced to the last observation frames. ADE [39] and FDE [39] are chosen as hand trajectory evaluation metrics, which respectively represent the overall accuracy and endpoint accuracy of trajectories. Detailed descriptions are available in the supplementary materials.

Interaction Hotspots Evaluation. For all datasets, we downsample each predicted heatmap to the size of 32$\times$32. We choose SIM [64], AUC-J [31], and NSS [54] as interaction hotspots evaluation metrics. For a detailed explanation of the metrics, please refer to the supplementary materials.

As shown in Tab. 1, our method achieves state-of-the-art performance in both trajectory prediction and interaction hotspots anticipation, with comparable leading margins across all three datasets. Taking the EGO4D dataset [28] with the richest variety of scenes as an example. In trajectory prediction, to ensure fair comparison, we additionally present results for OCT and BOT with only one sampling iteration. Under this single-sampling condition, BOT outperforms Segformer-b2, the best method in image/video understanding, with respective leads of 31.3% and 18.8% in ADE and FDE. Compared to OCT, BOT maintains advantages of 26.7% and 18.8% in ADE and FDE, respectively. These advantages persist even with 20 samplings, while the lead in FDE expands to 33.3%, underscoring the rationale behind BOT’s stochastic sampling mechanism. Regarding interaction hotspots, we achieve the best results across all metrics. Specifically, BOT surpasses the second-best approach (OCT) by 45.5% in SIM, 5.5% in AUC-J, and 93.2% in NSS, respectively. In summary, our method leverages the intrinsic correlations between hand trajectories and interaction hotspots more effectively, enhancing overall prediction accuracy of interaction intention, and exhibiting robustness across the three benchmark datasets.

To investigate the efficiency of the model, we provide statistics on inference time and parameter count (Tab. 2). For each method, we test the inference time of one sample on a single NVIDIA 3090 GPU. In comparison to OCT, although our model has a larger parameter count, it exhibits a similar inference time when sampling once. With multiple samplings, our model displays a notable advantage in inference time. This can be attributed to the improved efficiency of the Stochastic Unit in BOT, obviating the necessity for repetitive inference of complete trajectories compared to OCT. Given BOT’s superior inference capability, our approach demonstrates enhanced efficiency.

To further explore the subjective outcomes of models, we report the visualization results of three datasets, as shown in Fig. 6. Compared to other methods, our model provides more accurate prediction results in both hand trajectories and interaction hotspots. When there is significant movement of hands (column 1,3,8), the BOT’s predictions exhibit the highest accuracy. It indicates that our Spatial-Temporal Reconstruction Module can efficiently utilize the human motion information to exploit the hand interaction intention and thus accurately predict the hand trajectory. In cases where there are no identifiable interactive objects in the field of view (column 6), our method anticipates more accurate interaction intentions. It illustrates that our Bi-Progressive mechanism can eliminate the uncertainty in the interaction process by mining the intrinsic connection between the hand trajectory and the contact points, which results in a more accurate inference in the interaction region and the hand trajectory. Moreover, our predictions closely align with the actual scenario when both hands simultaneously exhibit trajectories and interaction hotspots (columns 4,5), which suggests the strategy of differentiating between left and right hands during the prediction of interaction intentions, thereby mitigating cross-hand interference. More visualization results are available in the supplementary materials.

Different joint approaches. To explore the impact of the Bi-Progressive mechanism on model performance, we experiment with five different joint configurations: Individual, Hand, Object, Bidirectional and Bi-Progressive. The results are shown in Tab. 3 and Fig. 8. It suggests that the network is challenging to accurately predict trajectories and interaction hotspots during individual predictions, and the separate incorporation of hands and objects can effectively improve the prediction accuracy of interaction hotspots. With the combination of hand and object information, the prediction performance of the trajectory and interaction region is greatly improved. However, there is still a gap compared to the Bi-Progressive mechanism. Further, Fig. 8 shows that the introduction of the progressive mechanism plays a pivotal role in mitigating the

accumulation of prediction errors over time.

Conditions in dual branches. The left portion in Fig. 9 delineates various conditions for the trajectory prediction branch. None signifies the absence of the fusion between two consecutive trajectory points, spatial denotes the exchange between the main and the condition input, point and heatmap represent different

modes of the condition, where the former is a coordinate and the latter is a heatmap, incorporating the potential distribution of the hand position. Experimental results indicate that the cross-attention block contributes to the temporal continuity of the trajectory, and global spatial features exert a more substantial influence on the prediction. Furthermore, employing an appropriate region to

depict the hand position contributes more significantly to the propagation of trajectory features. The right side of Fig. 9 illustrates conditions within the interaction hotspots prediction branch. It is evident that using heatmap as the condition for the C-VAE yields superior results compared to point. This implies that the contact point also exhibits a degree of uncertainty within a specific range.

Uncertainty in anticipation. Uncertainty is a crucial consideration in interaction intention anticipation. Following the logic of natural human behavior [70, 56, 49], both hand trajectories [45, 29, 33] and interaction hotspots [39, 41, 42] predictions inherently possess a degree of uncertainty.

To investigate the impact of uncertain region scale and the magnitude of uncertainty variations in hand trajectories, we modify $\alpha$ and $\beta$ in Eq. 17, with the resultant outcomes depicted in Fig. 10. $\alpha$ denotes the overall scale of the uncertain region, when $\alpha$ equals 0, the region’s size is zero, and the network output yields a unique result. As $\alpha$ increases, the uncertainty of trajectories grows, leading to improved accuracy due to random samplings. However, given the limitations in sampling times, as $\alpha$ continues to increase, the accuracy naturally decreases. $\beta$ determines the magnitude of trajectory uncertainty increasing from front to back, when $\beta$ equals 1, each trajectory point has the same level of uncertainty. Considering the escalating complexity and uncertainty in trajectory prediction over time, an elevation in $\beta$ initially corresponds to augmented accuracy, followed by a subsequent decline. This gradual increase in uncertainty also renders the FDE more sensitive to $\alpha$ and $\beta$, as shown in Fig. 11(a)-11(c). Regarding interaction hotspots, BOT exhibits relatively small uncertainty (Fig. 11(d)), as for a specific intention, the interaction area becomes increasingly influenced by object constraints. This is achieved by tightly controlling the uncertainty within a reasonably small range through the process of enabling C-VAE to generate feature maps and subsequently select points, thereby ensuring the network’s high confidence in predicting interaction regions.