Hypergraph-based Motion Generation with Multi-modal Interaction Relational Reasoning

The vehicle trajectory prediction in the dynamic realm of multi-vehicle interaction context of multi-lane highways involves determining the future movements of a target vehicle based on historical data and multi-modal predictions of its own state and the states of surrounding vehicles. This domain addresses two primary challenges: (i) multi-agent multi-modal trajectory prediction and (ii) prediction-guided motion generation after reasoning.

The objective of trajectory prediction is to estimate the future trajectories of the target vehicle and its surrounding vehicles, given their historical states. The historical states, spanning a time horizon $[1,\dots,T]$, are represented as $\mathbf{X}_{1:T}=\{\mathbf{X}_{1},\mathbf{X}_{2},\dots,\mathbf{X}_{T}\}\in\mathbb{R}^{T\times N\times C_{1}}$, where $N$ denotes the number of vehicles and $C_{1}$ denotes the number of features, including longitudinal and lateral positions and velocities. Each historical state $\mathbf{X}_{t}=\{\mathbf{x}_{t}^{i}|\forall i\in[1,N],\forall t\in[1,T]\}\in\mathbb{R}^{C_{1}}$ at time step $t$ captures these details for each vehicle $i$. Notably, the superscript refers to vehicle indices with $i=1$ representing the target vehicle, and the subscript to time steps, with $C_{1}=4$ for the input data.

The prediction model, $\mathbf{H}^{Pred}(\cdot)$, provides preliminary predictions of multi-modal trajectory candidates $\hat{\mathbf{X}}_{T+1:T+F}^{M}\in\mathbb{R}^{F\times N\times M\times C_{2}}$ for all the $N$ vehicles over the future time horizon $[T+1,\dots,T+F]$ with $M$ modes of driving behaviors. This model takes historical data $\mathbf{X}_{1:T}$ as the input and outputs future longitudinal and lateral positions, where $C_{2}=2$. The forecasted states $\hat{\mathbf{X}}_{f}^{M}=\{\mathbf{x}_{f}^{m,i}|\forall m\in[1,M],\forall i\in[1,N],\forall f\in[1,F]\}$ aim to estimate each vehicle’s future trajectory for each behavior mode $m$ at time step $T+f$. This is mathematically formulated as:

Based on that, the motion generation model $\mathbf{H}^{Gen}(\cdot)$ is further developed to generate plausible trajectories considering the implicit group-wise interactions, using both historical states $\mathbf{X}_{1:T}$ and preliminary multi-modal future trajectory candidates $\hat{\mathbf{X}}_{T+1:T+F}^{M}$ as the input. The generation model provides $K$ plausible trajectory $\hat{\mathbf{Y}}_{T+1:T+F}^{K}=\{\hat{\mathbf{Y}}_{T+1}^{K},\dots,\hat{\mathbf{Y}}_{T+F}^{K}\}\in\mathbb{R}^{F\times N\times K\times C_{2}}$ for all the $N$ vehicles for the next $F$ time steps. Each generated state $\hat{Y}_{T+f}^{K}=\{\hat{\mathbf{y}}_{T+f}^{k,i}|\forall k\in[1,K],\forall i\in[1,N],\forall f\in[1,F]\}\in\mathbb{R}^{C_{2}}$ represents the $k$-th generated longitudinal and lateral potisions of the $i$-th vehicle at time step $T+f$. The formulation for this generation problem is:

The proposed framework adopts an integrated architecture, as shown in Figure 3, which involves two major components:

• 1.

  GIRAFFE: Graph-based Interaction-awaRe Anticipative Feasible Future Estimator, which leverages graph representations to capture pair-wise interactions during both the historical and future time horizons, providing preliminary multi-modal trajectories prediction candidates for vehicles.

• 2.

  RHINO: Relational Hypergraph Interaction-informed Neural mOtion generator, which utilizes multi-scale hypergraph representations to model group-wise interactions and reason the interaction relations among the multi-modal behaviors. Built upon the preliminary multi-modal trajectories by GIRAFFE, RHINO will further generate plausible future trajectories for all vehicles in a probabilistic manner.

The subsequent sections will provide an in-depth explanation of the two principal frameworks.

In our context, a graph representation $\mathcal{G}$ is adopted by modeling $N$ vehicles as nodes $\mathcal{V}\in\mathbb{R}^{N}$ and the pair-wise interaction as the edges $\mathcal{E}\in\mathbb{R}^{|N\times N|}$. Further, The feature matrix $X\in\mathbb{R}^{N\times C}$ containing vehicle states (i.e., longitudinal and lateral position and speed) and the adjacency matrix $A\in\mathbb{R}^{N\times N}$ describing the interactions among nodes are further utilized to describe the graph. By that, we can define an Agent Graph as:

To better represent the interaction and relations of the predicted multi-agent multi-modal trajectory candidates with graphs, we expand each agent node to multiple nodes of the number of behavior modes based on our previous work [25], which further renders an Agent-Behavior Graph.

The transition from an Agent Graph $\mathcal{G}^{a}$ to an Agent-Behavior Graph $\mathcal{G}^{b}$ is by an expansion function $\mathbf{F}^{expand}(\cdot)$ as:

As shown in Figure 4, in this process, each agent node $v_{i}^{a}\in\mathcal{V}^{a}$ in the Agent Graph is expanded into $M$ behavior-specific nodes $v_{i}^{b}\in\mathcal{V}^{b}$, corresponding to the $M$ potential behavioral modes of the vehicle. These newly generated behavior nodes, which may exhibit significant interdependencies, are interconnected through edges $e_{ij}^{b}$, forming a more complex interaction structure. Consequently, the adjacency matrix $A^{b}$ is extended to accommodate the expanded node set, resulting in increased dimensionality. Here, $\Lambda$ is a behavior correlation matrix, and each element $\Lambda_{mn}$ in the matrix represents the correlation between behavior mode $m$ of one agent and behavior mode $n$ of another agent. The feature tensor $X^{b}$ of the behavior nodes encodes the possible motion states under each behavioral mode, capturing the multi-modal nature of vehicle behavior.

Based on that, a deep neural network is designed to capture interactions between the target vehicle and surrounding vehicles by $\mathcal{G}^{a}$, and to represent the output, which consists of predicted multi-agent multi-modal trajectories, by $\mathcal{G}^{b}$, as illustrated in Figure 5. Three key modules of GIRAFFE $\mathbf{H}^{Pred}(\cdot)$ are introduced below:

Unlike traditional graph representations, which are confined to pair-wise relationships, hypergraphs offer a more sophisticated and comprehensive framework for representing group-wise interactions. By connecting multiple vehicles that exhibit strong correlations through hyperedges, hypergraphs enable a more robust analysis and optimization of the complex network of interactions. The concept of an Agent Hypergraph to represent agents and their group-wise interactions is introduced as follows:

To convert an Agent Graph $\mathcal{G}^{a}$ into an Agent Hypergraph $\mathcal{H}^{a}$, we introduce a transformation function $\mathbf{F}^{transform}(\cdot)$. This function enables the shift from a pairwise interaction framework to a higher-order interaction model represented by the hypergraph. Formally, the transformation is expressed as:

In this transformation, the node set $\mathcal{V}^{a}$ and the feature tensor $X^{a}$ remain consistent between the graph and the hypergraph representations. However, the primary modification occurs in the edge formulation. The transformation replaces the pairwise edges of the original Agent Graph with hyperedges that can connect multiple nodes simultaneously. This redefinition of edges as hyperedges within the hypergraph $\mathcal{H}^{a}$ allows for the modeling of group-wise interactions, where a single hyperedge $u\in\mathcal{U}^{a}$ can link more than two nodes, capturing higher-order relationships among agents. The incidence matrix $H^{a}$ is updated to reflect this change, where each entry $H^{a}_{ij}$ indicates whether agent $v_{i}$ participates in hyperedge $u_{j}$. As a result, $\mathcal{H}^{a}$ can represent multi-agent interactions that involve multiple agents simultaneously, providing a richer and more flexible structure for modeling the dynamics of the system.

To enhance the understanding of complex group-wise interactions in multi-agent systems, it is essential to extend the traditional Agent Hypergraph model to account for the diverse behavioral modes of each agent. This is achieved by decomposing each agent node in the Agent Hypergraph $\mathcal{H}^{a}$ into multiple behavior-specific nodes, which correspond to the different modes of behavior each agent can exhibit. The result of this decomposition is the Agent-Behavior Hypergraph, denoted as $\mathcal{H}^{b}$.

To formally describe the process of transitioning from an Agent Hypergraph $\mathcal{H}^{a}$ to an Agent-Behavior Hypergraph $\mathcal{H}^{b}$, the expansion function $\mathbf{F}^{expand}(\cdot)$ is applied. This function decomposes each agent node into multiple behavior-specific nodes and updates the hyperedge structure accordingly. The behavior-specific nodes correspond to the different behavior modes, while the hyperedges represent the higher-order interactions among the behavior modes of different agents.

The node set $\mathcal{V}^{b}$ expands each agent $v_{i}^{b}$ into multiple behavior-specific nodes $v_{i,m}^{b}$, where each $m$ represents a different behavioral mode of the agent. The hyperedges $\mathcal{U}^{b}$ are formed between the behavior-specific nodes based on the group-wise interactions present in the original hypergraph, with the correlation between behavior modes captured by $\Lambda_{mn}$. The feature tensor $X^{b}$ captures the state of each behavior node, inheriting the feature data from the original hypergraph.The incidence matrix $H^{b}$ records whether a behavior-specific node $v_{i,m}^{b}$ is part of a hyperedge $u_{j,mn}^{b}$. In this extended framework, hyperedges can represent these aforementioned complex group-wise interactions by connecting behaviors of multiple vehicles that are influenced simultaneously by a shared context, as illustrated in Figure 6. Therefore, an Agent-Behavior Hypergraph is defined to model the multi-agent, multi-modal system for reasoning about group-wise interaction relations.

In addition to the expansion process, the transformation function $\mathbf{F}^{transform}(\cdot)$ converts an Agent-Behavior Graph $\mathcal{G}^{b}$ into an Agent-Behavior Hypergraph $\mathcal{H}^{b}$. This transformation replaces the pairwise edges of the graph with hyperedges that capture higher-order interactions between behavior modes across multiple agents. The transformation is guided by the adjacency matrix $A^{b}$ of the original graph and the behavior-mode correlation matrix $\Lambda_{mn}$. The transformation function $\mathbf{F}^{transform}(\cdot)$ is expressed as:

This structure allows for the connection of behavior nodes across different agents, enabling the representation of interactions among diverse behaviors of multiple agents in a shared context. Hence, the Agent-Behavior Hypergraph not only captures the individual behavior of each agent but also models how these behaviors interact and influence one another within a multi-agent system.

This research leverages two open-source datasets for the purpose of model training and validation: the Next Generation Simulation (NGSIM) dataset [47],[48] and the HighD dataset [49]. The NGSIM dataset provides a comprehensive collection of vehicle trajectory data, capturing activity from the eastbound I-80 in the San Francisco Bay area and the southbound US 101 in Los Angeles. This dataset encapsulates real-world highway scenarios through overhead camera recordings at a sampling rate of 10Hz. The HighD dataset originates from aerial drone recordings executed at a 25 Hz frequency between 2017 and 2018 in the vicinity of Cologne, Germany. Spanning approximately 420 meters of bidirectional roadways, it records the movements of approximately 110,000 vehicles, encompassing both cars and trucks, traversing an aggregate distance of 45,000 km. After data pre-processing, the NGSIM dataset encompasses 662 thousand rows of data, capturing 1,380 individual trajectories, while the HighD dataset comprises 1.09 million data entries, including 3,913 individual trajectories. For the purpose of training and evaluation of the model, the partition of the data allocates $70\%$ to the training set and $30\%$ to the test set. For the temporal parameters of the model, we adopt $T=30$ frames to represent the historical horizon and $F=50$ frames to signify the prediction horizon.

The experimental results for trajectory generation of the $K$ trajectories using the HighD dataset are presented in Figure 12. As can be found that, RHINO demonstrates strong generative capabilities, effectively producing plausible motion in a dynamic interactive traffic environment. To provide a more quantitative analysis, trajectory generation inaccuracies are illustrated in Figure 13. The generated longitudinal and lateral trajectories, along with the error box plots and heatmaps, are displayed. The box plot reveals that errors in both axes increase with the prediction time step by the natural of error propagation. However, the errors remain within an acceptable range, indicating decent model performance, which demonstrates high precision in trajectory generation. Notably, the model maintains a lower error margin for shorter prediction horizons, which is critical for short-term planning and reactive maneuvers in dynamic traffic environment.

The experiment focused on predicting vehicle trajectories within mixed traffic environments on highways by employing hypergraph inference to model group-based interactions. Through the application of hypergraph models at varying scales $s=2,3,5$, the experiment captured the evolution of multi-vehicle interactions across both historical and future horizons. The figures depict these dynamics through three distinct columns: the first column presents vehicle trajectories and the corresponding hyperedges, visualized as polygons that encapsulate groups of interacting vehicles. The second column illustrates the affinity matrix, where both rows and columns represent vehicles, and the strength of their relationships is indicated by the matrix values. The third column shows the incidence matrix, detailing the relationship between nodes and hyperedges, with each column representing a hyperedge and each vehicle’s involvement in that hyperedge marked by a 1 in the corresponding row.

The hypergraph-based approach is particularly effective in modeling complex, higher-order interactions that are beyond the scope of traditional pairwise models. By forming hyperedges that encompass multiple vehicles, the model captures the collective influence that a group’s behavior exerts on an individual vehicle. For instance, in the scenario shown in Figure 14, at scale $s=5$, when the target vehicle TAR initiates a lane change, the hypergraph reflects the interaction not only with a single neighboring vehicle but also with multiple surrounding vehicles, such as following vehicle F, preceding vehicle P, and preceding vehicle RP in the right lane. More examples are illustrated in A. This capability to model group-wise interactions across different scales is essential for accurately predicting vehicle trajectories in congested highway environments, where the actions of one vehicle can trigger ripple effects that influence an entire group. The hypergraph’s dynamic formation of hyperedges ensures that predicted trajectories remain adaptable and responsive to broader traffic conditions.

To evaluate the privileges of our proposed method, the state of art methods (i.e., Social-LSTM (S-LSTM) [20], Convolutional Social-LSTM (CS-LSTM) [31], Planning-informed prediction (PiP) [50], Graph-based Interaction-aware Trajectory Prediction (GRIP) [26], Spatial-temporal dynamic attention network (STDAN) [22] are compared.

The compared results presented in Table 2 and Figure 15. As can be found that, the proposed framework demonstrates good performance with respect to the RMSE across a prediction horizon of 50 frames when compared with existing baseline models. It exhibits a reduced loss in comparison to C-LSTM, CS-LSTM, PiP, and GRIP. These outcomes suggest that the proposed model effectively captures salient features pertinent to long-term predictions. In summary, the proposed framework outperforms baseline models on the HighD dataset and delivers commendable performance on the NGSIM dataset.

Since the RHINO adopts the GIRAFFE, we further compare the trajectory generation capability of RHINO with our previous work [25] and its enhanced version GIRAFFE. Both RHINO model and the enhanced GIRAFFE model consistently outperform the baseline models, demonstrating superior performance in various metrics. This suggests that our proposed approaches effectively address the limitations present in prevailing models by robustly capturing complex interactions.

Ablation study is conducted to provide more insights into the performance of our RHINO model, especially the impact of different components on the prediction performance by disabling the corresponding component from the entire RHINO. In particular, we consider the following four variants:

• 1.

  RHINO w/o HG (hypergraph) variant does not use the multi-scale hypergraphs representation but only adopts the pair-wise connected graph representations in the Hypergraph Relational Encoder.

• 2.

  RHINO w/o MM (multi-modal) variant does not adopt the multi-agent multi-modal trajectory prediction results and only use the single predicted future states for each agent as the input of the RHINO.

• 3.

  RHINO w/o PDL (posterior distribution learner) variant skips the Posterior Distribution Learner and directly input the graph embedding into the Motion Generator.

An investigation into the effects of model design variations, as presented in Table 3 and Figure 16. The removal of various components from RHINO invariably leads to performance degradation to varying degrees. Compared to the full RHINO, omitting the multi-scale hypergraphs results in an evident increase in prediction error across the prediction horizon, indicating that modeling and reasoning group-wise interactions using hypergraphs, rather than solely pair-wise interactions, enhances prediction accuracy which underscores the necessity of hypergraphs. Further, excluding the multi-agent multi-modal trajectory prediction input leads to a more substantial degradation in performance, highlighting the importance of incorporating multi-modal motion states and discussing the corresponding group-wise interactions among multiple driving behaviors of multiple agents. Lastly, the absence of the Posterior Distribution Learner module emphasizes its critical role in handling the stochasticity of each agent’s behavior. All these experiments justify the effectiveness of the full model.