Spatio-temporal Neural Structural Causal Models for Bike Flow Prediction

Traffic prediction is a fundamental problem in Intelligent Transportation System including the recent Bike-Sharing System, which has recently attracted significant attention. Since the real-world traffic network is non-Euclidean data, most methods construct GCN models to study spatio-temporal data prediction problems. utilized GCN with predefined graphs to model spatial correlations. To enrich spatial information, (Liu et al. 2020; Fang et al. 2021; Chai, Wang, and Yang 2018) establish multiple static topologies to effectively capture complex patterns. DCRNN(Li et al. 2018) replaces the linear transformation layer in GRU with diffusion graph convolution, which boosts the spatio-temporal representation ability of GRU. HGCN(Guo et al. 2021a) constructs the interaction between the micro and macro layers of GCN, which integrates the different scales of features of road segments and regions.

Recently, some researchers have paid attention to the strong dynamic correlations of traffic data in spatial and temporal dimensions. Therefore, it is crucial to model dynamic and nonlinear spatio-temporal correlations for accurate traffic prediction. GWNet(Wu et al. 2019) and AGCRN(Bai et al. 2020) randomly initialize the node embedding vectors, and then learn an adaptive matrix through gradient descent. Once the training of the model finishes, the adaptive matrix is fixed. So it cannot extract the dynamics from real data effectively. ASTGNN(Guo et al. 2021b) and GMAN(Zheng et al. 2020) dynamically extract the correlations of nodes by means of spatial attention. However, affected by the structure of predefined graphs, spatial attention can only change the weight of predefined graphs rather than the structure. CCRNN(Ye et al. 2021) proposes a coupled graph convolution with self-learned adjacency matrices varying from layer to layer, but the graph structure changes with the convolutional layer instead of time steps. DGCRN(Li et al. 2021) handles the dynamic relations by learning the matrix at each recurrent step, while it significantly depends on the node embedding layer initialized by random parameters. DMSTGCN(Han et al. 2021) learns the time-specific spatial dependencies of road segments to construct dynamic graphs, but ignores the fluctuations caused by the external environment. In addition, these methods do not take external conditions into consideration during the dynamic graph generation process, resulting in very limited effects of dynamic graphs.

The purpose of causal learning is to empower models the ability to pursue the causal effect (Zhang et al. 2020). Recent works are proposed to combine SCM (Pearl 2009; Schölkopf 2022) with deep learning models, which is called causal learning. Causal learning is widely used in the field of causal representation, CasualVAE (Yang et al. 2021) proposes a model with causal layer to transform exogenous factors into causal endogenous ones that correspond to causally related concepts in data. Shen (Shen et al. 2020)et al. use a SCM as the prior for bidirectional generative model which can generate data from any desired interventional distributions of the latent factors. (Zhang et al. 2020; Lin et al. 2022; Liu et al. 2022; Yue et al. 2020) use the backdoor criterion to eliminate the contextual confounding biases. Different from above works, the input data and extracted features do not satisfy the backdoor criterion in the field of traffic prediction. As shown in Fig. 1(a), we apply the frontdoor formula and use neural networks to model the sub terms, so our model is called neural structural causal model.

We formulate the causalities among bike flow $\mathbf{X}_{t}$, contextual condition $\mathbf{C}_{t}$, spatial neighborhood features $\mathbf{S}_{t}$, temporal dynamic features $\mathbf{T}_{t}$, spatio-temporal states $\mathbf{H}_{t}$, and predicted target $\mathbf{Y}_{t+1}$, with a structural Causal Model (SCM). As shown in Fig. 1(a), directed edges denote causal relationships between nodes. We believe that spatial neighborhood features $\mathbf{S}_{t}$ and temporal dynamic features $\mathbf{T}_{t}$ can be decoupled from spatio-temporal data $\mathbf{X}_{t}$, and they are integrated to form spatiotemporal states $\mathbf{H}_{t}$, which can describe dynamic spatio-temporal patterns in the data.

The contextual condition $\mathbf{C}_{t}$ is expressed as the common cause of $\mathbf{X}_{t}$ and $\mathbf{H}_{t}$, which may cause $\mathbf{H}_{t}$ to be more inclined to the general state and ignore the specific environment due to the limitation of the dataset, resulting in unfair bias of $\mathbf{H}_{t}$. However, the $\mathbf{C}_{t}$ have infinite possibilities and cannot be completely covered. we cannot use backdoor adjustments according to the path $\mathbf{X}_{t}\leftarrow\mathbf{C}_{t}\to\mathbf{H}_{t}$.

Fortunately, we apply the frontdoor criterion based on the path $\mathbf{C}_{t}\to\mathbf{X}_{t}\to\mathbf{S}_{t},\mathbf{T}_{t}\to\mathbf{H}_{t}\leftarrow\mathbf{C}_{t}$, cutting off the link $\mathbf{C}_{t}\to\mathbf{X}_{t}$, which gives $\mathbf{X}_{t}$ a fair opportunity to incorporate each contextual condition $\mathbf{C}$ into spatio-temporal state $\mathbf{H}_{t}$. Formally, we have:

where $P({{{\mathbf{X}^{\prime}}}_{t}})$ represents the prior distribution of the input data, and we propose an Input Gate to fit this prior distribution. $P({\mathbf{S}_{t}},{\mathbf{T}_{t}}|{\mathbf{X}_{t}})$ denotes the process of extracting spatio-temporal features from data and noise, and we propose a Dynamic Causality Generator to embed spatio-temporal causality into a dynamic causal graph. $P({\mathbf{H}_{t}}|{\mathbf{S}_{t}},{\mathbf{T}_{t}},{{{\mathbf{X}^{\prime}}}_{t}})$ denotes the generation of time-varying spatio-temporal states from features to describe the spatio-temporal patterns inherent in the data, and we propose a Spatio-temporal Evolutionary Graph Convolution to extract spatio-temporal states.

Under the condition of $\mathbf{X}_{t}$ in the factual scenario, $\mathbf{C}$ is set to $\mathbf{C}_{t+1}$ to infer the spatio-temporal state $\mathbf{H}^{\prime}_{t}$ in the counterfactual scenario, and then predict the future bike flow $\mathbf{Y}_{t+1}$ via $\mathbf{H}^{\prime}_{t}$. Formally, we have:

where $P\left(\mathbf{H}_{t}^{\prime}\mid\mathbf{X}_{t},\mathbf{C}_{t+1}\right)$ represents the counterfactual representation reasoning process, and $P\left(\mathbf{Y}_{t+1}\mid\mathbf{H}_{t}^{\prime},\mathbf{C}_{t+1}\right)$ represents the prediction process based on counterfactual representation.

Our purpose is to infer the spatio-temporal state in the case of $\mathbf{C}_{t+1}$ by focusing on the similar part between the external conditions of the future and history. Counterfactual representation reasoning module employs scaled dot-product attention to dynamically calculate relationships between each future and historical time step, and converts the encoded historical spatio-temporal states to future representations. Finally, future representations are used to initialize the decoder. As shown in Fig.2, a fully connected layer is used to generate future external features ${{\mathbf{F}}^{pred}}=\operatorname{FC}\left({{\mathbf{C}}_{t+1:t+Q}}\right)\in{{\mathbb{R}}^{N\times Q\times d}}$ and historical external features ${{\mathbf{C}}^{his}}=\operatorname{FC}\left({{\mathbf{C}}^{week}}||{{\mathbf{C}}^{day}}||{{\mathbf{C}}^{hour}}\right)\in{{\mathbb{R}}^{N\times P\times d}}$, which are taken as the query and key of the attention mechanism. The historical spatio-temporal states ${{\mathbf{H}}^{his}}={{\mathbf{H}}_{t-P+1:t}}\in{{\mathbb{R}}^{N\times P\times d}}$ are taken as the value. The counterfactual representation reasoning module is formulated as:

where $d$ is the number of feature channels. ${\mathbf{W}}^{Q}$, ${\mathbf{W}}^{K}$, and ${\mathbf{W}}^{V}$ are learnable parameters. Intuitively, for historical spatio-temporal states, the counterfactual representation reasoning module indicates to pay more attention to the parts whose external conditions are similar to the future. The future representations ${{\mathbf{H}}^{pred}}$ are input into a fully connected layer and then used to initialize the decoder.

Datasets: We collect two real-world datasets, NYC-Bike and BJ-Bike, each dataset contains the corresponding weather and time information. We split the dataset with a 30-minute interval. We select the first 60% of data as the training set, 20% as the validation set, and 20% as the test set.

Baselines: We compare STNSCM with recent state-of-the-art baselines. We summarize the models into four categories, including deep learning methods, predefined graph methods, adaptive graph methods, attention methods and dynamic graph methods. The main difference between adaptive graph and dynamic graph is whether the change of graph structure depends on input features.

Evaluation Metrics: We evaluate the performance of methods with Root Mean Square Error (RMSE), Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE).

For fairness, we deploy the same environment, loss function, periodic flow data, and external conditions for all models. We compare STNSCM with baselines for traffic prediction and the final average results are shown in Table 1.

We classify these methods according to whether the graph structure varies depending on input features. Results show our STNSCM outperforms baseline models consistently and overwhelmingly. In deep learning methods, Poor performances of indicate the limitation of failing to consider spatial correlation. STGODE(Fang et al. 2021) deepens networks to extract higher-order features through a tensor-based ordinary differential equation, but hampered by the amount of data, STGODE shows a worse performance. Besides, these methods only rely on the fixed graph structure while ignoring the dynamic characteristics.

The adaptive graph(Wu et al. 2019; Guo et al. 2021a; Ye et al. 2021; Han et al. 2021) is helpful in the short-term prediction (30min), but it is still static over time and fails to capture time-varying spatio-temporal dependencies, and the effect of long-term prediction (60min) is significantly reduced. The attention graph(Zheng et al. 2020; Guo et al. 2021b) can only change the weight of predefined graphs rather than the structure. DGCRN(Li et al. 2021) employs the pre-defined adjacency matrix to conduct the message-passing process for dynamic node status. Due to the incomplete connections in the data, the predefined graph itself may contain noise, hindering the generation of dynamic graphs.

Besides, these models do not have exclusive modules to handle contextual conditions, and we merely concatenate them with periodic flow data, so external information is not fully exploited. The qualitative prediction results are shown in Fig. 1(e). The main contribution of our model is stability and resistance to random fluctuations, which are also the most important capabilities of bike flow forecasting. Therefore, in the NYC-Bike test set, the bike flow distribution fluctuates greatly due to the external conditions, so our method outperforms all methods.

To verify effectiveness of key components in STNSCM, we conduct ablation experiments on NYC-Bike dataset, which are described as follows:

• •

  w/o ${{\mathbf{A}}^{geo}}$: It removes the geographical distance graph.

• •

  w/o ${{\mathbf{A}}^{trans}}$: It removes the transition probability graph.

• •

  w/o ${{\mathbf{A}}^{dyn}}$: It removes the dynamic causal graph.

• •

  EGG w/o SE: It removes the squeeze-excitation method from dynamic causality generator.

• •

  EGG w/o $\mathbf{H}$: It removes the spital-temporal state of the previous time step from dynamic causality generator.

• •

  EGG w/o $\mathbf{X}$: It removes the features output by input gate from dynamic causality generator.

• •

  w/o CR: It removes the counterfactual representation reasoning module from STNSCM. The last spital-temporal state of the encoder are copied to initialize the decoder.

• •

  IG w/ FC: It concatenates periodic flow data and external conditions to extract context-conditioned features only through fully connected layers.

• •

  w/o IG: It removes the input gate. The model merely input the bike flow tensor of the previous $P$ time steps.

The performance of all variants is summarized in Table 2. For the contribution of different graphs, the dynamic causal graph obviously plays a more prominent role. The introduction of the causal graph can significantly improve performance, as it provides implicit causality that cannot be extracted from static topological graphs. Meanwhile, the geographical distance graph and the transition probability graph are also necessary. The dynamic causal graph can collaborate with static topological graphs to better model complex transportation systems. We further visualized the change process of the contribution coefficient $\alpha$ in the spatio-temporal evolutionary graph convolution during the training period, as shown in Fig.3. In the encoder, the geographic distance graph and the transition probability graph have the same importance, while the input feature $\mathbf{X}_{t}^{(n-1)}$ of the graph convolution has a smaller coefficient. This indicates that the encoder needs to extract potential spatio-temporal dependencies in the transportation system by propagating and aggregating the node features from multi-view graphs. On the contrary, in the decoder, the input features $\mathbf{X}_{t}^{(n-1)}$ of the graph convolution account for a large proportion, which shows that the decoder needs to restore high-level spatio-temporal features to predict the future flow.

For the dynamic causality generator, the spatio-temporal state of the previous time step is crucial for modeling the dynamic spatio-temporal causality into the causal graph. We visualized the dynamic causal graph, as shown in Fig.4(c), the dynamic causal graph integrated with periodic features and external conditions directly model the interactive evolutionary process across regions.

In addition, external conditions dominate the model performance, otherwise, the model cannot govern the fluctuations caused by this factor. On the one hand, the performance of IG w/ FC and w/o IG is almost the same. This indicates that features extracted by our input gate are more effective. On the other hand, the model’s resistance to random fluctuations can be reflected in MAPE. Removing the input gate will greatly increase MAPE. The input gate can effectively fuse periodic flow data and external conditions to extract the context-conditioned features, which is also the basis for STNSCM to resist fluctuations caused by external environment.

The counterfactual representation reasoning module utilizes the attention mechanism to convert the encoded historical spatio-temporal states to future representations, which reduces information loss and improves the overall performance of the model.