STDCformer: A Transformer-Based Model with a Spatial-Temporal Causal De-Confounding Strategy for Crowd Flow Prediction

The main goal of the spatial representation module is to model the relationships between spatial regions and encode these relationships into representations. Based on the fundamental structure of spatial representation models, existing methods can be categorized into CNN-based, GNN-based, and Transformer-based spatial-temporal prediction approaches. CNN-based models partition spatial regions into regular grids and use spatial convolutions to achieve spatial representation. For example, DeepST[21] uses a deep neural network (DNN) to extract the spatial features of the urban grid. However, this approach is not suitable for spatial-temporal prediction in irregular regions. GNN-based spatial representation modules and Transformer-based spatial representation modules are separately the core component of existing STGNNs and ST Transformers.

GNN-based models. DCRNN [6] uses a diffusion convolution to extract graph structure signals by walking randomly on the graph; STGCN[16], T-GCN[10], and KST-GCN[7] use graph convolution networks to represent graph structures; STFGNN[15] uses a simplified graph matrix multiplication to aggregate spatial dependencies; Graph wavenet
citeRN13 generalizes the form of GCN to diffusion convolution to build graph convolution layer; ASTGCRN[5]proposes an adaptive graph convolution to learn node representations; ASTGCN[12]uses graph attention convolution to obtain spatial representations based on spatial attention; HGCN[14] uses spectral clustering and spatial gated graph convolution to obtain micro and macro spatial representations; MTGNN[30] uses a mix-hop propagation layer in the graph convolution module to select and aggregate information from neighboring nodes.

Transformer-based models. ASTTGN[20] leverages node2vec[31] to retain graph structure information, and inputs it as the embedding of the node sequence into the spatial transformer model. Then, a multi-head attention mechanism is used to capture dynamic and implicit spatial dependencies. STTN[17] inputs the representation obtained by graph convolution into the transformer structure to capture dynamic graph signals; GLSTTN [22] constructs a spatial transformer similar to STTN to learn spatial representations; Traffic transformer[8] constructs an encoder-decoder framework based on multi-head attention, and introduces the original graph structure as a mask in the decoder.

The primary goal of the temporal representation module is to model temporal features such as trends and periodicity in time series data. According to the fundamental structure of temporal representation models, existing methods can be categorized into RNN-based, CNN-based, and transformer-based spatial-temporal prediction models. RNN-based models primarily utilize Recurrent Neural Networks (RNNs)[32, 33] and their variants[34] to extract temporal features. CNN-based models extract temporal features by applying convolutions over the time dimension. Transformer-based temporal representation models, on the other hand, use self-attention mechanisms to capture dependencies across different time steps.

RNN-based models. Models such as ASTGCRN[5], T-GCN [10], and KST-GCN[7] all employ Gated Recurrent Units (GRU). DCRNN [6] utilizes GRU to construct a Diffusion Convolutional Gated Recurrent Unit for extracting temporal features. ST-MetaNet [11] employs a GRU as the first layer in a Seq2Seq framework. Traffic Transformer[8]uses a simple Long Short-Term Memory (LSTM) to obtain temporal embeddings for extracting temporal features. MASTGN [9]uses LSTM to process features after transformations based on spatial and temporal attention.

CNN-based models. STGCN[16] utilizes 1-D causal convolution and Gated Linear Units (GLU) to construct temporal convolutional layers. Graph Wavenet[13] uses Gated Temporal Causal Convolution to expand the temporal receptive field. STFGNN[15] proposes a new gated CNN module for extracting temporal features. ASTGCN [12] combines temporal attention with standard convolution to extract temporal features. HGCN [14] replaces the original gated temporal convolution with dilation convolution whose dilation factor is 2 to construct temporal gated convolution.

Transformer-based models. ASTTGN[20] employs an adaptive temporal transformer module to capture long-range temporal dependencies. STTN[17] construct a temporal transformer to capture bidirectional, long-range temporal dependencies. LSTTN [19] trains a transformer-based encoder to represent temporal features of subsequences through a self-supervised task of mask reconstruction.

Existing methods for spatial-temporal fusion can be divided into two categories: direct fusion and adaptive fusion. The former typically connects the temporal and spatial representation modules sequentially, interleaves them in a block or simply adding the two types of representations. In contrast, the latter involves inputting both representations into a learnable module for fusion.

Direct Fusion. In TGCN[10], the temporal representation module follows the spatial representation module, with the learned spatial representations being fed into the temporal representation module. While Graph Wavenet[13] places the temporal module before the spatial module. G-SWaN[35] first uses a WaveNet to obtain the temporal representation and then feeds it into the spatial graph transformer to learn the spatial representation. Similarly, Traffic Transformer[8] uses LSTM to obtain representations, which are then input into subsequent global and local spatial graph representation modules. STGCN[16] and HGCN [14] build the temporal gated convolutional layer and spatial convolutional layer into a ”sandwich” structure. MTGNN [30] interleaves the temporal and spatial convolution modules for spatial-temporal fusion, while STTN [17] interleaves the spatial transformer and temporal transformer. DCRNN[6] replaces matrix multiplications in the GRU with spatial diffusion convolutions, using spatial representations as recurrent units to learn temporal representations. Similarly, ASTGCRN[5] replaces the multilayer perceptron (MLP) in GRU with adaptive graph convolution. MASTGN [9] concatenates the representations transformed by spatial attention and internal attention, using this concatenated representation as input for the followed layers.

Adaptive Fusion. ASTTGN[20], MVSTT[36] and GMAN[37] use a gated fusion mechanism to adaptively perform weighted fusion of temporal and spatial representations. PDFormer[18] allocates different attention heads to the temporal and spatial representations for attention learning and then uses a multi-head attention mechanism to fuse the temporal and spatial representations.

Methods based on SCM can directly implement interventions on causal graphs, transforming causal problems into statistical language that can represent the data through front-door, backdoor adjustments and other methods. Backdoor adjustment realizes de-confounding by stratifying and balancing confounding factors, being adopted by many studies. Some works categorize confounding factors into pre-defined layers. For instance, STCTN [26] considers regional attributes in the region network as confounders, which may cause existing spatial-temporal prediction models to absorb spurious correlations between the historical and future data. To address this, spectral clustering is used to partition regional attributes, and an unbiased prediction model is constructed in an independent parameter space for each partitions to achieve de-confounded predictions. SEAD[38] aims to eliminate the confounding effects of social environments on pedestrian trajectories. This work assumes that the confounders follow a uniform distribution, and the joint distribution of confounders and causal variables is learned through a cross-attention module. CISTGNN [39] learns representations of confounders from external weather environments and applies backdoor criteria for de-confounding. STEVE[25] focuses on the finiteness of the number of confounders and the completeness of their categories, dividing confounders into invariant and variant layers. CaST [24] discretizes the confounders through a temporal environment codebook, so that the representations of confounder fall into the most appropriate layers. CTSGI[40] leverages images at pedestrian trajectory points as the source of confounder representations, extracting environmental representations through a semantic segmentation model, and then adjusting confounding effects using backdoor criteria. Front-door adjustment, on the other hand, applies to causal graphs where no back-door paths exist, providing a de-confounding strategy different from the backdoor criterion. STNSCM [41] uses GLUs to obtain the distribution of causal variables after intervention from historical observational data and external environment data. In this distribution, causal variables can combine with different confounders obeying an unbiased probability, allowing for de-confounded effect estimation. A similar strategy is used in vehicle trajectory prediction, where the contextual scene is treated as a confounder and eliminated through a counterfactual representation inference module based on front-door adjustment strategies [42]. CASPER [43] uses front-door adjustment to transform time-series completion tasks into summing over subcategories of input data representations, and calculates de-confounded causal effects through learnable prompt vectors.

The main idea of methods based on the potential outcome framework is to treat different observational units as samples and simulate interventions by processing these samples comprehensively, thus obtaining de-confounded causal effects. Sample re-weighting is one of the classical methods for achieving sample balance [44], which is used to assign different cities and regions to different treatment groups and calculate the causal effects corresponding to each treatment. For example, it has been used in studies to assess the impact of changes in POI on regional pedestrian flow[45]. With the development of representation learning, balancing confounders based on the obtained unified representations of the samples through representation learning can also ensure that the background attributes distribution across different experimental groups are similar. For instance, CAPE[46] generates representations for different locations from historical event sequences and measures the differences in the representations using the Integral Probability Metric (IPM), minimizing the overall differences between samples in different experimental groups to achieve representation balance. SINet[47] uses the Hilbert-Schmidt Independence Criterion (HSIC) as a regularizer to guide representation balance and remove the effect of hidden confounders. CIDER[48] applies the Wasserstein distance to balance representations across different administrative regions.

In the spatial-temporal prediction module, the proposed method constructs temporal and spatial representations based on transformer models, and employs adaptive spatial-temporal fusion. In the spatial-temporal de-confounding module, the de-confounding strategy proposed in this paper is based on SCM, utilizing the backdoor criterion for de-confounding. The distinctions between this method and the aforementioned related studies are as follows: 1) The strategy employed in this paper innovatively categorizes confounders into two abstract types: temporal confounder and spatial confounder. Furthermore, the probability distributions of the confounders are learned adaptively, which avoids the pre-setting of the number of categories and the distribution of confounders; 2) The constructed spatial-temporal de-confounded representations can simultaneously participate in both spatial-temporal representation learning and spatial-temporal fusion, maximizing information sharing between the two modules.

The foundation of the ability to infer future states from historical states for spatial-temporal prediction models is that the past has some form of ”influence” on the future. This influence determines the mapping relationship between historical input data and future output data, which is the basis of the predictability of spatial-temporal data. Causal inference can measure the influence of the past on the future using causal effects, aiming to build causal relationships between variables rather than merely correlations, enabling a more accurate and stable representation of the variables[49]. The underlying assumption is that causality is a more reliable form of association, where information flows along the causal links, transmitting the causal effect of the cause on the outcome. Based on this assumption, the spatial-temporal prediction process can be viewed as the process of estimating the causal effect of historical data representations on future data representations, inferring the outcome from the values of causes. However, the hidden confounding bias makes it impracticable to estimate the true causal effect directly from the observational data. This section takes the basic unit of spatial-temporal prediction, STT, as a starting point, to describe the causality underlying crowd flow prediction. We also propose a spatial-temporal backdoor adjustment strategy to learn a de-confounded representation space.

Spatial-Temporal Token (STT). Based on the form of input data, the basic unit in spatial-temporal data is the observation on a spatial entity $S_{i}$ at a specific time step $T_{j}$. In this paper, this smallest unit in spatial-temporal data is analogized to a token in text, referred to as the Spatial-Temporal Token (STT), represented as: $STT_{ij}=(S=S_{i}\in R^{1\times s},T=T_{j}\in R^{1\times t},V=IO_{ij}\in R^{1\times f})$. where $V$ is the variable to be predicted, and $S$ and $T$ describe its spatial and temporal characteristics, respectively. These can be considered as background attributes of STTs, and they influence the estimation of the causal effect $V_{past}\rightarrow V_{future}$.

Causal Graph underlying Spatial-Temporal Prediction. The prediction model infers future data representations from historical data representations, where the historical representation serves as the cause for the future representation. However, for each STT, the background attributes influence both the past and future IO representations, making $S$ and $T$ serve as confounder in the prediction process of $V$. The causal graph underlying spatial-temporal prediction can be illustrated as Fig. 6, which suggests that the causal effect from $V_{past}\rightarrow V_{future}$ may vary under different values of confounders. This is similar to how the effect of a drug may differ for patients of different ages in a medical trial. To estimate an accurate and fair causal effect, it is necessary to configure the trial groups in such a way that the distribution of confounders across all groups becomes uniform, i.e., to remove the confounding bias. The intervention operation on the causal graph provides an intuitive representation of de-confounding, where the arrow $C\rightarrow X$ is removed to make the distribution of $C$ independent of $X$. The causal effect after intervention is expressed as $P(Y|do(X))$.

Spatial-Temporal Backdoor Adjustment. Backdoor adjustment modifies the weights of observational samples with different values of confounders based on their distribution, simulating the balanced sample distribution under an intervention experiment, and is an effective way to de-confounding in observational data. It can be applied when the confounder satisfies the backdoor criterion for $X\rightarrow Y$. After backdoor adjustment, the causal effect of $X\rightarrow Y$ is expressed as: $P(Y|do(X))=\sum_{c}^{K}P(Y|X,C=c_{k})\cdot P(C=c_{k})$. where the confounder is discretized into $k$ categories. However, in spatial-temporal prediction, confounders are often hidden, making it difficult to predefine the value of $k$ and the distribution of each category. To address this issue, we consider the fundamental decision factors of human movement in the real world-namely, ”where” and ”when”-to construct the minimal descriptive set of human movement: ”when”, ”where”. We then categorize the confounders into two abstract types: temporal confounders $C_{T}$ and spatial confounders $C_{S}$. Furthermore, since STTs naturally contain the background variables $S^{1\times s}$ and $T^{1\times t}$, which can serve as information sources for $C_{S}$ and $C_{T}$, respectively. Based on this partition, we propose a spatial-temporal backdoor adjustment method as our de-confounding strategy, which is expressed as Eq. 2:

The relationship between past and future states is very abstract, if using a function to approximate this kind of relationship, the parameters of this mapping function will have different value in different temporal periods. Past and future can be viewed as a pair of Cross-Time spatial-temporal entity. Capturing the transformation relationship between these entities requires addressing the following two issues:

(1) Representation of spatial-temporal Entities’ Intrinsic Characteristics. Based on the observations in Fig. 2, we argue that spatial confounder and temporal confounder can jointly characterize the contextual differences between different STTs, which represent the intrinsic characteristics of tokens. Therefore, fusing the representations of temporal confounders with spatial confounder effectively captures the spatial-temporal characteristics of different spatial-temporal entities. For each STT that constitutes the Past and Future, we integrate their temporal and spatial confounder’s representations to obtain a Spatial-Temporal Embedding (STE). This embedding retains the temporal and spatial characteristics of each STT within the entire dataset, thereby representing the contextual features of different spatial-temporal entities.

(2) Querying Relationships Between Cross-Time Entities. The relationships between Cross-Time entities are diverse and dynamically evolving. Therefore, attention mechanism can be effective for capturing their similarities. Since this type of querying spans across time, we refer it to Cross-Time Attention (CTA). We use the STEs of Past and Future as the Key and Query, respectively, and employ the attention mechanism to perform the query. The spatial-temporal mapping is then implemented based on the results of queried attention.

For a $STT_{ij}=\left(S=S_{i}\in R^{1\times s},T=T_{j}\in R^{1\times t},V=IO_{ij}\in R^{1\times f}\right)$, the data embedding layer maps it into a representation vector $\overrightarrow{STR_{ij}}\in R^{1\times 2d}$, which is used to effectively integrate both the spatial-temporal characteristics and observational information of the STTs. The representation of the STT is constructed using the following two types of information:

1. 1.

   Observational Value. This mainly contains the observational feature of the STT. Therefore, temporal representation is used to preserve this part of the information. The flow observation value $V^{1\times f}$ is mapped into d-dimensional space by a temporal encoder, resulting in $V^{\prime}\in R^{1\times d}$ as the value representation.

2. 2.

   Spatial-Temporal Characteristics. This mainly contains the spatial-temporal location and structural information of the STT. Since the temporal and spatial confounders representation can portray the temporal and spatial properties of the STT, a spatial-temporal embedding fusing the information of two types of confounders can preserve this part of information. A confounder encoder is used to map $T_{j}\in R^{1\times t}$ and $S_{i}\in R^{1\times s}$ into a d-dimensional space, yielding $C_{T}\in R^{1\times d}$ and $C_{S}\in R^{1\times d}$, respectively. $S_{i}$ is first concatenated with its representation $Laplacian_{i}\in R^{1\times d_{lap}}$ in the Laplacian space of the adjacency graph, resulting in $S^{\prime}_{i}\in R^{1\times(s+d_{lap})}$, and then mapped. Since $C_{S}$ and $C_{T}$ capture the spatial-temporal attributes of the STT, they play a similar role to the token’s embedding. Therefore, $C_{S}$ and $C_{T}$ are fused through a convolution layer to obtain the Spatial-Temporal Embedding (STE), denoted as $STE\in R^{1\times d}$.

Finally, the value representation $V^{\prime}\in R^{1\times d}$ and the $STE\in R^{1\times d}$ are concatenated to form the representation $\overrightarrow{STR_{ij}}\in R^{1\times 2d}$. The transformation within the data embedding layer is described by the following formulas in Eq. 3, where $Conv(\cdot)$ denotes the 2D convolution with the ReLU activation function.

For the spatial-temporal representation within the historical time window, the observation time steps are defined as $T=(t-T^{p}+1,\ldots,t)$, and the spatial-temporal representation of the region $S_{i}$ is given by $STR_{S_{j}}=\left(\overrightarrow{STT_{1T_{j}}},\ldots,\overrightarrow{STT_{nT_{j}}}\right)\in R^{n\times 2d}$. For the observed spatial region $S=(1,\ldots,n)$, the spatial-temporal representation at time step $T_{j}$ is given by $STR_{S_{j}}=\left(\overrightarrow{STT_{1T_{j}}},\ldots,\overrightarrow{STT_{nT_{j}}}\right)\in R^{n\times 2d}$. Therefore, the spatial-temporal representation of the data within the prediction window is for all regions on all steps and is given by $STR_{ST}=\left(\overrightarrow{STT_{1T}},\ldots,\overrightarrow{STT_{nT}}\right)\in R^{T^{p}\times n\times 2d}$. It is then fed into the first layer of the Spatial-Temporal De-Confounded Attention Block in the STDC Encoder.

As illustrated in Fig. 9, the Spatial-Temporal De-Confounded Attention Block is composed of three modules: Spatial Attention, Temporal Attention, and Spatial-Temporal De-Confounding Fusion, which are designed to capture spatial representations, temporal representations, and spatial-temporal fusion, respectively. Spatial Attention and Temporal Attention use the self-attention mechanism to capture spatial and temporal dependencies between different STTs. Spatial-Temporal De-Confounding Fusion applies a spatial-temporal backdoor adjustment strategy to control spatial-temporal confounders, ensuring that the predictions are not influenced by confounding biases.

Spatial attention. The spatial attention module is designed to model the spatial dependencies between different regions within a given time window. In this module, the self-attention mechanism is employed to capture the similarities between STTs distributed across different spatial regions, facilitating the fusion of information across space. For each batch of data, $STR_{ST_{j}}=STR_{ST}[T_{j},:,:]\in R^{n\times 2d}$ is used to learn the spatial attention distribution for time step $T_{j}$. $Q_{S}^{T_{j}},K_{S}^{T_{j}},V_{S}^{T_{j}}\in R^{T_{P}\times d}$ are firstly learned for current input representation to learn the representations of Query, Key and Value. The spatial attention coefficient matrix $A_{S}^{T_{j}}\in R^{n\times n}$ is then computed. Formulas for this process are in Eq. 4:

Where $W_{ST_{j}}^{Q},W_{ST_{j}}^{K},W_{ST_{j}}^{V}\in R^{2d\times d}$. Finally, for time step $T_{j}$, the data representation after the spatial attention transformation is given by $STR_{ST_{j}}^{\prime}=\text{softmax}\left(A_{S}^{T_{j}}\right)\cdot V_{S}^{T_{j}}\in R^{n\times d}$. For the entire input time window, the representation is $STR_{S}^{\prime}=\text{softmax}\left(A_{S}^{T_{j}}\right)\cdot V_{S}^{T_{j}}\in R^{T_{P}\times n\times d}$.

Temporal attention. Temporal attention module is designed to model the temporal dependencies of a specific region across different time steps. In this module, the self-attention mechanism is employed to capture the similarities between STTs distributed across different time steps, facilitating the fusion of information across time. For each batch of data, $STR_{S_{i}T}=STR_{ST}[:,S_{i},:]\in R^{T_{P}\times 2d}$ is used to learn the temporal attention distribution for region $S_{i}$. $Q_{S_{i}}^{T},K_{S_{i}}^{T},V_{S_{i}}^{T}\in R^{T_{P}\times d}$ are firstly learned for current input representation to learn the representations of Query, Key and Value. The temporal attention coefficient matrix $A_{S}^{T_{j}}\in R^{T_{P}\times T_{P}}$ is then computed. Formulas for this process are in Eq. 5:

Where $W_{S_{i}T}^{Q},W_{S_{i}T}^{K},W_{S_{i}T}^{V}\in R^{2d\times d}$. Finally, for time step $T_{j}$, the data representation after the temporal attention transformation is $STR_{S_{i}T}^{\prime}=\text{softmax}\left(A_{S_{i}}^{T}\right)\cdot V_{S_{i}}^{T}\in R^{T_{P}\times d}$. The representation for the entire input time window is $STR_{T}^{\prime}=\text{softmax}\left(A_{S_{i}}^{T}\right)\cdot V_{S_{i}}^{T}\in R^{T_{P}\times n\times d}$.

Spatial-Temporal De-Confounded Fusion. The Spatial-Temporal De-Confounded Fusion module fuses the spatial and temporal representations obtained from the previous two modules with simultaneously controlling the confounders. According to the spatial-temporal backdoor adjustment strategy $P(Y|do(X))=P(Y|X,C=C_{S})\cdot P(C=C_{S})+P(Y|X,C=C_{T})\cdot P(C=C_{T})$, $P(X,C=C_{S})$ here, $P(X,C=C_{S})$ represents the joint distribution of spatial confounders and spatial representations, while $P(X,C=C_{T})$ represents the joint distribution of temporal confounders and temporal representations. Since $\overrightarrow{STT_{ij}}$ contains both $C_{S_{i}}$ and $C_{T_{j}}$, which are involved in the computation of spatial attention and temporal attention, respectively, $STR_{S}^{\prime}$ and $STR_{T}^{\prime}$ can be considered as the representations of $P(X,C=C_{S})$ and $P(X,C=C_{T})$. For each $STT_{ij}$, to obtain $P(C=C_{S})$ and $P(C=C_{T})$, $C_{S_{i}}$ and $C_{T_{j}}$ are summed and passed through a Sigmoid function, mapping the weight of the two confounding factors to the range from 0 to 1. Finally, after fusion, the final spatial-temporal representation $H\in R^{T_{P}\times n\times d}$ is obtained for all STTs in the input data, as shown by Eq. 6:

The STDC Decoder and STDC Encoder share the same structure of the Spatial-temporal De-Confounded attention block. The key difference lies in the input: the STDC Encoder takes the historical data’s ST representation as input, while the Decoder takes the future state ST representation after Past-to-Future mapping. Therefore, $STR_{ST^{\prime}}=(\overrightarrow{STT_{1T^{\prime}}},...,\overrightarrow{STT_{nT^{\prime}}})\in R^{T_{f}\times n\times 2d}$ is fed into the first layer of the Spatial-temporal De-Confounded attention block in the Decoder, where $T^{\prime}=(t+1,...,t+T_{f})$.

The Prediction Head maps the de-confounded representations of the future state from the Spatial-temporal De-Confounded attention block to the observational feature space. In this paper, a simple convolutional layer is used to map the vector $H_{future}\in R^{T_{f}\times n\times d}$ in the high-dimensional representation space to the observation space. It is formulated as $\hat{Y}=Conv2d(H_{future})\in R^{T_{f}\times n\times f}$. This directly implements multi-step forecasting for $T_{f}$ steps. The model is trained end-to-end with the Mean Absolute Error (MAE) as the loss function. The loss function is in Eq. 8:

Where $Y_{t}$ represents the ground truth.

In order to evaluate the model’s predictive ability for different features and the overall flow, we test the model’s predictions for Inflow, Outflow, and the combined Inflow-Outflow (IO). The results are shown in Table 3.

For different datasets, the model’s performance on the BKL dataset is superior to that on the MHT dataset, with smaller differences in prediction metrics. This is due to the fact that the MHT dataset exhibits an overall larger range of numerical variations. As the most famous and active central district of New York City, the complexity of crowd movement in MHT is higher, making prediction more challenging. For different model architectures, ST Transformers outperform most of the STGNNs and traditional models, indicating that the introduction of self-attention mechanisms is effective in capturing complex spatial-temporal dependencies. However, powerful STGNN models such as DCRNN and MTGNN demonstrate comparable or even stronger predictive performance than some ST Transformers. This suggests that STGNNs still have significant advantages in modeling temporal and spatial dependencies and, in certain cases, can achieve accurate predictions with lower cost.

Since the model is trained based on the overall crowd flow characteristics (IO), IO can be considered the primary evaluation metric. In terms of overall feature prediction, the proposed STDCformer achieves the highest IO prediction accuracy across both datasets, while PDFormer demonstrates the second-best performance. MTGNN, GMAN, and DCRNN show comparable performance to PDFormer. Regarding the prediction accuracy of individual features, STDCformer and PDFormer remain highly competitive. PDFormer outperforms our model in inflow prediction on the MHT dataset, but our model performs better in outflow prediction. This indicates that different models may focus on certain data features during training, influenced by the characteristics of the data. On one hand, it suggests that a key direction for future research is understanding how models balance their learning capacity for different features. On the other hand, it also implies that relevant application departments can select models based on specific strengths, depending on their practical needs.

The evaluation results of the zero-shot experiments under OOD conditions are shown in Table 4. STDCformer outperforms the baseline method in all metrics, indicating that STDCformer has better generalization ability. And it can be observed that on the same dataset, the differences in the metrics between models are more significant on the zero-shot task than on the supervised learning task, which implies that the de-confounded model has a greater advantage in generalization ability. The reason behind may be that although the distribution of spatial-temporal confounders in the ood dataset is shifted, frozen confounder encoder is still able to capture the difference of characteristics between STTs and use it to assign weights, which makes the weights in the de-confounding spatial-temporal fusion module is still able to de-confounding and make prediction.

The ability of the model to transfer to an OOD dataset in a zero-shot manner is of significant practical importance, as it allows for introducing models from other regions with minimal cost to support emergent decision-making while achieving acceptable performance. Although the accuracy of this zero-shot testing is higher compared to training from scratch on the OOD dataset, it is still acceptable for predictive needs in emergency situations, meeting the requirements of relevant planning and management departments.

To further validate the effectiveness of the components in STDCformer, we conducted ablation experiments. The MAE of the model after ablation is shown in Figure 12. We set up the following five ablated models:

1. 1.

   ’w/o DC’: The original model without the de-confounding module. Ablating this module demonstrates the importance of considering reweighting in the spatial-temporal fusion process.

2. 2.

   ’w/o MAP’: The original model without CTA-based mapping. Ablating this module highlights the importance of learning the transformation relationship between past and future states in the representation space.

3. 3.

   ’w/o SC’: The original model without the spatial confounder. Ablating this module shows the significance of integrating auxiliary information to characterize the spatial characteristics of different regions.

4. 4.

   ’w/o TC’: The original model without the temporal confounder. Ablating this module illustrates the importance of incorporating auxiliary information to represent the temporal characteristics of each time window.

5. 5.

   ’w/o LAP’: The original model without the Laplacian embedding. Ablating this module emphasizes the importance of retaining the global adjacency graph structure when characterizing spatial properties.

Overall, it can be observed that the model’s performance on the BKL dataset does not change significantly, with the original model achieving the best performance. This suggests that the backbone, spatial-temporal transformer, ensures the model’s basic capability on simple datasets, and the additional components we designed enhance performance on top of this foundation. In contrast, on the MHT dataset, which presents a greater prediction challenge, the contribution of each component to prediction accuracy provides a clear and intuitive comparison.

After removing the CTA-based mapping module, the model’s accuracy dropped the most, indicating that the differences between the spatial-temporal characteristics of past and future STTs are significant. The mapping relationship is highly complex and diverse, requiring the design of effective modules to uncover these dynamic relationships. The prediction performance also declined after removing the Laplacian embedding and the De-confounding module, demonstrating the importance of incorporating the region’s position within the global spatial structure and accounting for the weights of the temporal and spatial characteristics of the STTs in improving prediction accuracy. The model’s accuracy also showed an evident decline after removing the spatial confounder and temporal confounder, even falling below that of relatively weaker STGNN models. This demonstrates that modeling the auxiliary information as STT characteristics is beneficial for enhancing the model’s predictive capability.

We further investigate the sensitivity of the model’s predictions to hyperparameter settings. Since our primary goal is to explore the stability of the model’s performance across different parameter ranges and demonstrate that the model’s effectiveness is not benefit by randomness, we selected hyperparameter ranges that vary around the optimal parameter value. We chose values in a reasonable scope, because extreme values were excluded from consideration by any method.

The hyperparameters explored include: (1)Laplacian Dimension: exploring the relationship between spatial structural information and model performance through the dimension of Laplacian eigenvalues; (2) Layers: exploring the relationship between model complexity, feature abstraction, and model performance through the number of STDC attention blocks; (3) Dimension: exploring the relationship between the representation space dimension and model performance through the dimension of the hidden representation space; (4) Attention Head: exploring the relationship between the self-attention mechanism and model performance through the number of attention heads. Taking the MAE of the prediction results as an example, the results are shown in Figure 13. Overall, the model exhibits stable performance across different hyperparameter settings, indicating a strong basic predictive ability. The impacts of different hyperparameters are as follows:

1. 1.

   The dimension of Laplacian eigenvalues. Since the Laplacian eigenvalues need to be concatenated with other spatial confounders, we selected values close to and lower than the dimension of the hidden representation space with a scope of [6,8,10,12]. The best performance was achieved when the value was set to 8, suggesting that at this point, the spatial position features and spatial semantic features of different regions might have reached an optimal balance.

2. 2.

   Number of STDC Attention Blocks. While increasing the number of layers theoretically enables the model to learn more abstract representations, excessive layers can make the model over-complex, reducing efficiency or even leading to overfitting. Therefore, the maximum number of layers was set to 6. Overall, the predictive ability of the model improved with more layers, with the MHT and BKL datasets achieving optimal performance at 5 and 6 layers, respectively. The optimal values on these two datasets show little difference, however, for more complex datasets with higher prediction difficulty, more layers are required.

3. 3.

   Dimension of the hidden representation space. Since the original dimensions of the time-series observations are relatively low, mapping them to excessively high hidden dimensions may lead to information sparsity. Therefore, we chose a range of [32,64,128]. The optimal dimension for the hidden representation space on two datasets was both 64, suggesting that when the original 2-dimensional flow feature is scaled to a 64-dimensional space, an optimal trade-off is achieved between information retention and feature extraction.

4. 4.

   Number of Attention Heads. Different attention heads can capture attention coefficients with different semantics. Following existing studies, we set the range to [1,2,4,8]. Multi-head attention outperformed single-head attention, with the MHT and BKL datasets achieving optimal performance at 8 and 4 heads, respectively. This indicates that the semantic complexity of the MHT dataset is higher than that of the BKL dataset, which may be strongly correlated with the prediction difficulty.

Balance between Inflow and outflow. Crowd flow prediction requires capturing both inflow and outflow characteristics of a region, making it a multi-feature prediction task. Different models perform differently on inflow and outflow predictions and often fail to achieve a balanced performance. This imbalance arises because inflow and outflow can exhibit distinct patterns for the same region. For instance, during working hours, a company may load higher inflow than outflow, while the opposite may occur during after-work hours. Models’ predictive imbalance across features may be raised by biases in the distribution of features within the dataset, causing the model’s favor of certain features. Figure 14 illustrates the relative MAE ratio of predicting inflow and outflow by different models. We observe that STDCformer achieves the most balanced performance between inflow and outflow predictions across both datasets. This balance may be attributed to the incorporation of spatial characteristics, which help in constructing a semantic profile of the region (e.g., functionality). These semantics act as common driving factors behind inflow and outflow variations, enabling the model to capture changes in these features more effectively.

Balance across regions and across timesteps. As shown in Figure 15, using MHT dataset as an example, we examine the distribution of the model’s performance across spatial and temporal dimensions for each model. We gathered the statistic of prediction metrics separately for different regions and time windows, and report the mean and standard deviation. Taking the difference in MAE as an example, it can be observed that STDCformer achieves the smallest standard deviation in prediction across both regions and time steps. This indicates that STDCformer is able to maintain the best balance across spatial and temporal dimensions. The underlying reason for this performance is likely the incorporation of spatial-temporal confounders’ information, followed by a de-confounding process. This approach allows the model to distinguish samples from different spatial and temporal contexts, rather than treating each spatial-temporal sample equally in the parameter fitting process.

In general, all models perform well in capturing the upward and downward trends of crowd flow. However, STDCformer prefer to make predicted values within a relatively small range, likely due to the inherent restraint between the spatial confounder $C_{S}$ and temporal confounder $C_{T}$. Because STDCformer places significant importance on spatial characteristics, which are considered stable, thus leading to more conservative predictions. For traffic flow peaks that last for over two hours (as indicated by the red dashed boxes in Figures 16 and Figures 17), STDCformer provides relatively accurate predictions. On the other hand, PDFormer and GMAN tend to predict values that exceed the actual peak. However, in the case of sudden increases followed by sharp decreases in crowd flow (as indicated by the black dashed boxes in Figures 16 and Figures 17), STDCformer struggles to make more accurate predictions for the peak. This observation indicates two points, on the one hand, STDCformer is less sensitive to temporal changes, and its ability to predict flow shifts caused by unexpected events is relatively weaker. On the other hand, when the traffic flow data contains errors or noise (such as incorrect passenger counts for taxi orders), STDCformer is better at filtering out the noise.

To explore the physical meaning of the temporal and spatial confounders learned by the model, we analyzed the weights output from the spatial-temporal de-confounding layers. Here, $P\left(C_{S}\right)$ represents the weight of the spatial confounder in the STTs. Based on the observed phenomena, we hypothesize and provide an interpretation of the role and potential implications of the learned confounders. From the perspective of de-confounding, higher weights indicate that these confounders represent a lower proportion of the original data. Therefore, higher weights are needed to model an ideal unbiased distribution. From the perspective of spatial-temporal data fusion, higher weights suggest that, under the original data distribution, the model needs to pay more attention to the information embedded within this type of confounders.

1. 1.

   Observation 1: For all regions, $P\left(C_{S}\right)$ in the training dataset is no less than 0.5.

   Guess: The original data distribution leads the model to be more biased toward capturing temporal representations without a de-confounding strategy. Introducing higher weight of $P\left(C_{S}\right)$ allows the model to emphasize the role of spatial representations during the spatial-temporal fusion process.

   Interpretation: When predicting future crowd flow, the model considers the differences between regions and utilizes the position of each region within the global urban context for prediction. The learned ${P}\left({C}_{S}\right)$ in this work not only captures the spatial characteristics of regions but also includes the inter-region connectivity, which preserves the structural relationships between spatial regions from both an attribute and a positional perspective. On one hand, ${P}\left({C}_{S}\right)\geq 0.5$ highlights the importance of spatial information in spatial-temporal predictions, which is consistent with previous observations, such as the pre-experiment results from STGC-GNNs[50], which demonstrated the importance of spatial information, and the ablation study results from PDFormer[18], which also emphasizes this. On the other hand, ${P}\left({C}_{S}\right)\geq 0.5$ also implies that ${P}\left({C}_{S}\right)\geq{P}\left({C}_{T}\right)$, indicating that the temporal confounder has a higher weight in the original data distribution. This leads the model without de-confounding to rely more heavily on temporal features. The de-confounding process in this work adjusts this bias and corrects the overemphasis on temporal representations.

2. 2.

   Observation 2: When the IO value decreases, ${P}\left({C}_{S}\right)$ increases (as shown as Figure 18).

   Guess: The enhancement of ${C}_{S}$ can serve as an effective buffer when the functional characteristics of a region diminish.

   Interpretation: For a spatial region, a decrease in crowd flow suggests that its functionality is less being utilized, causing the region’s spatial portray increasingly vague. The differences between regions, as modeled by spatial confounders, are reduced. This leads the model to rely more heavily on temporal confounders, further amplifying the bias in the distribution of spatial and temporal confounders in the data. By explicitly incorporating spatial features as confounders, this work ensures that the spatial characteristics of each region remain relatively stable. Thus, when the weight of a region’s functionality in the observational data decreases, ${C}_{S}$ can counterbalance this by leveraging its interplay with ${C}_{T}$, preserving the utilization of spatial characteristics. Strengthening the use of spatial confounder’s information helps to retain the spatial information and prevent a decline in predictive accuracy.

   

   

   

3. 3.

   Observation 3: For regions in closer clusters, the values of their ${P}\left({C}_{S}\right)$ are also closer.

   Guess: The weight of ${C}_{S}$ for a region is significantly associated with its spatial characteristics and may serve as an indicator of the region’s functionality.

   Interpretation: As shown in Figure 20 and Figure 20, regions are clustered into six categories based on their POI features. It can be observed that regions with similar clustering results also exhibit closer average $P\left(C_{S}\right)$ values. The most likely reason is that these regions share similar functionality. To further explore this relationship, we selected several representative regions for detailed analysis.

   • •

     Roosevelt Island (Zone 194) and Randall’s Island (Zone 202). Both regions exhibit high $P\left(C_{S}\right)$ values and share similar functional zoning. Visualization of crowd flow data reveals that these regions are visited infrequently and often experience periods with no arrivals or departures. This is due to their highly singular and specialized functions, primarily providing recreational and leisure spaces. The confounding bias in the original data tends to downplay the importance of spatial characteristics for these regions. As a result, the model compensates by enhancing the spatial features for such areas, assigning them higher $P\left(C_{S}\right)$. Similarly, Highbridge Park (Zone 120) and The Battery Park (Zone 12 & Zone 13) also exhibit high $P\left(C_{S}\right)$ values due to their analogous characteristics. In contrast, Central Park (Zone 43), although functionally similar to these regions, has a relatively low $P\left(C_{S}\right)$ value. This can be attributed to its location near the center of Manhattan, which makes it highly accessible and surrounded by high human flow. As a result, the dependence of crowd flow on spatial functionality is reduced for this area.

   • •

     Midtown (e.g., Zone 164, Zone 48, Zone 68). The zones in midtown exhibit generally smaller $P\left(C_{S}\right)$, with values in the weekends lower than those on weekdays. This is due to the rich variety of functionalities in these areas, including numerous tourist attractions, department stores, and restaurants, which attract a large number of visitors. Consequently, the overall crowd flow is very high, and there is little difference between weekdays and weekends. For such regions, the overall spatial functionality is accessed relatively consistently, meaning there is less need to enhance the utilization of spatial characteristics. Instead, the model relies more on temporal trends for accurate predictions. A similar phenomenon can also be observed within the downtown area.

Once the window sizes for past and future time intervals are determined, the prediction task can be viewed as two sliding windows (past, value) along the time axis. Different windows can capture varying influences of the past on the future. For the same spatial region, the dependency between the future and the past also differs across time periods. Taking Zone 164 as an example, the cross-time mapping attention under different (past, future) windows is shown in Figure 21, where the gray box represents the past time window, and the green box represents the future time window. Since crowd flow in more recent time steps may be more similar, future time steps are likely to be more dependent on the last time step in the past window, which leads to assigning more attention to later time steps. When the future trend is very similar to the historical trend (Figure 21a) or completely dissimilar ((Figure 21b), the model tends to assign more attention to the most recent historical data. However, as shown in Figure 21c, as the prediction horizon increases, the similarity between the latter part of the future data and the historical data decreases. The model starts to focus more on the similarity of trends between past and future. Therefore, when the latter part of the future data shows a similar trend to some historical data segment, the model may capture this similarity and assign more attention to more distant historical data during prediction.

For different spatial regions during the same time period, the influence of the past on the future also varies. As shown in Figure 22, the values in the future window for Zone 125 exhibit relatively stable changes, remaining close to the value of the last historical time step. Consequently, the future time steps are more reliant on the values from the most recent historical time steps. While there is a similar trend of attention shifting toward more distant historical time steps, as observed earlier, the degree of this shift is less pronounced. These observations align with our hypothesis that the mapping relationships between the past and the future vary across different STTs.