Enhanced Urban Region Profiling with Adversarial Contrastive Learning

To capture both intra-view and inter-view region dependencies, we conduct message passing on the heterogeneous region graph $\mathcal{G}$. Our model integrates relation embeddings into the GCN for enhanced graph learning. Let $\mathcal{E}_{\mathbf{N}}^{(l)}=\left\{e_{1}^{(l)},e_{2}^{(l)},\ldots,e_{\mathbf{K}}^{(l)}\right\}$ and $H_{\Gamma}^{(l)}=\left\{h_{o}^{(l)},h_{t}^{(l)},h_{p}^{(l)},h_{g}^{(l)}\right\}$ denote the node and relation embeddings at the $l$-th layer, respectively. Here, $h_{o}$, $h_{t}$, $h_{p}$, and $h_{g}$ represent the relation embeddings for origin, target, POIs, and geographic neighbor relationships, respectively. Then, given the initial node embedding $\mathcal{E}_{\mathbf{N}}^{(0)}$ and relation embedding $H_{\Gamma}^{(0)}$, we update them at the $l-{th}$ layer as follows:

where $e_{u}^{(l)},e_{v}^{(l)}\in\mathcal{E}_{\mathbf{N}}^{(l)}$, $h_{r}^{(l)}\in H_{\Gamma}^{(l)}$, and $r$ denotes a specific relation type. Here, $\sigma(\cdot)$ represents the LeakyReLU activation function, and $\circ$ is the element-wise product. The learnable parameter for node embedding update in the $l$-th layer is denoted as $\mathbf{W}_{n}^{(l)}\in\mathbb{R}^{d\times d}$. Additionally, $\mathcal{N}_{u}^{\gamma}$ stands for the neighbor set of the region $r_{u}$ under relation type $\gamma$. To quantify the correlation between region vertex pair $\left(r_{i},r_{j}\right)$, we define the normalization weight $\phi_{u,\gamma}=\frac{1}{\left|\mathcal{N}_{u}^{\gamma}\right|}$, determined by the degrees of the vertices. Moreover, $\mathbf{W}_{\gamma}^{(l)}$ and $\mathbf{b}_{\gamma}^{(l)}$ are layer-specific parameters projecting relation embeddings from the previous layer to the same embedding space for subsequent use.

In addition, to mitigate feature smoothing in graph neural networks, we incorporate ResNet(He et al., 2016) into H-GCN for multi-layer graph aggregation. At the $l-$th layer, the aggregation process is expressed as follows:

where ${A}\in\mathbb{R}^{|\mathbf{K}|\times\mathbf{K}|}$ denotes the adjacency matrix, and ${D}\in\mathbb{R}^{|\mathbf{K}|\times\mathbf{K}|}$ is the corresponding diagonal matrix. The geographic neighbor similarity and adjacency matrices $\mathcal{S}_{n}$ are constructed based on neighbor relationships. To build the adjacency matrix based on human mobility data and POIs, we follow (Zhang et al., 2020) to build the POI similarity matrix $\mathcal{S}_{p}$, mobility origin similarity matrix $\mathcal{S}_{o}$ and mobility target similarity matrix $\mathcal{S}_{t}$. Subsequently, we employ the Top-$k$ method with $k$ set to 10 to construct these three adjacency matrices. Finally, we obtain the relation-focused region embeddings $\mathcal{E}_{N}=\left\{\mathcal{E}_{N}^{o},\mathcal{E}_{N}^{t},\mathcal{E}_{N}^{p},\mathcal{E}_{N}^{g}\right\}$ and the relation embeddings $H_{R}=\left\{h_{o},h_{t},h_{p},h_{g}\right\}$ under each relation type.

To autonomously generate effective positive samples based on the specificity of urban region embeddings for subsequent contrastive learning preparation, we perturb the region embeddings in latent space. In particular, given the region representations from $U$ relation-focused region embeddings as $\left\{\mathcal{E}_{N}^{i}\in\mathbb{R}^{|\mathbf{K}|\times d}\right\}_{i=1}^{U}$, we use $\mathcal{X}=\left\{\mathcal{X}_{o},\mathcal{X}_{t},\mathcal{X}_{p},\mathcal{X}_{g}\right\}$ to denote the positive example, where $\mathcal{X}_{i}\in\mathbb{R}^{|\mathbf{K}|\times d}$. Formally,

where $\Delta\boldsymbol{\theta}_{sp}$ is sampled from a Gaussian distribution with zero mean and standard deviation ${\sigma}$. $\eta$ represents the scale factor for perturbation magnitude, with $\sigma$ and $\eta$ serving as hyper-parameters. The result $\mathcal{X}$ is then fed into CM-sharing to obtain spatial learnable augmentation positive samples (LA-Pos), denoted as $\widetilde{\mathcal{X}}=\left\{\widetilde{\mathcal{X}}_{o},\widetilde{\mathcal{X}}_{t},\widetilde{\mathcal{X}}_{p},\widetilde{\mathcal{X}}_{g}\right\}$.

Generally, different attributes within the same region are interrelated. For instance, factors like unemployment, poverty, and educational levels in neighboring areas may influence crime rates in certain regions. To capture global information, we employ the multi-head self-attention mechanism(Vaswani et al., 2017). Formally, given the region representations from $U$ relation-focused region embeddings as $\left\{\mathcal{E}_{N}^{i}\in\mathbb{R}^{|\mathbf{K}|\times d}\right\}_{i=1}^{U}$. For each representation $\mathcal{E}_{N}^{i}$, we associate a key matrix $K_{i}\in\mathbb{R}^{|\mathbf{K}|\times d}$ , a query matrix $Q_{i}\in\mathbb{R}^{|\mathbf{K}|\times d}$ and a value matrix $V_{i}\in\mathbb{R}^{|\mathbf{K}|\times d}$ with it as follows:

where $\mathbf{W}_{i}^{Q}$, $\mathbf{W}_{i}^{K}$, and $\mathbf{W}_{i}^{V}$ are learnable parameters for the $i$-th head. Then, compute multi-head self-attention by the following equation:

where $\mathbf{W}_{O}$ is the learnable parameter for transformation, $\|$ denotes concatenation, and $\mathcal{H}$ is the number of heads. So far, we obtain $\mathcal{E}^{\prime}=\left\{\mathcal{E}_{o}^{\prime},\mathcal{E}_{t}^{\prime},\mathcal{E}_{p}^{\prime},\mathcal{E}_{g}^{\prime}\right\}$ which is considered as the relevant global region information. Subsequently, we introduce the learning-based linear interpolation to improve the embedding, formulated as follows:

where $\mathcal{E}_{i}\in\mathcal{E}$, $\mathcal{E}_{i}^{\prime}\in\mathcal{E}^{\prime}$, and $\mathbf{c}_{i}$ denotes the learning parameters. Finally, we obtain the relation-aware region embedding, denoted by $\widetilde{\mathcal{E}}=\left\{\widetilde{\mathcal{E}}_{o},\widetilde{\mathcal{E}}_{t},\widetilde{\mathcal{E}}_{p},\widetilde{\mathcal{E}}_{g}\right\}$. To ensure the model’s generalization ability, dropout layers are further employed.

Considering that the strong distinguishable properties of different region embeddings hinder the effectiveness of contrast training, inspired by (Suresh et al., 2021), stronger divergence and adversarial graph augmentation can improve contrast learning. As a result, we propose adversarial perturbations to generate hard negative pairs and hard positive pairs. This strategy facilitates overcoming noise while extracting high-level semantics in region embeddings, thus ensuring strong generalization. The studies suggest stronger divergence and adversarial graph enhancements improve contrast learning. The adversarial perturbation layer comprises the Trickster Generator and Deviation Copy Generator, with the detailed implementation processes depicted in Figure. 3.

The Trickster Generator aims to produce hard negative samples, denoted as ${\mathcal{X}}_{T}=\left\{\mathcal{X}_{T}^{o},\mathcal{X}_{T}^{t},\mathcal{X}_{T}^{p},\mathcal{X}_{T}^{g}\right\}$, that exhibit proximity to anchor $\widetilde{\mathcal{E}}$ in the latent embedding space while maintaining semantic distinctions. In particular, $\left\{\mathcal{X}_{T}^{i}\in\mathbb{R}^{|\mathbf{K}|\times d}\right\}_{i=1}^{U}$ is derived by introducing a slight perturbation $\delta$ to the anchor $\left\{\widetilde{\mathcal{E}}_{i}\in\mathbb{R}^{|\mathbf{K}|\times d}\right\}_{i=1}^{U}$, where the Euclidean norm of $\delta$ is constrained within the limit of $\epsilon$. Simultaneously, to alter the inherent semantics as extensively as feasible, we aim to minimize the conditional likelihood of ${\mathcal{X}}_{T}^{i}$ to the original similarity matrix $\left\{\mathcal{S}_{i}\in\mathbb{R}^{|\mathbf{K}|\times d}\right\}_{i=1}^{U}$. Formally,

where $\delta$ has the same size with relation-aware region embedding $\mathcal{E}_{N}^{i}$. Due to the intractability of exact minimization for neural networks, we implement it as follows:

Algorithm  1 presents the trickster generator’s pseudo code. ${\mathcal{X}}_{T}$ is an adversarial sample that puts pressure on the urban region embedding contrastive learning task, enhancing the robustness and generalization of the model.

In addition, we propose the Deviation Copy Generator intending to generate strong positive samples ${\mathcal{X}}_{D}=\left\{\mathcal{X}_{D}^{o},\mathcal{X}_{D}^{t},\mathcal{X}_{D}^{p},\mathcal{X}_{D}^{g}\right\}$, referred to as DevCopy. It is intended to exhibit semantics similar to the anchor $\widetilde{\mathcal{E}}$ while maintaining a far distance in the latent space. The relationship between them is illustrated in Figure 4. Specifically, a well-suited perturbation $\xi\in\mathbb{R}^{|\mathbf{K}|\times d}$ is introduced to $\widetilde{\mathcal{E}}$ to minimize its cosine similarity with $\widetilde{\mathcal{E}}$. However, the exact computation of $\eta$ under these constraints is intractable. To address this, we use two computational phases to approximate it. In the first stage, a perturbation is added to $\left\{\widetilde{\mathcal{E}}_{i}\in\mathbb{R}^{|\mathbf{K}|\times d}\right\}_{i=1}^{U}$ to minimize the contrastive learning loss $\mathcal{L}_{\text{cont }}$ (refer to Eq.21, where $\widetilde{\mathcal{E}}$ can be viewed as $\boldsymbol{e}_{x}$ and $\widetilde{\mathcal{X}}$ as $\boldsymbol{\bar{e}}_{x}$). Formally,

In the second step, we aim to maintain semantics by minimizing the KL divergence between the conditional likelihood $P(\mathcal{S}_{i}\mid\check{\mathcal{X}}_{i})$ and $Q(\mathcal{S}_{i}\mid\widetilde{\mathcal{E}}_{i})$, pulling them closer together. Formally,

Algorithm  2 presents the Deviation Copy Generator’s pseudo code. Finally, we also applied linear projections to ${\mathcal{X}}_{T}$ and ${\mathcal{X}}_{D}$ to obtain better results by applying separate projections(Xie et al., 2023) to the representations of different types of nodes.

In the contrastive learning layer, we utilize the InfoNCE Loss (van den Oord et al., 2019) to maximize the similarity between the anchor and DevCopy, while minimizing the similarity between the anchor and Trickster:

where $\varphi(i,j)=\exp(\operatorname{cosine}(i,j)/\tau)$ measures the correlation between two embeddings, $\bar{e}_{x}$ represents the spatially augmented positive sample (i.e., LA-Pos), and $Z$ is a set of randomly selected negative samples from the same batch.

As explained previously, incorporating the contrastive learning framework into urban region embeddings makes no sense if random non-targeted region representations are utilized as negative examples to train the model, since these fictitious negative examples are located far away from the positive examples in the embedding space, even if the positive examples are semantically irrelevant to the anchor. To make the contrast process more effective in guiding the training of the model, we additionally used the generated ”hard” negative and positive examples to optimize the contrastive learning loss. Firstly, we design the Trickster contrastive loss by introducing the Trickster ${\mathcal{X}}_{T}$ as an extra negative example to Eq. 21. Formally,

where $Z^{\prime}$ denotes $Z\cup\left\{{\mathcal{X}}_{T}\right\}$. We further enhance the training process by adding the hard positive sample ${\mathcal{X}}_{D}$ as follows,

Finally, the total contrastive loss is expressed as:

where $\alpha$ is a hyper-parameter.

In check-in and crime prediction tasks, we predict the number of crime and check-in events in each region for one year with the learned region embeddings. In practice, we apply the Lasso regression (Tibshirani, 1996) with metrics of Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and coefficient of determination ($R^{2}$) to measure the performance of the models. Table 3 reports the comparison results of the check-in and the crime prediction task. In particular, our approach improves MAE, RMSE, and $R^{2}$ in the check-in prediction task by 6.8%, 2.9%, 2.6%, and in the crime prediction task by 10.8%, 8.2%, 7.3% on dataset compared to the HREP approach which highlights the strength of our joint attentive supervised and contrastive learning framework. This framework encourages the encoder to extract region nonlinear dependency and capture high-level semantics in urban region embeddings that suffer from noise problems.

In contrast, traditional graph embedding methods (i.e. LINE, node2vec) perform poorly because of the local sampling approach, which can not fully express the relationships between nodes. The deep learning methods perform better than the traditional graph embedding methods in all three data sources. Models such as HDGE, ZE-Mob, and MV-PN utilize multi-scale graph structures and embedding methods to capture the multi-level features and complex relationships within urban areas. They do not consider the varying importance of nodes for modeling dependencies, which influences the performance. Furthermore, graph representation methods such as MVGRE, MGFN, and ROMER perform better than others. As they employ a multi-view fusion and attention mechanism that captures the dependencies between nodes, allowing the model to strike a good balance between local perception and information aggregation. While the HRPE model has demonstrated promising results by incorporating prompt learning to guide downstream tasks, none of the aforementioned models consider the impact of noise and incompleteness in urban region data on region representation learning.

We employ K-means to cluster the region embeddings and present the outcomes visually for intuitive interpretation. Fig. 6(a) illustrates that the community boards (Berg, 2007) partition the Manhattan administrative district into 12 regions based on land use. Accordingly, we split the study area into 12 clusters as well. The clustering outcomes should group regions with identical land use types. To further measure the region clustering results, inspired by (Yao et al., 2018), we use normalized mutual information (NMI) and adjusted Rand index (ARI). Figure 5 reports the comparison results of the land use classification task. Compared to the ROMER approach, more than 3.7% improvement in NMI and more than 4.4% improvement in ARI are achieved in the land classification task.

For visual evaluation of the land use classification task, we depict the clustering results of five baselines and our model in Fig. 6, where regions in the same cluster are marked with the same color. Notably, the clustering results obtained by our method show the most ideal degree of consistency with the real boundaries of the ground conditions. This observation suggests that the region embeddings learned by our model better represent regional functions. The reason is that our method performs well at exploiting high-level semantics in noisy or incomplete urban data, capturing universal human mobility patterns, and generating accurate urban region embeddings.

To better verify the role of each component, this paper conducts an ablation study in land usage classification, check-in prediction tasks, and crime prediction. We have designed the variants of EUPAC in the following way:

• •

  w/o Spatial Augmentation (w/o SA): The spatial learnable augmentation layer using the replacement data augmentation approach to generate positive samples.

• •

  w/o Adversarial Contrastive Learning (w/o ACL): We remove the adversarial contrastive module from EUPAC.

• •

  w/o Attentive Supervised Learning (w/o ASL): We remove the attentive supervised module from EUPAC.

• •

  w/o Trickster (w/o T): ${\mathcal{X}}_{T}$ is removed from EUPAC, with all other settings unchanged.

• •

  w/o DevCopy (w/o D): ${\mathcal{X}}_{D}$ is removed from EUPAC, with all other settings unchanged.

Fig. 7 shows the ablation results of EUPAC and its variants. These results highlight the critical importance of effective data augmentation for graph contrastive learning. Without our proposed spatial learnable augmentation layer, the performance is worse than when adversarial contrastive learning is excluded. This indicates that the model’s robustness is significantly improved by adding appropriate spatially augmented perturbations to the latent space. Additionally, using only adversarial contrastive learning yields unsatisfactory results, underscoring the necessity of our framework, which combines attentive supervised and adversarial contrastive learning. Moreover, the model’s discriminative performance is enhanced by dynamically adding Trickster and DevCopy. As training progresses, the difficulty of augmented samples increases, further strengthening the model’s discriminative ability. Finally, the integration of these modules in our final model produces the best outcomes.

To further validate the contribution of our proposed adversarial contrastive learning in addressing noise issues, rather than merely complicating the structure, we added noise to the data through replacement and compared the performance of different methods under the influence of noise in downstream tasks. As shown in Table 4, although some of the state-of-the-art methods in recent years are structurally simpler, they perform very poorly when confronted with noisy data, with ROMER’s performance decreasing by approximately 9%. In contrast, our EUPAC model is very robust, with a performance degradation of only about 2.5%. Additionally, the ablation results indicate that within the adversarial contrastive module, the Trickster Generator exhibits stronger noise resistance than the Deviation Copy Generator.

We systematically tune important hyperparameters in EUPAC. The weighting parameter $\alpha$ in Eq. 24 affects the sum of positive and negative sample losses in contrastive learning, which we explored within $\{0.5,0.6,0.7,0.8,0.9,0.95\}$, with $\alpha$=0.50 yielding the best performance. This balance turns out to be crucial for effective contrastive learning within our designed adversarial self-supervised module. For the joint optimization parameter $\beta$ in Eq. 25, varying within $\{0.05,0.1,0.15,0.2,0.3,0.4,0.5\}$, the model excels with $\beta$ = 0.15, highlighting the dominant role of contrastive learning in mining high-level region semantics from noisy urban data.