Boosting House Price Estimations with Multi-Head Gated Attention

In the intricate landscape of data science, similarity is a critical underpinning for various algorithms and methodologies. This sub-subsection aims to illuminate the key metrics ubiquitously employed to quantify similarity, laying the groundwork for the following analyses.

• •

  Euclidean Distance: A foundational metric in geometry, Euclidean distance provides a straightforward measure of similarity by calculating the straight-line distance between two points in an Euclidean space.

  $$d(P_{1},P_{2})=\sqrt{(x_{2}-x_{1})^{2}+(y_{2}-y_{1})^{2}}$$

  (1)

• •

  Cosine Similarity: This metric is invaluable in high-dimensional spaces, measuring the cosine of the angle between two vectors. It is especially pertinent in text analysis and natural language processing.

  $$\text{Cosine Similarity}=\frac{C\cdot D}{\|C\|\times\|D\|}$$

  (2)

• •

  Jaccard Index: A set-based metric, the Jaccard Index is helpful for categorical data, quantifying the ratio of the intersection to the union of two sets.

  $$J(C,D)=\frac{|C\cap D|}{|C\cup D|}$$

  (3)

• •

  Identity Similarity: This is a binary similarity measure used to ascertain whether or not two data points are identical. Unlike continuous similarity measures, the Identity Similarity scores one if the data points are similar and 0 if they differ. This measure is handy in scenarios requiring exact matching or where data is categorical. Mathematically, it is expressed as:

• •

  Gaussian Kernel: Also known as the Radial Basis Function (RBF) with Gaussian form, this metric is a cornerstone in non-linear data transformations. Unlike other metrics that measure distance directly, the Gaussian Kernel calculates similarity by mapping the original data points into a higher-dimensional space through a Gaussian function. This allows it to capture complex, non-linear relationships between data points. Mathematically, it is expressed as:

  $$K(x,y)=\exp\left(-\frac{\|x-y\|^{2}}{2\sigma^{2}}\right)$$

  (4)

  The parameter $\sigma$ controls the spread of the Gaussian function, thereby influencing the similarity measure. A smaller $\sigma$ will result in a narrower Gaussian function, making the similarity measure more sensitive to the distance between data points.

These metrics serve as the backbone for various algorithms and offer a nuanced understanding of how data points relate to each other in complex spaces, with the Gaussian Kernel standing out for its ability to capture non-linear relationships.

Spatial interpolation is a critical technique for predicting unknown values at unobserved locations based on known values at observed locations, finding applications in diverse fields such as geostatistics, environmental science, and real estate. The effectiveness of spatial interpolation is intrinsically tied to the choice of similarity measures. For instance, Euclidean distance can be employed in a straightforward approach like ”inverse distance weighting” (IDW) [44], where the influence of a neighbouring point on the interpolated value is inversely proportional to its Euclidean distance from the target location. On the other hand, the Gaussian Kernel [45] offers a more nuanced approach by transforming the Euclidean distance into a measure of similarity, thereby capturing complex, non-linear spatial relationships. This is especially useful in advanced geostatistical methods like kriging [46]. Therefore, the choice between straightforward measures like Euclidean distance and more complex ones like the Gaussian Kernel can significantly impact the quality of spatial interpolation, exemplifying the broader applicability and importance of similarity measures in data science.

Attention mechanisms [47] has emerged as a cornerstone in many deep learning models, predominantly in sequence-to-sequence tasks such as machine translation and speech recognition. The essence of Attention is to emulate the human ability to focus on specific segments of input data, much like how we selectively concentrate on some aspects of a visual scene or a conversation. Among the diverse attention mechanisms, Soft Attention is a mechanism that computes a weighted sum of all input values. These weights, indicative of the relevance of each input, are typically determined using a softmax function, ensuring a normalised distribution where the weights sum up to one. The continuous nature of these weights makes soft Attention inherently differentiable, rendering it particularly amenable to gradient-based optimisation techniques [48]. On the other hand, intricate Attention operates more selectively. Instead of distributing focus across all inputs, it zeroes in on a specific subset, effectively sidelining the others. Given its discrete selection process, traditional backpropagation struggles with optimising intricate Attention. Yet, this challenge is surmountable with techniques like the reinforce algorithm [49]. The Gated Attention mechanism [50] bridges the gap between these two. It adeptly amalgamates information from diverse sources and employs gating tools to ascertain the relevance of each source. This approach can be perceived as a harmonious blend of the soft and hard attention paradigms, encapsulating the strengths while mitigating their limitations [51].

The Geo Multi-head, Gated Attention mechanism is intricately designed to capture the spatial relationships between a target house and its neighbouring properties. This involves using a Gaussian kernel function to calculate geographic similarity scores between the target house and its neighbours. Equation 5 demonstrates how the geographic score between the target house $G_{i}$ and its neighbouring house $G_{i,j}$ is computed using the Gaussian kernel function.

Here, $\rho=\frac{\sigma^{2}}{2}$ and $\text{geo\_dist}(G_{i},G_{i,j})$ represents the geodesic distance between $G_{i}$ and $G_{i,j}$. The vector of similarity scores $L$ is then transformed into a hidden representation $H^{\prime}$ through a fully-connected layer, as described in Equation 6:

In this equation, $W^{\prime}$ and $b^{\prime}$ are the learned weights and bias terms, respectively. The attention weights $a_{\text{geo}}$ are computed using a softmax layer, as formulated in Equation 7:

Then, using our defined gated attention mechanism (Equation 8), we apply it to the attention weights:

Subsequently:

Where:

• •

  $x$ is the input value, in this case, the original attention weight $a_{\text{geo}}(G_{i},G_{i,j})$.

• •

  $W_{g}$ represents the learned weight matrix associated with the gate.

• •

  $b_{g}$ denotes the bias term.

• •

  $\sigma$ is the sigmoid function, ensuring the output value of the gate lies in the [0,1] range.

With this, the Geo Gated Attention Vector $v_{\text{ggeo}}(G_{i})$ is computed as a weighted sum of the features of the neighbouring houses using the modified attention weights $a^{\prime}_{\text{geo}}$:

In this equation, $\Delta d_{i,j}$ represents the geographic distance between house $i$ and its neighbour $j$. Similarly, $y_{i,j}$ signifies the price of the neighbor $j$, and $\oplus$ denotes the concatenation operation. The dimensionality of $v_{\text{geo}}(G_{i})$ is derived from the summation of dimensions where $G_{i,j}\in\mathbb{R}^{2}$, $A_{i,j}\in\mathbb{R}^{T}$, $\Delta d_{i,j}\in\mathbb{R}^{1}$, and $y_{i,j}\in\mathbb{R}^{1}$. Consequently, the vector $v_{\text{geo}}(G_{i})$ can be viewed as a weighted sum of vectors $G_{i,j}$, concatenated with $\Delta d_{i,j}$ and $y_{i,j}$, and weighted using the normalised geo gated attention coefficients which are determined during the training process.

The Euclidean Multi-head Gated Attention mechanism is precisely engineered to emphasise the most relevant structural similarities between a target house and its neighbouring properties. This mechanism employs the Euclidean distance to compute the similarity scores between the target house and its neighbours. The Euclidean distance between the target house $A_{i}$ and a neighboring house $A_{i,j}$ is computed as shown in Equation 11:

where $d(A_{i},A_{i,j})$ is the Euclidean distance indicating similarity between houses based on structural attributes, $A_{i}$ represents the structural features of the target house $i$, $A_{i,j}$ denotes the structural features of the $j^{th}$ neighboring house to $i$, $a_{i,p}$ and $a_{i,j,p}$ are specific structural attributes of houses $i$ and $j$, respectively, and $T$ is the total number of structural attributes considered.

After computing the Euclidean distances, we construct a vector of similarity scores $L$, which is then transformed into a hidden representation $H$ through a fully-connected layer, as described in Equation 12:

In Equation 12, $W$ and $b$ are the learned weights and bias terms, respectively. The attention weights $a_{\text{euc}}$ are computed using a softmax layer, as formulated in Equation 13:

The essence of the gated attention mechanism is to refine the attention weights by introducing an additional modulation step. This modulating factor, or ”gate”, is typically represented as a value between 0 and 1 and is applied element-wise to the attention weights. The purpose is to amplify or diminish the original attention values based on the model’s learned parameters.

Given this, the gated attention can be defined as:

Where:

• •

  $x$ is the input value, in this case, the original attention weight $a_{\text{euc}}(A_{i},A_{i,j})$.

• •

  $W_{g}$ represents the learned weight matrix associated with the gate.

• •

  $b_{g}$ denotes the bias term.

• •

  $\sigma$ is the sigmoid function, ensuring the output value of the gate lies in the [0,1] range.

Subsequently, the gated attention mechanism can be formalised as:

Here, $\odot$ denotes element-wise multiplication. Thus, the attention weight is modulated by its gating value, allowing the model to allocate attention more selectively to houses exhibiting the most congruent features.

The Vector with Euclidean Gated Attention denoted as $v_{\text{geuc}}(A_{i})$, represents a cumulative weighted mix of attributes from the surrounding homes. This process uses the gated attention coefficients $a^{\prime}_{\text{euc}}$ and is illustrated in Equation 16:

Within Equation 16, $y_{i,j}$ defines the price of the $j^{th}$ neighboring home of house $i$, while $\oplus$ denotes the concatenation action. The size of $v_{\text{euc}}(A_{i})$ stands at $T+1$ given that $A_{i,j}$ resides in $\mathbb{R}^{T}$ and $y_{i,j}$ is part of $\mathbb{R}^{1}$. The composition of $v_{\text{euc}}(A_{i})$ involves initially multiplying the combined vector $[A_{i,j}\oplus y_{i,j}]$ for each $j^{th}$ neighbor of house $i$ by its respective gated attention coefficient $a^{\prime}_{\text{euc}}(A_{i},A_{i,j})$, producing an individual weighted vector for every $j^{th}$ neighbor. An overall summation is then applied to these vectors for all $n$ adjacent homes to house $i$. Consequently, the elements within $v_{\text{euc}}(A_{i})$ represent a comprehensive weighted sum of the structural attributes and the valuations of the nearby homes of house $i$. The gated attention coefficients undergo refinement during the learning phase.

The final aggregation stage shown in Figure 2 E involves collecting and combining the attention vectors from each head of the attention mechanism and applying the gated attention based on the normalized gates weights and biases. It is important to note that this process is unique for each attention mechanism, namely Geo and Euclidean, and it results in the formation of two separate aggregated attention vectors.

To ensure the effectiveness of the attention mechanism in both Geo and Euclidean interpolation, it is crucial to normalise the gating weights and biases using a softmax function, as shown in Equation 17. By normalizing the gating weights and biases, they fall within the range of 0 to 1, which makes them more easily interpretable.

After normalising the gating weights and biases, we perform element-wise multiplication with each attention and then aggregate them. The resulting vector that shows the aggregated gated geographic attention, denoted as $v_{\text{agg\_ggeo}}$, is presented in Equation 18.

Where $\text{gate}_{\text{norm, geo},i}$ represents the softmax-normalized gating weights and biases, and $v_{\text{geo},i}$ refers to the attention vectors from the Geo attention heads.

In a similar vein, the aggregated gated Euclidean attention vector $v_{\text{agg\_geuc}}$ is represented by Equation 19:

Here, $\text{gate}_{\text{norm, euc},i}$ signifies the softmax-normalized gating weights, and $v_{\text{geuc},i}$ portrays the gated attention vectors emergent from the Euclidean attention heads.

In conclusion, the consolidated Geo attention vector $v_{\text{agg\_ggeo}}$ and the Euclidean attention vector $v_{\text{agg\_geuc}}$ are computed using an element-wise multiplication between the softmax-normalized gating weights and their corresponding attention vectors as illustrated in Figure 2 F derived from the Geo and Euclidean attention heads, respectively. This approach ensures an accurate integration of the significance associated with each feature and reflects the complex spatial relationships inherent within the Geo and Euclidean contexts.

After training, the model was evaluated using standard regression metrics such as Root Mean Squared Error (RMSE) and Mean Absolute Error (MALE). These metrics serve specific purposes in assessing the model’s performance:

• •

  RMSE (Root Mean Squared Error): Provides a measure of the model’s prediction error, penalising more significant errors more severely than smaller ones. It is advantageous when significant errors are undesirable in the prediction task.

• •

  MALE (Mean Absolute Logarithmic Error): This metric expresses the average magnitude of the relative errors between predicted and actual values while disregarding their direction. It is beneficial when dealing with exponential growth, or underestimation is more critical than overestimation.

These metrics collectively offer a comprehensive evaluation of the model’s performance in predicting house prices, allowing for the assessment of the model’s accuracy and its goodness of fit to the actual data.

Table 3 provides an exhaustive evaluation of multiple machine-learning models In an exhaustive evaluation of machine learning models on real estate datasets from Italy (IT), King’s County (KC), Porto Alegre City in Brazil (POA), and Beijing (BJ), the best performance was consistently demonstrated by XGBoost (XGB). Specifically, XGB recorded the best MALE values of 0.1350 in IT, 0.1160 in KC, 0.1613 in POA, and 0.0723 in BJ. Notably, the average performance for XGB was stable and closely aligned with these best values, indicating high reliability across diverse geographic datasets. CatBoost and LightGBM also performed strongly, closely trailing XGB in each dataset. For instance, CatBoost had the best MALE values of 0.1362 in IT, 0.1131 in KC, 0.1793 in POA, and 0.0782 in BJ. LightGBM posted the best MALE deals of 0.1381 in IT, 0.1164 in KC, 0.172 in POA, and 0.0790 in BJ. The average performances of CatBoost and LightGBM were also impressively stable and nearly matched their respective best values. Conversely, Support Vector Machines (SVM) significantly underperformed, with its best MALE values being 0.4072 in IT, 0.1331 in KC, 0.2232 in POA, and a dismal 0.2234 in BJ. K-Nearest Neighbors (KNN), a traditional algorithm, also lagged, particularly in the BJ dataset, where it posted a best MALE of 0.1116. In summary, XGB takes the lead across all datasets regarding best and average performance metrics, closely followed by CatBoost and LightGBM, which also show highly stable average performances. Conversely, SVM and traditional models like KNN are less effective, particularly in complex, geographically diverse datasets.

We conducted a series of tests to evaluate our model and compare it with ANN and ASI models. The dataset was split into three subsets, with 70% assigned for training, 20% for testing, and 10% for validation. Table 4 shows the performance metrics of our model, ASI, and ANN, compared across four different datasets from various locations: Italy (IT), King’s County (KC), Porto Alegre (POA), and Beijing (BJ).

Our model performed better than the ASI model in the IT dataset. It has a superior MALE that’s approximately 1.52% lower and an RMSE that’s approximately 0.36% lower, with figures of 0.1312 and 45,797, respectively. These results highlight our model’s ability to interpret the IT dataset more accurately.

In the KC dataset, our model’s robustness shines with a notable 13.3% improvement in RMSE compared to the ASI model. This translates to a lower RMSE of approximately 107,993 for our model compared to ASI’s 124,557. This underscores our model’s consistent efficiency across various geographical contexts.

Our model also outperforms the ASI model in the POA and BJ datasets, demonstrating its versatility and accuracy in handling diverse data types. In the POA dataset, our model achieves a MALE of approximately 1.44% lower and an RMSE of approximately 13.45% lower compared to ASI. In the BJ dataset, our model achieves a MALE of approximately 2.67% lower and an RMSE of 2.31% lower compared to ASI. These metrics align with those observed in the IT and KC datasets, reaffirming the broad applicability of our model.

Our model incorporates Multi-Head Gated Attention mechanisms, which enable it to assimilate diverse spatial cues between houses and their surroundings. This fosters more precise and robust predictions as our model comprehensively grasps spatial dynamics.

In conclusion, our model has proven superior to the ASI model across all evaluated metrics and datasets. The advanced Multi-Head Gated Attention architecture plays a pivotal role in aggregating contextual cues, ultimately enhancing overall predictive accuracy.

In our experiment, we wanted to see how custom house embeddings generated by our Multi-Head Gated Attention model would affect the performance of various baseline machine learning models. These embeddings were created based on structural and geographical information and enhanced the feature space for algorithms like Linear Regression, KNN, Decision Tree, Random Forest, SVM, LightGBM, CatBoost, and XGBoost. We evaluated the models’ performance using four different geographical datasets: Italy (IT), King’s County (KC), Porto Alegre (POA), and Beijing (BJ), and assessed the Best and Average MALE and RMSE scores.

Our results showed that our custom embeddings significantly positively impacted the predictive performance of the baseline models. For example, when the CatBoost model was augmented with our custom embeddings, it achieved the lowest RMSE score in the IT dataset at 45,708, outperforming even our original Multi-Head Gated Attention model. However, we found that the improvement magnitude was inconsistent across all datasets. The IT dataset, which combines data from various cities with significant geographical and Euclidean distances between them, showed only a modest enhancement of around 1.3% in RMSE when deploying CatBoost with custom embeddings compared to the baseline.

We discovered that the unique spatial complexities inherent in each dataset could impact the effectiveness of the custom embeddings. For instance, in the KC dataset, CatBoost with custom embeddings demonstrated significant gains over its baseline, whereas, in IT, the improvements were more restrained. We also found that even simpler models like Linear Regression could benefit substantially from the enriched feature space the embeddings provide. In the IT dataset, the best MALE improved by approximately 65.8%, the average MALE improved by approximately 66.0%, the best RMSE improved by approximately 39.8%, and the average RMSE improved by approximately 39.8%.

In the CatBoost model for the IT dataset, the best MALE improved by approximately 2.4%, and the average MALE improved by approximately 2.9%. The best RMSE improved by approximately 0.5%, and the average RMSE improved by approximately 1.0%. This indicates a positive trend in reducing MALE and RMSE values, which is crucial for achieving better model performance in predictive tasks like house price prediction.

In the KC dataset, the implementation of custom embeddings reflected varying degrees of improvement across different machine-learning models. The CatBoost model illustrated an enhancement in the best MALE value by approximately 5%, although the average MALE value experienced a minor deterioration by approximately 0.44%. On the brighter side, a more noticeable improvement was observed in the RMSE values, where the best RMSE value improved by approximately 8.20%, and the average RMSE value improved by approximately 8.04%.

The POA dataset manifested a significant leap in performance metrics upon integrating custom embeddings. Specifically, the CatBoost model, when augmented with custom embeddings, demonstrated a robust improvement in both MALE and RMSE values. The best MALE value improved by an impressive margin of approximately 23.77%, while the average MALE value improved by approximately 22.66%. Concurrently, the RMSE metrics also exhibited substantial enhancements, with the best RMSE value improving by approximately 13.40%, and the average RMSE value improving by approximately 13.45%.

In the BJ dataset, we observed that models trained on embeddings generally perform better on average values, reflecting a more consistent performance across varying data points. However, the best values achieved in MALE and RMSE metrics were slightly better when models were trained on original data. This suggests that while embeddings generally enhance model performance, there might be specific instances or datasets where traditional feature sets could yield better or comparable results.

The present study introduces a Multi-Head Gated Attention model that exhibits superior performance compared to baseline models when applied to various datasets, including IT and POA. This model utilizes distinct weights and biases within each attention head to capture various contextual relationships within the data, showcasing its exceptional capabilities in spatial interpolation tasks. This approach provides a more comprehensive understanding of the underlying spatial dynamics. Our model’s multi-head gated attention mechanism exceeds traditional singular attention approaches by integrating various spatial and structural features from the data. This integration is essential as it moderates the influence of outliers, which is expected in a vast and diverse metropolis like Beijing, where extreme data points can skew the analysis. By employing this sophisticated mechanism, the model ensures the delivery of accurate and nuanced house price predictions that genuinely reflect the complex intricacies of Beijing’s housing market, setting a new benchmark for robustness and reliability. The box plots in Figure 3(a,b) effectively illustrate each dataset’s spatial and structural features. Specifically, Figure 3(b) reveals that Kings County (KC) has a compact urban form, indicated by a median geodesic distance of just under 0.65 km, which is also supported by a low median normalized Euclidean distance shown in Figure 3(a), highlighting high structural homogeneity among houses. In contrast, Beijing (BJ) portrays a more dispersed housing structure with a median geodesic distance of approximately 0.45 km, as indicated in Figure 3(b), and a median normalized Euclidean distance of approximately 0.150, as shown in Figure 3(a). These distances indicate a significant variation in structural features, suggesting a housing landscape that includes densely packed urban areas and more spread-out suburban or peri-urban zones. The Italian (IT) region demonstrates a median geodesic distance of around 0.50 km, reflecting less uniformity and greater architectural diversity, as further evidenced by a median normalized Euclidean distance of around 0.110. Moving to Porto Alegre (POA), the dataset displays a distinctive spatial composition, with a median geodesic distance that suggests moderately dense housing and a median normalized Euclidean distance of approximately 0.100. This places POA in a unique position between the densely packed environment of KC and the varied spatial arrangements of BJ and IT. The moderate variation in POA’s housing structures signifies an urban design that merges densely built areas with open suburban spaces, reflecting its rich historical development and cultural diversity. Employing the multi-head gated attention mechanism for the POA dataset allows for an in-depth exploration of the city’s complex architectural styles and spatial dynamics. When juxtaposed with the consistent architecture of KC and the diverse spatial distributions of BJ and IT, our model’s multifaceted approach yields a deep understanding of the nuances within POA’s urban clusters and the distinctive nature of its rural homes. As a result, our model stands out as a sophisticated and precise analytical tool, uniquely equipped to navigate and predict the intricate dynamics of the housing market with extraordinary accuracy and insight.

The improvements highlighted in Table 4 emphasises the progress made by our model compared to the ASI model. Our model achieved improvements of 1.35% and 1.46% in MALE and RMSE, respectively, for the IT dataset, 1.79% and 13.34% for the KC dataset, 2.16% and 1.92% for the POA dataset, and 2.67% and 1.73% for the BJ dataset. These results demonstrate the superiority of our model across different datasets and spatial configurations. The multi-head gated attention mechanism played a significant role in achieving these improvements. It captures diverse contextual relationships within the data by leveraging weights and biases in each head, especially when dealing with regions with a more varied architectural landscape and pronounced geographical diversity. The improvements in the KC dataset are significant, as it has a high degree of architectural uniformity. However, the model could still capture minute differences and nuances, leading to a 13.34% improvement in RMSE. For the BJ dataset, which has a more dispersed housing layout and a vast spatial range, the model achieved a 2.67% improvement in MALE and a 1.73% improvement in RMSE, highlighting the model’s ability to accurately capture the essence of each area despite the considerable differences in spatial dynamics and architectural styles. The advances in the IT dataset were also noteworthy, with the model achieving a 1.35% improvement in MALE and a 1.46% improvement in RMSE despite the unique spatial layout of the region compared to KC. These results demonstrate the robustness and reliability of our model in providing accurate predictions against the ASI model across diverse datasets and spatial configurations.

In Table 5, we present a comparative analysis of our model embeddings against the benchmarks outlined in Table 3. Additionally, the results from the regression layer of our model are presented in Table 4. The results underline the substantial advancements made by our model and the generated embeddings. Rigorous evaluations across various validation sets demonstrate the superior performance of our model in handling complex spatial datasets. Furthermore, the efficiency of the generated embeddings emphasises our model’s role in reducing data complexity so that simple models like linear regression can outperform ensembling models.

In the IT dataset, our model achieved a Mean Absolute Logarithmic Error (MALE) of 0.1312 and a Root Mean Square Error (RMSE) of 45,797. These results represent a 2.89% improvement in MALE and a 0.46% improvement in RMSE over the best baseline model, XGBoost, which recorded a MALE of 0.1350 and an RMSE of 46,008.

Furthermore, the embeddings in our model outperformed the regression layer of our model and the base benchmarking in terms of RMSE, with the Catboosting model achieving the best result of 45,708. This indicates a slight improvement over our model’s performance.

These results can be attributed to the challenging nature of predicting housing prices accurately in this dataset, where various factors come into play. Our model’s success suggests that its embeddings effectively capture the price variations associated with the diverse housing landscape, as evident from the wide distribution of Euclidean distances in Figure 3(a). This distribution reflects the influence of different cities in one dataset, especially Italian cities, which exhibit various housing structures from the south to the north of italy.

Transitioning to the KC dataset, our model displayed a MALE of 0.110 and an RMSE of 107993. This corresponds to a percentage improvement of 2.81% and 10.27% in MALE and RMSE, respectively, compared to the best baseline model, CatBoost. CatBoost had a MALE of 0.1131 and an RMSE of 120351. However, the embeddings seem to mark the best results over our model, and the base benchmarking with 0.1103 MALE value and 106954 RMSE scored in the linear regression model shows an improvement in comparison to our model in both metrics, further emphasising the power of our generated embeddings.

Furthermore, the significant improvement observed in the Kings County (KC) dataset demonstrates our model’s enhanced capability in dense housing and architectural uniformity regions. Our model boosts the prediction accuracy for the most relevant houses and creates diverse contextual frameworks that underscore the interrelationships between houses, even in areas of uniformity. Additionally, creating embeddings encapsulating these relationships further improves the model’s performance.

Our model exhibits exceptional performance on the Porto Alegre (POA) dataset, achieving the lowest Mean Absolute Logarithmic Error (MALE) at 0.136 and Root Mean Square Error (RMSE) at 92,020. This performance surpasses the XGBoost baseline’s MALE of 0.1613 and RMSE of 100,212, indicating a 15.67% improvement in MALE and an 8.17% improvement in RMSE. The model’s superior embeddings are instrumental in this achievement, effectively streamlining intricate urban data for linear regression without losing essential details, as indicated in Tables 5 and 4. Figures 3(a,b) potentially reveal the spatial complexity of POA, with its moderately dense urban fabric intertwined with suburban and rural patches. Despite this diversity posing challenges for predictive models, our embeddings adeptly encode these complexities, effectively representing the multifaceted housing styles and values within POA. Our model’s predictive precision stems from its algorithmic sophistication and nuanced understanding of the region’s unique urban tapestry.

Lastly, base benchmarking for the Beijing (BJ) dataset performs better than our model’s regression layer. However, our embeddings demonstrate better results, suggesting they are more generalized than the base benchmarking outcomes. The embeddings score the best MALE of 0.072 and the best RMSE of 7,713, compared to 0.073 and 0.0732 MALE, and 7,797 and 7,779 RMSE with our model and linear regression model using our embeddings, respectively. Our embeddings’ average values from cross-validation are 0.0733 MALE and 7,786 RMSE, while the base benchmarking average values are 0.074 MALE and 7,836 RMSE, showing a close similarity to the embeddings.

Examining the housing market in Beijing presents several challenges, including managing diverse and often extreme data points typical of a large metropolis. The median distance to the nearest 60 homes in Beijing, as depicted in Figure 3(b), is approximately 0.45 km, highlighting an extensive and varied housing layout. The city’s diverse architectural styles add another layer of complexity to the dataset. Our model, equipped with a Multi-Head Gated Attention mechanism, is adept at handling these challenges. This mechanism effectively regulates the influence of outliers, ensuring a nuanced and accurate representation of Beijing’s housing landscape. The embeddings generated by our model are particularly noteworthy for their ability to generalize across Beijing’s diverse housing market. While the base benchmarking results provide valuable insights, our model’s embeddings capture a broader range of intricacies, ensuring they are statistically sound and meaningfully representative of the real-world scenario.

This quantitative comparison highlights the considerable enhancements of our model. The marked performance uplift in the BJ dataset accentuates our model’s potential in real estate price prediction tasks. Additionally, the comparative analysis with the original attention-based interpolation model by Viana et al. [15] on the KC and POA datasets further amplifies the strengths of our model. Our model’s ability to efficaciously reduce data dimensionality while retaining crucial information has led to significant improvements in MALE and RMSE across all datasets. This proficiency in compressing high-dimensional data into more digestible forms has enabled algorithms like linear regression to compete and outperform complex ensemble models like LightGBM, CatBoost, and XGBoost.