Utilizing Model Residuals to Identify Rental Properties of Interest: The Price Anomaly Score (PAS) and Its Application to Real-time Data in Manhattan

The borough of Manhattan in New York City (NYC) has consistently ranked among the most expensive and competitive locations in the United States. With a population of 8.4 million in NYC, approximately 1.6 million individuals reside in Manhattan competing for 913,000 housing units available [1, 2]. The competitiveness of the Manhattan housing market has been even more heightened due to by influx of individuals returning to the city post COVID, mainly due to the return to office policy of corporation [11]. Simultaneously, the expiration of the 421-a tax abatement - a program that allows tax exemption for a period of time for condo and co-op - in June 2022 has led to a decline in the number of constructions, which has further worsened the housing supply [7]. Given the heightened competitiveness of the Manhattan rental market, there arises a crucial need for a reliable tool that can assist both tenants and landlords in identifying properties that are overpriced and underpriced. Such a tool would empower the the market participants to make informed decisions when looking for or putting up a place to rent, ensuring that opportunities are maximized.

Accurate and current market data plays a pivotal role in the precise estimation of rental prices. Real-time market data provides necessary granularity to incorporate the market dynamics into the model. By leveraging real-time information, predictive models can offer more valuable insight into the rental price estimation through adapting swiftly to evolving circumstances, thus enhancing their reliability and relevance. Especially for a volatile and dynamic market such as Manhattan, real-time data is even more beneficial in ensuring that the predictive models remain responsive to nuances of the market, fostering their efficacy as valuable tool to assist market participants to make informed decisions.

This paper aims to introduce a new approach for identifying overpriced and underpriced rental properties by integrating real-time data scraped from StreetEasy and introducing a novel metric grounded in prediction residuals. In contrast to conventional approaches, our methodology embraces the dynamic nature of real-time data, ensuring a more responsive and up-to-date analysis. Furthermore, the research introduces a metric based on prediction residuals, enhancing the precision of identifying deviations from fair market values. Through identifying the machine learning models that could provide the most accurate prediction of the rental prices, the paper will also provide insight into the most important factors that influence the rental prices in Manhattan and explore variations in prices across different neighborhoods in the area.

Historically, studies focusing on real estate markets have predominantly utilized national or local housing surveys such as American Community Survey data. However, these sources are often deficient in offering temporal and spatial specificity, which restricts their effectiveness in monitoring immediate fluctuations within local markets. With the migration of rental housing advertisements to online platforms, an increasing corpus of scholarly work has turned to web scraping methodologies to analyze rental market dynamics. Many of these studies have primarily relied on Craigslist as a source of data [6]. However, the streamlined listing process on Craigslist, characterized by minimal validation or verification procedures, results in a proliferation of fraudulent or deceptive postings. Therefore, this study will use data from StreetEasy, a prominent and popular online real estate listing website. The listings on StreetEasy are constantly monitored and verified for accuracy, which helps ensure that the information is up-to-date and factually correct [9]. In addition, given the high data quality it resembles the actual population of Manhattan allowing for very valuable experimentation.

The StreetEasy website contains the majority of properties for rent in all boroughs of New York City, for this data collection process the focus was specifically on Manhattan. Based on a custom-built web-scraping script that uses selenium, the data of all 773 pages of apartments for rent in Manhattan, or 8,273 properties was collected. Initial features derived were title, address, price, details, and listing_by. These features were then post-processed to derive additional features like location, rental_type, beds and baths. Street names were also parsed out and used to see if features would be deemed useful in the model training process as shown in Figure 1.

As shown in Figure 2, the Upper West Side stands out as the neighborhood with the largest number of listings, reflecting its popularity and density amongst rental properties. Figure 3 illustrates the distribution of rental prices across the top 10 neighborhoods out of all 53 that have the most listings. Out of the top 10 popular neighborhoods, the highest median rental price is in Lincoln Square. Lincoln Square also has the highest deviation between median and mean price; in addition to the right skewed distribution this suggests that there are a number of luxury rental properties that are significantly driving up the average rental price, indicating a market with a wide range of property values and likely some very high-end options.

When trying to understand the relative distributions between rental property bedroom size, it can be shown that there are significant outliers within the data in Table LABEL:fig:tableI when looking at the maximum rental prices per bedroom size. A 7-bedroom rental has a maximum price of $175,000 per month and minimum of $34,000, while somewhere out there a 1-bedroom at its peak price in Manhattan is going for $38,000 per month. When taking a deeper look, there are two condos at this maximum price for rent both of which residing on Fifth Ave, which makes sense given that Fifth Avenue is a prominent location in Manhattan. Overall, given the summary statistics it can be concluded that the dataset represents the relative market landscape well given the counts of the types of rentals and also accounts for those properties which deviate significantly from the majority.

In preparation for modeling feature selection and engineering was performed to have adequate independent continuous and discrete variables to model the price which is a continuous dependent variable. Out of all of the features in the current dataset, unique parsed street names were not used as a feature, since there were 600 unique street names this could add a curse of dimensionality to the feature space. The features selected were beds, baths, location and rental_type. Out of the selected features, beds and baths were converted to continuous features while location and rental type were one hot encoded as discrete ensuring to drop the first category of the variables to reduce correlations amongst variables for a total of 61 features.

As a starting point, linear regression was fitted to the dataset on a log transformed dependent variable and without to understand whether the data would follow the assumptions. For the purpose of making inferences on coefficient values and their statistical significance, it was found that the data did not fit these assumptions and thus linear regression was not selected. Instead of using linear models to derive feature impact, SHAP analysis was used to perform the same type of inference one could make similar to linear regression coefficient values. After testing many other models utilizing 10-fold cross-validation on 80% of the dataset retaining a 20% holdout set; gradient boosting trees (XGBoost) was selected. Gradient boosting trees combine many weak learned decision trees in a sequential manner to combine overall prominent predictions. XGBoost is an optimized version which includes additional regularization. It was used as the main model to base feature importance and analyze residuals for the purpose of identifying underpriced or overpriced properties based on the magnitude of the residuals using PAS values mentioned later. As shown in Table II, the gradient boosting trees had the highest $R^{2}$ signifying that the independent variables explain approximately 79% of the variance or variability represented in the rental prices overall.

In order validate factors that may indicate a property of interest due to price discrepancies that are beyond the overall distribution, further confirmation among the feature impact on price and its alignment with the real world setting is conducted. An example to understand whether feature distributions resemble a well proportioned dataset relative to the market is validated by cross-checking with other literature. Upon analyzing the features which contribute to the price prediction in the selected model, it can be shown in Figure 4 as the value of baths or beds increases, there is a high increase on the SHAP value which is a direct impact on model output in this case price. The XGBoost model is suggesting that rentals in Tribeca or Midtown have a much higher impact on the price as opposed to East Harlem or Central Harlem. This suggests that rental units in East or Central Harlem are much less than those in Tribeca or Midtown in terms of the variance that the model learns when it comes to the overall distributions contributing to price. Although not mentioned in the above figure, South Harlem, Hamilton Heights, Yorkville, Hell’s Kitchen and Washington Heights also have negative SHAP values, indicating lower price impact from rentals which reside in these locations. The observed outcome is consistent with the prevailing understanding of rental prices in Manhattan. As indicated by Corcoran, one of the prominent residential brokerages in New York City, Soho/Tribeca emerged as the highest-priced neighborhood for rental properties in October 2023 [4]. Concurrently, Washington Heights, Hamilton Heights, and East Harlem were identified as Manhattan neighborhoods ranking among the top 10 most affordable areas in New York City, according to information reported by Timeout [3].

Unlike most machine learning approaches that focus on the overall error of the test set to ensure that the model is performing well on what is unseen, here we focus on the overall error of all apartments. After hyperparameter tuning to find the most optimal model given the highest R-squared and lowest RMSE as mentioned earlier, the optimal model is then fit on the entire dataset without splitting the data. The catch here is to find the optimal model which predicts the unseen data utilizing cross-validation, then to use those optimal model parameters to refit to the entire dataset to then analyze the residuals holistically. Once this has been achieved, the residuals can be analyzed and the Price Anomaly Score can be applied for a full analysis of the dataset.

In this section, we introduce the Price Anomaly Score (PAS), designed to capture the extent of deviation from expected prices, indicating potential overvaluation or undervaluation of properties. This score utilizes the concept of Z-score application from previous literature to improve on identifying pricing discrepancies; it incorporates the magnitude, Z-score and fine tuning of boundaries from a parameter q.

In Figure 5, the distribution of PAS values follows more of a normal distribution than the Z-score. When further comparing these two distributions, the valuation labeling is added in the legend to overlay the PAS boundaries using a q-value of 0.4 for both plots. This is conducted to show the counts of each metric relative to the PAS boundary selection. The tails of each distribution are significantly different which signals that a boundary based on a scalar value for Z-score alone may not capture all properties according to their market valuations. When using PAS, the importance of the error produced by the model is being scaled with an assumption that not all errors are created equal. If the Z-score is to be used alone to set boundaries for valuation, a small error for a very unusual property (high Z-score) could be considered more significant than the same error for a more typical property. This is due to the fact that the Z-score alone gives a measure of how far off a particular property’s rental price is from the mean rental price, assuming a normal distribution of rental prices. However when utilizing the Price Anomaly Score, it signifies the predictive anomaly; how much the actual price differs from what the model predicts it should be, given all the factors the model takes into account. When PAS is used to set labels for valuation it scales the Z-score based on the error magnitude of the model, providing a different distinction between what is considered ’fair’ priced and what is not.

In Figure 6 it can be shown that a logarithmic relationship exists between Z-score and PAS values. A larger Z-score magnifies the error making it much more significant when it comes to properties that are already outliers. Errors in prediction for properties that are closer to the average price (Z-score around 0) have less impact on PAS; this allows for more attention to be focused on outliers where model predictions are least accurate.

For renters or real estate professionals, both the Z-score and PAS values have their positives and negatives for usage as metrics when identifying overpriced or underpriced properties. The Z-score is a purely statistical measure which is able to handle outliers in terms of price deviation from the mean, however it does not necessarily account for why a property might be an outlier. Using the Z-score for valuation labeling can be beneficial if the objective is to measure how far a property’s residual deviates from the average of residuals in terms of standard deviations. However, when it comes to identifying the overall impact in the context of market dynamics (location, beds, baths) the Price Anomaly Score incorporates the predictive error which can suggest whether an outlier is statistically rare or genuinely mispriced in the housing market.

An example of PAS usage can be shown in Figure 7 where the dimensionality of the dataset is reduced. Here it can be shown relative to the top two eigenvectors with the highest magnitude, the deviation from the predicted price can be visualized. This simplifies the dataset into a format where patterns can be more easily discerned. The properties are color-coded by the number of bedrooms; however the principal components do not suggest a strong clustering by this attribute alone. The highlighted property at ”558 Broome Street #7” serves as a case study showcasing a significant discrepancy between the market price and models prediction, resulting in a negative PAS value and its classification as ’Underpriced’.

PAS values are heavily dependent on the relevance of the data fed into the predictive model. Both PAS and Z-score depend on the data however the PAS values effectiveness is tied to the performance and construction of the underlying model and quality of the data used in the model. The choice between these two metrics heavily depends on the specific requirements and constraints of the analysis or business objective.

Despite the promising results obtained from the machine learning models used in this study, it is crucial to acknowledge several limitations and challenges that may impact the reliability and generalizability of the findings. Firstly, the study’s scope is confined by the exclusive use of data from StreetEasy. This limits the analysis to the information available or provided by the platform and therefore does not include any real estate that is not rented on StreetEasy but may be transacted on another platform. Moreover, Manhattan’s real estate markets, known for their dynamic characteristics, are susceptible to rapid changes influenced by economic conditions, external events, and other unpredictable factors, posing challenges for the selected model to adapt to sudden market shifts. Since the model uses data from a snapshot of data in time and therefore may not take into account some of these time-dependent factors. Additionally, Manhattan real estate may not be representative of real estate more broadly in the United States or the world, so the results of this study should not be generalized to broader areas.

Additionally, there are some challenges with the proposed residual feature metrics. The PAS metric relies on the assumption of normality in the distribution of the residuals. Therefore, if the rental prices do not follow a normal distribution, it may impact the validity of the metric and the subsequent classification of overvalued and undervalued properties. Lastly, the user-chosen threshold ”q” for PAS introduces subjectivity into the determination of property valuation. The definition of what constitutes an over or underpriced real estate asset may be inherently subjective, and therefore the optimal threshold might vary based on user preferences or specific real estate market conditions. This study does its best to take this subjectivity into account by allowing for a moving threshold. In light of these constraints, it is essential to interpret the study results with these limitations in mind, recognizing the nuanced nature of real estate dynamics.