Shifting the Paradigm: Estimating Heterogeneous Treatment Effects in the Development of Walkable Cities Design

In our study, we employed a virtual reality (VR)-based conjoint experiment to assess individuals’ preferences and behaviors in response to urban design interventions. This approach features three stages: specifying attributes and their levels, designing 3D neighborhood simulations, and developing an online questionnaire to capture perceptual and experiential responses within a VR environment.

The choice of attributes and levels of VR-based conjoint experiment design were based on earlier works[14]. The experiment focuses on the street block level, chosen for its analogous influence on walking behavior to that of a neighborhood, as supported by literature[26]. This scale allows for detailed 3D modeling and enables participants to closely engage with the urban environment’s features.

Attributes impacting walking behavior—land use mix-diversity, walking facilities, sidewalks, and trees—were identified based on prior research, along with connectivity and open space[14, 27]. Five attributes were selected for the street block experiment, each with two levels: land use mix (exclusively residential or mixed-use), block connectivity (high or low), road size (two lanes with narrow pedestrian zones or one lane with wide pedestrian zones), open space (presence or absence), and green/trees (presence or absence of trees).

From the potential 32 ($2^{5}$) attribute combinations, we applied a fractional factorial design to reduce the number while preserving orthogonality, ensuring statistical independence among attributes. This approach resulted in eight distinct attribute profiles, facilitating efficient estimation of main effects without attribute correlation[28]. This design strategy enhances our experimental setup’s capacity to discern the individual contributions of various urban design elements to walking behavior, streamlining the process of identifying effective interventions for promoting walkability.

In the subsequent phase, the eight attribute combinations were translated into corresponding virtual reality (VR) environments. To accomplish this, we constructed the foundational 3D model to represent the Dutch street block by using SketchUp Pro. The dimensions of the experiment area within the street block spanned 300 meters in length and 240 meters in width. While maintaining a consistent road width, we introduced variations in road type: (1) two lanes dedicated to cars with a narrow pedestrian sidewalk, and (2) a single lane for cars with a wider pedestrian pathway.

Regarding the land use mix attribute, we established two levels: the first level entailed an exclusively residential street block, while the second level incorporated a mix of residential and commercial areas such as shops and supermarkets dispersed within the residential area. In terms of the block connectivity attribute, we manipulated the number of intersection points within the street block. Additionally, we introduced variations in the presence or absence of open space, as well as the inclusion or exclusion of street trees to represent the green attribute. By utilizing the 3D sketch models and altering attribute levels accordingly, we generated a total of eight distinct VR environments.

Subsequently, all eight VR environments were imported into Twinmotion, a rapid 3D rendering software[29]. Within Twinmotion, we integrated materials, trees, traffic elements, facilities, and human figures into each of the 3D sketch models, enhancing their realism and creating more interactive experiences. Then, we established a consistent walking perspective and exported all virtual reality environments as video files. To ensure coherence, every video depicting the virtual reality environments adhered to identical parameters, including the walking route, viewing direction, geographical location, duration of sunlight, seasonal conditions, and weather conditions. Each video had a standardized duration of 90 seconds.

Furthermore, to maintain uniformity in the conveyed information, all scenarios incorporated identical 3D objects, such as buildings, trees, and facilities, to represent the respective characteristics. Consequently, participants were presented with consistent attributes throughout all scenarios, including uniform tree colors, building styles, and facility materials. This approach ensured that participants evaluated the environments based on the attributes utilized in their construction, while keeping all other factors constant.

The questionnaire employed in this study is divided into two distinct sections. The first part focuses on individuals’ perceptions of their existing neighborhood and personal characteristics, while the second part encompasses the virtual reality (VR) environment, comprising videos and related questions intended to elicit participants’ perceptions of the VR environments. Within the VR environments section of the questionnaire, participants are prompted to evaluate their experiences of the virtual environments while viewing the corresponding videos. They are asked to provide ratings based on the emotions evoked by each video. To manage the length of the questionnaire, four out of the eight dynamic 3D videos showcasing the VR environments are randomly presented to each respondent. The participant’s perception of each virtual reality environment is assessed through two sections of questions.

The first section encompasses two inquiries pertaining to the environment’s quality, specifically: (1) ”How satisfied are you with the overall quality of this virtual environment?”; and (2) ”How satisfied are you with the walking friendliness of this virtual environment?” Participants are asked to rate their satisfaction using a 7-point Likert scale, ranging from ”not at all satisfied” to ”fully satisfied.” To capture the participants’ feelings during the virtual walk-through experience, the preference rating method introduced by Birenboim et al.(2019) is employed[20]. The second section of the questionnaire focuses on the emotional responses evoked by the virtual environment. Four dimensions of emotions associated with perceived walkability during the virtual walk-through experience are examined: happiness, comfort, annoyance, and security. Participants are requested to indicate the extent to which they experienced each of these emotions. The questions are presented as statements, such as ”I felt happy/comfortable/annoyed/secure.” For each item, respondents provide their responses on a 7-point Likert scale, ranging from ”completely disagree” (1) to ”completely agree” (7), as illustrated in Table 5. Additionally, the second section includes inquiries concerning the perceived benefits derived from the virtual environment.

The data for this study was sourced from earlier works by Liao et al. (2022)[14]. Participants were recruited for this study from a nationally representative consumer panel in the Netherlands, as well as through popular social media platforms such as Twitter (X) , LinkedIn, and Facebook. Upon introducing the virtual reality (VR) environments, respondents were informed that they would be presented with scenarios depicting a typical Dutch street block in a virtual setting. Subsequently, participants were requested to evaluate the overall quality and walking friendliness of the virtual scenarios, as well as provide their subjective emotional responses while viewing these scenarios.

A total of 308 individuals completed the online questionnaire, with 272 respondents sourced from the consumer panel and an additional 36 participants recruited through social media channels. To ensure the reliability and robustness of the data, respondents who provided repetitive answers to each question or who completed the VR portion of the questionnaire in less than 8 minutes were excluded from the analysis. Following the data cleaning process, a final sample of 295 respondents remained for further analysis. All participants were exposed to four 3D-videos, resulting in a total of 1,180 ratings recorded for each item of interest. Consequently, the final dataset consists of observations for each respondent pertaining to 4 virtual walking trips.

Rubin’s potential outcomes framework [21] suggests that in studies with a binary treatment, we can only observe the actual outcomes for each subject under the specific treatment they received. To understand a treatment’s effect, we must compare what actually happened to a subject with the treatment to what would have happened without it or with a different treatment. This hypothetical scenario is called the counterfactual outcome. The challenge, especially in observational studies, is accurately estimating these counterfactual outcomes. This difficulty arises because the characteristics of the groups receiving different treatments can vary significantly, leading to biased comparisons. In simpler terms, certain traits may make some people more likely to receive a specific treatment, complicating the task of isolating the treatment’s true effect.

We formalize our problem using a sample dataset which has $n$ independent and identically distributed examples $(X_{i},Y_{i},W^{\eta}_{i})$, $i=1,\ldots,n$, where $X_{i}\in\mathcal{X}$ represents individual features, $Y_{i}\in\mathbb{R}$ is the observed outcome, and $W^{\eta}_{i}\in\{0,1\}\text{ and }\eta=1,2,\ldots,H$ stands for $H$ treatment options and their assignment status. We posit the existence of potential outcomes $\{Y_{i}(W^{\eta}_{i}=1),Y_{i}(W^{\eta}_{i}=0)\}$ corresponding to the outcome we would have observed given the treatment assignment $W^{\eta}_{i}=1$ or $W^{\eta}_{i}=0$ respectively, such that $Y_{i}=Y_{i}(W^{\eta}_{i})$. One common quantity researchers interested is the average treatment effect for treatment $\eta$, which is $\tau=\mathbb{E}[Y_{i}(W^{\eta}=1)-Y_{i}(W^{\eta}=0)]$. Here, in contrast, we want to understand how treatment effects vary with the observed covariates $X_{i}$, and consider the conditional average treatment effect (CATE):

G-formula, propensity score matching and targeted maximum likelihood estimation (TMLE) are popular statistical methods used in epidemiology and other fields for causal inference and adjusting for confounding in observational studies. Here is a brief introduction to each method and their extension to joint treatments and likert scale outcomes.

To benchmark the algorithms, we simulate a dataset containing 5 covariates and 5 treatments, with the outcome variable being a Likert scale ranging from 1 to 7. The data simulation process is designed to explore the impact of these treatments and covariates on the outcome variable, providing insights into potential causal relationships.

The dataset consists of 5000 samples, each with 5 covariates $(X_{1},X_{2},X_{3},X_{4},X_{5})$ generated from a normal distribution $\mathcal{N}(1,1)$. The treatment effects are uniformly set to 1 for all five treatments, indicating an equal impact of each treatment on the outcome variable. The treatment assignments are influenced by both the covariates and a confounding parameter, with a baseline probability adjusted based on the sum of the first three covariates and a confounding ratio.

The covariates for each sample are generated as follows:

where $X_{i}$ represents the $i$-th covariate for each sample.

The probability of receiving each treatment is calculated using a combination of the covariates and a confounding factor (confound):

where:

• •

  $j\in\{1,2,3,4,5\}$, with only the first three covariates contributing to the unbalanced confounding,

• •

  $\alpha$ represents a small coefficient (e.g., $10^{-5}$) applied to the covariates,

• •

  $\beta$ corresponds to the confound ratio, it has been set to $0.15,0.5$ and $0.85$ to indicate scenarios with low, medium and high level of confounding.

• •

  $\text{expit}(x)=\frac{1}{1+e^{-x}}$ is the logistic function, and

• •

  $T_{j}$ is the treatment assignment for the $j$-th treatment.

The response variable ($Y$) is generated considering the baseline response, the effect of treatments, the impact of covariates, and random noise:

where:

• •

  $T_{j}$ denotes the effect of the $j$-th treatment,

• •

  $X_{i}$ represents the $i$-th covariate,

• •

  $\epsilon\sim\mathcal{N}(0,0.1)$ is the noise term, and

• •

  the clip function ensures $Y$ remains within the Likert scale range of 1 to 7.

We generate 50 sets of simulation datasets under each confounding scenario and compute the estimated conditional average treatment effect $\hat{\Psi}$ with the true treatment effect $\Psi$, which can be calculated using the counterfactual $Y$ as explained in previous sections, the benchmark is computed as percentage error: $(\hat{\Psi}-\Psi)/\Psi$.

The experimental framework facilitated the derivation of 32 ($\sum_{i=1}^{5}{C^{5}_{n}}$) possible combinations of conditional average treatment effects (CATE, refer to the Methodology section for a comprehensive explanation). Illustrated in Figure 3, the absolute standardized mean differences (ASMD) post-propensity score matching reveal a notable contraction in the discrepancies between covariates across treatment and control groups. This phenomenon indicates a moderate degree of confounding present within the VR dataset under examination.

The presence of unbalanced confounders between treatment and control groups introduces potential bias in the estimated CATE from our interventions, which may significantly affect the validity and reliability of causal inferences drawn from the study.

In the context of our study, where diverse demographic groups’ perceptions of walkability are evaluated through design interventions, such imbalances could skew the interpretation of how environmental changes impact pedestrian behavior. Specifically, if certain demographic characteristics or pre-existing conditions are disproportionately represented in either group, the observed effects may not accurately reflect the interventions’ true impact but rather the underlying differences between the groups. This could lead to erroneous conclusions about the effectiveness of design interventions in improving urban walkability. To mitigate this issue, our study utilizes the Targeted Maximum Likelihood Estimation (TMLE) algorithm, an analytical approach designed for precision in the face of measured confounding variables.

The results presented in Table 3 offer a nuanced understanding of how varied urban design interventions influence perceived walkability among different demographic segments.

With the application of a single intervention, Land use mix (LM) showed a modest but positive effect (CATE: 0.92%, SE: 0.602%), indicating that even isolated alterations to land use configurations can significantly enhance walkability perceptions. Block connectivity (BC) further exhibited a beneficial impact (CATE: 1.11%, SE: 0.547%), reinforcing the value of interconnected block designs in urban settings. In contrast, adjustments solely to Road size (RS) led to a marginal decrement in walkability (CATE: -1.21%, SE: 0.489%), suggesting that without additional supportive measures, enlarging road dimensions may adversely affect walkability.

Scenarios featuring Open space (OS) as the sole intervention demonstrated a marked positive effect (CATE: 3.83%, SE: 0.716%), underscoring the critical role open areas play in promoting walkability. Similarly, the exclusive addition of Green/Trees (GT) correlated with a favorable outcome (CATE: 2.03%, SE: 0.525%), highlighting the integral contribution of green elements to the urban environments.

Upon the integration of two or more interventions, combinations involving OS without LM yielded the highest CATE, approximately 4%, manifesting the pronounced impact of open spaces when combined with other urban design elements, except LM, whereupon the CATE diminished to 3.486%. Intriguingly, scenarios incorporating two or more interventions alongside LM experienced a decrease in CATE, illustrating a potential dilution or negative interaction effect when LM is part of multiple intervention strategies.

These findings suggest a complex interplay between various urban design interventions and their collective impact on walkability. The substantial positive effects of OS and GT, even as standalone interventions, reinforce the importance of integrating natural and open spaces within urban landscapes to enhance pedestrian experiences.

Conversely, the observed decrease in CATE with certain combinations implies the need for careful consideration in selecting and implementing intervention mixtures to avoid counterproductive outcomes.

In Figure 4, we present an analysis of the impact of urban design interventions, contingent on the extent of concurrent interventions. The number of interventions—ranging from one, indicating the exclusive presence of the intervention under consideration, to five, denoting the hypothetical application of all interventions—serves as a proxy for the complexity of urban design strategies. The Conditional Average Treatment Effect (CATE) is computed for select interventions, adjusting for potential confounders and the influence of other concurrent interventions based on Table 3. It reveals a spectrum of responsiveness across interventions: certain measures are more heavily influenced by the number of concurrent interventions than others. This variability underscores a complex interplay among urban design measures and highlights the importance of a calibrated approach in policy formulation and the execution of urban development agendas.

First, the analysis reveals distinct patterns of influence as the number of interventions increases. LM shows an incremental positive trend in average CATE values with additional interventions, starting from a negligible effect at a single intervention (CATE = 0.92%) to a substantially larger effect with five interventions (CATE = 3.49%). This pattern suggests that interventions promoting mixed land use could have compounding beneficial impacts.

Notably, OS initially indicates an effect (CATE = 3.83%) for a single intervention, yet keeps stable above $3\%$ as the number of interventions rises. It consistently exhibits the highest CATE values across all levels of intervention, implying that creating or preserving open spaces could be the most effective singular urban design intervention in our case study.

RS, BC and GT parallels LM with increasing positive CATE values, while GT demonstrates a unique profile with the CATE starting at a higher baseline (CATE = 2.03% for a single intervention) and also enhancing as the number of interventions grows. The precision of these effects is demonstrated by the 95% confidence intervals, which are notably tight for OS, suggesting a high level of confidence in its positive impact.

Overall, the evidence advocates for a strategic approach to urban design, where specific combinations and numbers of interventions can be tailored to maximize the beneficial effects of urban features on the targeted outcomes.