Caliper Synthetic Matching: Generalized Radius Matching with Local Synthetic Controls

Suppose we have $n$ independent and identically distributed observations, with $n_{T}$ treated units and $n_{C}$ control units. For each unit $i$, let $Z_{i}\in\{0,1\}$ denote its binary treatment status, $Y_{i}\in\mathbb{R}$ denote its observed real-valued outcome, and $\mathbf{X}_{i}\equiv\{X_{1i},\dots,X_{pi}\}^{T}\in\mathbb{R}^{p}$ denote its $p$-dimensional real-valued covariate vector. We use the potential outcomes framework and denote the observed outcome for unit $i$ as $Y_{i}\equiv(1-Z_{i})Y_{i}(0)+Z_{i}Y_{i}(1)$, for potential outcomes $Y_{i}(1)$ and $Y_{i}(0)$ under the stable unit treatment value assumption (Imbens and Rubin,, 2015). We make the standard conditional ignorability assumption:

$\displaystyle(Y(1),Y(0))\perp\!\!\!\perp Z\mid\mathbf{X},$

so that conditioning on the observed covariates is sufficient to identify the causal effect of $Z$. Under a population sampling framework, we write $\epsilon_{i}\equiv Y_{i}(Z_{i})-f_{Z_{i}}(\mathbf{X})$, where $f_{0}(\mathbf{X})\equiv E[Y(0)|\mathbf{X}]$ and $f_{1}(\mathbf{X})\equiv E[Y(1)|\mathbf{X}]$ are the true conditional expectation functions of the potential outcomes under control and treatment, respectively. We write the set of all treated units’ indices as $\mathcal{T}=\{i:Z_{i}=1\}$, the set of all control units’ indices as $\mathcal{C}=\{i:Z_{i}=0\}$, and $t\in\mathcal{T}$, $j\in\mathcal{C}$ as individual treated and control units respectively. We denote the set of the indices of the control units matched to a treated unit $t\in\mathcal{T}$ as $\mathcal{C}_{t}=\{j\in\mathcal{C}:\text{ unit }j\text{ is matched to unit }t\}$. Finally, we denote the size of a set $\mathcal{S}$ as $|\mathcal{S}|$.

In this paper, we will focus on estimating the sample average treatment effect on the treated (SATT):

$\displaystyle\tau=\frac{1}{n_{T}}\sum_{t\in\mathcal{T}}\left(Y_{t}(1)-Y_{t}(0)\right),$

where $n_{T}=|\mathcal{T}|$ denotes the number of treated units.

Under a population sampling framework, the SATT approaches the overall population average treatment effect on the treated (PATT) as the number of treated units increases. While matching methods can easily be extended to estimate sample and population average treatment effects (SATEs and PATEs), we focus on the SATT to clarify key ideas and simplify exposition.

To motivate the importance of locality and joint balance, we provide a pair of toy examples. Figure 1 plots the covariates $X_{1}$ and $X_{2}$ of control units $c_{i}$ and treated units $t_{j}$ for two scenarios. The colors and contours visualize $f_{0}(\cdot)$, which takes on greater values within the innermost contour. To conduct causal inference, we use the observed outcomes of the control units $c_{i}$ to impute the unobserved counterfactual outcomes of the treated units $t_{j}$.

For Example 1 (Figure 1(a)), accurate causal inference is not possible due to a lack of overlap: we cannot accurately impute outcomes for $t_{1}$ and $t_{2}$ because we do not observe any nearby evaluations of $f_{0}(\cdot)$. Many causal inference methods, however, could fail to detect this problem. For example, a matching or balancing approach that works with marginal distributions could exactly balance both $X_{1}$ and $X_{2}$ by assigning weights of 1 to all four units. Similarly, an outcome model may fit a flat response surface to the observed outcomes of $c_{1}$ and $c_{2}$. Because the usual marginal balance checks and outcome models would appear good, the analyst would be left unaware that these analyses significantly underestimate counterfactual outcomes, i.e., overestimate the SATT. Approximate exact matching approaches would also fail here, as there are no control units close to the treated units. The failure to find local matches, however, would alert the analyst to the presence of a problem. Therefore, it is important to have a metric on how good controls each treatment can obtain. In our method, we use nearest neighbor distance as this quality metric.

Example 2 (Figure 1(b)) highlights the role of transparent local matches. Because $f_{0}(\cdot)$ is smooth, using only the outcomes of local control units (i.e., controls $c_{3}$ and $c_{4}$ within the dotted circles) leads to accurate counterfactual estimates. Local matches also produce transparent estimates; unlike using a black-box outcome model, it is immediately clear how each control unit contributes to each treated unit’s counterfactual estimate. Finally, local matches encourage joint balance. In Example 2, if we assigned weights of 1 to $c_{1}$ and $c_{2}$, and 0 to $c_{3}$ and $c_{4}$, we would have perfect marginal balance, but poor local match quality and a poor counterfactual. It is worth sacrificing some marginal balance to upweight the closer units $c_{3}$ and $c_{4}$, since doing so greatly improves joint balance and the resulting treatment-effect estimates.

Having illustrated the importance of using distance matching to achieve local matches, we now turn to the selection of the distance metric. The Mahalanobis distance has been commonly used in the literature, but it has drawbacks, including sensitivity to outliers and the curse of dimensionality. Outliers can distort the covariance matrix upon which Mahalanobis distance relies, leading to poor match quality. Moreover, with an increase in the number of covariates, the effectiveness of Mahalanobis matching may diminish due to the curse of dimensionality, where units in high-dimensional spaces appear far apart, complicating the search for suitable matches.

To partially address these issues, we propose a more flexible metric, the scaled distance, which permits analysts to specify the desired tightness of matching for each covariate in turn. Regions of low overlap can then be handled via an adaptive caliper method as detailed in Section 3.3. The challenge of dimensionality is also reduced, given prior knowledge on the relative importance of the covariates, as covariates of interest can be scaled to achieve higher level of control; see Section 3.1.

Lastly, while these toy examples demonstrate the benefits of adhering to the ”spirit of exact matching” by finding local matches to enhance causal estimates and identify potential biases, it is important to acknowledge that in some, possibly even many, situations, focusing on joint balance in this manner may be unnecessary or even detrimental. For example, if the conditional expectation function is additive in its covariates, i.e., $f_{0}(\mathbf{X})=\sum_{j=1}^{p}g_{j}(X_{j})$ for functions $g_{j}(\cdot):\mathbb{R}\to\mathbb{R}$, marginal balance suffices (Zubizarreta,, 2015). In such settings, attempting to balance the full joint covariate distribution is excessively conservative, as it protects estimates from bias that does not exist, namely, bias due to imbalances in covariate interactions.

Matching methods have a long history in observational causal inference (Stuart,, 2010). Rather than attempt an exhaustive review, we briefly trace how these methods have operationalized the spirit of exact matching over time and provide further details in Section 3 as we develop the method introduced in this paper.

Early work in matching incorporated locality via nearest-neighbor (Rubin,, 1973), caliper (Cochran and Rubin,, 1973; Rosenbaum and Rubin, 1985b, ), and radius matching (Dehejia and Wahba,, 2002) approaches. These methods were typically combined with dimension reduction, e.g., via propensity scores (Rosenbaum and Rubin,, 1983), to circumvent the challenge of near-exact matching with multiple covariates, though some approaches, e.g., Mahalanobis distance matching (Rubin and Thomas,, 2000) and genetic matching (Diamond and Sekhon,, 2013), directly operated on a scaled version of the original covariate space. Also see Aikens et al., (2020), who use set-aside data to fit a prognostic score, and then match on that score. Regardless, to evaluate and improve the quality of a matched dataset, researchers would typically conduct iterative balance checks, revising their matching scheme if it led to poor marginal mean balance.

To circumvent the need for these iterative balance checks, Iacus et al., (2012) introduced Coarsened Exact Matching (CEM), which we discuss in further detail in Section 3.2. CEM fixes a user-specified level of joint balance, operationalized by a given degree of coarsening, and returns the resulting sample. By making local matches a primary rather than a secondary criterion, CEM enjoys the desirable transparency and joint balance properties of exact matching.

More recently, researchers have developed matching methods that flexibly learn notions of locality and similarity rather than using user-specified defaults. Genetic matching (Diamond and Sekhon,, 2013) learns to more precisely match covariates that appear to be important for overall covariate balance, while “almost-exact” matching methods for discrete (Dieng et al.,, 2019; Wang et al.,, 2021) and continuous (Morucci et al.,, 2020; Parikh et al.,, 2022) data aim to more precisely (or exactly) match covariates that are more predictive of the potential outcomes. Matching After Learning to Stretch (Parikh et al.,, 2022) minimizes predictive error over a hold-out training set to identify a similarity metric, and synthetic controls (Abadie et al.,, 2010) minimize the resulting predictive error with respect to a set of held-out pre-treatment outcome variables to obtain optimal variable importance weights in a time-series setting.

Work outside of matching has also noted the importance of locality in observational causal inference. For example, Abadie and L’hour, (2021) augments the popular synthetic controls methodology with a penalty for using control units far from the treated unit. In a similar setting, Ben-Michael et al., 2021b tunes the extent to which synthetic controls may extrapolate away from the control units. More generally, Kellogg et al., (2021) explicitly trades off the bias from extrapolating beyond local matches with the bias from linearly interpolating between distant units. While these approaches do not attempt to directly emulate exact matching, they highlight the use of local data for principled causal inference.

In another direction, one can directly optimize weights with respect to overall balance criterion, rather than engage in the iterative cycle of matching with a subsequent balance check on the result. For three examples of many, see stable balance weights (SBW; Zubizarreta,, 2015), entropy balance weights (Hainmueller,, 2012), or covariate balancing propentiy scores (Imai and Ratkovic,, 2014).

In the absence of exact matches, matching algorithms find control units as “close” as possible to their treated counterparts to improve joint covariate balance and reduce potential bias (Rosenbaum and Rubin, 1985a, ). One popular distance measure for “closeness” is propensity score distance:

$\displaystyle d^{(e)}(\mathbf{X}_{i},\mathbf{X}_{j})=|e(\mathbf{X}_{i})-e(\mathbf{X}_{j})|,$

(1)

where the propensity score $e(\cdot)=P(Z_{i}=1\mid\mathbf{X}_{i})$ represents the probability that a unit is treated, given its covariates (Rosenbaum and Rubin,, 1983).

While matching units with similar propensity scores leads to principled causal estimates, it does not construct intuitive matched sets (King and Nielsen,, 2019). Propensity score distance is not formally a distance metric on $\mathbb{R}^{p}$,²²2For example, $d^{(e)}(\mathbf{X}_{i},\mathbf{X}_{j})=0$ does not imply that $\mathbf{X}_{i}=\mathbf{X}_{j}$. For a simple introduction to distance metrics on $\mathbb{R}^{p}$, see Appendix B.1. so it can violate our natural understanding of “closeness.” For example, two units that are “close” in terms of propensity score distance, which simply means they have similar chances of getting treatment, may have very different covariate values. If a researcher matches two such units, it can be unclear whether they should trust this match, fit a better propensity-score model, or find a closer propensity-score match.

Formal distance metrics provide a more natural approach for assessing similarity between units. Researchers are generally familiar with the covariates in their data, so directly attaching a distance metric to the space of covariates builds upon existing intuitions. Multidimensional distance metrics typically take the form of a scaled Euclidean (i.e., $L_{2}$) distance metric:

$\displaystyle d^{(2)}_{V}(\mathbf{X}_{i},\mathbf{X}_{j})=\sqrt{(\mathbf{X}_{i}-\mathbf{X}_{j})^{T}(V^{T}V)(\mathbf{X}_{i}-\mathbf{X}_{j})}$

(2)

or scaled $L_{\infty}$ distance metric:

$\displaystyle d^{(\infty)}_{V}(\mathbf{X}_{i},\mathbf{X}_{j})=\sup_{k=1,\dots,p}|V(\mathbf{X}_{i}-\mathbf{X}_{j})|_{k},$

(3)

for a given $p\times p$ symmetric positive definite matrix $V$. The matrix $V$ scales the raw differences in the covariates and their two-way interactions. For example, if $V_{11}$ were large, then the resulting distance metric would magnify, i.e., upweight, differences in the first covariate. We consider both scaled $L_{2}$ and scaled $L_{\infty}$ distance metrics in this paper.

A variety of scaled distance metrics have been proposed in the literature. One popular scaled $L_{2}$ distance metric is Mahalanobis distance, which uses $V^{T}V=\Sigma^{-1}$, the inverse covariance matrix estimated from the control group (Rubin,, 1980). Other $L_{2}$ approaches, such as genetic matching (Diamond and Sekhon,, 2013) restrict the scale matrix $V$ to be diagonal and directly optimize it.

In this paper, we set $V$ to a diagonal matrix determined by a covariate-wise caliper specified by a researcher, as formalized in Proposition B.3. A covariate-wise caliper, $\pi_{k}$, represents the maximum allowable difference in the $k$-th covariate for matching purposes. When the $L_{\infty}$ metric $d_{V}^{\infty}$ is set to $c$, the $k$-th covariate difference between the two units is at most $c\pi_{k}$. The is equivalent to scaling each covariate $k$ by $V_{kk}=1/\pi_{k}$ and then matching using the $L_{\infty}$ distance with threshold $c$. Generally, we advocate setting $\pi$ so that any pair of units with distance bounded by $c=1$ is considered to be a “reasonably similar” comparison.

Using $d_{V}^{\infty}$ with a covariate-wise caliper is strongly connected to Coarsened Exact Matching (CEM) (Iacus et al.,, 2012), as the distance metric provides researchers with the desired degree of covariate-wise control. In CEM, continuous variables are initially coarsened into discrete bins; for example, an age variable may be segmented into bins of 0-25, 25-50, 50-75, and 75-100 years. Observations are then precisely matched based on these coarsened covariates. By setting a covariate-wise caliper to 12.5, we allow a treated unit to be matched with a control unit if their age difference is no more than 12.5 years. This approach generalizes the concept of coarsening, as coarsening, given the above bins, would not match a 26-year-old with a 24-year-old, but our method could. Further details are discussed in Section 4.5.

Section 2.2 argued the importance of local matching. Here we discuss ways of achieving local matching by mainly revisiting the literature.

Given a chosen measure of “closeness,” there are many ways to select the closest matched units. One popular approach is nearest-neighbors matching, where each treated unit is matched with the control unit(s) closest to it. This can be done either greedily for each treated unit (Rubin,, 1973) or optimally over all treated units (Rosenbaum,, 1989). Nearest-neighbors matching can be conducted after some preprocessing steps, such as learning an optimal distance metric (Diamond and Sekhon,, 2013; Parikh et al.,, 2022).

While nearest-neighbors approaches are intuitive, they can quietly fail to achieve covariate balance. While nearest-neighbors approaches guarantee that each treated unit is matched with its closest control, the closest control may still be quite far off. Large distances between treated units and their matched controls, i.e., low match quality, can lead to poor joint covariate balance.

To combat this problem, many methods apply calipers coupled with nearest-neighbor matching. Calipers are a distance $c$ beyond which matches are forbidden. Calipers modify the distance metric as:

$\displaystyle d(\mathbf{X}_{i},\mathbf{X}_{j})=\begin{cases}d(\mathbf{X}_{i},\mathbf{X}_{j})&\text{if }d(\mathbf{X}_{i},\mathbf{X}_{j})\leq c\\ \infty&\text{if }d(\mathbf{X}_{i},\mathbf{X}_{j})>c\end{cases}$

Using calipers, nearest-neighbor approaches can avoid problems associated with poor match quality, using the closest matches only if they are “close enough.” The simplest form of nearest neighbor matching does not, however, take full advantage of data rich areas of the sample: if there are many controls, using all of them rather than the strictly closest could improve precision.

Calipers are frequently applied to the propensity score, rather than to covariate distance. Unfortunately, units that are local with respect to assignment probability may not be actually local with respect to their covariates; see King and Nielsen, (2019).

The distance metric and the caliper are quite connected; in particular, if we scale our distance metric, we can simply scale the caliper by the same amount. This means $c=1$ indicates a caliper of one unit on the distance metric, which under the infinity-norm means no covariate differs by more than one “width” as defined by $\pi$. Thus, without loss of generality, we generally assume an initial value of $c=1$ hereafter, unless otherwise noted.

In this paper, we directly use all control units within a given caliper of each treated unit, which maximizes the number of local matches used to produce causal estimates. We allow matching with replacement, meaning a control unit close to multiple treatment units will match to all of then. This matching approach is known as radius matching (Dehejia and Wahba,, 2002) with replacement, though previous proposals mostly use radius matching on propensity score distances. We relegate discussion of caliper selection to Appendix A.1.

Matching with a fixed caliper could lead to a substantial fraction of the treated units being dropped, which can significantly change the target estimand. In particular, dropping difficult-to-match treated units, while potentially improving the quality of the resulting estimate, changes the estimand from the SATT to the feasible sample average treatment effect (FSATT):

$$\tau_{\mathcal{F}}=\frac{1}{|\mathcal{F}|}\sum_{t\in\mathcal{F}}Y_{t}(1)-Y_{t}(0),$$

where $\mathcal{F}$ denotes the set of indices of treated units with at least one control unit within $c$ $d_{V}^{(\infty)}$ units, i.e., $\mathcal{F}=\{t\in\mathcal{T}:\exists\ j\in\mathcal{C}\text{ with }d_{V}^{(\infty)}(\mathbf{X}_{t},\mathbf{X}_{j})\leq c\}$. The FSATT can differ from the SATT if the excluded units have systematically different treatment impacts than the kept units.

Instead of shifting the estimand based on a selected caliper, we propose instead assigning each treated unit $t$ an adaptive caliper $c_{t}$, with

$$c_{t}=\max\{c,\alpha d_{t}\}\mbox{ with }d_{t}\equiv\min_{j:Z_{j}=0}d(\mathbf{X}_{t},\mathbf{X}_{j}),$$

(4)

where $c$ is our global minimum caliper that we default to 1 given our distance metric, $d_{t}$ is the distance between unit $t$ and its nearest control-unit neighbor (Dehejia and Wahba,, 2002), and $\alpha\geq 1$ is an inflation factor that allows for catching all control units that are similarly close to the closest unit.

The adaptive caliper guarantees all treated units are matched to at least one control. The floor $c$ value allows for taking advantage of data-rich environments; for treated units with many controls, we will not shrink to an overly small caliper but instead keep all those controls measured as “reasonably similar” as a comparison group.

In data-rich contexts, the adaptive caliper may also be selected so that the resulting matched sets work well with synthetic controls (introduced below), e.g., we can set $c_{t}$ to be the smallest caliper such that treated unit $t$ has $p+1$ within-caliper controls, where $p$ is the dimension of the covariate space.

In principle, with adaptive calipers our matched dataset allows for direct estimate of the SATT as all treated units are preserved. In practice, some treated units may have very poor matches, which would be indicated by large $c_{t}$ values. Because we have a direct measure of match quality via $c_{t}$, we can monitor the impact of these poor matches on our overall estimand, and possibly deliberately trim based on our diagnostics.

An important step in any matching procedure is to assess the resulting matches. Classic approaches would be to conduct balance checks by comparing marginal covariate distributions between the treated and control groups (usually people will look at standardized mean differences). Such marginal balance checks may reveal significant departures from joint balance, but cannot confirm when joint balance is approximately achieved, as demonstrated in Toy Examples 1 and 2. Checking low-dimensional summaries of joint balance can also fail to assess overlap or identify subsets of the treated units for which it may be easier or more difficult to estimate treatment effects.

Using a distance-metric caliper, on the other hand, directly ensures good covariate balance (e.g., see Proposition 4.2), if the caliper is fixed. We can extend this idea to using the unit-specific caliper values $c_{t}$ across all treated units to assess the estimate-estimand tradeoff, creating balance-sample-size frontier plots (King et al.,, 2017) to show how dropping poorly matched treated units (i.e., those with large $c_{t}$) affects both potential bias and the SATT estimate for the remaining sample. Also see Aikens et al., (2020); Aikens and Baiocchi, (2023) for other approaches to making diagnostic plots. We can also summarize characteristics of units with different $c_{t}$ values to better understand regions with poorer overlap.

Given matched units from the design phase of a matching analysis, the final step is to produce an estimate. With high-quality matches, the SATT may be, in principle, estimated with a simple average:

$$\hat{\tau}^{avg}=\frac{1}{n_{T}}\sum_{t\in\mathcal{T}}\left(Y_{t}-\frac{1}{|\mathcal{C}_{t}|}\sum_{j\in\mathcal{C}_{t}}Y_{j}\right).$$

This is quite clear, but when matching is imperfect, it can be much lower performing than other methods that do further adjustment.

In this paper, instead of using the average weight $\frac{1}{|\mathcal{C}_{t}|}$ for each matched control, we calculate weights with the synthetic control method (SCM; Abadie et al.,, 2010) within the matched sets. For each treated unit $t$, we find convex weights, i.e., weights $w_{jt}\geq 0$ that sum to 1, for its matched control units that minimize covariate imbalance as measured by a scaled distance metric $d_{V}(\cdot,\cdot)=d_{V}^{(2)}(\cdot,\cdot)$ or $d_{V}^{(\infty)}(\cdot,\cdot)$:

	$\displaystyle\mathbf{w}_{t}:=\operatorname*{arg\,min}_{\{w_{jt}:j\in\mathcal{C}_{t}\}}$	$\displaystyle\hskip 5.69054ptd_{V}(\mathbf{X}_{t},\sum_{j\in\mathcal{C}_{t}}w_{jt}\mathbf{X}_{j})$
	s.t.	$\displaystyle\sum_{j\in\mathcal{C}_{t}}w_{jt}=1$
		$\displaystyle 0\leq w_{jt}\leq 1\text{ for }j\in\mathcal{C}_{t},$

where we have written the weight for control unit $j$ associated with treated unit $t$ as $w_{jt}$. The “synthetic control” unit for treated unit $t$ gets covariates $\sum_{j\in\mathcal{C}_{t}}w_{jt}\mathbf{X}_{j}$ and outcome $\sum_{j\in\mathcal{C}_{t}}w_{jt}Y_{j}$, and the ATT estimate is taken as a simple difference-in-means between the outcomes of the treated units and their synthetic controls:

$\displaystyle\hat{\tau}_{t}^{SC}=\frac{1}{n_{T}}\sum_{t\in\mathcal{T}}\big{(}Y_{t}-\sum_{j\in\mathcal{C}_{t}}w_{jt}Y_{j}\big{)}.$

See Appendix B.6 for further detail.

Our approach deviates from the standard SCM setup in a few ways. First, while Abadie et al., (2010) introduces SCM as a quadratic programming problem using scaled $L_{2}$ distance, we also allow SCM to be implemented as a linear programming problem using scaled $L_{\infty}$ distance. Second, while synthetic controls are typically used in time-series settings with past outcomes as additional covariates, we directly apply them in our setting without focus on past outcomes. Finally, the SCM introduced in Abadie et al., (2010) includes an outer optimization to learn an “optimal” scaling matrix $V$, whereas we simply use the $V$ matrix implied by the given covariate-wise caliper. This follows other approaches to the SCM. See, e.g., Ben-Michael et al., 2021b .

Using synthetic controls in the analysis phase provides two primary benefits. First, synthetic controls are highly transparent. Because synthetic controls explicitly produce a counterfactual for each treated unit, researchers can directly check whether each counterfactual seems reasonable.

Second, as we prove, synthetic controls naturally reduce bias within calipers as they are mathematically equivalent to linear interpolation. While linear interpolation over long distances can lead to bias, interpolating over short distances, such as within a caliper, typically improves results due to the local linearity of smooth outcome functions.

To recap, we propose matching by first selecting an interpretable distance metric that can be used to assess similarity between units based on their covariate profiles. This metric should be interpretable and justifiable by those using it.

Once a metric is in place, identify sets of local controls for each treated unit, possibly adaptively adjusting the caliper (radius) defining locality unit by unit. Finally, build synthetic comparison units by using a variant of the synthetic control procedure for each unit to obtain further reductions in bias. The final estimate is simply the average of the pairwise differences between treated and synthetic control. Finally, run some diagnostics, examining measures such as how the estimated impact shifts with the trimming of hard-to-match units.

We denote the use of synthetic controls within adaptive $L_{\infty}$ calipers as Caliper Synthetic Matching (CSM). We generally use a scaled $L_{\infty}$ distance for its simplicity, its connections to exact matching, and its connections to CEM. Adapting the caliper enables clear diagnostic plots of the estimate-estimand tradeoff and the synthetic controls produce interpretable local bias corrections.

Iacus et al., (2011) introduces the Monotonic Imbalance Bounding (MIB) class of matching methods. MIB matching methods directly control covariate balance between the treated and matched control groups, independently for each covariate. As a result, MIB matching methods enjoy desirable properties such as bounded covariate imbalance and bounded estimation error, under reasonable assumptions.

Although matching using Mahalanobis distance alone is not MIB, matching using a distance-metric caliper is MIB as long as the caliper for each covariate may be tuned without affecting the caliper for any other covariate (Iacus et al.,, 2011). Any such covariate-wise caliper can be satisfied by a scalar caliper on an appropriately scaled distance metric. For example, if $p=2$ and we want to ensure $|X_{t1}-X_{j1}|\leq 2$ and $|X_{t2}-X_{j2}|\leq 5$, we may define $V=\begin{bmatrix}\frac{1}{2}&0\\ 0&\frac{1}{5}\end{bmatrix}$ so that restricting $d^{(\infty)}_{V}(\mathbf{X}_{t},\mathbf{X}_{j})\leq 1$ satisfies the desired caliper. By the triangle inequality, the bound on the distance translates to bounds on the individual covariates. This idea is formalized by Proposition 4.1, which shows that matching using a distance-metric caliper is a member of the MIB class.

Denote the weighted mean for covariate $k$ for the treated and control units after matching as $\bar{X}^{\mathbf{w}}_{Tk}$ and $\bar{X}^{\mathbf{w}}_{Ck}$, respectively, where the weight is obtained from matching. More detailed definitions can be found in Appendix B.3. We then have:

Proposition 4.1 not only shows that changing $\pi_{k}$ for one variable does not affect the imbalance on the other variables, but also shows that covariate balance is controlled by $V$. Again, note that the choice to bound $d^{(2)}_{V}(\mathbf{X}_{t},\mathbf{X}_{j})$ and $d^{(\infty)}(\mathbf{X}_{t},\mathbf{X}_{j})$ by 1 is arbitrary, e.g., bounding $d^{(\infty)}_{V}(\mathbf{X}_{t},\mathbf{X}_{j})\leq 1$ is equivalent to bounding $d^{(\infty)}_{V}(\mathbf{X}_{t},\mathbf{X}_{j})\leq c$ for $c>0$ and $V=diag\{\frac{c}{\pi}\}$.

With adaptive calipers, CSM is MIB for the feasible treated units within $\mathcal{F}$ but $d_{V}(\mathbf{X}_{t},\mathbf{X}_{j})\leq 1$ does not hold for other treated units, and thus the bound does not apply. That said, the adaptive calipers still provide transparency about the extent to which CSM leaves the MIB class in terms of number of units and the amount they deviate as recorded by their caliper sizes. In the subsequent subsections, we focus on cases when all treated units have controls within a set caliper and thus are feasible, i.e., $\mathcal{M}_{T}=\mathcal{F}$, to focus on several properties of the MIB matching methods.

Having shown the bounded marginal balance, we now show how distance-metric caliper matching methods control joint covariate imbalance. Write the empirical joint covariate distributions of the treated and control units as:

	$\displaystyle f_{T}(\mathbf{x})$	$\displaystyle\equiv\frac{1}{n_{T}}\sum_{t\in\mathcal{T}}\delta_{\mathbf{X}_{t}}(\mathbf{x})$
	$\displaystyle f_{C}(\mathbf{x})$	$\displaystyle\equiv\frac{1}{n_{T}}\sum_{t\in\mathcal{T}}\sum_{j\in\mathcal{C}_{t}}w_{jt}\delta_{\mathbf{X}_{j}}(\mathbf{x}),$

where $\delta_{\mathbf{X}}(\cdot)$ represents a Dirac delta function at $\mathbf{X}$, i.e., $\delta_{\mathbf{X}}(\mathbf{y})=1$ if $\mathbf{y}=\mathbf{X}$ and $0$ otherwise. These empirical distributions are simply weighted sums of point masses located at the units’ values.

To demonstrate how distance-metric calipers control the difference between $f_{T}$ and $f_{C}$, we use Wasserstein distance. Formally, the $q$-Wasserstein distance between probability distributions $P$ and $Q$ with distance metric $d(\cdot,\cdot)$ is:

$\displaystyle\mathcal{W}_{q}(P,Q)=\inf_{\pi\in\Pi(P,Q)}E_{\pi}\big{[}d(\mathbf{X},\mathbf{Y})^{q}\big{]}^{1/q},$

where the infimum is over all couplings (joint distributions) $\pi$ that have marginal distributions of $P$ and $Q$ ($Pi(P,Q)$). More intuitively, Wasserstein distance is also known as “earth-mover’s distance”: viewing $P$ and $Q$ as piles of soil each with total mass 1, $W_{q}(P,Q)$ measures the minimum “cost” to move soil to make $P$ distribute as $Q$ (or vice-versa), where the “cost” of moving $k$ units of soil from $\mathbf{x}$ to $\mathbf{y}$ is $k\cdot d(\mathbf{x},\mathbf{y})$. Wasserstein distance therefore uses a distance metric between points in $\mathbb{R}^{p}$ to measure distance between full probability distributions on $\mathbb{R}^{p}$.

With this notation in hand, Proposition 4.2 shows that radius matching bounds the Wasserstein distance between $f_{T}$ and $f_{C}$.

We make a few remarks about Proposition 4.2. First, while the notation is technical, the intuition is straightforward. Distance-metric calipers control how far each treated unit’s covariates can be from its matched controls’ covariates. Since $f_{T}$ and $f_{C}$ are weighted sums of the point masses associated with these covariates, the calipers must also control the distance between $f_{T}$ and $f_{C}$. For an exact caliper $c=0$, Proposition 4.2 simply states that exact matching guarantees that $f_{T}$ coincides with $f_{C}$. In practice, however, reducing the caliper size to $c=0$ drops all of the treated units, leaving the proposition to be vacuously true; $c$ must be chosen with the estimate-estimand tradeoff in mind. To the best of our knowledge, this is the first theoretical result for a matching method’s bound on joint balance. Iacus et al., (2011) did show that CEM improved joint balance, but only through simulation.

The Wasserstein distance depends on the chosen distance metric. In the proposition, $\mathcal{W}^{(l)}_{q}(f_{T},f_{C})$ is dependent on $d_{V}$. Thus, if our $V$ changes, the Wasserstein distance also changes. However, the core idea of the proof remains intact: the bounds on the covariates due to matching ensure a joint balance between the covariates of the matched controls and treatments, thereby providing local control and helping to avoid those adverse situations discussed in Section 2.2.

Control over joint covariate imbalance naturally implies control over marginal imbalances. For example, Proposition 4.3 shows how distance-metric calipers also bound the distance between the ($p$-dimensional) empirical weighted covariate means of the treated and matched control units, respectively denoted $\bar{\mathbf{X}}_{T}$ and $\bar{\mathbf{X}}_{C}$.

The above proposition is in effect a joint version of Proposition (b), which gives bounds on the covariates in turn.

Lastly, we note that other distance metrics between $P$ and $Q$ could be chosen. For instance, Iacus et al., (2011) uses the $L_{1}$ distance, demonstrating that Coarsened Exact Matching reduces the joint imbalance between the control and treatment groups using an empirical example. We use $L_{1}$ distance for its computational simplicity and use the Wasserstein distance because its properties facilitate easier proof construction, as it inherently utilizes a distance metric.

In summary, distance-metric calipers enable precise control of joint covariate imbalance. While these bounds are not necessarily small in practice, they guarantee that observed imbalance cannot be too great, even in the worst case. This leads to a variety of desirable properties which we illustrate in the following sections.

Because MIB matching methods bound the distance between the covariates of matched units, they naturally bound the distance between smooth functions $f:\mathbb{R}^{p}\to\mathbb{R}$ of those covariates as well. Write the control potential outcome for unit $i$ as $Y_{i}(0)=f_{0}(\mathbf{X}_{i})+\epsilon_{i}$. Then assuming that $f_{0}(\cdot)$ is smooth (i.e., Lipschitz) immediately bounds the bias of any FSATT estimate produced by a method using a distance-metric calipers. (See Appendix B.2 for technical details about Lipschitz functions in $\mathbb{R}^{p}$.)

Proposition 4.4 states that for distance-metric caliper matching methods, worst-case bias is proportional to the caliper size $c$.³³3Proposition 4.4 applies to a slightly different set of methods than Proposition 1 from Iacus et al., (2011), which proves a similar bias bound for MIB matching methods (see Appendix B.3). As in Proposition 4.2, setting $c=0$ shows that exact matching leads to unbiased estimates, but in practice we must use $c>0$ to navigate the estimate-estimand tradeoff without dropping all of the treated units.

Of course, we rarely know the Lipschitz constant $\lambda$ in practice, so Proposition 4.4 does not generate empirical bias bounds. Nonetheless, Proposition 4.4 illustrates how distance-metric calipers control bias. If each treated unit is close to all of its matched controls in covariate space, its expected counterfactual outcome must be close to their expected outcomes, for reasonable (i.e., smooth) outcome functions. As a result, any weighted average of the control units’ outcomes cannot differ too much in expectation from the treated unit’s true counterfactual outcome, regardless of the specific form of $f_{0}(\cdot)$.

In practice, we can ideally get estimates that beat this bound by assuming local linearity and adjusting further for residual imbalance. We discuss this in the next section.

While using distance-metric calipers bounds bias, using synthetic controls within these calipers actively reduces bias. Specifically, synthetic controls naturally remove linear bias by conducting local linear interpolation. This means that if $f_{0}(\cdot)$ is linear, synthetic controls would completely eliminate bias if the weights achieved perfect balance on $X$. For nonlinear $f_{0}(\cdot)$, bias will be reduced to higher-order nonlinear trends, which are, in general, less influential within tight calipers for smooth functions. Proposition 4.5 more precisely shows how exact synthetic controls eliminate linear bias.

In Proposition 4.5, the $o(c)$ term represents higher-order terms which go to zero more quickly than does the caliper $c$ as $c$ shrinks toward zero. Compare to the bias bound of $\lambda c$ without the synthetic control step in Propostion 4.4: we have lost the $\lambda c$ term, leaving something smaller (asymptotically). In practice, shrinking $c$ to zero drops all treated units, but the notation highlights how implementing synthetic controls within calipers takes advantage of local linearity. In particular, while linear interpolation across large distances can lead to significant interpolation bias (Kellogg et al.,, 2021), restricting the donor-pool units to be within a caliper distance from the treated unit reduces the impact of nonlinearities.

As discussed in Section 3.2, radius matching methods have many similarities with coarsened exact matching (CEM) (Iacus et al.,, 2012), which we celebrate by matching two of the three letters. In particular, radius matching and CEM both possesses the imbalance-bounding and bias-bounding guarantees discussed in the previous sections.

To clarify the benefits of radius matching, we consider an example with two covariates $X_{1}$ and $X_{2}$, as in Figure 2. The coarsening of each variable is made equal sized. We visualizes the CEM grid defined by the coarsening of the two covariates, along with an equivalently sized $L_{\infty}$ caliper around treated unit $t_{1}$.

Because $t_{1}$ lies near a boundary defined by the covariate coarsening, CEM does not match $t_{1}$ to $c_{1}$, while the $L_{\infty}$ caliper does. On the other hand, CEM matches $t_{1}$ to $c_{2}$ because they lie in the same caliper grid, even though the two units lie more than one $L_{\infty}$ unit apart from each other. Figure 2 shows how, given a fixed caliper size, CEM guarantees that treated units lie within two $L_{\infty}$ units of their matched controls whereas the $L_{\infty}$ caliper guarantees the distance is no more than one unit. By centering calipers on each treated unit, radius matching matches each treated unit to all nearby control units while guaranteeing imbalance and bias bounds that are twice as tight as those guaranteed by CEM. In other words, when each method is “equally sized” in terms of how large a treated unit’s cachement area for possible control is, radius matching has tighter control of bias.

As a result, we can present the following proposition, which highlights the improved guarantee of bias bounds. Specifically, it improves the upper bound of the bias for CSM compared to CEM.

Naturally, there are tradeoffs for these improved bias bounds. Computationally, radius matching requires computing distances between each treated unit and the control units, an operation of order $n_{t}n_{c}$, unlike CEM which only requires a frequency tabulation of order $n_{t}+n_{c}$. Non-uniform calipers are also slightly easier to implement via covariate coarsenings, though for many common covariates one can directly transform the covariate and use a uniform caliper to replicate a non-uniform caliper.⁴⁴4E.g., rather than non-uniformly coarsening income as { $0-20k, $20k-50k, $50k-100k, $100k+ }, it may be more reasonable to log-transform the income covariate and use a uniform caliper to avoid, e.g., concluding that an individual earning $20k is as far from an individual earning $20.1k as they are from an individual earning $50k. Overall, however, radius matching preserves the transparency and interpretability of CEM while significantly improving on its useful bias and imbalance bounding properties.

We also note that CSM offers the flexibility to select units that are challenging to match effectively, a feature that CEM lacks. In CEM, after specifying the desired coarsening size, an analyst can only determine if a treated unit has found a match by checking if there is a control within the same stratum. Conversely, with the adaptive caliper, we can evaluate the treated units based on the size of their adaptive calipers, which indicate the extent of their matching difficulty. By contrast, CEM allows for a clean inferential strategy by viewing the bins as strata in a post-stratified experiment; with CSM inference is less clear cut, as we will discuss below.

Reducing caliper sizes reduces the potential bias of the resulting SATT estimate, as shown by Proposition 4.4, but may increase the estimate’s variance due to reducing the number of controls and, without adaptive calipers, the number of treated units. To formalize this idea, we introduce the (conditional) mean-squared error for the SATT $\tau$:

$\displaystyle CMSE=E[(\hat{\tau}-\tau)^{2}|\mathbf{X},\mathbf{Z}],$

where the expectation is conditioned on the observed covariates and treatment assignments. Then, using a working model of the potential outcomes of $Y_{i}(z)=f(X_{i})+z\tau_{i}+e_{i}$, where $e_{i}$ is a 0-centered noise term, we can use standard algebraic manipulation to show that:

$\displaystyle\hat{\tau}-\tau$

$\displaystyle=Bias+Error$

(5)

where

$$\begin{split}Bias&=\frac{1}{n_{T}}\sum_{t\in\mathcal{T}}\sum_{j\in\mathcal{C}_{t}}w_{jt}\big{(}f_{0}(\mathbf{X}_{t})-f_{0}(\mathbf{X}_{j})\big{)}\\ Error&=\frac{1}{n_{T}}\sum_{t\in\mathcal{T}}(\epsilon_{t}-\sum_{j\in\mathcal{C}}w_{jt}\epsilon_{j}).\end{split}$$

(6)

We then have

$\displaystyle CMSE$

$\displaystyle=Bias^{2}+Var,$

where, using our working model,

$\displaystyle Var$

$\displaystyle=\frac{1}{n_{T}^{2}}\sum_{t\in\mathcal{T}}\sigma_{t}^{2}+\frac{1}{n_{T}^{2}}\sum_{j\in\mathcal{C}}(\sum_{t\in\mathcal{T}}w_{jt})^{2}\sigma_{j}^{2},$

(7)

with $\sigma_{i}$ representing the population sampling variance of unit $i$ conditional on its covariates $\mathbf{X}_{i}$ (Kallus,, 2020).

If we assume homoskedasticity of the residuals within treatment arms (i.e., $\sigma_{i}=\sigma_{C}$ in the control group) and the conditions of Proposition 4.4 with caliper $c$, we can bound CMSE as:

	$\displaystyle CMSE$	$\displaystyle\leq(\lambda c)^{2}+\Big{(}\frac{\sigma^{2}_{T}}{n_{T}}+\frac{\sum_{j\in\mathcal{C}}(\sum_{t\in\mathcal{T}}w_{jt})^{2}\sigma^{2}_{C}}{n_{T}^{2}}\Big{)}$
		$\displaystyle=(\lambda c)^{2}+\frac{\sigma_{T}^{2}}{ESS(\mathcal{T})}+\frac{\sigma^{2}_{C}}{ESS(\mathcal{C})},$		(8)

where the effective sample size (ESS) of a set $\mathcal{S}$ of units with weights $w_{i}$ is

$\displaystyle ESS(\mathcal{S})=\frac{(\sum_{i\in\mathcal{S}}w_{i})^{2}}{\sum_{i\in\mathcal{S}}w_{i}^{2}},$

(9)

with $w_{i}=\sum_{t\in\mathcal{T}}w_{it}$ for $i\in\mathcal{C}$, and $w_{i}=1/n_{T}$ for $i\in\mathcal{T}$.

Equation 8 clarifies the relationship between caliper size and the bias-variance tradeoff. For a fixed estimand, where we do not drop or add treated units as caliper size changes,increasing $c$ naturally exposes the resulting estimate to more bias. Increasing $c$ also generally reduces variance by dispersing weight across more control units, increasing the effective sample size of the control group.

That said, in contexts where $|\mathcal{C}|\gg|\mathcal{T}|$, the overall variance of the average impact estimate could be dominated by the variance associated with the treated units (of order $1/ESS(\mathcal{T})$, which remains unchanged even if more control units are added (which shrinks $1/ESS(\mathcal{C}$. To be more explicit, consider a single treated unit $t$ with matched controls $j\in\mathcal{C}_{t}$ all given uniform weights. Here the variance associated with the treated unit is $\sigma^{2}/1$ while the variance associated the controls is $\sigma^{2}/|\mathcal{C}_{t}|$ (assuming homoskedasticity). Even if $\mathcal{C}_{t}$ were large, we still have the initial $\sigma^{2}$ term. Once $\mathcal{C}_{t}$ reaches 4 or 5, increasing caliper size would have diminishing returns on variance reduction, suggesting that it may typically be better, in terms of CMSE, to use smaller calipers.

On the other hand, when matching with replacement, a single control $j$ may be assigned significant weight for multiple treated units $t$. In these cases, $\sum_{t\in\mathcal{T}}w_{jt}$ could be greater than 1 for some control units, driving $ESS(\mathcal{C})$ down. In other words, if the same controls get reused heavily enough, the variance associated with the control units may not be dominated by the variance associated with the treated units, even if the initial control pool is large.

How to estimate standard errors for our matching method requires special attention. Abadie and Imbens, through a series of papers (2006, 2008, 2011), argued that the bootstrap method was inadequate and developed an asymptotically valid approach for the $M$-nearest neighbor estimator as the numbers of treatment and control units increase infinitely, with $M$ remaining fixed. Otsu and Rai, (2017) introduced a weighted bootstrap methodology that relies on overlap of treated and control covariates.

We instead draw from the survey sampling literature (e.g., Potthoff et al.,, 1992) and plug in effective sample sizes and variance estimates into our Equation 8. This has been successfully applied in various different types of weighting estimators in causal inference (Ben-Michael et al.,, 2022; Lu et al.,, 2023; Keele et al.,, 2023).

Our approach differs from the Abadie and Imbens literature in several aspects: First, we utilize synthetic control weights instead of average $M$-nearest-neighbors weights. Second, we condition on the treated units as defined by their covariates, assume a stochastic model for the units’ outcomes, and make a working assumption of homoskedasticity. Third, we operate in a scenario where the number of treatment units ($n_{T}$) is low, while the number of control units ($n_{C}$) is high. As Ferman, (2021) noted, the asymptotics in this setting differ from those Abadie and Imbens described, with $n_{T}$ increasing to infinity.

Our variance estimator is the following plug-in estimator:

$\displaystyle\widehat{Var}(\hat{\tau})$

$\displaystyle=S^{2}\left(\frac{1}{n_{T}}+\frac{1}{\text{ESS}(\mathcal{C})}\right)$

(10)

where $S^{2}$ is an estimate of the homoskedastic residuals (assumed the same for treatment and control groups), and $\text{ESS}(\mathcal{C})$ follows 9. In particular, $S^{2}$ is a pooled variance estimator, pooling across clusters:

$$S^{2}=\frac{1}{N_{C}}\sum_{t}|\mathcal{C}_{t}|s^{2}_{t}\text{ with }N_{C}=\sum_{t}|\mathcal{C}_{t}|.$$

where each cluster’s residual variance is

$$s^{2}_{t}=\frac{1}{|\mathcal{C}_{t}|-1}\sum_{j\in\mathcal{C}_{t}}e_{tj}^{2},$$

with

$$e_{tj}=Y_{tj}-\bar{Y}_{t},\text{ where }\bar{Y}_{t}=\frac{1}{|\mathcal{C}_{t}|}\sum_{j\in\mathcal{C}_{t}}Y_{tj}.$$

We drop all clusters $\mathcal{C}_{t}$ with $|\mathcal{C}_{t}|=1$, including for the calculation of $N_{C}$. Although we could use the weights from the synthetic control to prioritize “good” controls, we opt for uniform weights over all matched units as the key element is control unit similarity, which is not necessarily optimized by the synthetic step.

Our estimates of $s_{j}^{2}$ are inflated by differences in $f_{0}(\mathbf{X}_{i})$ driven by variation in the $\mathbf{X}_{i}$ within each cluster. In other words, if the control units are widely dispersed and the expected outcome $f_{0}(X_{i})$ changes rapidly as $\mathbf{X}_{i}$ changes, then the variation in $f_{0}(X_{i})$ will be absorbed by the $s_{j}^{2}$ terms, resulting in overly large standard errors. This phenomenon was observed in our simulations, as discussed in Section 5.4. This inflation is not necessarily bad, however, as we would expect it to grow roughly in proportion to the size of the bias term, which is also driven by these differences (see the $f_{0}(X_{t})-f_{0}(X_{i})$ terms in Equation 6). One can thus view these SEs as predicting overall error (see, e.g., the discussion of standard errors as predicting overall error in Sundberg, (2003) and discussion of expanding of standard errors to include bias in Weidmann and Miratrix, 2021b ).

Our inference goal is the Root Mean Squared Error (RMSE) of our estimate, conditioned on the treatment group’s covariates, represented by $\sqrt{E[(\hat{\tau}-\tau)^{2}|\mathbf{X}_{\mathcal{T}}]}$. Other options are possible, as is well illustrated by the literature. Kallus, (2020) focuses on the conditional standard error, essentially the variance term. Abadie and Imbens, along with Otsu and Rai, (2017), target the unconditional coverage. Ferman, (2021) aims at the unconditional type-I error rate. In particular, it is important to note, as Imbens and Rubin, (2015) mentions, that the conditional and unconditional standard errors differ.

We compare CSM to a variety of popular matching, balancing, outcome regression, propensity score, and doubly robust methods, as described in Table 1.⁵⁵5We initially also included CEM using synthetic controls within each cell and caliper matching using simple averages within each caliper, but, for clarity, we now exclude these results since their performances typically lie between the performances of CSM and CEM.

We use default settings for all of the algorithms to standardize comparisons. We implement BART, TMLE, and AIPW in R using defaults from the dbarts (Dorie,, 2023), tmle (Gruber and Van Der Laan,, 2012), and AIPW (Yongqi Zhong et al.,, 2021) packages, respectively. For SuperLearner, the linear models include a simple mean, linear regression, and generalized linear regression models; the machine-learning models include the linear models as well as generalized additive models, random forests, BART, and XGBoost (Chen and Guestrin,, 2016).

To standardize comparisons for the matching methods, for each dataset we use the distance metric implied by the covariatewise caliper generated by coarsening each numeric covariate into five equally spaced bins. I.e., we use $d_{V}^{(\infty)}$ and a diagonal $V$ with entries $V_{kk}=5/(max(X_{k})-min(X_{k}))$. We do not tune the covariatewise caliper, assuming zero domain knowledge about variable importance. We conduct CEM using the same uniform coarsening of each covariate into five bins.

To build intuition for situations in which CSM may perform well, we first show results from the simulation based on the toy examples in Section 2.2. We simulate the covariates for 50 treated units each from multivariate normal distributions centered at $(0.25,0.25)$ and $(0.75,0.75)$, and 225 control units each from multivariate normal distributions centered at $(0.25,0.75)$ and $(0.25,0.75)$, all with covariance matrices $\begin{bmatrix}0.1^{2}&0\\ 0&0.1^{2}\end{bmatrix}$. We then add 100 control units distributed uniformly on the unit square to ensure we do have some overlap. We finally generate outcomes for each unit $(x_{1},x_{2})$ as:

$$Y=f_{0}(x_{1},x_{2})+Z\cdot\tau(x_{1},x_{2})+\epsilon,$$

where $f_{0}$ is the density function for a multivariate normal distribution centered at $(0.5,0.5)$ with covariance matrix $\begin{bmatrix}1&0.8\\ 0.8&1\end{bmatrix}$, $\tau(x_{1},x_{2})=3x_{1}+3x_{2}$ is the true treatment effect, and $\epsilon\sim N(0,0.5^{2})$ is homoskedastic noise. Figure 3 plots a sample simulated dataset.

Figure 4 shows the root mean-squared error and absolute bias of the point ATT estimates for the various methods.

We see that CSM performs well, even compared to complex machine-learning methods.

We emphasize that the goal of this simulation is not to demonstrate CSM’s performance, but rather to illustrate settings in which we may expect good performance. In the toy example, the interaction between the two covariates drives both the control potential outcome function $f_{0}(X_{1},X_{2})$ and the implicit propensity score. Because the covariates are interacted with each other, joint covariate balance is very important. As a result, those methods that do not target joint balance or otherwise do not have access to the interaction term between $X_{1}$ and $X_{2}$ perform poorly due to high levels of bias.

The datasets from Kang and Schafer, (2007), Hainmueller, (2012), and the 2016 American Causal Inference Conference competition (Dorie et al.,, 2017) have canonically been used to compare methods for observational causal inference. Table 2 contains basic information about the settings we used for each dataset. Please see the original papers for further details.

Figure 5 summarizes the results of our simulations using these datasets and illustrates the tradeoffs made by CSM.

We see that CSM tends to outperform the simple matching, balancing, and outcome or propensity-score modeling approaches, but that it is outperformed by the complex machine-learning approaches known to do well on these types of datasets (Dorie et al.,, 2017). CSM is designed to maximize performance while preserving the simple intuitions underlying exact matching. Under such constraints, it is remarkable that CSM performs nearly as well as black-box machine-learning approaches on the Kang and Schafer, (2007) and Hainmueller, (2012) datasets.

The trends in Figure 5 highlight important concepts regarding locality in observational causal inference. As the number of covariates increases, the matching methods tend to deteriorate in performance relative to the complex modeling methods. Caliper-based matching methods only use local information. This behavior protects them against incorrect extrapolation, but also prevents them from correctly extrapolating in cases where overlap is unnecessary.

In many simulated (and real) datasets, extrapolating marginal effects can improve counterfactual predictions. For example, suppose increasing $X_{1}$ clearly increases outcomes among controls which have low values of $X_{2}$, and we would like to predict counterfactual outcomes for treated units with high values of $X_{2}$. A linear model fit to the controls may extrapolate and assume that increasing $X_{1}$ always increases outcomes regardless of the value of $X_{2}$, while an exact matching estimator would not do so. Increasing the number of covariates exaggerates this behavior, since models extrapolate more marginal trends. Figure 5 shows us that this type of model-based marginal extrapolation improves results for these simulated datasets, since in the considered DGPs there are few high-dimensional interactions to falsify such extrapolation.

Matching aims to produce transparent, model-free causal inferences. As shown by Figure 5, this transparency generally comes with a cost in terms of performance. The simulations show that while avoiding model-based extrapolation controls bias in the worst case (e.g., Proposition 4.4), it can be too conservative in settings where such extrapolation is helpful. Whether these costs are outweighed by the ability to clearly explain how counterfactual predictions are made depends on the particular causal inference setting. Nonetheless, CSM performs competitively in these canonical simulations, making it an attractive option in settings where explainability is paramount.

We next assess the variance estimator described in Section 4.7. We revisit the example from Section 5.2 but vary the degree of overlap. Specifically, we increase the overlap by increasing the proportion of control units distributed uniformly on the unit square and reducing the proportion of controls centered at (0.25, 0.75) and (0.75, 0.25). For the range of overlap, see the three illustrative datasets in Figure 6. These scenarios correspond to the same distribution of treated units, but different pools of potential controls to select matches from. For each degree of overlap, we generate 500 trials.

Our primary estimation target is the SATT.

To capture tuning $c$ for bias reduction, we use an adaptive strategy for the caliper size, shrinking the caliper to select no more than five units in regions with great density, and expanding it to include at least one unit in regions of low density. With larger calipers, the synthetic control optimization tends to select only a few units of the set when we have a low-dimensional covariate set, and the selected units tend to be more distant from the treated unit than others within the neighborhood defined by the caliper; for future work we could follow Ben-Michael et al., 2021b and regularize the weights to take more advantage of rich areas of control units.

Table 3 presents the results. The column “avg $\widehat{SE}$” shows the estimated standard error averaged over 500 iterations. The “avg $SE$” column shows the true conditional standard error,⁶⁶6This value is calculated as the standard deviation of $\hat{\tau}-bias-\tau$ across the simulations in order to remove variation due to simulation iterations having different true SATT values. Overall, we observe that our method slightly overestimates the true standard error due to the inflation caused by absorbing variation in $f_{0}$, as discussed in Section 4.7.

As expected, both the true standard error and bias go down as overlap increases, due to the steadily increasing effective sample size and improved ability to find close matches. In particular, when there is low overlap, we re-use many control units, and only at high overlap is our control group getting close to the treatment group in size. Additionally, with high overlap, there are more control units near the concentration of treated units, allowing for closer matches and thus less bias.

Finally, the “Coverage” column shows the coverage of the 95% confidence interval. The CSM method effectively maintains a small degree of bias and gives reasonably estimated standard errors, resulting in CI coverage close to the nominal 95% across all the scenarios.

An alternate form of inference would some form of bootstrap. Unfortunately, we found that bootstrap methods, including the weighted bootstrap in Otsu and Rai, (2017) applied to our context, and the naïve casewise bootstrap, did not work effectively. We leave further development of bootstrap inference to future work.

We start our CSM analysis by choosing $L^{\infty}$ as the distance metric. Next, we set the covariate-wise calipers. For the standardized test scores $X_{1}$ to $X_{3}$, we set the covariate-wise calipers to 0.2, meaning we want matched schools to differ at most 0.2 standard deviations in test scores. For $X_{4}$ (indicator of being in Sao Paolo), we set the covariate-wise caliper to $1/1000$, a small value to ensure the matching of $X_{4}$ is exact.

The distance-metric caliper $c$ is usually set to a default of 1, as done in Proposition 4.1, to give direct interpretation to the covariate-wise calipers $\bm{\pi}$. In practice, however, given $\bm{\pi}$ we can tune $c$ to guarantee each treated unit gets enough but not too many matched controls, thereby optimally trading off data use and potential bias (see Section 6.3). When we set $c<1$, we obtain tighter matches than initially planned with $\bm{\pi}$. When we set $c>1$, our matches can be worse than initially planned.

To tune $c$, we plot histograms of the distances of the top-1, top-2, and top-3 matches for each treated unit, and look for a natural break. We see that, for most treated units, even the third worst match has a distance of around 0.35 or less. We can thus focus on bias reduction, and set our distance-metric caliper $c$ to $0.35$. See Appendix A.1 for further details.

Some of our treated units have no controls within this set minimum caliper. For these units, we use an adaptive caliper, matching the units to their nearest neighbor. We then apply the synthetic control method to obtain weights for each set of matched controls. Finally, we use these weighted controls to construct the point estimate and estimated standard error, using the inferential method outlined in Section 4.7.

Our chosen caliper results in 49 (91%) treated units having at least one matched control within a radius of $c=0.35$. These are our “feasible” treated units. Among the 49 feasible treated units, all matched controls are within 0.2 standardized scores for the years 2007, 2008, and 2009. Figure 7 illustrates how the differences in marginal means for three matching variables vary when adding infeasible treated units in order of increasing adaptive caliper size, until all 54 treated units are included.

By Proposition 4.3, the marginal mean difference of each covariate must be within $c\cdot\pi_{k}$, where $\pi_{k}$ is the covariate caliper of the $k$-th covariate, and $c$ is the distance-metric caliper. For $X_{1}$ to $X_{3}$, this value is $0.35\cdot 0.2=0.07$. For $X_{4}$, this value is $0.35\cdot\frac{1}{1000}=3.5\times 10^{-4}$. The leftmost point in each plot in Figure 7 shows the difference in marginal means between the treated units and their synthetic controls for the 49 feasible treated units, with differences all well below the corresponding $c\cdot\pi_{k}$ values. This result is not surprising: $c\cdot\pi_{k}$ represents a worst-case guarantee, and imbalances within each matched control set can cancel out.

An important use of Figure 7 is to provide a diagnostic on covariate balance. The plot assesses the approximate joint balance through marginal covariate balance while including increasingly infeasible units. While joint and marginal balance is guaranteed for feasible units, it is not guaranteed when more infeasible units are added. In this dataset, the marginal balances remain good even when all infeasible treated units are included.

The synthetic control step after matching ideally improves the comparability of the controls matched to each treated unit, but can also reduce the effective sample size of the control group by downweighting some units. We next assess this bias-variance trade-off, comparing SCM with classic radial matching (where we take the simple average of all close controls as the counterfactual for each treated unit in turn) and one-nearest-neighbor (1-NN) matching. Both can be viewed as weights on the controls:

1. 1.

   Simple average weight: let $w_{jt}=\frac{1}{|\mathcal{C}_{t}|}$ for each control unit. This is the default weighting method used in the matching literature after matches are made.

2. 2.

   One-nearest-neighbor⁷⁷7If unit $t$ has $k$ unites tied for nearest neighbor, we assign each a weight of $w_{jt}=\frac{1}{k}$.: let $w_{jt}=\begin{cases}1&\text{if unit }j\text{ is closest to unit }t\\ 0&\text{otherwise.}\end{cases}$

   This weight corresponds to 1-to-1 matching.

The results for SCM, average, and 1-NN weights on the feasible treated units are shown on Table 5. We analyze the bias-variance trade-off through two metrics: the mean covariate difference (a proxy for bias, Equation 6) and the effective sample size (ESS, a proxy for variance, Equation 7). The mean covariate difference, shown in the first column of Table 5, is the first-order term in Taylor’s expansion of bias. It is also referred to as the extrapolation bias in Kellogg et al., (2021), representing how far the weighted control is from the treated unit in $d_{V}^{\infty}$ in the covariate space. SCM performs better than simple average and 1-NN since it is optimized to minimize this bias.

Under homoskedasticity, the variance of an estimator is inversely associated with ESS, meaning a larger ESS implies smaller variance. ESS will be larger when more controls are used and when weights are more evenly distributed among those selected. Average weights achieve the highest ESS by assigning non-zero weights to all 202 distinct controls within the caliper distance of at least one of the 49 feasible units. In contrast, 1-NN uses only 49 distinct controls, as each treated unit is matched to a unique nearest neighbor. SCM falls between these scenarios, assigning zero weights to some control unit otherwise within caliper distance, but using multiple controls where possible. Overall, SCM performs well in the bias-variance trade-off by achieving the lowest imbalance (bias) and moderate variance.

We finally present the estimated treatment effect on the feasible units and also show the influence of infeasible treated units on the ATT estimate. Before interpreting the results, recall that, given our within study design, we expect to see no treatment impact. In other words, assuming our estimation is sound, we should expect an “impact estimate” of zero.

Figure 8 presents the main results of the analysis, providing a series of confidence intervals given a steady increase in the number of units kept in the analysis. Each confidence interval is constructed using the method in Section 4.7. The leftmost confidence interval, representing the feasible set, covers zero, indicating no significant effect as desired. As we include more infeasible units, the ATT varies slightly, but is overall stable, and all confidence intervals include 0. This stability aligns with Figure 7, which shows covariate balances only vary slightly as more infeasible units are included.

The SATT estimates from CSM and the other two methods are presented in Table 6. Using the point estimates and standard errors, all methods conclude no treatment effect. The standard error follows a trend where 1-NN is the largest, Average is the smallest, and CSM is in the middle, consistent with the trend we observed in Section 6.3: CSM uses more data than 1-NN, but sacrifices some precision to ideally reduce bias.