Distributionally Robust Instrumental Variables Estimation

Consider the following standard linear instrumental variables regression model with $X\in\mathbb{R}^{p},Z\in\mathbb{R}^{d}$ where $d\geq p$, and $\beta_{0}\in\mathbb{R}^{p},\gamma\in\mathbb{R}^{d\times p}$:

$\displaystyle\begin{split}Y&=\beta^{T}_{0}X+\epsilon,\\ X&=\gamma^{T}Z+\xi.\end{split}$

(2)

In (2), $X$ are the endogenous variables, $Z$ are the instrumental variables, and $Y$ is the outcome variable. The error terms $\epsilon$ and $\xi$ capture the unobserved (or residual) components of $Y$ and $X$, respectively. We are interested in estimating the causal effects $\beta_{0}$ of the endogenous variables $X$ on the outcome variable $Y$ given independent and identically distributed (i.i.d.) samples $\{X_{i},Y_{i},Z_{i}\}_{i=1}^{n}$. However, $X$ and $Y$ are confounded through some unobserved confounders $U$ that are correlated with both $Y$ and $X$, represented graphically in the directed acyclic graph (DAG) below:

Mathematically, the unobserved confounding can be described by the moment condition

$$\mathbb{E}\left[X\epsilon\right]\neq\mathbf{0}.$$

As a result of the unobserved confounding, the standard ordinary least squares (OLS) regression estimator of $\beta_{0}$ that regresses $Y$ on $X$ is biased. To address this problem, the IV estimation approach leverages access to the instrumental variables $Z$, also often called instruments, which satisfy the moment conditions

	$\displaystyle\text{rank}(\mathbb{E}\left[ZX^{T}\right])$	$\displaystyle=p,$		(3)
	$\displaystyle\mathbb{E}\left[Z\epsilon\right]=\mathbf{0},\mathbb{E}\left[Z\xi^{T}\right]$	$\displaystyle=\mathbf{0}.$		(4)

Under these conditions, a popular IV estimator is the two stage least squares (TSLS, sometimes also stylized as 2SLS) estimator (Theil,, 1953). With $\Pi_{Z}\mathrel{\mathop{\mathchar 58\relax}}=\mathbf{Z}(\mathbf{Z}^{T}\mathbf{Z})^{-1}\mathbf{Z}^{T}$ and $\mathbf{X,Y,Z}$ matrix representations of $\{X_{i},Y_{i},Z_{i}\}_{i=1}^{n}$ whose $i$-th rows correspond to $X_{i},Y_{i},Z_{i}$, respectively, the TSLS estimator $\hat{\beta}^{\text{IV}}\mathrel{\mathop{\mathchar 58\relax}}=(\mathbf{X}^{T}\Pi_{Z}\mathbf{X})^{-1}\mathbf{X}^{T}\Pi_{Z}\mathbf{Y}$ minimizes the objective

$\displaystyle\min_{\beta}\frac{1}{n}\|\mathbf{Y}-\Pi_{Z}\mathbf{X}\beta\|^{2}.$

(5)

In contrast, the standard OLS estimator $\hat{\beta}^{\text{OLS}}$ solves the problem

$\displaystyle\min_{\beta}\frac{1}{n}\|\mathbf{Y}-\mathbf{X}\beta\|^{2}.$

(6)

When the moment conditions (3) and (4) hold, the TSLS estimator is a consistent estimator of the causal effect $\beta_{0}$ under standard assumptions (Wooldridge,, 2020), and valid inference can be performed by constructing variance estimators based on the asymptotic distribution of $\hat{\beta}^{\text{IV}}$ (Imbens and Rubin,, 2015). Although not the most common presentation of TSLS, the optimization formulation in (5) provides intuition on how IV estimation works: when the instruments $Z$ are uncorrelated with the unobserved confounders $U$ affecting $X$ and $Y$, the projection operator $\Pi_{Z}$ applied to $\mathbf{X}$ “removes” the confounding from $X$, so that $\Pi_{Z}\mathbf{X}$ becomes (asymptotically) uncorrelated with $\epsilon$. Regressing $\mathbf{Y}$ on $\Pi_{Z}\mathbf{X}$ then yields a consistent estimator of $\beta_{0}$.

The validity of estimation and inference based on $\hat{\beta}^{\text{IV}}$ relies critically on the moment conditions (3) and (4). Condition (3) is often called the relevance condition or rank condition, and requires $\mathbb{E}\left[ZX^{T}\right]$ to have full rank (recall $p\leq d$). In the special case of one-dimensional instrumental and endogenous variables, i.e., $d=p=1$, it simply reduces to $\mathbb{E}\left[ZX\right]\neq 0$. Intuitively, the relevant condition ensures that the instruments $Z$ can explain sufficient variations in the endogenous variables $X$. In this case, the instruments are said to be relevant and strong. When $\mathbb{E}\left[ZX^{T}\right]$ is close to being rank deficient, i.e., the smallest eigenvalue $\lambda_{p}(\mathbb{E}\left[ZX^{T}\right])\approx 0$, IV estimation suffers from the so-called weak instrument problem, which results in many issues in estimation and inference, such as small sample bias and non-normal statistics (Stock et al.,, 2002). Some $k$-class estimators, such as limited information maximum likelihood (LIML) (Anderson and Rubin,, 1949), are partially motivated to address these problems. Condition (4) is often referred to as the exclusion restriction or instrument exogeneity (Imbens and Rubin,, 2015), and instruments that satisfy this condition are called valid instruments. When an instrument $Z$ is correlated with the unobserved confounder that confounds $X,Y$, or when $Z$ affects the outcome $Y$ through an unobserved variable other than the endogenous variable $X$, the instrument becomes invalid, resulting in biased estimation and invalid inference of $\beta_{0}$ (Murray,, 2006). These issues can often be exacerbated when the instruments are weak, when there is heteroskedasticity (Andrews et al.,, 2019), or the data is highly leveraged (Young,, 2022).

Although many works have been devoted to addressing the problems of weak and invalid instruments, there are fundamental limits on the extent to which one can test for these issues. Given the popularity of IV estimation in practice, it is therefore desirable to have estimation and inference procedures that are robust to the presence of such issues. Our work is precisely motivated by these considerations. Compared to many existing robust approaches to IV estimation, we take a more agnostic approach via distributionally robust optimization. More precisely, we argue that many common challenges in IV estimation can be viewed as uncertainties about the data distribution, i.e., deviations from the ideal model that satisfies IV assumptions, which can be explicitly taken into account by choosing an appropriate ambiguity set in a DRO formulation of the standard IV estimation. To demonstrate this perspective more concretely, we now examine some common problems in IV estimation and show that they can be viewed as distributional shifts under a suitable metric, and therefore amenable to a DRO approach.

Consider now the following one-dimensional version of the IV model in (2)

$\displaystyle\begin{split}Y&=X\beta_{0}+\epsilon\\ X&=Z\gamma+\xi,\end{split}$

where we assume that $X,Y$ are confounded by an unobserved confounder $U$ through

$$\epsilon=Z\eta+U,\quad\xi=U.$$

Note that in addition to $U$, there is also potentially a direct effect $\eta$ from the instrument $Z$ to the outcome variable $Y$. We focus on the resulting model for our subsequent discussions:

$\displaystyle\begin{split}Y&=X\beta_{0}+Z\eta+U\\ X&=Z\gamma+U.\end{split}$

(7)

The standard IV assumptions can be succinctly summarized for (7). The relevance condition (3) requires that $\gamma\neq 0$, while the exclusion restriction (4) requires that $Z$ is uncorrelated with $U$ and that in addition $\eta=0$. Assume that $U,Z$ are i.i.d. standard normal. $X,Y$ are then determined by $\eqref{eq:simple-IV}$. We are interested in the shifts in data distribution, appropriately defined and measured, when the exogeneity and relevance conditions are violated.

Next, we consider the distributional shift resulting from heteroskedastic errors, which are known to yield the TSLS estimator inefficient and the standard variance estimator invalid (Baum et al.,, 2003). Some $k$-class estimators, such as the LIML and the Fuller estimators, also become inconsistent under heteroskedasticity (Hausman et al.,, 2012).

The preceding discussions demonstrate that distributional uncertainties resulting from violations of common model assumptions in IV estimation are well captured by the 2-Wasserstein distance on the distributions of the conditional variables $(\tilde{X},\tilde{Y})$. We therefore propose to construct an ambiguity set in (1) using a Wasserstein ball around the empirical distribution on $(\tilde{X},\tilde{Y})$. We provide details of this framework in the next section.

Motivated by the intuition that common challenges to IV estimation in practice, such as violations of model assumptions, can be viewed as distributional uncertainties on the conditional distributions of $(\tilde{X},\tilde{Y})=(X\mid Z,Y\mid Z)$, we propose the Distributionally Robust IV Estimation (DRIVE) framework, which solves the following DRO problem given a dataset $\{(X_{i},Y_{i},Z_{i})\}_{i=1}^{n}$ and robustness parameter $\rho$:

$\displaystyle\textbf{(DRIVE Objective)}\quad\min_{\beta}\sup_{\{\mathbb{Q}\mathrel{\mathop{\mathchar 58\relax}}D(\mathbb{Q},\tilde{\mathbb{P}}_{n})\leq\rho\}}\mathbb{E}_{\mathbb{Q}}\left[({Y}-{X}^{T}\beta)^{2}\right],$

(12)

where $\tilde{\mathbb{P}}_{n}(\mathcal{X}\times\mathcal{Y})$ is the empirical distribution on $(X,Y)$ induced by the projected samples

$$\{\tilde{X}_{i},\tilde{Y}_{i}\}_{i=1}^{n}\equiv\{(\Pi_{\mathbf{Z}}\mathbf{X})_{i},(\Pi_{\mathbf{Z}}\mathbf{Y})_{i}\}_{i=1}^{n}.$$

Here $\mathbf{X}\in\mathbb{R}^{n\times p},\mathbf{Y}\in\mathbb{R}^{n},\mathbf{Z}\in\mathbb{R}^{n\times d}$ are the matrix representations of observations, and $\Pi_{\mathbf{Z}}=\mathbf{Z}(\mathbf{Z}^{T}\mathbf{Z})^{-1}\mathbf{Z}^{T}$ is the projection matrix onto the column space of $\mathbf{Z}$. $D(\cdot,\cdot)$ is a metric or divergence measure on the space of probability distributions on $\mathcal{X}\times\mathcal{Y}$. Therefore, in our DRIVE framework, we first regress both the outcome $Y$ and covariate $X$ on the instrument $Z$ to form the $n$ predicted samples $(\Pi_{\mathbf{Z}}\mathbf{X},\Pi_{\mathbf{Z}}\mathbf{Y}$). Then an ambiguity set is constructed using $D$ around the empirical distribution $\tilde{\mathbb{P}}_{n}$. This choice of the reference distribution $\mathbb{P}_{0}$ is a key distinction of our work from previous works that leverage DRO in statistical models. In the standard regression/classification setting, the reference distribution is often chosen as the empirical distribution $\hat{\mathbb{P}}_{n}$ on $\{X_{i},Y_{i}\}_{i=1}^{n}$ (Blanchet et al.,, 2019). In the IV estimation setting where we have additional access to instruments $Z$, we have the choice of constructing ambiguity sets around the empirical distribution on the marginal quantities $\{(X_{i},Y_{i},Z_{i})\}_{i=1}^{n}$, which is the approach taken in Bertsimas et al., (2022). In contrast, we choose to use the empirical distribution on the conditional quantities $\{(\tilde{X}_{i},\tilde{Y}_{i})\}_{i=1}^{n}$. This choice is motivated by the intuition that violations of IV assumptions can be captured by conditional distributional shifts, as illustrated by examples in the previous section.

The choice of the divergence measure $D(\cdot,\cdot)$ is also important, as it characterizes the potential distributional uncertainties that DRIVE is robust to. In this paper, we propose to use the 2-Wasserstein distance $W_{2}(\mu,\nu)$ between two probability distributions $\mu,\nu$ (Mohajerin Esfahani and Kuhn,, 2018; Gao and Kleywegt,, 2023). One advantage of the Wasserstein distance is the tractability of its associated DRO problems (Blanchet et al.,, 2019), which can often be formulated as regularized regression problems with unique solutions. See also Appendix A.1. In Section 2.2, we provided several examples that demonstrate the 2-Wasserstein distance is able to capture common distributional uncertainties in the IV estimation setting. Alternative distance measures of probability distributions, such as the class of $\phi$-divergences (Ben-Tal et al.,, 2013), can also be used instead of the Wasserstein distance. For example, Kitamura et al., (2013) use the Hellinger distance to model local perturbations in robust estimation under moment restrictions, although not in the IV estimation setting. In this paper, we focus on the Wasserstein DRIVE framework based on $D=W_{2}$, and leave studies of DRIVE with other choices of $D$ to future works.

We next begin our formal study of Wasserstein DRIVE. In Section 3.2, we will show that the Wasserstein DRIVE objective is dual to a convex regularized regression problem. As a result, the solution to the optimization problem (12) is well-defined, and we denote this estimator by $\hat{\beta}_{\mathrm{DRIVE}}$. In Section 4, we show $\hat{\beta}_{\mathrm{DRIVE}}$ is consistent with potentially non-vanishing choices of the robustness parameter and derive its asymptotic distribution.

It is well-known in the optimization literature that min-max optimization problems such as (12) often have equivalent formulations as regularized regression problems. This correspondence between regularization and robustness already manifests itself in the ridge regression, which is equivalent to an $\ell_{2}$-robust OLS regression (Bertsimas and Copenhaver,, 2018). Importantly, the regularized regression formulations are often more tractable in terms of solving the resulting optimization problem, and also facilitate the study of the statistical properties of the estimators. We first show that the Wasserstein DRIVE objective can also be written as a regularized regression problem similar to, but distinct from, the standard TSLS objective with ridge regularization. Proofs can be found in Appendix F.

Note that the robustness parameter $\rho$ of the DRO formulation (12) now has the dual interpretation as the regularization parameter in (13). This convex regularized regression formulation implies that the min-max problem (12) associated with Wassertein DRIVE has a unique solution, thanks to the strict convexity of the regularization term $\sqrt{\rho(\|\beta\|^{2}+1)}$, and is easy to compute despite not having a closed form solution. In particular, (13) can be reformulated as a standard second order conic program (SOCP) (El Ghaoui and Lebret,, 1997), which can be solved efficiently with off-the-shelf convex optimization routines, such as CVX. More importantly, we leverage this formulation of Wasserstein DRIVE as a regularized regression problem to study its novel statistical properties in Section 4.

The equivalence between Wasserstein DRO problems and regularized regression problem is a familiar general result in recent works. For example, Blanchet et al., (2019) and Gao and Kleywegt, (2023) derive similar duality results for distributionally robust regression with $q$-Wasserstein distances for $q>1$. Compared to previous works, our work is distinct in the following aspects. First, we apply Wasserstein DRO to the IV estimation setting instead of standard regression settings, such as OLS or logistic regression. Although from an optimization point of view there is no substantial difference, the IV setting motivates a new asymptotic regime that uncovers interesting statistical properties of the resulting estimators. Second, the regularization term in (13) is distinct from those in previous works, which often use $\|\beta\|_{p}$ with $p\geq 1$. This seemingly innocuous difference turns out to be crucial for our novel results on the Wasserstein DRIVE. Lastly, compared to the proof in Blanchet et al., (2019), our proof of Theorem 3.1 is based on a different argument using the Sherman-Morrison formula instead of Hölder’s inequality, which provides an independent proof of the important duality result for Wasserstein distributionally robust optimization.

The regularized regression formulation of the Wasserstein DRIVE problem in (13) resembles the standard ridge regularized (Hoerl and Kennard,, 1970) TSLS regression:

$\displaystyle\min_{\beta}\frac{1}{n}\sum_{i}({Y}_{i}-\tilde{X}_{i}^{T}\beta)^{2}+\rho\|\beta\|^{2}\Longleftrightarrow\min_{\beta}\frac{1}{n}\|\mathbf{Y}-\Pi_{\mathbf{Z}}\mathbf{X}\beta\|^{2}+\rho\|\beta\|^{2}.$

(14)

We therefore refer to (13) as the square root ridge regularized TSLS. However, there are three major distinctions between (13) and (14) that are essential in guaranteeing the statistical properties of Wasserstein DRIVE not enjoyed by the standard ridge regularized TSLS. First, the presence of square root operations on both the risk term and the penalty term; second, the presence of a constant in the regularization term; third, an additional projection on the outcomes in $\Pi_{\mathbf{Z}}\mathbf{Y}$. We further elaborate on these features in Section 4.

In the standard regression setting without instrumental variables, the square root ridge

$\displaystyle\min_{\beta}\sqrt{\frac{1}{n}\|\mathbf{Y}-\mathbf{X}\beta\|^{2}}+\sqrt{\rho(1+\|\beta\|^{2})}$

(15)

also resembles the “square root LASSO” of Belloni et al., (2011):

$\displaystyle\min_{\beta}\sqrt{\frac{1}{n}\|\mathbf{Y}-\mathbf{X}\beta\|^{2}}+\lambda\|\beta\|_{1}.$

(16)

In particular, both can be written as dual problems of Wasserstein DRO problems (Blanchet et al.,, 2019). However, the square root LASSO is motivated by high-dimensional regression settings where the dimension of $X$ is potentially larger than the sample size $n$, but $\beta$ is very sparse. In contrast, our study of the square root ridge is motivated by its robustness properties in the IV estimation setting, where the dimension of the endogenous variable is small (often one-dimensional). In other words, variable selection is not the main focus of this paper. A variant of the square root ridge estimator in (15) was also considered in the standard regression setting by Owen, (2007), who instead uses the penalty term $\|\beta\|_{2}$.

As is well-known in the regularized regression literature (Fu and Knight,, 2000), when the regularization parameter decays to 0 at a rate $O_{p}(1/\sqrt{n})$, the ridge estimator is consistent. A similar result also holds for the square root ridge (15) in the standard regression setting as $\rho\rightarrow 0$. However, in the IV estimation setting, our distributional uncertainties about model assumptions, such as the validity of instruments, could persist even in large samples. Recall that $\rho$ is also the robustness parameter in the DRO formulation (12). Therefore, the usual requirement that $\rho\rightarrow 0$ as $n\rightarrow\infty$ cannot adequately capture distributional uncertainties in the IV estimation setting. In the next section, we study the asymptotic properties of Wasserstein DRIVE when $\rho$ does not necessarily vanish. In particular, we establish the consistency of Wasserstein DRIVE leveraging the three distinct features of (13) that are absent in the standard ridge regularized TSLS regression (14). This asymptotic result is in stark contrast to the conventional wisdom on regularized regression that regularized regression achieves lower variance at the cost of non-zero bias.

Recall the linear IV regression model in (2)

$\displaystyle\begin{split}Y&=\beta^{T}_{0}X+\epsilon,\\ X&=\gamma^{T}Z+\xi,\end{split}$

where $X\in\mathbb{R}^{p},Z\in\mathbb{R}^{d}$, and $\beta_{0}\in\mathbb{R}^{p},\gamma\in\mathbb{R}^{d\times p}$ with $d\geq p$ to ensure identification. In this section, we make the standard assumptions that the instruments satisfy the relevance and exogeneity conditions in (3) and (4), $\epsilon,\xi$ are homoskedastic, the instruments $Z$ are not perfectly collinear, and that $\mathbb{E}\|Z\|^{2k}<\infty,\mathbb{E}\|\xi\|^{2k}<\infty,\mathbb{E}|\epsilon|^{2k}<\infty$ for some $k>2$. The results can be extended in a straightforward manner when we relax these assumptions, e.g., only requiring that exogeneity holds asymptotically. Given i.i.d. samples from the linear IV model, recall the regularized regression formulation of the Wasserstein DRIVE objective

$\displaystyle\min_{\beta}\sqrt{\frac{1}{n}\sum_{i}(\Pi_{\mathbf{Z}}\mathbf{Y}-\Pi_{\mathbf{Z}}\mathbf{X}\beta)_{i}^{2}}+\sqrt{\rho_{n}(\|\beta\|^{2}+1)},$

(17)

where $\Pi_{\mathbf{Z}}=\mathbf{Z}(\mathbf{Z}^{T}\mathbf{Z})^{-1}\mathbf{Z}^{T}\in\mathbb{R}^{n\times n}$, and $\Pi_{\mathbf{Z}}\mathbf{Y}\in\mathbb{R}^{n}$ and $\Pi_{\mathbf{Z}}\mathbf{X}\in\mathbb{R}^{n\times p}$ are $\mathbf{Y}\in\mathbb{R}^{n},\mathbf{X}\in\mathbb{R}^{n\times p}$ projected onto the instrument space spanned by $\mathbf{Z}\in\mathbb{R}^{n\times d}$.

Therefore, Wasserstein DRIVE is consistent as long as the limit of the regularization parameter is bounded above by $\overline{\rho}=\lambda_{p}(\gamma^{T}\Sigma_{Z}\gamma)$. In the case when $\Sigma_{Z}=\sigma^{2}_{Z}I_{d}$ for $\sigma_{Z}^{2}>0$, the upper bound $\overline{\rho}$ is proportional to the square of the smallest singular value of the first stage coefficient $\gamma$, which is positive under the relevance condition (3). Recall that $\rho_{n}$ is the radius of the Wasserstein ball in the min-max formulation of DRIVE in (12). Theorem 4.1 therefore guarantees that even when the robustness parameter $\rho_{n}\equiv\rho\neq 0$, which implies the solution to the min-max problem is different from the TSLS estimator ($\rho=0$), the resulting estimator always has the same limit as long as $\rho\leq\lambda_{p}(\gamma^{T}\Sigma_{Z}\gamma)$.

The significance of Theorem 4.1 is twofold. First, it provides a meaningful interpretation of the robustness parameter $\rho$ in Wasserstein DRIVE in terms of problem parameters, more precisely the variance covariance matrix $\Sigma_{Z}$ of $Z$ and the first stage coefficient $\gamma$ in the IV regression model. The maximum amount of robustness that can be afforded by Wasserstein DRIVE without sacrificing consistency is $\lambda_{p}(\gamma^{T}\Sigma_{Z}\gamma)$, which directly depends on the strength and variance of the instrument. This relation can be described more precisely when $\Sigma_{Z}=\sigma^{2}_{Z}I_{d}$, in which case the bound is proportional to $\sigma^{2}_{Z}$ and $\lambda_{p}(\gamma^{T}\gamma)$. Both quantities improve the quality of the instruments: $\sigma^{2}_{Z}$ improves the proportion of variance of $X$ and $Y$ explained by the instrument vs. noise, while a $\gamma$ far from rank deficiency avoids the weak instrument problem. Therefore, the robustness of Wasserstein DRIVE depends intrinsically on the strength of the instruments. The quantity $\gamma^{T}\Sigma_{Z}\gamma$ is not unfamiliar in the IV setting, as it is proportional to the inverse of the asymptotic variance of the standard TSLS estimator when errors are homoskedastic. This observation suggests an intrinsic connection between robustness and efficiency in the IV setting. See the discussions after Theorem 4.2 for more on this point.

More importantly, Theorem 4.1 is the first consistency result for regularized regression estimators where the regularization parameter does not vanish with sample size. Although regularized regression such as ridge and LASSO is often associated with better finite sample performance at the cost of introducing some bias, our work demonstrates that, in the IV estimation setting, we can get the best of both worlds. On one hand, the ridge type regularization in Wasserstein DRIVE improves upon the finite sample properties of the standard IV estimators, which aligns with conventional wisdom on regularized regression. On the other hand, with a bounded level of the regularization parameter $\rho$, Wasserstein DRIVE can still achieve consistency. This is in stark contrast to existing asymptotic results on regularized regression (Fu and Knight,, 2000). Therefore, in the context of IV estimation, with Wasserstein DRIVE we can achieve consistency and a certain amount of robustness at the same time, by leveraging additional information in the form of valid instruments. The maximum degree of robustness that can be achieved also has a natural interpretation in terms of the strength and variance of the instruments.

Theorem 4.1 also suggests the following procedure to construct a feasible and valid robustness/regularization parameter $\hat{\rho}$ given data $\{X_{i},Y_{i},Z_{i}\}_{i=1}^{n}$. Let $\hat{\gamma}=(\mathbf{Z}^{T}\mathbf{Z})^{-1}\mathbf{Z}^{T}\mathbf{X}$ be the OLS regression estimator of the first stage coefficient $\gamma$ and $\hat{\Sigma}_{Z}$ an estimator of $\Sigma_{Z}$, such as the heteroskedasticity-robust estimator (White,, 1980). We can use $\hat{\rho}\leq\lambda_{p}(\hat{\gamma}^{T}\hat{\Sigma}_{Z}\hat{\gamma})$ to construct the Wasserstein DRIVE objective, i.e., any value bounded above by the smallest eigenvalue of $\hat{\gamma}^{T}\hat{\Sigma}_{Z}\hat{\gamma}$. Under the assumptions in Theorem 4.1, $\lambda_{p}(\hat{\gamma}^{T}\hat{\Sigma}_{Z}\hat{\gamma})\rightarrow\lambda_{p}({\gamma}^{T}\Sigma_{Z}{\gamma})$, which guarantees that the Wasserstein DRIVE estimator with parameter $\hat{\rho}$ is consistent. In Appendix E, we discuss the construction of feasible regularization parameters in more detail. We demonstrate the validity and superior finite sample performance of DRIVE based on these proposals in simulation studies in Section 5.

One may wonder why Wasserstein DRIVE can achieve consistency with a non-zero regularization $\rho$. Here we briefly discuss the phenomenon that the limiting objective (18)

$\displaystyle\min_{\beta}\sqrt{(\beta-\beta_{0})^{T}\gamma^{T}\Sigma_{Z}\gamma(\beta-\beta_{0})}+\sqrt{\rho(\|\beta\|^{2}+1)}$

has a unique minimizer at $\beta_{0}$ for bounded $\rho>0$. The first term $\sqrt{(\beta-\beta_{0})^{T}\gamma^{T}\Sigma_{Z}\gamma(\beta-\beta_{0})}$ achieves its minimum value of 0 at $\beta=\beta_{0}$. When $\rho$ is small, the effect of adding the regularization term $\sqrt{\rho(\|\beta\|^{2}+1)}$ does not overwhelm the first term, especially when its curvature at $\beta_{0}$ is large. As a result, we may expect the minimizer to not deviate much from $\beta_{0}$. While this intuition is reasonable qualitatively, it does not fully explain the fact that the minimizer does not change for small $\rho$. In the standard regression setting, the same intuition can be applied to the standard ridge regularization, but we know shrinkage occurs as soon as $\rho>0$. The key distinction of (17) turns out to be the square root operations we apply to the loss and regularization terms, which endows the objective with a geometric interpretation, and ensures that the minimizer does not deviate from $\beta_{0}$ unless $\rho$ is above some positive threshold. We call this phenomenon the “delayed shrinkage” of the square root ridge, as shrinkage does not happen until the regularization is large enough. We illustrate it with a simple example in Fig. 1, where the minimizer of the limiting square root objective does not change for a bounded range of $\rho$.

Lastly, we comment on the importance of projection operations in Wasserstein DRIVE. A crucial feature of the Wasserstein DRIVE objective is that both the outcome and the covariates are regressed on the instrument to compute their predicted values. In other words, the objective (12) uses $\Pi_{\mathbf{Z}}\mathbf{Y}-\Pi_{\mathbf{Z}}\mathbf{X}\beta$ instead of $\mathbf{Y}-\Pi_{\mathbf{Z}}\mathbf{X}\beta$. For standard IV estimation ($\rho=0)$, there is no substantial difference between the two objectives, since their minimizers are exactly the same, due to the idempotent property $\Pi^{2}_{\mathbf{Z}}=\Pi_{\mathbf{Z}}$. In fact, in applications of TSLS, the outcome variable is often not regressed on the instrument. However, Wasserstein DRIVE is consistent for positive $\rho$ only if the outcome $Y$ is also projected onto the instrument space. In other words, the following problem does not yield a consistent estimator when $\rho>0$:

$\displaystyle\min_{\beta}\sqrt{\frac{1}{n}\|\mathbf{Y}-\Pi_{\mathbf{Z}}\mathbf{X}\beta\|^{2}}+\sqrt{\rho(\|\beta\|^{2}+1)}.$

The reason behind this phenomenon is that ${\frac{1}{n}\|\Pi_{\mathbf{Z}}\mathbf{Y}-\Pi_{\mathbf{Z}}\mathbf{X}\beta\|^{2}}$ is a GMM objective

$\displaystyle(\frac{1}{n}\sum_{i}Z_{i}(Y_{i}-\beta^{T}X_{i}))^{T}(\frac{1}{n}\mathbf{Z}^{T}\mathbf{Z})^{-1}(\frac{1}{n}\sum_{i}Z_{i}(Y_{i}-\beta^{T}X_{i})),$

which when $n\rightarrow\infty$ achieves a minimal value of 0 at $\beta_{0}$, while the limit of ${\frac{1}{n}\|\mathbf{Y}-\Pi_{\mathbf{Z}}\mathbf{X}\beta\|^{2}}$ does not vanish even at $\beta_{0}$. In the former case, the geometric properties of the square root ridge ensure that the minimizer of the regularized objective is $\beta_{0}$.

Having established the consistency of Wasserstein DRIVE with bounded $\rho$, we now turn to the characterization of its asymptotic distribution. In general, the asymptotic distribution of Wasserstein DRIVE is different from that of the standard IV estimator. However, we will also examine several special cases relevant in practice where they coincide.

Recall that the maximal robustness parameter $\rho$ of Wasserstein DRIVE while still being consistent is equal to the smallest eigenvalue of $\gamma^{T}\Sigma_{Z}\gamma$, which is proportional to the inverse of the asymptotic variance of the TSLS estimator. Therefore, as the efficiency of TSLS increases, so does the robustness of the associated Wasserstein DRIVE estimator. The “price” to pay for robustness when $\rho>0$ is an interesting question. It is clear from Fig. 1 that the curvature of the population objective decreases as $\rho$ increases. Since the objective is not continuous at $\beta_{0}$, however, a generalized notion of curvature is needed to precisely characterize this behavior. Note that the asymptotic distribution of the TSLS estimator minimizes the objective

$$(\mathcal{Z}+\Sigma_{Z}\gamma\delta)^{T}\Sigma_{Z}^{-1}(\mathcal{Z}+\Sigma_{Z}\gamma\delta),$$

Theorem 4.2 implies that in general the asymptotic distributions of Wasserstein DRIVE and TSLS are different when $\rho>0$. However, there are still several cases relevant in practice where their asymptotic distributions do coincide, which we discuss next.

In particular, the just-identified case with $d=p=1$ covers many empirical applications of IV estimation, since in practice we are often interested in the causal effect of a single endogenous variable, for which we have a single instrument. The case when $\beta_{0}\equiv\mathbf{0}$ is also very relevant, since an important question in practice is whether the causal effect of a variable is zero. Our theory suggests that the asymptotic distribution of Wasserstein DRIVE should be the same as that of the TSLS when the causal effect is zero and $d>1$, even for $\rho>0$. Based on this observation, intuitively, we should expect that Wasserstein DRIVE and TSLS estimators to be “close” to each other. If the estimators or their asymptotic variance estimators differ significantly, then $\beta_{0}$ may not be identically 0. We can design statistical tests by leveraging this intuition. For example, we can construct test statistics using the TSLS estimator and the DRIVE estimator with $\rho>0$, such as the difference $\hat{\beta}^{\text{DRIVE}}-\hat{\beta}^{\text{TSLS}}$. Then we can use bootstrap-based tests, such as a bootstrapped permutation test, to assess the null hypothesis that $\hat{\beta}^{\text{DRIVE}}-\hat{\beta}^{\text{TSLS}}=0$. If we fail to reject the null hypothesis, then there is evidence that the true causal effect $\beta_{0}=\mathbf{0}$.

Corollary 4.3 can be seemingly pessimistic because it demonstrates that the asymptotic distribution of Wasserstein DRIVE could be the same as that of the TSLS in special cases. However, recall that Wasserstein DRIVE is formulated to minimize the worst-case risk over a set of distributions that are designed to capture deviations from model assumptions. Therefore, there is not actually any a priori reason that it should coincide with the TSLS when $\rho>0$. In this sense, the fact that the Wasserstein DRIVE is consistent with $\rho>0$ and may even coincide with TSLS is rather surprising. In the latter case, the worst-case distribution for Wasserstein DRIVE in the large sample limit must coincide with that of the standard population distribution, which may be worth further investigation.

The asymptotic results we develop in this section provide the basis on which one can perform estimation and inference with the Wasserstein DRIVE estimator. In the next section, we study the finite sample properties of DRIVE in simulation studies and demonstrate that it is superior in terms of estimation error and out of sample prediction compared to other popular estimators.

We use the data generating process

$\displaystyle\begin{split}Y&=X\beta_{0}+Z\eta+U\\ X&=\gamma Z+U\\ Z&=U\beta_{UZ}+\epsilon_{Z},\\ \end{split}$

where $U,\epsilon_{Z}\sim\mathcal{N}(0,\sigma^{2})$ and we allow a direct effect $\eta$ from the instruments $Z$ to the outcome $Y$. Moreover, the instruments $Z$ can also be correlated with the unobserved confounder $U$ ($\beta_{UZ}\neq 0)$. We fix the true parameters and generate independent datasets from the model, varying the degree of instrument invalidity. In Table 1, we report the MSE of estimators averaged over 500 repeated experiments. We control the degree of instrument invalidity by varying $\eta$, the direct effect of instruments on the outcome, and $\beta_{UZ}$, the correlation between unobserved confounder and instruments. Results in Table 1 are based on data where $\|\gamma\|\gg 0$ is large. We see that when instruments are strong, Wasserstein DRIVE performs as well as TSLS when instruments are valid, but performs significantly better than OLS, TSLS, anchor, and TSLS ridge when instruments become invalid. This suggests that DRIVE could be preferable in practice when we are concerned about instrument validity.

We further investigate the empirical performance of Wasserstein DRIVE when instruments are potentially invalid or weak. We present box plots (omitting outliers) of MSEs in Fig. 2. The Wasserstein DRIVE estimator with regularization parameter $\rho$ based on bootstrapped quantiles of the score function consistently outperforms OLS, TSLS, anchor ($k$-class), and TSLS with ridge regularization. Moreover, the selected penalties increase as the direct effect of $Z$ on $Y$ or the correlation between the unobserved confounder $U$ and the instrument $Z$ increases, i.e., as the model assumption of valid instruments becomes increasingly invalid. See Fig. 5 in Appendix Appendix E for more details. This property is highly desirable, because based on the DRO formulation of DRIVE, $\rho$ represents the amount of robustness against distributional shifts associated with the estimator, which should increase as the instruments become more invalid (larger distributional shift). Box plots of estimation errors in Fig. 2 also verify that even when instruments are valid, the finite sample performance of Wasserstein DRIVE is still better compared to the standard IV estimator, suggesting that there is no additional cost in terms of MSE when applying Wasserstein DRIVE, even when instruments are valid.

We now turn our attention to a different task that has received more attention in recent years, especially in the context of policy learning and estimating causal effects across heterogeneous populations (Dehejia et al.,, 2021; Adjaho and Christensen,, 2022; Menzel,, 2023). We study the prediction performance of estimators when they are estimated on a dataset (training set) that has a potentially different distribution from the dataset for which they are used to make predictions (test set). We demonstrate that whenever the distributions between training and test datasets have significant differences, the prediction error of Wasserstein DRIVE is significantly smaller than that of OLS, IV, and anchor ($k$-class) estimators.

We conduct our numerical study using the classic dataset on the return of education to wage compiled by David Card (Card,, 1993). Here, the causal inference problem is estimating the effect of additional school years on the increase in wage later in life. The dataset contains demographic information about interviewed subjects. Importantly, each sample comes from one of nine regions in the United States, which differ in the average number of years of schooling and other characteristics, i.e., there are covariate shifts in data collected from different regions. Our strategy is to divide the dataset into a training set and a test set based on the relative ranks of their average years of schools, which is the endogenous variable. We expect that if there are distributional shifts between different regions, then predicting wages using education and other information using conventional models trained on the training data may not yield a good performance on the test data.

Since each sample is labeled as working in 1976 in one of nine regions in the U.S., we split the samples based on these labels, using number of years of education as the splitting variable. For example, we can construct the training set by including samples from the top 6 regions with the highest average years of schooling, and the test set to consist of samples coming from the bottom 3 regions with the lowest average years of schooling. In this case, we would expect the training and test sets to have come from different distributions. Indeed, the average years of schooling differs by more than 1 year, and is statistically significant.

In splitting the samples based on the distribution of the endogenous variable, we are also motivated by the long-standing debates revolving around the use of instrumental variables in classic economic studies (Card and Krueger,, 1994). A leading concern is the validity of instruments. In the case of the study on educational returns, the validity of estimation and inference require that the instruments (proximity to college and quarter of birth) are not correlated with unobserved characteristics that may also affect their earnings. The following quote from Card, (1999) illustrates this concern:

“In the case of quasi or natural experiments, however, inferences are based on difference between groups of individuals who attended schools at different times, or in different locations, or had differences in other characteristics such as month of birth. The use of these differences to draw causal inferences about the effect of schooling requires careful consideration of the maintained assumption that the groups are otherwise identical.”

When this assumption is violated, the estimates based on a particular subpopulation becomes unreliable for the wider population, and we evaluate the performance based on how well they generalize to other groups of the population with potential distributional or covariate shifts. In Table 2, we compare the test set MSE of OLS, IV, Wasserstein DRIVE, anchor regression, ridge, and ridge regularized IV estimators. We see that Wasserstein DRIVE consistently outperforms other estimators commonly used in practice.