Estimation of Local Average Treatment Effect by Data Combination

By Assumption 1.1, we suppose that the joint distribution of the potential variables and covariates in each assignment regime is invariant to each other, and equal to the joint distribution in the population of interest. This means that the participants in each regime are random draws from the population of interest. We can still identify LATE even if we weaken Assumption 1.1 to its conditional version, but the direct estimation is no longer possible. See Appendix A for more discussion. Assumption 1.2 indicates that the assignment regimes are different to each other for any level of the covariates.

Assumption 2.1 states that $Z^{(k)}$ are randomly assigned within a subpopulation sharing the same level of the covariates. The mean independence between the potential variables and the treatment assignment conditional on covariates is sufficient for identifying LATE, but we maintain Assumption 2.1 as it is typical to assume the full conditional independence in the causal inference literature. Assumption 2.2 relates the potential variables to their realized counterparts. Assumption 2.3 excludes defiers, those who never follow the given treatment assignment, from our analysis. Assumption 2.4 is necessary for $Z^{(k)}$ to be a valid IV of $D^{(k)}$. Figure 1 illustrates the data generating process under the 2RD.

Assumption 2 is the direct extension of the standard assumptions used for the LATE estimation [1, 18, 51, 38] to the 2RD. However, we omit the condition referred to as positivity $0<P(Z^{(k)}=1|\bm{X}^{(k)})<1$ as it is unnecessary in the 2RD. Therefore, it is possible to set $P(Z^{(0)}=1|\bm{X}^{(0)})=0$ as long as Assumption 1.2 is satisfied. We term the design with $P(Z^{(1)}=1|\bm{X}^{(1)})>0$ for any $\bm{X}^{(1)}$ and $P(Z^{(0)}=1|\bm{X}^{(0)})=0$ for any $\bm{X}^{(0)}$ as the one-experiment 2RD (1E2RD) since an experiment is conducted only for those with $k=1$, and a natural state without any intervention is observed for those with $k=0$.

Although [58] does not mention anything on this point, it is of great practical importance since the 1E2RD is much easier to implement than the general 2RD, and extends the plausibility of our setting. In the 1E2RD, we only have to conduct one experiment or collect one set of observational data just like the standard setting for causal inference. All we need in addition is a dataset collected from those without any intervention. Such data may be available at no additional cost, for example, when there is appropriate open data published by the government.

Hereafter, we abuse the notation by omitting the superscript on the potential variables and covariates unless this causes confusion.

The proof can be found in Appendix B. Notably, this form coincides with the identification result of ATE in [58], which means that we can also estimate $\mu(\bm{X})$ from the same combination of datasets proposed in [58] under appropriate assumptions. This is a rather powerful result in practice since $Z^{(k)}$ is not required despite the presence of noncompliance. Moreover, it is obvious that $p_{d}^{(k)}:=P(D^{(k)}=1)$ and $P(\bm{X}|D^{(k)}=1)$ are sufficient to identify the denominator of (1) since $E[D^{(k)}|\bm{X}]$ can be identified as $\frac{P(\bm{X}|D^{(k)}=1)p_{d}^{(k)}}{P(\bm{X})}$ by applying the Bayes’ theorem.

The point is that we can identify $\mu(\bm{X})$ as a combination of functions depending only on covariates $\bm{X}$ in our setting. This property allows us to use a direct estimation method for $\mu(\bm{X})$. We can identify LATE as $\mu(\bm{X})=\frac{E[Y|Z=1,\bm{X}]-E[Y|Z=0,\bm{X}]}{E[D|Z=1,\bm{X}]-E[D|Z=0,\bm{X}]}$ under the standard assumptions [1, 18, 51, 38], but we cannot rewrite it as a combination of functions depending only on covariates (see Appendix A).

This corollary shows that we can reduce the number of necessary datasets in the 1E2RD under one-sided noncompliance. This fact not only facilitates the data collection, but also benefits the estimation since the denominator is now just a propensity score, thus estimable accurately by, for example, the Positive-Unlabeled (PU) learning [15, 13, 14, 28] with the logistic loss.

Although one-sided noncompliance is often associated with RCTs, some observational studies also fit with the framework [19, 27]. In the case of one-sided noncompliance, we can also consider our problem as the estimation of the average treatment effect on the treated (ATT) since LATE is equal to ATT under one-sided noncompliance and the other standard assumptions [19, 12].

We assume that the joint samples of $(Y^{(k)},D^{(k)},Z^{(k)},\bm{X})$ are not available. By Theorem 1, the following separate datasets and the estimate of $p_{d}^{(k)}$ are sufficient for estimating $\mu(\bm{X})$:

for $k=0,1$. This setting is much easier to apply to real-world situations than the standard setting where the joint samples are required for every individual.

We can interpret this data collection scheme in two ways. First, it can be regarded as a version of the repeated cross-sectional (RCS) design [36, 6, 20, 42, 30] with $k=0,1$ representing the time points before and after the assignment regime switches, respectively. The 1E2RD is also possible in this case by setting $P(Z^{(0)}=1|\bm{X}^{(0)})=0$. This means collecting data for $k=0$ at some point before an experiment is conducted. Generally, although panel data are advantageous for statistical analyses because they follow the same individuals over multiple time periods, RCS data have some advantages over panel data. RCS data are easier and cheaper to collect in many cases. Consequently, they are often more representative of the population of interest and larger in sample size than panel data.

However, there is a concern regarding the validity of Assumption 1.1 when we use RCS data since the potential variables may change over time. We need some sort of side information to be confident about the use of RCS data in our setting as we cannot directly test whether the potential variables remain unchanged.

The second interpretation is to collect data with $k=0,1$ at the same time by randomly splitting the population of interest. If an experimenter is able to surely perform random splitting, this approach is favorable since we do not have to worry about the validity of Assumption 1.1. The implementation of random splitting is easy in the case of RCTs.

One specific example of our setting is O2O marketing. To measure the effect of an online video ads on sales in physical stores, we usually cannot link the data of those who watched the ads with that of those who shopped at the stores. However, it is relatively easy to separately collect data from those who watch the ads and the purchasers. In this case, data collection can be performed by either RCS design or random splitting.

Another example is when estimating the effect of a treatment that takes a certain period of time to take effect. For instance, it is desirable to use panel data to estimate the effect of job training on future earnings. However, individuals may gradually drop out of the survey, and the probability of the attrition can depend on unobserved variables [21, 37]. Therefore, it is easier to collect RCS data than to construct balanced panel data. Because we do not need to observe the outcome and treatment status simultaneously, we are more likely to be able to collect even larger and more representative data than with a normal RCS design.

A naive estimation method for $\mu(\bm{X})$ is to separately estimate the components and combine them as

where a hat denotes an estimator. $E[D^{(k)}|\bm{X}]$ can be estimated by the PU learning [15, 13, 14, 28] with the logistic loss by using $\{\bm{x}_{di}^{(k)}\}_{i=1}^{n_{d}^{(k)}}$ as positive data, and the covariates in $\{(y_{i}^{(1)},\bm{x}_{i}^{(1)})\}_{i=1}^{n^{(1)}}$ and $\{(y_{i}^{(0)},\bm{x}_{i}^{(0)})\}_{i=1}^{n^{(0)}}$ as unlabeled data. Separate estimation is easy to implement, but usually does not provide a good estimate since it has four possible sources of error.

One advantage of the separate estimation is that the model selection can also be naively performed by choosing the best model for each component. We can easily calculate the model selection criteria, such as the mean squared error (MSE) for each component from the separately observed datasets. However, the resulting $\widehat{\mu}_{\mathrm{sep}}$ may perform poorly because it does not necessarily minimize the model selection criterion in terms of the true $\mu$ [43].

We can alleviate the drawback of the separate estimation by directly estimating the numerator and denominator in (2). Let $T:=KD^{(1)}-(1-K)D^{(0)}$ and $U:=KY^{(1)}-(1-K)Y^{(0)}$ be the auxiliary variables for the notational and computational convenience, and assume $P(K=1|\bm{X})=0.5$. Then, we can rewrite the expression in Theorem 1 as $\mu(\bm{X})=\nu(\bm{X})/\pi(\bm{X})$, where $\nu(\bm{X}):=E[U|\bm{X}]$ and $\pi(\bm{X}):=E[T|\bm{X}]$. To estimate $\nu(\bm{X})$ and $\pi(\bm{X})$, we can construct combined datasets

from $\{\bm{x}_{di}^{(k)}\}_{i=1}^{n_{d}^{(k)}}$ and $\{(y_{i}^{(k)},\bm{x}_{i}^{(k)})\}_{i=1}^{n^{(k)}}$, respectively, where $n_{t}:=n_{d}^{(1)}+n_{d}^{(0)}$ and $n_{u}:=n^{(1)}+n^{(0)}$.

We can approximate the expectation of a product of any function of $\bm{X}$ and $T$ or $U$ by the simple sample average using the above datasets since we have

An estimator of $\nu(\bm{X})$ can be obtained by any regression method using the above combined dataset, while we need a little twist to obtain an estimator of the PSD $\pi(\bm{X})$. See Appendix C for the direct estimation methods for the PSD.

We can apply the direct least squares (DLS) method [58] originally proposed for ATE estimation under complete compliance. Motivated by the drawbacks of the separate estimation, the DLS directly estimates $\mu(\bm{X})$, which is advantageous not only in performance but also in computational efficiency.

Theorem 2 immediately follows from Theorem 1. In what follows, we show that the minimizer of the problem (3) can be estimated without going through the estimation of the conditional mean functions $\nu$ and $\pi$. Since $\left(H_{f}(\bm{X})-g(\bm{X})\right)^{2}\geq 0$ for any square integrable function $g\in L^{2}$, we have $H_{f}(\bm{X})^{2}\geq 2H_{f}(\bm{X})g(\bm{X})-g(\bm{X})^{2}$ by expanding the square. The equality holds at $g(\bm{X})=H_{f}(\bm{X})$ for any $f$, which maximizes $2H_{f}(\bm{X})g(\bm{X})-g(\bm{X})^{2}$ with respect to $g$. Hence,

where $J(f,g):=E\left[2H_{f}(\bm{X})g(\bm{X})-g(\bm{X})^{2}\right]$. Rewriting $J(f,g)$ yields

While the objective functional in (3) requires the conditional mean estimators, this form can be estimated based on the sample averages as

We can implement the DLS estimation of $\mu$ with an arbitrary regularization term $\Omega$ in practice:

where $F$ and $G$ are model classes for $f$ and $g$, respectively.

Although any model can be trained by optimizing the model parameters to minimize the above objective functional, a practically useful choice is a linear-in-parameter model. Set $F=\{f_{\bm{\alpha}}:\bm{x}\mapsto\bm{\alpha}^{\top}\bm{\phi}(\bm{x})|\bm{\alpha}\in\mathbb{R}^{q_{f}}\}$ and $G=\{g_{\bm{\beta}}:\bm{x}\mapsto\bm{\beta}^{\top}\bm{\psi}(\bm{x})|\bm{\beta}\in\mathbb{R}^{q_{g}}\}$, where $\bm{\phi}$ and $\bm{\psi}$ are $q_{f}$- and $q_{g}$-dimensional basis functions, respectively. Using the $\ell_{2}$-regularizer, we have

where

and $\lambda_{f}$ and $\lambda_{g}$ are some positive constants. The advantage of this formulation with the linear-in-parameter models and $\ell_{2}$-regularizer is that we have an analytical solution. The solution to the inner maximization is given by

where $\bm{I}_{q_{g}}$ denotes the $q_{g}\times q_{g}$ identity matrix. We can then obtain the DLS estimator of $\bm{\alpha}$ by substituting $\widehat{\bm{\beta}}$ and solving the outer minimization:

For the model selection, we can choose a model that minimizes $\widehat{J}(\widehat{f},\widehat{g})$ evaluated on a validation set. However, as pointed out in [58], one cannot tell if $\widehat{J}(\widehat{f},\widehat{g})$ is small because $\widehat{f}$ is a good solution to the outer minimization, or $\widehat{g}$ is a poor solution to the inner maximization. For this reason, $\widehat{\mu}_{\mathrm{dls}}(\bm{x}):=\widehat{\bm{\alpha}}_{\mathrm{dls}}^{\top}\bm{\phi}(\bm{x})$ can be unstable, and one cannot be confident about which model is the best. The increased dimensionality of the search space because of the need to simultaneously select models for $f$ and $g$ can also make the model selection based on $\widehat{J}(\widehat{f},\widehat{g})$ challenging.

We generated synthetic data with the different shapes of $\mu$, the covariates’ dimension $q_{x}$, and the size of training samples. The mean and standard deviation of the MSE over 100 trials are summarized in Table 2. We conducted the Wilcoxon signed-rank test since the IWLS was highly unstable, which made it difficult to use the $t$-test.

Although we did not trim the PSD for the DWLS estimator, its performance was sufficiently stable to outperform the others in almost all the cases. The DWLS had larger errors than the DLS only when $\mu$ was constant and $q_{x}=1$. This may be because the benefit of direct estimation outweighed the difficulties of hyperparameter tuning of the DLS since the one-dimensional constant $\mu$ is the easiest to learn. This result supports the effectiveness of our approach: avoiding the minimax formulation but without the inverse PSD.

On the other hand, the IWLS estimator often worked terribly poorly even with the weight trimming. The IWLS might be affected by the small PSD most severely since it uses the inverse PSD both in the estimation and hyperparameter tuning. The performance of the DLS was not as poor, but it had a larger MSE than SEP in many cases, indicating that hyperparameter tuning did not work as well as the DWLS.

Figure 2 shows the relation between the squared error of each estimator and the PSD for the linear $\mu(\bm{X})$ when $q_{x}=5$ and $n=10000$. The area with a dense plot is colored in light. The squared error of the DWLS shown in Figure 2a is kept very small except it is slightly larger when the PSD is close to zero. This result demonstrates the robustness of our method against the near-zero PSD, whereas the performance of the other estimators is affected more severely by the small PSD. We can observe the instability of the IWLS again in Figure 2b, which shows the sporadic high errors over the entire PSD. The squared error of the SEP displays the similar pattern. In Figure 2d, the light-colored area is at a high position, indicating that the hyperparameter tuning of the DLS does not work well.

We used the dataset of the National Job Training Partnership Act (JTPA) study¹¹1https://www.upjohn.org/data-tools/employment-research-data-center/national-jtpa-study. This is one of the largest RCT dataset for job training evaluations in the US with approximately 20000 participants, and it has been used in the several previous studies on causal inference [8, 2, 12]. We used the doubly robust (DR) estimator [38] trained with all joint samples as the pseudo true LATE since we do not know the true treatment effect for a real-world dataset.

The results are summarized in Table 2. We report the mean, max, and min of $\widehat{\mu}(\bm{x})$ in addition to the root mean squared error (RMSE) to see the behavior of each estimator. The IWLS had the smallest RMSE, but the difference from the DWLS was not statistically significant. The mean, max, and min of the DWLS and IWLS also indicate they have similar performance. The stability of the IWLS in contrast to the synthetic experiment is because there were no samples with a small PSD. The relatively good performance of the SEP should be because of the same reason. The performance of the DLS was highly unstable, reflecting the difficulty of the model selection based on the DLS objective.