Causal Meta-Mediation Analysis: Inferring Dose-Response Function From Summary Statistics of Many Randomized Experiments

Nowadays it is common in the internet industry to develop algorithms that power online products using historical data. The one that improves evaluation metrics from historical data will be tested against the one that has been in production to assess the lift in key performance indicators (KPIs) of the business in online A/B tests. Here we refer to metrics calculated from historical data as offline metrics and metrics calculated in online A/B tests as online metrics. In many cases, offline evaluation metrics are different from online business KPIs. For instance, a ranking algorithm, which powers search pages in e-commerce platforms, typically optimizes for relevance by predicting purchase or click probabilities of items. It could be tested offline (offline A/B tests) for rank-aware evaluation metrics, for example, normalized discounted cumulative gain (NDCG), mean reciprocal rank (MRR) or mean average precision (MAP), which are calculated from the test set of historical purchase or click-through feedback of users. Most e-commerce platforms, however, deem sitewide gross merchandise value (GMV) as their business KPI and test for it online. There could be various reasons not to directly optimize for business KPIs offline or use business KPIs as offline evaluation metrics, such as technical difficulty, business reputation, or user loyalty. Nonetheless, the discrepancy between offline evaluation metrics and online business KPIs poses a challenge to product owners because it is not clear that, in order to maximize online business KPIs, which offline evaluation metric should be adopted to guide the offline development of algorithms.

The challenge essentially asks for the causal effects of increasing offline evaluation metrics on business KPIs, e.g., how business KPIs would change for a 10% increase in an offline evaluation metric. The offline evaluation metric in which a 10% increase could result in the most significant lift in business KPIs should be the north star to guide algorithm development. Algorithms developed offline power online products, and online products contribute to the success of the business (see Figure 1). By noting that online products can be measured by online evaluation metrics, the online counterparts of offline evaluation metrics, we decompose the problem into two parts. The offline A/B test literature (see, e.g, Gilotte et al. (2018)) works out the first part (the black arrow): counterfactual estimators of offline evaluation metrics to bridge the inconsistency between changes of offline and online evaluation metrics. We focus on the second part (the red arrow): the causality between online products (assessed by online evaluation metrics) and the business (assessed by online business KPIs). The offline evaluation metric whose online counterpart causes the most significant lift in online business KPIs should be the north star. Hence, the question for us becomes, how business KPIs would change for a 10% increase in an online evaluation metric.

Randomized controlled trials, or online A/B tests, are popular to measure the causal effects of online product change on business KPIs. Unfortunately, they cannot answer our question directly. In online A/B tests, in order to compare the business KPIs caused by different values of an online evaluation metric, we need to fix the metric at its different values for treatment and control groups. Take the ranking algorithm as an example. If we could fix online NDCG of the search page at 0.22 and 0.2 for treatment and control groups respectively, then we would know how sitewide GMV would change for a 10% increase in online NDCG at 0.2. However, this experimental design is impossible, because most online evaluation metrics depend on users’ feedback and thus cannot be directly controlled.

We address the question by developing a novel approach of causal inference. We model the causality between online evaluation metrics and business KPIs by dose-response function (DRF) in potential outcome framework (Imbens, 2000; Imbens and Hirano, 2004). DRF originates from medicine and describes the magnitude of the response of an organism given different doses of a stimulus. Here we use it to depict the value of a business KPI given different values of an online evaluation metric. Different from doses of stimuli, values of online evaluation metrics cannot be directly manipulated. However, they could differ between treatment and control groups in experiments of treatments other than algorithms—user interface/user experience (UI/UX) design, marketing, etc. This could be due to the “fat hand” (Peysakhovich and Eckles, 2018; Yin and Hong, 2019) nature of online A/B tests that a single intervention can change many causal variables at once. A change of the tested feature, which is not algorithm, could induce users to change their engagement with algorithm-powered online products, so that values of online evaluation metrics would change. For instance, in an experiment of UI design, users might change their search behaviors because of the new UI design, so that values of online NDCG, which depends on search interaction, would change, even though ranking algorithm does not change. The evidence suggests that online evaluation metrics could be mediators that (partially) transmit causal effects of treatments on business KPIs in experiments where treatments are not necessarily algorithm-related. Hence, we formalize the problem as the identification, estimation, and test of mediator DRF.

In mediation analysis literature, there are two popular identification techniques: sequential ignorability (SI) and instrumental variable (IV). SI assumes each potential mediator is independent of all potential outcomes conditional on the assigned treatment, whereas IV permits dependence between unknown factors and mediators but forbids the existence of direct effects of the treatment. Rather than making these stringent assumptions, we leverage trial characteristics to explain average direct effect (ADE) in each experiment so that we can tease it out from average treatment effect (ATE) to identify the causal mediation. The utilization of trial characteristics means we have to use data from many trials because we need variations in trial characteristics. Hence, we develop our framework as a meta-analysis and propose an algorithm that only uses summarized results from many existing experiments and gain the advantage of easy implementation to scale.

Most meta-analyses rely on summarized results from different studies with different raw data sources. Therefore, it is almost impossible to learn more beyond the distribution of ATEs. Fortunately, the internet industry produces plentiful randomized trials with consistently defined metrics, and thus presents an opportunity for performing a more complicated meta-analysis. Literature is lacking in this area while we create the framework of causal meta-mediation analysis (CMMA) to fill in the gap.

Another prominent strength of our approach in real application is, for a new product that has been shipped online but has few A/B tests, it is plausible to explore the causality between its online metrics and business KPIs from many A/B tests of other products. The values of online metrics of the new product can differ between treatment and control groups in experiments of other products (”fat hand” (Peysakhovich and Eckles, 2018; Yin and Hong, 2019)), which makes it possible to solve for mediator DRF of the new product without its own A/B tests.

Note that, our approach can be applied to any evaluation metric that is defined at experimental-unit level, like metrics discussed in offline A/B test literature. The experimental unit means the unit for randomization in online A/B tests. For example, in search page experiments, the experimental unit is typically the user. Also, the evaluation metric can be any combination of existing experimental-unit-level metrics.

To summarize, our contributions in this paper include:

1. (1)

   This is the first study that offers a framework to choose the north star among all available offline evaluation metrics for algorithm development to maximize business KPIs when offline evaluation metrics and business KPIs are different. We decompose the problem into two parts. Since the offline A/B test literature works out the first part: counterfactual estimators of offline evaluation metrics to bridge the inconsistency between changes of offline and online metrics, we work out the second part: inferring causal effects of online evaluation metrics on business KPIs. The offline evaluation metric whose online counterpart causes the most significant lift in business KPIs should be the north star. We show the implementation of our framework on data from Etsy.com.

2. (2)

   Our novel approach CMMA combines mediation analysis and meta-analysis to identify, estimate, and test mediator DRF. It relaxes standard SI assumption and overcomes the limitation of IV, both of which are popular in causal mediation literature. It extends meta-analysis to solve causal mediation while the meta-analysis literature only learns the distribution of ATE. We demonstrate its effectiveness by simulation and show its performance is superior to other methods.

3. (3)

   Our novel approach CMMA uses only trial-level summary statistics (i.e., meta-data) of many existing trials, which makes it easy to implement and to scale up. It can be applied to all experimental-unit-level evaluation metrics or any combination of them. Because it solves for causality problem of a product by leveraging trials of all products, it could be particularly useful in real applications for a new product that has been shipped online but has few A/B tests.

We draw on two strands of literature: mediation analysis and meta-analysis. We briefly discuss them in turn.

We follow the literature of potential outcomes (Rubin, 2003; Imai et al., 2010; Small, 2012) to set up our framework.

As illustrated in Figure 2, we suppose there are many experiments with different treatments, and there is a mediator $M$ that can be affected by any treatment $T^{s}$ and will in turn influence outcome $Y$. Each treatment may also affect $Y$ directly. But we are particularly interested in recovering their shared casual channel, the link between $M$ and $Y$, marked in red in the figure. In the ranking algorithm example, $M$ is online evaluation metric of search page, $Y$ is online business KPI. A challenge to identify the red link emerges if a confounder $U$ exists. $U$ could be (user engagement of) any other web-page/module or user preference in the ranking algorithm example. In the literature, there are two approaches to solve this challenge, SI and IV. SI requires that $U$ is observed and measured, and there are no other unmeasured/unobserved confounders after controlling for $U$. The standard IV approach allows unmeasured/unobserved $U$, but assumes no direct links between $T^{s}$s and $Y$. An IV method proposed by Small (2012) relaxes requirements of standard approach, and assumes all $T^{s}$s share a single direct link to $Y$. Our method allows unmeasured/unobserved $U$ and direct links from each $T^{s}$ to $Y$.

Suppose there are $K$ randomized trials in total. For each trial, there exists one treatment group and one control group¹¹1Experiments with multiple treatment arms can be considered as multiple trials with one treatment group and one control group.. To simplify the discussion, here we assume experimental units are first randomly assigned to different trials, and then randomly assigned into the treatment or control group of that trial. However, our approach CMMA allows the same unit to participate in multiple trials. We will go back to this point in Section 5.1.

We consider the following model for potential outcomes.

We use a one-hot vector $\bm{s}=(s_{1},\cdots,s_{K})$ to encode the trial assignment where $(0,\cdots,s_{k}=1,\cdots,0)$ indicates the assignment to trial $k$, and a binary variable $t\in\{0,1\}$ to encode treatment assignment, with $t=1$ if assigned to the treatment group. For experimental unit $i$ with trial assignment $\bm{s}$ and treatment assignment $t$, the random variable $M_{i}^{(t,\bm{s})}$ represents the potential mediator, and the random variable $Y_{i}^{(t,\bm{s},m)}$ represents the potential outcome that would be observed if $i$ were to receive or exhibit level $m$ of the mediator through some hypothetical mechanism.

We are interested in mediator DRF, which represents the value of $Y_{i}^{(t,\bm{s},m)}$ given different values ($m$) of $M_{i}^{(t,\bm{s})}$ for the same $t$ and $\bm{s}$. This effectively is the red arrow in the Figure 2 for individuals. Let $\mu_{i}(m)$ be the mediator DRF for individual $i$. Our goal is to estimate average mediator DRF: $\mathbb{E}[\mu_{i}(m)]$, with which we can compute the percentage change of $\mathbb{E}[\mu_{i}(m)]$ for a 10% increase in $m$ ceteris paribus. The expectation here is taken over the population of $i$ and so are other expectations in this paper if not specified. Here we consider polynomial mediator DRF: $\mu_{i}(m)=\sum_{p=1}^{P}\beta_{i,p}m^{p}$, which can capture the nonlinearity of the causality. Estimating $\mathbb{E}[\mu_{i}(m)]$ means to estimate $\beta_{p}=\mathbb{E}[\beta_{i,p}]$ for $p=1,2,\cdots,P$. Vectors $\bm{\tau_{i}}$ and $\bm{\gamma_{i}}$ are in $\mathbb{R}^{K}$. Each element of the vector, $\tau_{i,k}$ or $\gamma_{i,k}$, represents direct effect of the treatment on mediator or outcome in trial $k$, and is assumed not to depend on $m$.

Vectors $\bm{\phi}$ and $\bm{\theta}$ are also in $\mathbb{R}^{K}$, representing trial fixed effects. Random variables $M_{i}^{*}$ and $Y_{i}^{*}$ represent idiosyncratic individual characteristics in the values of potential mediator and potential outcome. Assume $\mathbb{E}[M_{i}^{*}]=\mathbb{E}[Y_{i}^{*}]=0$ ²²2This assumption can always be satisfied by reparameterizing $M_{i}^{*},Y_{i}^{*}$ and $\bm{\phi},\bm{\theta}$.. Let $\mathcal{M}$ be the support of the distribution of mediator, and $\mathcal{S}$ be the set containing all possible trial assignments.

We only observe realized data $\{M_{i},Y_{i},T_{i},\bm{S}_{i}\}$. The observed mediator $M_{i}:=M_{i}^{(T_{i},\bm{S}_{i})}$, and observed outcome $Y_{i}:=Y_{i}^{(T_{i},\bm{S}_{i},M_{i})}$.

The specification has two implications. First, it implies that being in a particular trial will not affect mediator DRF. If mediator DRF is not trial independent, then having many trials only adds noises rather than provides additional information, and thus defeats the purpose of conducting a meta-analysis.

Second, the specification implies that there are no interaction effects between treatment and mediator on the outcome. It means the individual direct effect in each trial is irrelevant of the value of mediator. It is common in the literature of causal mediation (see, e.g., NPSEM-IE of Pearl (2001) and FRCISTG of Robins (2003)).

When mediator $M_{i}$ is not independent of $Y_{i}$ (which is the case here), it is not trivial to recover $\mathbb{E}[\mu_{i}(m)]$ through observed data. Figure 3 illustrates the challenge using simulated data. The grey lines in Figure 3 are simulated individual mediator DRFs following Definition 1, which represents the true causality between $M_{i}$s and $Y_{i}$s. The blue line is the average mediator DRF. After randomly assign individuals to a trial, we can compute their observed mediator values $M_{i}$ and outcome values $Y_{i}$ when in the control group, which are depicted by black scattered points. The black line shows the result from fitting the observed points by a widely-used non-parametric machine learning algorithm, locally estimated scatterplot smoothing (LOESS). Although the black line fits the data almost perfectly, it significantly deviates from the true underlying causality: the average mediator DRF.

We conduct Monte Carlo simulations to study the finite sample performance of our estimator. The details of our simulation set-up are described in Appendix C. The R code is available in our GitHub repository: https://github.com/znwang25/cmma.

We use Limited Information Maximum Likelihood (LIML) estimator as a benchmark to evaluate our CMMA estimator. An LIML estimator that is specified according to the simulation setup should have the best performance theoretically. We include two common estimators in the literature of mediation analysis into the comparison: Sobel (2008) and LSEM (Baron and Kenny, 1986), which are derived under identification approaches discussed in Section 2. Sobel (2008) assumes complete mediation, and LSEM (Baron and Kenny, 1986) relies on SI assumption (Imai et al., 2010). Both assumptions are false under our setting. We also implement the Full Sample 2SLS estimator prescribed in Theorem 1, which should produce similar results to CMMA.

We set the sample size per trial ($N_{per}$), to 200, 500, and 1000 and perform 100 simulations for each setting. Table 1 reports average biases and 95% confidence interval coverage of the estimators. The performance of CMMA is quite good and largely comparable to LIML’s result. When the sample size per trial is small, our estimator is slightly biased but the bias is much smaller than those of Sobel and LSEM estimators. As $N_{per}$ increases, the bias of our estimator decreases to zero, whereas the biases of the Sobel and LSEM estimators remain roughly the same. As the sample size is usually enormous in A/B tests (on average, one trial has millions of observations in the real data we obtained from an internet company), the bias of CMMA will be negligible in practice. Table 1 also shows that, as $N_{per}$ increases, the 95% confidence interval coverage of our estimator converges to the nominal coverage. This means that the OLS variance estimated in the trial-level regression is valid for hypothesis testing when $N_{per}$ is sufficiently large. In addition, the point estimate of Full Sample 2SLS estimator is numerically equal to CMMA, which empirically validates the equivalence claim made in Section 5.1.

In Table 2, we examine the performance of CMMA when each assumption fails. The results show that, the failure of Assumption 2 does not seem to affect the estimator’s unbiasedness, whereas with failures of Assumption 3 or 4, CMMA is no longer unbiased. Comparing results across rows, it seems to suggest that violating Assumption 4 has worse consequence in terms of bias. Fortunately, Assumption 4 is testable as discussed in Section 4.

We also test the performance of the Wald test with a higher order of $\mu(m)$. Table 3 shows that Wald tests can successfully select the correct model. For example, in the second row of Table 3, the true mediator DRF is a quadratic function of $m$: $4m+2m^{2}$. Wald tests in all simulations reject the null hypothesis that coefficients for the second-degree term and the third-degree term are zeros, whereas 97% of simulations fail to reject the hypothesis that coefficient for the third-degree term is zero. The result suggests that the highest order of ${\tt ATE}^{(M^{p})}_{k}$ should be 2. The last column of Table 4 shows that, with a correctly specified model, we can accurately estimate all the underlying parameters.

We apply the approach on three most popular rank-aware evaluation metrics: NDCG, MRR, and MAP, to show, for ranking algorithms that power search page of Etsy.com, which one could lead to the most significant lift of sitewide GMV. Since the offline A/B test literature (Gilotte et al., 2018) bridges the inconsistency between changes of offline and online evaluation metrics, we only focus on, how sitewide GMV would change for 10% lifts in online NDCG, MRR, and MAP of search page respectively. All metrics in the application, unless otherwise noted, are online metrics. Please note the approach has not been deployed in Etsy. This work is not intended to apply to, nor is it a prediction of, actual live performance metrics or performance changes on Etsy or any other property.

In the internet industry, the algorithms developed offline power online products and online products contribute to the success of a business. In many cases, offline evaluation metrics, which guide algorithm development, are different from online business KPIs. It is important for product owners to pick the offline evaluation metric guided by which the algorithm could maximize online business KPIs. By noticing that online products could be assessed by online counterparts of offline evaluation metrics, we decompose the problem into two parts. Since the offline A/B test literature works out the first part: counterfactual estimators of offline evaluation metrics that move the same way as their online counterparts, we focus on the second part: inferring causal effects of online evaluation metrics on business KPIs. The offline evaluation metric whose online counterpart causes the most significant lift in online business KPIs should be the north star. We model online evaluation metrics as mediators and formalize the problem as to identify, estimate, and test mediator DRF. Our novel approach CMMA combines mediation analysis and meta-analysis and has many advantages over the two strands of literature. In particular, it takes as inputs only summary statistics from multiple past A/B tests, and thus it is easy to implement in scale. We apply the approach on Etsy’s real data to uncover the causality between three most popular rank-aware online evaluation metrics and GMV, and show how we successfully identify MRR as the offline evaluation metric for GMV maximization.