Virtual Control Group: Measuring Hidden Performance Metrics

The fields of Survey theory and causal inference produced statistical methods to try and replicate randomized experiments when only observational data is available. We describe in this section some of the methods of which we have considered. The method tries to estimate W(x), Y(x) or both based on the observed covariates and then draw conclusions over the mean of an unobserved outcome of the treated group.

Many estimation methods are based on the Propensity Score which is the probability of a unit to be treated and defined by [8].

Propensity Score matching is a well-known approach used widely in practice. Each unit’s propensity score is estimated. Then units from the treated group are matched to units from the untreated based on their nearest neighbor in terms of the propensity score. The matching can be 1:1 also called pair matching or it can be k-nearest neighbor matching in which every item in the treated group is matched to k-nearest neighbors with replacement in the untreated group. Let $\mathcal{I}_{\pi}$ be the set of units in the untreated group which are the k-nearest neighbors by the propensity score of some unit in the treated group. Let $y_{i}$ be the true outcome of unit i. The set $\mathcal{I}_{\pi}$ is then used to estimate the mean outcome of the treated group:

Let $\hat{m}(x)$ be a Logistic Regression model fitted on the outcome of the untreated group. Outcome score matching is done by matching units from the treated group with units from the untreated group with nearest values of $\hat{m}(x)$. Let $\mathcal{I}_{y}$ be the set of units in the untreated group which are the k-nearest neighbors by their predicted outcome of some unit in the treated group. The mean outcome is then estimating the mean the same way as in Equation 2.

We used matching on a single continuous covariate i,e the outcome score since according to [1] for the case when only a single continuous covariate is used to match, the efficiency loss can be made arbitrarily close to zero by allowing a sufficiently large number of matches.

We have also considered inverse propensity weighting estimator over the population of nonrespondents [5]

A different group of estimators are regression estimators. These estimators model the outcome conditioned on the covariates using regression and use this model to estimate the missing outcomes. Let the outcome model denoted by $\hat{m}(x)$ and $\mathcal{U}_{t}$ the set of units in the treated group then the mean predicted outcome estimator is

Doubly robust methods take into account both the model predictions for $y_{i}$ with inverse-probability weights. These methods are highly efficient when the y-model is true yet remain asymptotically unbiased when the y-model is misspecified [6]. We have considered such regression model for which the Logistic Regression was trained with non-respondents weights ${\pi}^{-1}(1-\pi)$ and refer to to it as $\hat{m}_{w}(x)$ so that the weighted mean predicted outcome estimator is :

In contrast to Machine Learning settings causal inference requires reporting of confidence intervals on top of the prediction. This is highly important so that the analyst can make a proper decision and know when prediction is of low quality. We use bootstrapping to estimate the standard error of all our estimators as recommended by many authors including [2]. In bootstrapping data variability estimation the dataset is sampled many times with replacement to produce a sample with the same size. For each sample the estimator is being calculated. The standard deviation of the estimations is calculated and reported as the estimation of the standard error of the estimator. From this standard error (SE) the 95% confidence interval can be simply derived by

A simulation study was done to study the performance of different estimators for the potential mean outcome of the treated group. Design (1) is based on similar setting described by [7] and design (2) is similar to one described in [2]

Treatment Design 1: $W(X)=\mathbbm{1}{[a_{1}X_{1}+a_{2}X_{2}+a_{3}X_{3}>w]}$ Outcome Design 1 : $Y(X)=\mathbbm{1}{[b_{1}X_{1}+b_{2}X_{2}+b_{3}X_{3}+\epsilon>y]}$ Outcome Design 2 : $Y(X)=\mathbbm{1}{[b_{1}X_{1}^{2}+b_{2}X_{2}+b_{3}X_{3}+\epsilon>y]}$ $X_{i}\sim\mathcal{N}(0,1),a_{i}\sim\mathcal{U}(0,1),b_{i}\sim\mathcal{U}(0,1),\epsilon\sim\mathcal{N}(0,\sigma)$ $w\sim\mathcal{U}(-1,1),y\sim\mathcal{U}(-1,1)$

We use $\sigma$ to control the outcome model noise and thus the error rate of the estimation. For each design we sample 1000 data sets and estimate the mean outcome using all the estimators described in section 3.1.

The results are summarized in Table 1, Table 2 and Table 3. All results in this paper are presented in percentage format (actual value times 100) for reader convenience. The method of Mean predicted Outcome gave best results in our simulation studies in terms or Root mean square error and Mean Absolute error. We’ve noticed that an increase in the number of samples from 10K to 100K resulted in an improvement for the misspecified model but in quantities much lower than $\sqrt{n}$. The 10-NN outcome matching was the second runner up with low errors suggesting the benefits of matching over more than one unit. Methods not taking into account the outcome such as Inverse probability weighting and propensity score matching gave very bad results which makes them an unreasonable choice for the problem settings.

[6] found that the y-model and -model based estimators are sensitive to misspecification models as not observing the direct confounders and rather observing some non-linear function of them. We have observed the same results for the methods tested. We’ve also noticed that introducing non-linearity by itself is enough to degrade the mean population estimation even if the covariates are observed.

We’ve evaluated the confidence interval validity produced by bootstrapping. We ran the simulation again with 100 samples of design 1. Each sample was of size 10K. Standard errors were estimated using 100 bootstrapped samples for each iteration. We have calculated the distribution ratio between the error and the estimated standard error and also the 95% coverage rate presented in Tables 4, 5. For the correctly specified model all confidence intervals were estimated relatively good besides for the IPW confidence interval which underestimated the error with only about 24% coverage. For the design 2 with misspecified outcome model our estimation of the confidence interval degraded with about  75% coverage. The only good estimation of the coverage rate was produced by the propensity score matching estimator.

To further investigate the estimator we’ve conducted an analysis over the public dataset “Credit Card Fraud Detection” from the Kaggle website. The dataset contains transactions made by credit cards in September 2013 by European cardholders. This dataset presents transactions that occurred in two days, with 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions. It contains only numeric input variables which are the result of a PCA transformation. Due to confidentiality issues, the original features were not provided. Features V1, V2, … V28 are the principal components obtained with PCA, the only features which have not been transformed with PCA are ’Time’ and ’Amount’. Feature ’Time’ contains the seconds elapsed between each transaction and the first transaction in the dataset. The feature ’Amount’ is the transaction Amount, Feature ’Class’ is the response variable and it takes value 1 in case of fraud and 0 otherwise.

We performed feature selection using the Anova test and selected the top 10 features. To simulate a policy and treated group we sample at each iteration 4 features and create a mock policy using a Gauss Naive Bayes model. The data is split 50:50 to a train set and test set. The model is trained on the train set. The treated group size is fixed to 100 units. The treated group is assigned to the units with top model scores in the test set. Following that we run all the estimators and evaluate the results which are summarized in Table 6.

For the empirical results 10-NN score matching outperformed all other methods in terms of RMSE and MAE. IPW remains a bad choice for our domain but surprisingly 1-NN propensity score matching performance increased compared to our initial simulation studies.

The empirical studies along with the simulation studies give strong evidence that these methods can be used in practice to get estimates of fraud prevalence and catch fraud trends.

We evaluate the performance of bootstrapping in estimating the confidence interval of our estimators. We sample the Kaggle dataset 100 times for each estimation method. Each sample contains 900 non-fraud units and 100 fraud units. We train a Gaussian Naive Bayse model on 10% of the data with 4 random features. We then predict the model score over the entire set. We add random uniform noise $\mathcal{U}(0,1)$ to the Bayse Model score to get a higher variance in the positive ratio within the treated group. We select a random treated group size n with distribution $\mathcal{U}(50,80)$ and then assign the top n scores to the treated group. The results are summarized in Table 7 and Figure 7.

It seems that bootstrapping for the use case of Kaggle fraud dataset with Naive Bayse policy overestimated the confidence interval and yielded conservative estimation.