Learning with Selectively Labeled Data from Multiple Decision-makers

Define $x^{+}=\max\{x,0\}$, $x\land y=\min\{x,y\}$ and $x\lor y=\max\{x,y\}$ for $x,y\in\mathbb{R}$. We use the symbol $[N]$ as a shorthand for the set $\{1,2,\cdots,N\}$ and $|\cdot|$ to denote the absolute value of number or cardinality of set.

The weight functions $\{w_{k}\}_{k=1}^{K}$ differ between point-identified and partial-identified settings, requiring separate estimation. For $\operatorname{mode}=\operatorname{point}$, we first estimate $\operatorname*{\mathrm{Cov}}(D,Z\mid X)$ and $\operatorname*{\mathrm{Cov}}(DY_{k},Z\mid X)$ in eq. (4.2), then compute their ratio. For example, $\operatorname*{\mathrm{Cov}}(DY_{k},Z\mid x)$ can be expressed as $\mathbb{E}[DZY_{k}\mid x]-\mathbb{E}[DY_{k}\mid x]\cdot\mathbb{E}[Z\mid x]$, with conditional expectations estimated via regression on $X=x$. Alternatively, forest methods, such as Generalized Random Forests, can directly estimate the conditional covariance ratio in one step [Athey et al., 2019].

For $\operatorname{mode}=\operatorname{partial}$, we estimate the bounds $l_{k}$ and $u_{k}$ for each $k\in[K]$ as defined in Theorem 3, involving conditional expectations $\mathbb{E}[DY_{k}\mid x,z]$ and $\mathbb{E}[D\mid x,z]$ via regression. These bounds are then used to construct $w_{k}(x)$.

The empirical surrogate risk in Equation 5.3 defines a cost-sensitive classification problem. Once the weight function $\{w_{k}\}_{k=1}^{K}$ are estimated, we can then solve for the score function $\boldsymbol{h}$ , which can be implemented using $\operatorname{PyTorch}$ by defining a cost-sensitive loss function with $\operatorname{softmax}$ being the output layer. We can also incorporate regularization in Equation 5.3 to reduce over-fitting. In the binary setting, we can simply solve a weighted classification problem with $\operatorname*{\mathrm{sgn}}[\hat{w}(X)]$’s as the labels and $|\hat{w}(X)|$ as weights, and this can be easily implemented by off-the-shelf ML packages that take additional weight inputs. For example, considering the surrogate risk in Equation 6.3 with $\phi$ as the logistic loss, we can simply run a logistic regression with the aforementioned labels and weights.

In Algorithm 1, we divide the dataset into $L$ folds and apply the cross-fitting to estimate these weight functions. This approach has been widely used in statistical inference and learning with ML nuisance function estimators [e.g., Chernozhukov et al., 2018, Athey and Wager, 2021], which effectively alleviates the over-fitting bias by avoiding using the same data to both train weight function estimators and evaluate the weight function values.

Consider a dataset consisting of observable variables $\mathbf{X}=(X_{1},\dots,X_{p})$, and an unobservable variable $\mathbf{U}=(U_{1},U_{2},\dots,U_{q})$, following the joint Gaussian distribution. We also introduce the variable $Z$ that represents the random assignment of decision-makers, which is uniformly drawn from the set $\mathcal{Z}=\{1,2,\dots,J\}$. The missingness decision $D\in\{0,1\}$ is modeled as Bernoulli distributed variables with parameters $p_{D}:=\mathbb{P}(D=1\mid X,U,Z)$, and the true label $Y^{\star}\in\{1,\dots,K\}$ is modeled as a categorical variable with parameters $p_{k}:=\mathbb{P}(Y=k\mid X,U,Z)$ for $k\in[K]$. The complete data generating process is summarized in Equation 7.1.

Here we define the functions $\operatorname{expit}:v\mapsto 1/(1+\exp(-v))$ and $\text{softmax}:\mathbf{v}\mapsto\exp(\mathbf{v})/\sum_{k=1}^{K}\exp(v_{k})$ for any vector $\mathbf{v}\in\mathbb{R}^{K}$. In both UC and NUCEM models, the parameter $\alpha_{D}\in(0,1)$ controls the impact of unobservable variables $\mathbf{U}$ on the probability of missingness $p_{D}$, while the parameter $\alpha_{Y}\in(0,1)$ adjust the magnitude of $\mathbf{U}$ affecting the distribution of outcome $Y^{*}$. Overall, the pair of $(\alpha_{D},\alpha_{Y})$ jointly determine the degree of selection bias in our data generating process.

In this experiment, we simulate a full dataset with sample size $N=10000$, feature dimensions $p=q=5$, the number of instruments level $J=5$, and the label classes $K=3$. Each data point in the dataset is a tuple $\{(X_{i},U_{i},Z_{i},D_{i},Y_{i}^{*})\}_{i=1}^{N}$. The observed dataset consists of a tuple of variables $\{(X_{i},Z_{i},D_{i},Y_{i})\}_{i=1}^{N}$ with $Y_{i}:=Y_{i}^{\star}$ if $D_{i}=1$ and $Y_{i}:=\text{NaN}$ if $D_{i}=0$ for each $i\in[N]$.

The missingness model NUCEM captures one of the setting of No Unmeasured Common Effect Modifier, and this ensures the satisfaction of sufficient condition in Proposition 2, and therefore the conditional probabilities $\{p_{k}\}_{k=1}^{K}$ can be exactly identified according to Theorem 1. Meanwhile, the model UC captures a general setting of Unmeasured Confounding in which the point-identification condition cannot be realized in this situation.

We divide our dataset into two parts: a training set and a testing set, using a 7:3 split ratio, denoted as $\mathcal{S}_{\text{train}}$ and $\mathcal{S}_{\text{test}}$. In the training set, five different meta-learning methods are trained on selectively labeled data, using a simple neural network classifer with softmax output layer. We then evaluate the performance of these methods on the testing set with fully labeled data. The five methods under consideration are as follows:

• •

  The first two methods act as baselines. The SelectedSample method involves running multiclass classification algorithm solely on the labeled portion of the training set $\{i\in S_{\text{train}}:D_{i}=1\}$. For the second method, SelectedSample(IPW), we firstly estimate the inverse propensity weighting (IPW) for the labeled subset, and then implement the weighted multiclass classification for the labeled training set. These baseline methods establish a “lower bound” for any other meta learning algorithm, as any enhanced method with the use of selectively labeled data should least beat the performance of these two methods.

• •

  The third method, FullSample, runs a multiclass classification algorithm on the entire training set. This represents an ideal but impractical scenario since we cannot actually observe the true label $Y_{i}^{*}$ for the missing data (where $D_{i}=0$). This method serves as an ’upper bound’ for learning performance, indicating the highest possible effectiveness if full information were available.

• •

  Finally, our two proposed methods correspond to the point- and partial- identification settings, named PointLearning and PartialLearning respectively. We anticipate that the performance of these methods will fall between the established “lower” and “upper” bounds, representing a realistic estimation of effectiveness under selective label conditions. Similar to the method SelectedSample(IPW), our method involve estimating a series of weight functions $w_{k},k=1,\dots,K$ to correct for selection bias (refers to the details in Section 6). In this experiment, we estimate these weght functions using Histogram Gradient Boosting with a 5-fold cross-fitted approach. We selected all hyperparameter through 5-fold cross-validation. The cost-sensitive classification problem is solved by a simple neural network with a softmax output layer.

It’s important to note the random assignment of decision-maker, denoted as $Z$, play different roles in these methods. In our proposed method, $Z$ is treated as an instrumental variable, aiding in correcting for selection bias, while in the baseline methods (SelectedSample, SelectedSample(IPW), and FullSample), $Z$ is only one of the observed features.

Figures 4 and 5 report the testing accuracy of each method on the data generated by the NUCEM and UC models in Equation 7.1 respectively. All the experiments are repeated 100 times with confounding strengths $\alpha_{D},\alpha_{Y}\in\{0.3,0.5,0.7\}$. Each boxplot then shows the distribution of the testing accuracy for distinct methods under different combinations of confounding strength $\alpha_{D}$ and $\alpha_{Y}$.

Transition from the top to the bottom panel, $\alpha_{Y}$ increases from $0.3$ to $0.7$, indicating a stronger influence of the unobservable variable $U$ on the true label $Y^{\star}$. As we can see from Figures 4 and 5, the accuracy of each of five methods decreases as $\alpha_{Y}$ increases, reflecting the growing challenge in accurately predicting outcomes as the influence of unobservable factors intensifies. Moreover, transition from the left to the right panel, $\alpha_{D}$ rises from $0.3$ to $0.7$, indicating an enhanced role of the unobservable variable $U$ in determining the missingness decision $D$. A larger value of $\alpha_{D}$ within the range $(0,1)$ suggests that the missingness decision $D$ is predominantly influenced by the unobservable $U$, hinting at a potential larger difference of the distribution among selected group and missing group. This is visually corroborated in Figures 4 and 5, where the performance disparity between the SelectedSample and FullSample methods widens as $\alpha_{D}$ increases, illustrating the growing challenge in correcting the selection bias.

The results in Figures 4 and 5 demonstrate the effectiveness of our proposed methods in handling selection bias in the data generating process. In each combinations of $(\alpha_{Y},\alpha_{D})$, our Partial Learning method consistently outperforms the baseline methods, including the Selected Sample, and Selected Sample(IPW) methods. Moreover, the perforamnce of Partial Learning is close to the Full Sample method, which is the upper bound of the performance. The performance of the Point Learning method is also competitive, although it is slightly less robust than the Partial Learning method. The results in Figure 4 are consistent with those in Figure 5, except that under this setting, the point identification requirement in Theorem 1 is not satisfied, and the Point Learning method is therefore less effective than the Partial Learning method.

Below we provide a detailed analysis of the performance of each method.

• •

  Selected Sample (IPW): As we see from Figures 4 and 5, the performance of the Selective Sample (IPW) approach is close to the baseline of naively running multiclass classification algorithms on the selected subsample (named Selected Sample), and it is outperformed by our proposed algorithms. This is not surprising since the propensity score $P(D=1\mid X,Z)$ cannot correct selection bias due to the existence of unobserved variables $U$.

• •

  PointLearning: Our proposed Point Learning method shows promising results in the bottom right panel of Figure 4 with parameter $(\alpha_{Y},\alpha_{D})=(0.7,0.7)$, in which case the unobserved confounding strength is large. Although the NUCEM model in Equation 7.1 satisfies Proposition 2, which enables the point identification of the conditional probabilities and therefore the classification risk $\mathcal{R}(h,w)$, Point Learning does not outperform the baseline methods when the unmeasured confounding strength is not so large. This limitation is attributed to the complexities involved in accurately estimating the weight function $\{w_{k}\}_{k=1}^{K}$ as outlined in Equation 4.3.

  The crux of the challenge lies in estimating the ratio of two conditional covariances, $\operatorname*{\mathrm{Cov}}(DY_{k},Z\mid X)$ and $\operatorname*{\mathrm{Cov}}(D,Z\mid X)$ for each $k\in[K]$. Estimating conditional covariances is inherently more difficult than estimating conditional expectations, primarily because the process of calculating the ratio of these estimates is fraught with potential inaccuracies. If the denominator is not precisely estimated, the resultant ratio can become unstable, leading to exaggerated or diminished values. Such inaccuracies can severely impact the effectiveness of the downstream cost-sensitive classification, undermining the robustness of Point Learning under these conditions.

• •

  Partial Learning: The cornerstone of our learning framework is Partial Learning. As evidenced by the graphs in Figures 4 and 5, Partial Learning significantly enhances the accuracy of performance predictions while maintaining low variance, showcasing its potential for real-world applications. Remarkably, this improvement in prediction accuracy is consistent across almost all combinations of confounding strengths $(\alpha_{Y},\alpha_{D})$, as well as the decision models NUCEM and UC.

  Reflecting on the conditions outlined in Theorem 3, Partial Learning only requires the conditional mean function $\mathbb{E}[Y_{k}^{\star}\mid U,X=x],k=1,\dots,K$ to be almost surely bounded, aside from the basic instrumental variable (IV) conditions. This requirement is generally more feasible in real-world scenarios than the point-identification assumptions detailed in Theorem (referenced as Theorem 1). The success of Partial Learning can be attributed to two key strategies: Firstly, we adopt a robust optimization approach to develop a prediction rule that remains effective under the least favorable conditions of $\eta^{\star}(X)$, as detailed in our minimax risk function formulation (see Equation 5.2). Secondly, the weighting functions $\{w_{k}\}_{k=1}^{K}$ are based on the summation of several conditional expectations, which is inherently more stable than the ratio of conditional covariances required for point identification.

Overall, the compelling performance of Partial Learning demonstrated in Figures 4 and 5 convinces us of its practical viability. It stands as a reliable method that consistently outperforms baseline approaches, even in situations where the presence of selection bias (missing not at random) in the data generating process is uncertain.

In this dataset the label of interest is fully observed, so we choose to synthetically create selective labels on top of the dataset. Specifically, we simulate $10$ decision-makers (e.g., bank officers who handle the loan applications) and randomly assign one to each case. We simulate the decision $D$ from a Bernoulli distribution with a success rate $p_{D}$ that depends on an “unobservable” variable $U$, the decision-maker identity $Z$, and the ExternalRisk variable (which serves as an algorithmic assistance to human decision-making). We blind the true label $Y^{\star}$ for observations with $D=0$. Specifically, we construct $U$ as the residual from a random forest regression of $Y^{\star}$ with respect to $X$ over the whole dataset, which is naturally dependent with $Y^{\star}$. We then specify $p_{D}:=\mathbb{P}(D=1\mid X,U)$ according to

Here the $\operatorname{expit}$ function is given by $\operatorname{expit}(t)=1/(1+\exp(-t))$. The parameter $\alpha_{D}\in(0,1)$ controls the impact of $U$ on the labeling process and thus the degree of selection bias. We can easily verify that the sufficient condition in Proposition 2 is guaranteed and therefore the point-identification of classification risk $\mathcal{R}(h,w)$ is realizable under the NUCEM model in Equation 7.2.

We randomly split our data into training and and testing sets at a $7:3$ ratio. Similar to what we discuss in Section 7.1, on the training set, we apply five types of different methods: Selected Sample, SelectedSample(IPW), FullSample, PointLearning and PartialLearning. The first three methods serve as the benchmark of the last two methods. For each type of method, we try multiple classification algorithms including AdaBoost, Gradient Boosting, Logistic Regression, Random Forest, and SVM. Our proposed method also need to estimate some unknown weight functions, which we implement by $K=5$ fold cross-fitted Gradient Boosting. Again, all hyperparameters are chosen via $5$-fold cross-validation, and we evaluate the classification accuracy of the resulting classifiers on the testing data.

Similarly, we remark that the decision-maker assignment $Z$ plays different roles in different methods. Our proposed methods (PointLearning and PartialLearning) treatment the decision-maker assignment $Z$ as an instrumental variable to correct for selection bias. In contrast, the baseline methods (SelectedSample, SelectiveSample(IPW) and FullSample) do not necessarily need $Z$. However, for a fair comparison between our proposals and the baselines, we still incorporate $Z$ as a classification feature in the baseline methods, so they also use the information of $Z$.

Figures 6 and 7 presents the testing accuracy of each method over 50 experiment replications for $\alpha\in\{0.5,0.7,0.9\}$ under both the NUCEM and UC models defined in Equation 7.2. First, we observe that the performance of SelectedSample(IPW) is comparable to the baseline SelectedSample, which applies binary classification algorithms directly to selectively labeled data. Notably, as the strength of unmeasured confounding ($\alpha_{D}$) increases, the gains from using our proposed methods—especially PartialLearning—over the baseline methods (SelectedSample and SelectedSample(IPW)) become more pronounced. Interestingly, the PartialLearning method outperforms even under the NUCEM model, where the point-identification condition is satisfied. As discussed in Section 7.1, this may be because the PointLearning method relies on estimating a conditional variance ratio, which is challenging to estimate accurately in practice. In contrast, the PartialLearning method requires only the estimation of conditional expectations, making it more stable and robust. This highlights the advantage of the PartialLearning method: it is robust to violations of the point-identification assumption and achieves more stable performance even when point-identification holds.