On Comparing Fair Classifiers under Data Bias

Fairness in AI and machine learning is an important problem because automated decision-making in highly sensitive domains such as justice, healthcare, banking, and education can proliferate social and economic harms [17, 29, 4, 45]. Fair and representative data is a foremost requirement to train and validate fair models. However, most collected and labeled datasets invariably contain historical and systemic biases [5], resulting in unfair and unreliable performance after deployment [58, 13, 61]. Even many standard datasets used to evaluate and compare bias mitigation techniques are not perfectly fair and representative [21, 48].

Recent literature in fair classification has studied the failure of fairness and accuracy guarantees under distribution shifts (e.g., covariate shift, label shift) between the training and test distributions and proposed fixes under reasonable assumptions [57, 20, 54, 59, 44, 22, 10, 56]. In this paper, we consider a simple, natural data bias model that incorporates under-representation, over-representation, and label biases [12], and study the change in accuracy and fairness of various classifiers as we inject bias in their training data. The central question we ask is: Among the various fair classifiers available in the standard fairness toolkits (e.g., [7]), which ones are less vulnerable to data bias?

The definitions of fairness prevalent in literature fall into two broad types: individual fairness and group fairness [24, 6, 50, 27, 50, 33, 62, 8]. Individual fairness promises similar treatment to similar individuals. Our focus is group-fair classification, which promises equal or near-equal outcomes across different sensitive groups (e.g., race, gender). Examples of group fairness include Equal Opportunity (equal false negative rates across groups) and Statistical Parity (equal acceptance rate across groups). Various bias mitigation techniques have been proposed for group-fair classification motivated by different applications and stages of the machine learning pipeline [69, 29, 45]. They are broadly categorized into three types: 1. Pre-Processing, where one transforms the input data, independent of the subsequent stages of training, and validation [37, 25, 15], 2. In-Processing, where models are trained by fairness-constrained optimization [14, 39, 64, 63, 66, 1, 16], and 3. Post-Processing, where the output predictions are adjusted afterward to improve fairness [38, 33, 52] . These techniques appear side by side in standard fairness toolkits with little normative prescription about which technique to choose under data bias. Furthermore, it’s unclear what happens to such standard fair classifiers when they are trained and exposed to varying amounts of data bias over time in the training or deployment environments. We attempt to close this gap by providing an auditing method to compare different fair classifiers under a simple model of data bias.

To examine the effect of data bias on group-fair classifiers, we use a theoretical model by Blum & Stangl [12] for under-representation and label bias. It is a simple model for demographic under-/over-sampling and implicit label bias seen in real-world data [58, 13, 61, 32, 41]. Given a train-test data split, we create biased training data by injecting under-representation and label bias in the training data (but not the test data). We then study the error rates and (un)fairness metrics of various pre-, in-, and post-processing fair classifiers on the original test data, as we vary the amount of bias in their training data. We inject two types of under-representation biases in the training data: $\beta_{\text{pos}}$-bias, where we undersample the positive labeled population of the minority group by a multiplicative factor of $\beta_{\text{pos}}$, and $\beta_{\text{neg}}$-bias, where we undersample the negative labeled population of the minority group by a multiplicative factor of $\beta_{\text{neg}}$. We examine the effect of increasing the parameters $\beta_{\text{pos}}$ and $\beta_{\text{neg}}$ separately as well as together. We also investigate the effect of label bias parameter $\nu$, where we flip the positive labels of the minority group to negative ones with probability $\nu$. We cannot hope to re-create real-world data bias scenarios via this synthetic pipeline [13, 61], but our findings on stability/instability of various classifiers can be somewhat useful towards designing fair pipelines in the presence of various data biases.

The robustness of machine learning models when there is a mismatch between their training and test distributions has been studied under various theoretical models of distribution shifts, e.g., covariate shift and label shift. Recent works have studied fair classification subject to these distribution shifts and proposed solutions under reasonable assumptions on the data distribution [19, 57, 12, 54, 59, 20, 10, 22, 44, 31]. As pointed out in the introduction, our goal is different, so we defer the critical contributions of these works to Appendix A.1. However, later in Section 6, we discuss different distribution shifts and their relation to the data bias model of Blum & Stangl [12] used in our paper.

Our work builds on the under-representation and label bias model proposed in Blum & Stangl [12]. Under this model, they prove that the optimal fair classifier that maximizes accuracy subject to fairness constraints (equal opportunity constraints, to be precise) on the biased data distribution gives the maximum accuracy on the unbiased data distribution. For under-representation bias, the above result can be achieved by equalized odds and equal opportunity constraints but not by demographic parity constraints. Along similar lines, Maity et al. [44] derive necessary and sufficient conditions about when performance parity constraints in the presence of subpopulation shifts give better accuracy on the test distribution. Jiang et al. [36] have also considered the label bias model, although we look at the aggregate effect over varying amounts of label bias. Our observations on label bias also corroborate various findings by other works investigating the role of label noise in fair classification [43, 28, 60, 42]. Under-representation bias can also be studied as a targeted Sample Selection Bias mechanism [18]. There is some work done on fair sample selection bias [22, 67], which we elaborate on in Appendix A.1.

Parallel to our work, a recent preprint by Akpinar et al. [2] proposes a simulation toolbox based on the under-representation setting from Blum & Stangl [12] to stress test four different fairness interventions with user-controlled synthetic data and data biases. On the other hand, our work uses the under-representation framework from Blum & Stangl [12] to extensively examine the stability of different types of fair classifiers on various synthetic and real-world datasets. While Akpinar et al. [2] focus on proposing a simulation framework to extensively test all findings of Blum & Stangl [12], we focus on using the under-representation framework to highlight the differences in performance between various fair classifiers and theoretically investigating the stability of one of the fair classifiers.

We compare various fair classifiers in the presence of under-representation and label bias, which complements some expository surveys in fair machine learning [69, 45, 49, 26] and some benchmarking on important aspects like the choice of data processing, splits, data errors, and data pre-processing choices [29, 10, 53, 35]. We elaborate on these works in the Appendix A.2.

For our analysis, we select a comprehensive suite of commonly used fair classifiers implemented in the open-source AI Fairness 360 (AIF-360) toolkit [7]. From the available choice of pre-processing classifiers, we include two: 1. Reweighing (‘rew’) [37], and 2. Adaptive Reweighing (‘jiang_nachum’) [36] . Since ‘jiang_nachum’ is not a part of AIF-360, we use the authors’ implementation¹¹1https://github.com/google-research/google-research/tree/master/label_bias. Among the in-processing classifiers in AIF-360, we choose three: 1. Prejudice Remover classifier (‘prej_remover’) [39], 2. Exponentiated Gradient Reduction classifier (‘exp_grad’), and its deterministic version (‘grid_search’) [1] and 3. the method from Kearns et al. (‘gerry_fair’) [40] . From post-processing classifiers, we choose three: 1. the Reject Option classifier (‘reject’) [38], 2. the Equalized Odds classifier (‘eq’) [33], and 3. the Calibrated Equalized Odds algorithm (‘cal_eq’) [52] . We omitted the use of some classifiers in the AIF360 toolkit for two reasons: resource constraints and their extreme sensitivity to the choice of hyperparameters.

All fair classifiers use a base classifier, which it uses in its optimization routine to mitigate unfairness. We experiment with two base classifiers: a logistic regression (LR) model and a Linear SVM classifier with Platt’s scaling [51] for probability outputs. We use the Scikit-learn toolkit to implement the base classifiers [47]. The ‘prej_remover’ and ‘gerry_fair’ classifiers do not support SVM as a base classifier; hence, we only show results for those on LR.

The fairness metric we use in the main paper is Equalized Odds (EODDS) [33], which is defined as the difference of False positive and False negative Rates, and we measure an equally weighted combination of both to report our results $0.5*|\operatorname{Pr}(\hat{Y}=0|Y=1,S=1)-\operatorname{Pr}(\hat{Y}=0|Y=1,S=0)|+0.5*|\operatorname{Pr}(\hat{Y}=1|Y=0,S=1)-\operatorname{Pr}(\hat{Y}=1|Y=0,S=0)|$, where $\hat{Y}\in\{0,1\}$ denotes the predicted label, $Y\in\{0,1\}$ denotes a binary class label, and $S\in\{0,1\}$ denotes a binary sensitive attribute. Blum & Stangl [12] suggest that equal opportunity and equalized odds constraints on biased data can be more beneficial than demographic parity constraints. Therefore, we use the equalized odds constraint for ‘jiang_nachum’ and ‘exp_grad’. Different classifiers have been proposed for different fairness metrics and definitions. However, we often compare different methods with one or two standard metrics in practice. Since most of the methods used in this paper were trained for Equalized Odds, we report Equalized Odds results in the main text and report average Equal Opportunity Difference (EOD) and Statistical Parity Difference (SPD) results over under-representation bias settings to report a complete picture of the effect on data bias on fairness and error rate. EOD is defined as the True Positive Rate difference: $|\operatorname{Pr}(\hat{Y}=1|Y=1,S=1)-\operatorname{Pr}(\hat{Y}=1|Y=1,S=0)|$, while SPD is defined as $|\operatorname{Pr}(\hat{Y}=1|S=1)-\operatorname{Pr}(\hat{Y}=1|S=0)|$. For both metrics, values closer to $0$ mean better fairness, whereas values closer to $1$ indicate extreme parity difference between the two sensitive groups $S=0$ and $S=1$.

We train on multiple datasets to perform our analysis. We consider four standard real-world datasets from fair classification literature: the Adult Income dataset [23], the Bank Marketing dataset [46], the COMPAS dataset [3], and the German Credit dataset [23]. We also consider a synthetic dataset setup from Zeng et al. [65], which consists of a binary label $y\in\{0,1\}$ and a binary sensitive attribute setting $s\in\{0,1\}$. We take the original train-test split for a given dataset, or in its absence, a 70%-30% stratified split on the subgroups. Appendix B gives specific details for all datasets.

To perform our analysis, we use the data bias model from Blum & Stangl [12] and inject varying amounts of under-representation and label bias into the original training data before giving it to a classifier. We summarize our experimental setup in Algorithm $1$, presented in Appendix C. Let $Y\in\{0,1\}$ represent the label, and $S\in\{0,1\}$ represent the sensitive attribute. Let $\beta_{pos}$ be the probability of retaining samples from the subgroup defined by the favorable label $Y=1$ and underprivileged group $S=0$ ($10$-subgroup). Similarly, let $\beta_{neg}$ be the probability of retaining samples from the unfavorable label $Y=0$ and underprivileged $S=0$ group ($00$-subgroup). We inject under-representation bias into the training data by retaining samples from the $10$-subgroup and the $00$-subgroup with probability $\beta_{pos}$ and $\beta_{neg}$, respectively. We consider ten different values each for $\beta_{pos}$ and $\beta_{neg}$ varying over $\{0.1,0.2,.....,1.0\}$. This results in $100$ different settings ($10~{}\beta_{pos}$ factors x $10~{}\beta_{neg}$ factors) for training data bias. We separately inject ten different levels of label bias by flipping the favorable label of the underprivileged group to an unfavorable one ($10\rightarrow 00$) with a probability $\nu$ varying over $\{0.0,0.1,........,0.9\}$. The test set for all settings is the original split test set, either provided at the original dataset source or taken out separately beforehand. This results in $110$ different datasets corresponding to different data bias settings, and consequently $110$ different results each for a particular fair classifier. Finally, the training of all classifiers and the procedures to create biased training data using $\beta_{pos}$, $\beta_{neg}$, and $\nu$ is performed $5$ times to account for randomness in sampling and optimization procedures.

We also note that some fair classifiers in our list, like ‘exp_grad’ and ‘cal_eq’, are randomized classifiers. Therefore, repeating the entire pipeline multiple times and reporting means and standard deviations normalizes the random behavior of these methods to some extent, apart from the measures taken during the implementation of these methods in widely used toolkits like the one we are using, AIF360 [7]. The code to train and reproduce the results is available here ²²2https://github.com/mohitsharma29/comparision_data_bias_fairness.

In this section, we present our experimental results about the stability of fairness and accuracy guarantees of various fair classifiers after injecting under-representation and label bias in their training data. Stability means whether the error rate and unfairness show high variance/spread in response to increasing or decreasing under-representation or label bias. A stable behavior is indicated by small changes in error rate and unfairness, as $\beta_{pos},\beta_{neg}\text{ or }\nu$ change.

We first plot the unfairness (Equalized Odds) and error rates of various fair classifiers on the original training splits. Figure 1 shows the results of various fair classifiers with a logistic regression model on original dataset splits without any explicitly injected under-representation or label bias. While most fair classifiers exhibit an expected behavior of trading off some accuracy to minimize unfairness, some classifiers perform worse in mitigating unfairness than the unfair LR model. We attribute this reason to the strong effects of different kinds of data preprocessing on the performance of fair classifiers, as noted in previous works [29, 11]. Other results with SVM as a base classifier, Equal Opportunity Difference, and Statistical Parity Difference are given in Appendix D.

We can also inspect Figure 1 and other figures later in the paper by looking at the four quadrants formed by the blue horizontal and vertical lines corresponding to the unfairness and error rate of the base LR model. Following a Cartesian coordinate system, most fair classifiers, when trained on the original training set without explicit under-representation or label bias, lie in the fourth quadrant, as they trade-off some accuracy to mitigate unfairness. Any method lying in the first quadrant is poor, as it is worse than the base LR model in terms of both error rate and unfairness. Finally, any method lying in the third quadrant is excellent as it improves over both error rate and unfairness compared to the LR model.

Figure 1 shows how various fair classifiers perform without explicit under-representation and label bias. In this figure, we try to summarize all the available pieces of results for a given dataset in various ways to get a holistic picture. We show the stability of error rates and Equalized Odds unfairness and look at the average stability of the other two metrics: Equal Opportunity and Statistical Parity Difference, since many times in practice, we care about the performance of methods against a fixed set of metrics. We now look at the individual effects of both under-representation factors ($\beta_{pos}$, $\beta_{neg}$ and label bias ($\nu$)), with the LR model as a reference, denoted by the vertical and the horizontal error rate and Equal Opportunity Difference (EOD) lines respectively. Figure 2(a) shows those results for the Compas dataset. The results for the Adult, Synthetic, Bank Marketing, and Credit datasets are given in Appendix E. We first observe that many fair classifiers exhibit a large variance in the unfairness-error rate plane and often become unfair as the bias factors increase. This is also indicated by how the fair classifiers jump from quadrant four to quadrant one, as $\beta_{pos}$, $\beta_{neg}$, or $\nu$ increase. In fact, for the ‘prej_remover’ classifier, the results lie outside the plot limits, indicating its instability. Even when its average SPD is measured in Figure 2(c) (‘prej_remover’ is implicitly optimized for SPD), we observe that it does not perform well.

However, ‘rew’, ‘jiang_nachum’, and ‘exp_grad’ fair classifiers remain stable across all three kinds of biases for both error rate and EOD and show a minimal spread across both the error rate and unfairness (EOD) axes. They tend to remain in the same quadrant with varying bias factors. This remains true on other datasets as well (as shown in Appendix E). Furthermore, it’s interesting to note that a simple logistic regression model tends to stay relatively stable in the presence of under-representation and label biases, and on many occasions, more fair than many fair classifiers in our analysis.

We can also look at the combined effect of $\beta_{pos}$ and $\beta_{neg}$ in Figure 2(b) with the help of heatmaps. Each cell in the heatmap denotes a possible $\beta_{pos},\beta_{neg}$ setting, thus covering all the $100$ possible settings. Darker color values in cells denote lower error rates and unfairness. For this analysis, to examine stability with heatmaps, we look for uniformity of colors in the entire grid, which means that the error rate and unfairness values do not change with different $\beta_{pos},\beta_{neg}$ values.

Figure 2(b) again confirm the stability of ‘rew’, ‘jiang_nachum’ and ‘exp_grad’ fair classifiers even when they are trained on datasets with combined $\beta_{pos}$ and $\beta_{neg}$ biases. However, ‘jiang_nachum’ and ‘exp_grad’ emerge as stable and better classifiers in terms of their unfairness values than the ‘rew’ classifier because they have darker uniform colors across their entire error rate and unfairness grid. The ‘eq’ classifier also remains stable for unfairness, except on the Credit dataset (shown in Appendix E). Amongst the classifiers optimized for Equalized Odds (‘jiang_nachum’, ‘exp_grad’, ‘eq’ and ‘cal_eq’), ‘jiang_nachum’ and ‘exp_grad’ emerge out to be most stable ones in terms of error rate and unfairness.

To quantitatively summarize the findings from the heatmap plots and to show this statistic with other choices of base classifiers (LR and SVM) and unfairness metrics (SPD and EOD), we look at the average performance across $100$ different settings of $\beta_{pos}$ and $\beta_{neg}$ in Figure 2(c) for the Compas dataset. An indication of stability with these results is whether the mean error rate and unfairness across $100$ different under-representation settings stay low and whether this mean performance has a low standard deviation. Because we run the whole experimental setup $5$ times with different random seeds for reproducibility, the values represent the mean value of the mean across $5$ runs for each $\beta_{pos},\beta_{neg}$ setting, and its standard deviation over the $100$ settings. We embolden the top-$3$ mean and top-$3$ standard deviation values for each column, where the top-$3$ means the lowest $3$ error rate and the unfairness metrics (SPD and EOD).

For all datasets, the observations across all emboldened values corroborate our findings from the heatmap plots. For both unfairness metrics (SPD and EOD), ‘exp_grad’ and ‘jiang_nachum’ often appear in the top-$3$ mean performance across all settings, whereas ‘rew’, ‘jiang_nachum’ and ‘exp_grad’ often appear in the top-$3$ smallest standard deviation values, indicating their stability. Furthermore, ‘rew’ and ‘jiang_nachum’ often appear in the top-$3$ mean and standard deviation list for error rate. ‘rew’ often appears in the top-$3$ mean performance for Statistical Parity Difference (SPD). As expected, the unfair base classifier often appears in the top-$3$ mean error rate list since its objective is always to minimize error rates without regard for unfairness. Finally, when looking over all the datasets’ results, we find that ‘exp_grad’ and ‘jiang_nachum’ often emerge to be the most stable classifiers for other metrics like EOD and SPD, despite not being trained to minimize those metrics, further solidifying the evidence for their stable behavior.

Another interesting observation across the results of all datasets is that the deterministic version of the ‘exp_grad’ fair algorithm, the ’grid_search’ classifier [1], is very unstable compared to the ’exp_grad’ classifier, which is the most stable amongst the lot, indicating that in the face of data bias, randomness might help. Other results, such as Figure 2 (a,b) with SPD as the unfairness metric, cannot be included in the manuscript.

Finally, it is worth noting that the stability of ‘rew’ and ‘exp_grad’ has also been reported in another benchmarking study about testing various fair classifiers in the presence of varying sensitive attribute noise [30]. This indicates towards a potentially stable performance from reweighing-based approaches in the presence of varying kinds of data biases and noises in the training data.

The following theorems give a theoretical justification for the stability of test accuracy when the reweighing classifier [37] is trained even on data injected with extreme under-representation such as $\beta_{\text{pos}}\rightarrow 0$ or $\beta_{\text{neg}}\rightarrow 0$.

The proof for this theorem is given in Appendix F. The proof involves bounding subgroup and group/label probabilities using $\alpha$, and with a reasonable assumption that $X^{\prime}|Y^{\prime}=y,S^{\prime}=s$ in $D_{\beta}\sim X|Y=y,S=s$ in $D$. The main take-away from Theorem 1 is the following: when we reweigh a given non-negative loss function using the Reweighing scheme from Kamiran & Calders [37] on samples from $D_{\beta}$, we always lie in some constant ‘radius’ of the expected loss on the true distribution $D$ for any classifier $f$. This does not tell us anything about the expected loss on $D$ itself, but rather, about the ability of the reweighed loss $L^{\prime}$ to recover from any arbitrary under-representation bias $\beta$. However, the constant factors depend on $\alpha$, the worst case imbalance between the subgroups for a given distribution.

Using Theorem 1, we prove the following bound on the expected loss of the reweighing classifier that is obtained by empirical minimization of the reweighed loss on the biased distribution $D_{\beta}$ but tested on the original distribution $D$.

The proof for this theorem is given in Appendix F. It uses a generalized version of the Hoeffding’s inequality [34], proof of Theorem $2$ of Zhu et al. [68] and Theorem  1. Note that Theorem 2 implies that given any $\epsilon>0$ and $\delta>0$, we can get the guarantee

with probability at least $1-\delta$, by simply choosing the number of samples $N$ from $D_{\beta}$ in the empirical reweighed loss minimization to be large enough so that

It is important to note that Theorems 1 and 2 hold for data without any injected bias, i.e., $\beta_{\text{pos}}=\beta_{\text{neg}}=1$, where such theoretical guarantees for the reweighing classifier [37] were not known earlier.

In this section, we discuss where our results stand in the context of related works on distribution shifts and data bias in real-world AI/ML models and make a few practical recommendations based on our work.

We show that many state-of-the-art fair classifiers exhibit a large variance in their performance when their training data is injected with under-representation and label bias. We propose an experimental setup to compare different fair classifiers under data bias. We show how the Bayes Optimal unfair and fair classifiers react with under-representation bias and propose a reweighing scheme to recover the true fair Bayes Optimal classifiers while optimizing over biased distributions with a known bias. We give a theoretical bound on the accuracy of the reweighing classifier [37] that holds even under extreme data bias. Finally, we discuss our work in the broader context of data bias literature and make practical recommendations.

A limitation of our work is that we use a simple model for injecting under-representation and label bias, with which one cannot hope to address the root cause behind different kinds of biases in real-world data. Thinking about the causal origin of data bias instead may allow one to model more types of biases and obtain better fixes for them under reasonable assumptions. Theoretically explaining the stability of the ‘exp_grad’ [1], and ‘jiang_nachum’ [36] fair classifiers with under-representation bias is also an interesting future work.