How Robust is Your Fairness?
Evaluating and Sustaining Fairness under Unseen Distribution Shifts

Machine learning fairness can be generally categorized into individual fairness and group fairness (Du et al., 2020). Individual fairness requires similar inputs to have similar predictions (Dwork et al., 2012). Compared with individual fairness, group fairness is a more popular setting and thus the focus of our paper. Given input data $X\in{\mathbb{R}}^{n}$ with sensitive attributes $A\in\{0,1\}$ and their corresponding ground truth labels $Y\in\{0,1\}$, group fairness requires a learned binary classifier $f(\cdot;\theta):{\mathbb{R}}^{n}\rightarrow\{0,1\}$ parameterized by $\theta$ to give equally accurate predictions (denoted as $\hat{Y}\coloneqq f(X)$) on the two groups with $A=0$ and $A=1$. Multiple fairness criteria have been defined in this context. Demographic parity (DP) (Edwards & Storkey, 2016) requires identical ratio of positive predictions between two groups: $P(\hat{Y}=1|A=0)=P(\hat{Y}=1|A=1)$. Equalized Odds (EO) (Hardt et al., 2016) requires identical false positive rates (FPRs) and false negative rates (FNRs) between the two groups: $P(\hat{Y}\neq Y|A=0,Y=y)=P(\hat{Y}\neq Y|A=1,Y=y),\forall y\in\{0,1\}$. Based on these fairness criteria, quantified metrics are defined to measure fairness. For example, the EO distances (Madras et al., 2018) is defined as follows:

Many methods have been proposed to mitigate model bias. Data pre-processing methods such as re-weighting (Kamiran & Calders, 2012) and data-transformation (Calmon et al., 2017) have been used to reduce discrimination before model training. In contrast, (Hardt et al., 2016) and (Zhao et al., 2017) propose post-processing methods to calibrate model predictions towards a desired fair distribution after model training. Instead of pre- or post-processing, researchers have explored to enhance fairness during training. For example, (Madras et al., 2018) uses a adversarial training technique and shows the learned fair representations can transfer to unseen target tasks. The key technique, adversarial training (Edwards & Storkey, 2016), was designed for feature disentanglement on hidden representations such that sensitive (Edwards & Storkey, 2016) or domain-specific information (Ganin et al., 2016) will be removed while keeping other useful information for the target task. The hidden representations are typically the output of intermediate layers of neural networks (Ganin et al., 2016; Edwards & Storkey, 2016; Madras et al., 2018). Instead, methods, like adversarial debiasing (Zhang et al., 2018) and its simplified version (Wadsworth et al., 2018), directly apply the adversary on the output layer of the classifier, which also promotes the model fairness. Observing the unfairness due to ignoring the worst learning risk of specific samples, Hashimoto et al. (2018) proposes to use distributionally robust optimization which provably bounds the worst-case risk over groups. Creager et al. (2019) proposes a flexible fair representation learning framework based on VAE (Kingma & Welling, 2013), that can be easily adapted for different sensitive attribute settings during run-time. Sarhan et al. (2020) uses orthogonality constraints as a proxy for independence to disentangles the utility and sensitive representations. Martinez et al. (2020) formulates group fairness with multiple sensitive attributes as a multi-objective learning problem and proposes a simple optimization algorithm to find the Pareto optimality. Another line of research focuses on learning unbiased representations from biased ones (Bahng et al., 2020; Nam et al., 2020). Bahng et al. (2020) proposes a novel framework to learn unbiased representations by explicitly enforcing them to be different from a set of pre-defined biased representations. Nam et al. (2020) observes that data bias can be either benign or malicious, and removing malicious bias along can achieve fairness. Li & Vasconcelos (2019) jointly learns a data re-sampling weight distribution that penalizes easy samples and network parameters. Li et al. (2019) scaled by higher-order power to re-emphasize the loss of minority samples (or nodes) in distributed learning. Agarwal et al. (2018) formulates a fairness-constrained optimization to train a randomized classifier which is provably accurate and fair. Quadrianto et al. (2019) casts the sensitive information removal problem as a data-to-data translation problem with unknown target domain.

When many fairness metrics and debiasing algorithms are designed for general learning problems as aforementioned, there are a line of research and applications focusing on fairness-encouraged computer vision tasks. For instance, Buolamwini & Gebru (2018) shows current commercial gender-recognition systems have substantial accuracy disparities among groups with different genders and skin colors. Wilson et al. (2019) observe that state-of-the-art segmentation models achieve better performance on pedestrians with lighter skin colors. In (Shankar et al., 2017; de Vries et al., 2019), it is found that the common geographical bias in public image databases can lead to strong performance disparities among images from locales with different income levels. Nagpal et al. (2019) reveal that the focus region of face-classification models depends on people’s ages or races, which may explain the source of age- and race-biases of classifiers. On the awareness of the unfairness, many efforts have been devoted to mitigate such biases in computer vision tasks. Wang et al. (2019) shows the effectiveness of adversarial debiasing technique (Zhang et al., 2018) in fair image classification and activity recognition tasks. Beyond the supervised learning, FairFaceGAN (Hwang et al., 2020) is proposed to prevent undesired sensitive feature translation during image editing. Similar ideas have also been successfully applied to visual question answering (Park et al., 2020).

Recently, several papers have investigated the fairness learning problem under distributional shift (Mandal et al., 2020; Zhang et al., 2021; Rezaei et al., 2021; Singh et al., 2021; Dai & Brown, 2020). Although these works are relevant with ours, there are significant differences in the problem settings. Zhang et al. (2021) studied the problem of enforcing fairness in online learning, where the training distribution constantly shifts. The authors proposed to adapt the model to be fair on the current known data distribution. In contrast, our work aims to generalize fairness learned on current distribution to unknown and unseen target distributions. In our setting, the algorithm can not access any training data from the unknown target distributions. Rezaei et al. (2021) studied to preserve fairness under covariate shift. However, their method requires unlabeled data from the target distribution. In other words, they assume the target distribution to be known. In contrast, our method is more general and works on unknown target distributions. Singh et al. (2021) also studied to preserve fairness under covariate shift. However, their method is based on model adaptation and requires the existence of a joint causal graph to represent the data distribution for all domains. Our method, however, does not require such requirement and generally works on any unseen target distributions. Dai & Brown (2020) studies fairness under label distributional shift, while we focus on covariate shift.

It is inefficient to directly optimize the loss curvature through Eq. (2), since it involves high order gradients.¹¹1The Hessian matrix itself involves second order gradients, and backpropagation through Eq. (2) requires even higher order gradient on top of the Hessian matrix SN. To solve this problem, we use a one-shot power iteration method (PIM) for practical approximation of ${\mathcal{C}}(x)$ during training. First we rewrite ${\mathcal{C}}(x)$ with the following form: ${\mathcal{C}}(x)=\sigma(H(x))=\|H(x)v\|$, where $v$ is the dominant eigenvector with the maximal eigenvalue, which can be calculated by power iteration method. In practice, we estimate the dominant eigenvector $v$ by the gradient direction: $\tilde{v}\coloneqq\frac{\text{sign}(g)}{\|\text{sign}(g)\|}\approx v$, where $g=\nabla_{\theta}{\mathcal{L}}(x)$. This is because previous works have observed a large similarity between the dominant eigenvector and the gradient direction (Miyato et al., 2018; Moosavi-Dezfooli et al., 2019). We further approximate Hessian matrix by finite differentiation on gradients: $H(x)v\approx\frac{\nabla_{\theta}{\mathcal{L}}(x+hv)-\nabla_{\theta}{\mathcal{L}}(x)}{h}$ where $h$ is a small constant. As a result, the final approximation of curvature smoothness is

Experiments are conducted on three datasets widely used to evaluate machine learning fairness: Adult (Kohavi, 1996), and CelebA (Liu et al., 2015), and Communities and Crime (C&C) (Redmond & Baveja, 2002).³³3Traditional image classification datasets (e.g., ImageNet) are not directly applicable since they lack fairness attribute labels. Adult dataset has 48,842 samples with basic personal information such as education and occupation, where 30,000 are used for training and the rest for evaluation. The target task is to predict the person’s annual income, and we use “gender” (male or female) as the sensitive attribute. The features in Adult dataset are of either continuous (e.g., age) or categorical (e.g. sex) values. We use one-hot encoding on the categorical features and then concatenate them with the continuous ones. We use data whitening on the concatenated features. CelebA has over 200,000 images of celebrity faces, with 40 attribute annotations. The target task is to predict gender (male or female) and the sensitive attributes to protect are “chubby” and “eyeglasses”. We randomly select $10,000$ as training samples and $1,000$ as testing samples. All images are center-cropped and resized to $64\times 64$, and pixel values are scaled to $[0,1]$. C&C dataset has 1,994 samples with neighborhood population statistics, where 1,500 are used for training and the rest for evaluation. The target task is to predict violent crime per capita, and we use “RacePctBlack” (percentage of black population in the neighborhood) and “FemalePctDiv” (divorce ratio of female in the neighborhood) as sensitive attributes. All features in C&C dataset are of continous values in $[0,1]$. To fit in the fairness problem setting, we binarilize the target and sensitive attributes with the top-$30\%$ largest value as the threshold (As a result $\text{P}[A=0]=30\%$ and $\text{P}[Y=0]=30\%$). We also do data-whitening on C&C.

For C&C and Adult datasets, we use two-layer MLPs for $f_{s}$, $h_{t}$ and $h_{a}$. Specifically, suppose the input feature dimension is $d$, then the dimensions of hidden layers in $f_{s}$ and $h_{t}$ are $d\rightarrow 100\rightarrow 64$ and $64\rightarrow 32\rightarrow 2$, respectively. $h_{a}$ has identical model structure with $h_{t}$. For all three sub-networks, ReLU activation function and dropout layer with $0.25$ dropout ratio are applied between the two fully connected layers. For CelebA dataset, we use ResNet18 as backbone, where the first three stages are used as $f_{s}$ and the last stage (together with the fully connected classification layer) is used as $h_{t}$. The auxiliary adversarial head $h_{a}$ has the same structure as $h_{t}$.

We compare CUMA with the following state-of-the-art in-distribution fairness algorithms. Adversarial debiasing (AdvDebias) (Zhang et al., 2018) is one of the most popular fair training algorithm based on adversarial training (Ganin et al., 2016). Madras et al. (2018) proposes a similar framework termed Learned Adversarially Fair and Transferable Representations (LAFTR), by replacing the cross-entropy loss used in (Zhang et al., 2018) with a group-normalized $\ell_{1}$ loss, which is shown to work better on highly unbalanced datasets. We also include normal (fairness-ignorant) training as a baseline.

We report the overall accuracy on all test samples in the original test sets. To measure in-distribution fairness, we use $\Delta_{EO}$ on the original test sets. To measure robust fairness under distribution shifts, we use $\Delta_{EO}$ on test sets with distribution shifts. See the following paragraph for the details in constructing distribution shifts.

Adult dataset contains US Census data collected before 1996. We use the 2014 and 2015 subset of Folktables dataset (Ding et al., 2021), which contain US Census data collected in 2014 and 2015 respectively, as the test sets with distribution shifts. This simulates the real-world temporal distributional shifts.

On CelebA dataset, we train the model on $10,000$ “young” face images. We use another $1,000$ “young” face images as in-distribution test set and $1,000$ “not young” face images as the test set with distribution shifts. This simulates the real-world scenario when the model is trained and used on people with different age groups. We also construct another test set with a different type of distribution shift, by applying strong JPEG compression on the original $1,000$ “young” test images, following (Hendrycks & Dietterich, 2019). This simulates the scenario when the model is trained on good quality images while the test images has poor visual quality.

For C&C dataset, we construct two artificial distribution shifts by adding random Gaussian and uniform noises, respectively, to the test data. Specifically, the categorical features in C&C dataset are first one-hot encoded and then whitened into float-value vectors, where noises are added. Both types of noises have mean $\mu=0$ and has standard derivation $\sigma=0.03$ .

Unless further specified, we set the loss trade-off parameter $\alpha$ to 1 in all experiments by default. We use Adam optimizer (Kingma & Ba, 2014) with initial learning rate $10^{-3}$ and weight decay $10^{-5}$. The learning rate is gradually decreased to 0 by cosine annealing learning rate scheduler (Loshchilov & Hutter, 2016). On both Adult and C&C datasets, we train for $50$ epochs from scratch for all methods. On CelebA dataser, we first normally train a model for 100 epochs, and then finetune it for 20 epochs using CUMA. For fair comparison, we train for 120 epochs on CelebA for all baseline methods. The constant $h$ in Eq. (3) is set to $1$ by default.