Fairness Testing of Deep Image Classification with Adequacy Metrics

In this work, we focus on the fairness testing of deep learning models, specifically, deep neural networks (DNNs) for image classification applications. A deep neural network $F$ consists of multiple hidden layers between an input and an output layer. It can be denoted as a tuple $M=(I,L,\Phi,TS)$ where

• •

  $I$ is the input layer;

• •

  $L=\{L_{j}|j\in\{1,\dots,J\}\}$ is a set of hidden layers and the output layer, the number of neurons in the $j$-th layer is $|L_{j}|$, and the $k$-th neuron in layer $L_{j}$ is denoted as $n_{j,k}$ and its output value with respect to the input x is $v_{j,k}(x)$;

• •

  $\Phi$ is a set of activation functions, e.g., Sigmoid, Hyperbolic Tangent (TanH), or Rectified Linear Unit (ReLU);

• •

  $TS$: $L\times\Phi\to L$ is a set of transitions between layers. As shown in Equation 1, the neuron activation value, $v_{j,k}(x)$, is computed by applying the activation function to the weighted sum of the activation value of all the neurons within its previous layer, and the weights represent the strength of the connections between two linked neurons.

  (1)

  $$v_{j,k}(x)=\phi(\sum_{l=1}^{|L_{j-1}|}\omega_{j-1,k,l}\cdot v_{j-1,l}(x))$$

A classification DNN can be defined as $M:X\to Y$ which transforms a given input $x\in X$ to an output label $y\in Y$ by propagating layer by layer as above.

We denote the fairness sensitive attribute of interest as $SA$ (e.g., race). Note that for image classification, $SA$ is hidden from $X$. We define $HF:X\to SA$ as a function which returns the sensitive attribute of a given sample $x\in X$. We further define $X_{A}\subset X$ as the samples satisfying $HF(x)=A$ where $x\in X,A\in SA$. To change the sensitive attribute for a sample $x$, we define a transformation function $T_{A\to B}:X\to X$ which transforms a sample from $X_{A}$ to $X_{B}$ while preserving other information. Then, we define individual fairness of an image classification model $M$ as follows (Thomas et al., 2019).

A variety of robustness testing criteria for DNN has been proposed (Pei et al., 2017; Ma et al., 2018a; Kim et al., 2019; Wang et al., 2021; Feng et al., 2020; Gerasimou et al., 2020). We briefly introduce the following representative robustness testing metrics. Readers are referred to (Pei et al., 2017; Ma et al., 2018a; Kim et al., 2019; Wang et al., 2021; Feng et al., 2020; Gerasimou et al., 2020) for details.

Different from the previous robustness testing works, we aim to propose a set of testing adequacy metrics for individual fairness specially designed for image classification. In particular, we aim to achieve the following research objectives:

• •

  How can we design testing adequacy metrics which are well correlated with the model’s fairness?

• •

  How can we select test cases which can effectively fix the model’s fairness issues?

The first question is how to realize the domain transformation function $T_{A\to B}$. Note that this is straightforward for structured or text data which can be done by replacing the protected feature or token with a value from a predefined domain (Zhang et al., 2020, 2021). However, for image data, the sensitive attribute of interest is hidden from the input feature space. We thus follow  (Yucer et al., 2020) and adopt CycleGAN (Zhu et al., 2017) to transform images across different protected domains as follows.

As shown in Figure 2, CycleGAN provides a mechanism to transfer from $A$ to $B$ domains and $B$ to $A$ domains respectively with two corresponding generative models $T_{A\to B},T_{B\to A}$. Similar with traditional GAN (Goodfellow et al., 2014), it contains two discriminators $D_{A}$ and $D_{B}$ to distinguish whether the input is ‘real’, i.e., a sample is generated by generative model or from the original dataset.

The loss function consists of three parts, the first one is adversarial loss, which is defined as follows,

(2)

$$\begin{split}L_{GAN}(T_{A\to B},D_{B},A,B)&=\mathbb{E}[\log(1-D_{B}(T_{A\to B}(X_{A})))]\\ &+\mathbb{E}[\log D_{B}(X_{B})]\end{split}$$

The transformer $T_{A\to B}$ aims to synthesize a picture satisfying the distribution of $X_{B}$ based on the seed from $X_{A}$, while the goal of discriminator $D_{B}$ is to distinguish the raw images $X_{B}$ from the artificial ones $T_{A\to B}(X_{A})$. $T_{B\to A}$ and $D_{A}$ have the same definition of adversarial loss. Since domain transformation needs to modify other information as little as possible in the process of changing the sensitive attribute, the second part of the objective function is thus to ensure that the generated image is identical with the raw one, which is defined as follows:

(3)

$$\begin{split}L_{IDE}(T_{A\to B})&=\mathbb{E}[\|T_{A\to B}(X_{A})-X_{A}\|_{p}]\end{split}$$

In addition, it also needs to ensure that $T_{A\to B}(X_{A})$ is in the data distribution of $X_{B}$. To this end, CycleGAN introduces cycle consistency loss as the core of its joint optimization objective to make the synthesized image more realistic. It is defined as follows.

(4)

$$\begin{split}L_{CYC}(T_{A\to B},T_{B\to A})&=\mathbb{E}[\|T_{B\to A}(T_{A\to B}(X_{A}))\|_{p}]\\ &+\mathbb{E}[\|T_{A\to B}(T_{B\to A}(X_{B}))\|_{p}]\end{split}$$

The intuition is that the pair of well-trained generators can recover the original image through the reconstruction process, i.e., for a given $x\in X_{A}$, $T_{B\to A}(T_{A\to B}(x))=x$. Overall, the complete loss function of CycleGAN is defined as follows.

(5)

$$\begin{split}L&=L_{GAN}(T_{A\to B},D_{B},A,B)+L_{GAN}(T_{B\to A},D_{A},B,A)\\ &+\gamma(L_{IDE}(T_{A\to B})+L_{IDE}(T_{B\to A}))\\ &+\eta L_{CYC}(T_{A\to B},T_{B\to A})\end{split}$$

where $\gamma$ and $\eta$ are hyperparameters to balance among these three losses. In the “Raw Data” frame of Figure 1, we show an example race transformer (Caucasian to African) from VGGFace.

The next step is to select the fairness-related neurons, i.e., those neurons in the DNN which may have a significant impact on the fairness of the model’s decisions. The benefit is that, rather than blindly selecting neurons in certain layers (Ma et al., 2018a) or only the model’s output (Feng et al., 2020), it enables us to test the model’s fairness in a more fine-grained and focused way as these neurons are more correlated with the model’s fairness.

Our key idea of neuron selection is by significance testing (Kruskal and Wallis, 1952) of whether a neuron is fairness-related as follows. Formally, we make the following null hypothesis on a given neuron $n_{j,k}$.

(6)

$$\begin{split}&H_{0}=n_{j,k}\ \text{is}\ \text{not}\ \text{fairness-related}\\ \end{split}$$

In addition, we use a standard parameter (a.k.a., significance level), $\alpha$, to control the probability of making errors, e.g., rejecting $H_{0}$ when $H_{0}$ is true. We then use Kruskal-Wallis test (Kruskal and Wallis, 1952) (also know as H-test) to test the above hypothesis. The intuition is that we could identify a fairness-related neuron by looking at the difference between the activation distribution on those fair samples and unfair samples. The larger the difference, the more fairness-related is the neuron. Specifically, given the training dataset $X$ and $T$, we could collect the activation value difference on neuron $n_{j,k}$ over $X$:

(7)

$$XD=\{\|v_{j,k}(x)-v_{j,k}(x^{\prime})\||\forall x\in X,x^{\prime}=T(x)\}.$$

We further divide $XD$ into two orthogonal subsets $XD_{f},f=0,1$, according to the prediction results $M(x)$ and $M(x^{\prime})$ are equal or not, respectively.

We first sort $XD$ in ascending order and denote the rank of i-th element in $XD_{f}$ as $r_{f}^{i}$. Then we add the ranks in each subset $XD_{f}$ to obtain the rank sum, denoted as $R_{f}=\sum_{i}r_{f}^{i}$. When the original hypothesis $H_{0}$ is true, the average rank of each subset should be close to that of all samples, i.e., $(|XD|+1)/2$, and we thus use the following equation to compute the rank variance between subsets:

(8)

$$RVS=\sum_{f=0,1}|XD_{f}|(\frac{R_{f}}{|XD_{f}|}-\frac{|XD|+1}{2})^{2}$$

In order to eliminate the influence of dimension, we then calculate the average of rank variance of all samples, which is defined as follows.

(9)

$$\begin{split}ARV&=\frac{1}{|XD|-1}\sum_{f=0,1}\sum_{i=1}^{|XD_{f}|}({r_{f}^{i}}-\frac{|XD|+1}{2})^{2}\\ &=\frac{|XD|(|XD|+1)}{12}\end{split}$$

Note that the freedom degree of sample variance is $|XD|-1$. Thus, the Kruskal-Wallis rank-sum statistic H is given by,

(10)

$$H=\frac{RVS}{ARV}=\frac{12}{|XD|(|XD|+1)}\sum_{f=0,1}\frac{R_{f}^{2}}{|XD_{f}|}-3(|XD|+1)$$

When $|XD|$ is large, $H$ approximately obeys a chi-square distribution (Pearson, 1900) with the freedom degree of $1$, $H\sim\mathcal{X}^{2}(1)$. Therefore, the critical value of Kruskal-Wallis testing, $H_{c}$, corresponding to $\alpha$ is determined according to the chi-square distribution table. That is, if the computed statistic $H>H_{c}$, we reject $H_{0}$ and conclude that the neuron $n_{j,k}$ is fairness-related.

Next, we propose a set of testing metrics to measure the adequacy of DNN fairness. Note that different from the robustness testing metrics (Ma et al., 2018a; Gerasimou et al., 2020; Wang et al., 2021; Feng et al., 2020; Kim et al., 2019), fairness testing metrics are based on the behavioral differences between an instance pair, i.e., $x$ and $x^{\prime}$ (which only differ in certain sensitive attributes). Note that one important desirable property on the metrics is that they should be well correlated with the model’s fairness.

Our fairness metrics will satisfy the above property as we define our metrics on the basis of selected fairness-related neurons. Specifically, let $NF_{j}\subset\{n_{j,1},\dots,n_{j,|L_{j}|}\}$ be a set of fairness-related neurons within layer $L_{j}$. We denote the activation value vector and boolean activation patterns over neurons in $NF_{j}$ with respect to the input $x$ as $\vec{v}(x,NF_{j})$ and $\vec{a}(x,NF_{j})$, respectively. Note that in $\vec{a}(x,NF_{j})$, $1$ and $0$ represent whether the value of $n_{j,k}\in NF_{j}$ before and after ReLU is the same or not, respectively. We first characterize the model differences between the inputs $x$ and $x^{\prime}$ at the layer level as the decision is propagated layer by layer.

The above layer-level metrics are based on the statistical results on a set of neurons in a layer. In addition, we also provide a more fine-grained metric based on a single neuron.

Finally, we could define the coverage for each testing metric by dividing its value range (based on $X$ and the domain transformer $T$) into $Z$ equal bins, denoted as $B^{z}$, and then compute the coverage ratio as follows:

(15)

$$\frac{\sum_{i=1}^{|C|}|\{B_{i}^{z}|\exists x\in X:C_{i}(x,T(x))\in B_{i}^{z}\}|}{Z*|C|}$$

where $|C|$ is the dimension for collecting the metric value. For layer-level similarity, we traverse layer by layer to calculate its value, thus $|C|=|J|$, and the value range is $[0,1]$ for Tanimoto and Cosine coefficient and $[-1,1]$ for Spearman correlation. For neuron distance, we select $topK$ neurons each layer to analyse, thus $|C|=|J|*topK$, and the upper bound is computed by the training data (Ma et al., 2018a).

The above metrics enables us to evaluate a model’s fairness adequacy. The follow-up questions are 1) how to generate diverse test cases to improve the fairness adequacy, and then 2) how to select the most valuable test cases to enhance the model’s fairness by augmented training, which has been proved to be useful in enhancing the robustness or fairness of DNN (Ma et al., 2018a; Zhang et al., 2020, 2021).

We address the first question by introducing the following test case generation methods, both from the spirit of fairness testing literature and traditional digital image processing approaches. Given a fair sample pair $(x,x^{\prime})$, where $x$ and $x^{\prime}$ are only different in certain sensitive attributes, we aim to maximize the output differences between them after applying perturbation $p$ on them, which is formally defined as follows:

(16)

$$\text{argmax}_{p}\{M(x+p)-M(x^{\prime}+p)\}$$

We introduce the following perturbation methods for approximating the above goal.

After generating test cases to increase the testing adequacy, i.e., the coverage of metrics, we then address the second question which selects a subset of test cases for augmented training to improve the model’s fairness more efficiently. In particular, we prefer to select a set of test cases with diverse metric values inspired by (Wang et al., 2021). Note that the neuron distance for each sample is a vector, of which each value is collected from each neuron independently which is difficult to quantify accurately. We thus only utilize the layer-level metrics, i.e., Tanimoto coefficient, cosine similarity, and Spearman correlation, for the selection.

Specifically, we adopt the K-Multisection Strategy (KM-ST) in our work, which uniformly sample the value space of a given metric. Formally, let $C_{min}$ and $C_{max}$ be the minimum and maximum value of crition $C$ collected from all the test cases, $n$ is the total number of test cases we need, and $k$ is the number of divided sections. We first divide the value range $[C_{min},C_{max}]$ into $k$ equal sections with the interval of $d=(C_{max}-C_{min})/k$, and then randomly select $n/k$ samples from each section independently to consist the final augmenting dataset.

Datasets and Models We adopt $4$ open-source datasets in our evaluation. Two of them are used for training the domain transformer (refer to Section 3.1) of the protected attribute gender and race respectively, and the remaining two are the experiment subjects.

• •

  CelebA (Liu et al., 2015, [n.d.]) is a widely used large-scale face recognition dataset. It contains $202,599$ face images from $10,177$ celebrities around the world. Each image has $5$ landmark locations and $40$ binary attributes, e.g., male, big lips, and pale skin. We use CelebA for gender transformation.

• •

  BUPT-Transferface (Wang et al., 2019a; Deng, [n.d.]) is a dataset of face images which aims to bring racial awareness into studies on face discrimination and achieving algorithm fairness. We use it for race transformation.

• •

  VGGFace (Parkhi et al., 2015, [n.d.]) consists of over $2,600,000$ images for $2,622$ identities from IMDB. The dataset is collected from multiple search engines, e.g., Google and Bing, with limited manual curation. We evaluate DeepFAIT on VGGFace.

• •

  FairFace (Karkkainen and Joo, 2021, [n.d.]) contains $108,501$ face images collected from the YFCC100m dataset. It is manually labeled with gender, race, and age groups and balanced on race. It divides the age into $9$ bins. We train an age classifier and take it as another benchmark for evaluating DeepFAIT.

We utilize two state-of-the-art DNN architectures as the benchmark models.

• •

  VGG (Simonyan and Zisserman, 2015) is an advanced architecture for extracting abstract features from image data. VGG utilizes small convolution kernels (e.g., $3*3$ or $1*1$) and pooling kernels (e.g., $2*2$) to significantly increase the expressive power of the model.

• •

  ResNet (He et al., 2016) improves traditional sequential DNNs by solving the vanishing gradients problem when expanding the number of layers. It utilizes short-cuts (also called skip connections), which adds up the input and output of a layer and then transforms the sum to the next layer as input.

The details of our pre-trained models are shown in Table 1.

In the training process of the transformation model, we need to ensure that only the given sensitive attribute changes to avoid the impact of other sensitive attributes. Therefore, when we train the race (respectively gender) transformation model, the image pairs that we construct are made sure to have the same gender (respectively race).

We aim to answer the following 3 research questions through our evaluation.

Table 3 lists the number of total neurons and the selected fairness-related neurons of the deepest $5$ layers both on VGGFace and FairFace with respect to race and gender. We can observe that a considerable percentage of the neurons show correlation with fairness, for instance, the proportion is ranging from $70.34\%$ ($65.23\%$) to $100.00\%$ ($99.41\%$) on VGGFace with respect to race (gender). Further, Figure 4 shows the H distribution of these $5$ layers. Note that the H values within each layer follows a long-tailed distribution, i.e., the H value of most neurons centralize on lower values, whereas a few values are pretty large, which shows significant correlation with fairness.

We thus have the following answer to RQ1,

We evaluate the testing coverage on $3$ data combinations. Table 4 shows the coverage computed on $4$ ordered layers, i.e., FC7, FC6, Conv5_3, and Conv4_2, of VGG-16 with respect to the sensitive attribute gender. Row Fair and Ori. are the coverage of $10,000$ fair pairs and adding all the discriminatory ones (since its size is smaller than $10,000$) of original dataset, and row Fair+GG is the adequacy of the original fair data augmented with $10,000$ discriminatory instance pairs generated by the gradient-based strategy.

First, it can be observed that for layer-level statistic, the coverage on the layers that are close to the output layer is significantly higher. For instance, the tanimoto coefficient of fair images drops from $69.50\%$ in FC7 to $57.60\%$ and $16.40\%$ in FC6 and Conv5_3 and reaches $0.20\%$ in Conv4_2. One possible explanation is that the deeper layers in the DNN are able to extract the more identity-related information. In addition, we also observe that: 1) a deeper layer is more sensitive to the original discriminatory samples, e.g., compared with the fair images, the absolute distance adequacy of the whole original testing dataset increases by $2.95\%$, $7.03\%$, $7.87\%$, and $9.69\%$ for Conv4_2, Conv5_3, FC6, and FC7 respectively; 2) the generated test cases show similar layer sensitivity, e.g., when we augment the original fair data with the one generated using the gradient-based strategy, the coverage of relative distance from FC7 to Conv4_2 increases by $10.24\%$, $7.46\%$, $3.92\%$, and $0.11\%$, respectively. It is obvious that the selection of layers will affect the effectiveness of the testing. We thus suggest to choose the deeper layers to measure the adequacy of test cases.

Next, we conduct a correlation analysis on the coverage metrics and the individual fairness of the experimented models. We adopt the model mutation technique developed in (Wang et al., 2019b; Ma et al., 2018b) to obtain a significant number of models with different behaviors efficiently for the study. In this work, we generate $10$ models for each mutation operator, e.g., Gaussian Fuzzing, Weight Shuffling, Neuron Switch, and Neuron Activation Inverse. Besides, since it is almost impossible to obtain meaningful images through random sampling in the input space, we measure the individual fairness of the model based on the ratio of non-discriminatory instances in the original testing dataset. The ranges of the accuracy and fairness scores of these $40$ models are in the range of $[94.00\%,97.28\%]$ and $[64.37\%,89.10\%]$, respectively. In order to ensure the fairness of the experiment, we randomly selected $10,000$ non-discriminatory image pairs and obtain the coverage on the deepest hidden layer, i.e., FC7.

We show the Pearson product-moment correlation (Pearson, 1920) results on VGGFace with sensitive attribute gender in Figure 5. The number in each cell is the correlation value between the metrics of the corresponding row and column, which ranges from $\-1$ to $1$. Note that all the correlation is significant, i.e., $p<0.05$.

From Figure 5, we have the following observations. First, the three layer-level criteria are significantly positively correlated with the fairness score, i.e., with a minimum correlation coefficient of $0.85$. It indicts that if the non-discriminatory instances has a higher coverage on the layer statistics, the DNN is fairer. This is intuitively expected, i.e., a fairer model can tolerate greater behavioral differences. Moreover, the two neuron distances show highly negative correlation with the individual fairness, i.e., with the value of $0.69$ and $0.90$. Our explanation is that for a fair model, the output difference of the neurons is small (so as to ensure that the final prediction does not change). Second, it is obvious that Tanimoto, cosine, and Spearman similarity have strong positive correlations with each other, while there is a moderate negative correlations between layer-level and neuron-level criteria.

In traditional software testing, coverage is not only a measure of testing adequacy, but also an effective tool for revealing bugs (Zhang et al., 2019) (i.e., by generating tests with high coverage). In this work, a bug refers to whether individual discriminatory samples exist in a given DNN model. Therefore, we conduct an effectiveness evaluation by comparing the coverage of testing criteria on the fair dataset and the dataset augmented with the individual discriminatory samples. The latter is obtained in two ways. One is the discrimination in the original testing data (column Fair+UnFair), and the other is discriminatory instances generated using $3$ different approaches, e.g., gradient-based generation (column Fair+GG), random generation (column Fair+RG), and Gaussian-noise injection (column Fair+GI). The parameters for generation are shown in Table 2.

For each dataset and each sensitive attribute, we random select $10,000$ image pairs for all the testing subsets including fair and discriminatory data (if the number is less than $10,000$, we take all of them) in the original set, and generated individual discriminatory pairs. The coverage result of the penultimate layer is presented in Table 5. We repeat the procedure $5$ times and report the average coverage to avoid the effect of randomness. It can be observed that compared with the original fair pairs, the coverage of all the criteria has a significant increase after adding the discriminatory ones. Furthermore, compared with the discriminatory pairs in the original set and generated with image processing technology, fairness testing methods lead to higher coverage on all the criteria in most of the cases, except for the absolute distance and relative distance of VGG-16 with respect to gender. This is in line with our expectation, i.e., the criteria are more sensitive with individual discriminatory pairs generated by fairness testing, especially by the gradient-based method. In addition, we further conduct a comparison with DeepImportance (Gerasimou et al., 2020). It is worth noting that DeepImportance reports similar layer sensitivity in its evaluation. For a fair comparison, we also take the $10$ most important neurons, and the The total number of important neurons cluster combinations on VGG-16 and ResNet-50 are $131,072$ and $1,024$, respectively. We observe that the coverage of DeepImportance also increases when discriminatory samples are added. One possible reason is that the generation of unfair test case will push the seed towards the decision boundary (Zhang et al., 2020, 2021), which will also reduce the robustness of the seed. However, it is also observed that the DeepImportance is less sensitive than DeepFAIT in most of the cases. In addition, since DeepImportance is only computed on individual samples, it cannot distinguish the similarity between each transformed sample and the original one, when sensitive attributes have multiple values.

We have the following answer to RQ2,

Figure 6 shows the results on dataset Fairface. Since we randomly select $10\%$ of the generated unfair samples for data augmentation, we present the average improvement of $5$ repeats. All models after augmented training have only a slight loss of accuracy, i.e., with a maximum decrease of $1.44\%$. From left to right, each bar represents the completely random strategy (RA) and KM-ST on Tanimoto Coefficient (TC), Cosine Similarity (CS), and Spearman Correlation (SC), respectively. We observe that compare with completely random: 1) applying KM-ST on cosine similarity and Spearman correlation are capable of reducing more discrimination in the model, especially applying it based on the cosine similarity, which has the greatest improvement of $5.15\%$ and $4.08\%$ with respect to race and gender, respectively. The reason is that compared with completely random method, KM-ST uniformly samples in each smaller subspace, which can better ensure that the obtained unfair samples are representative; 2) applying KM-ST on Tanimoto coefficient has $1.16\%$ and $1.73\%$ less fairness improvement with respect to race and gender, respectively. The reason is that compared with cosine similarity and Spearman coefficient, Tanimoto coefficient only considers the activation of neurons in each layer, while ignores some more fine-grained information such as the activation value.

We have the following answer to RQ3,