NeuronFair: Interpretable White-Box Fairness Testing through Biased Neuron Identification

First, we draw AS curve and compute AUC value to measure DNNs’ discrimination. Then, based on the measurement results, we identify the key neurons that cause unfair decisions as biased neurons.

NeuronFair generates IDIs in two phases, i.e., a global generation phase and a local generation phase. The global phase aims to acquire diverse IDIs. The IDIs’ diversity in the global phase is crucial since these instances serve as seeds for the local phase. Instead, to guarantee the IDIs’ quantity, the local phase aims to search for as many IDIs as possible near the seeds.

We intend to solve the challenge of $A_{s}$ modification to generalize NeuronFair to unstructured data. Here we take image data as an example. Attributes of an image are determined by pixels with normalized values between 0 and 1, i.e., the input domain of images is $\mathbb{I}\in[0,1]$. Motivated by the adversarial attack, we design a generalization framework to implement the image’s $A_{s}$ modification, which modifies $A_{s}$ through adding a small perturbation to most pixels, as shown in Fig. 4.

We consider a fairness testing scenario for face detection, which determines whether the input image contains a face. The face detector consists of a CNN module (i.e., Fig. 4 (i)) and a FCN module (i.e., Fig. 4 (ii)). As shown in Fig. 4, for a given face image $x$ and a detector $f(x,\Theta)$, there are three steps: \footnotesize1⃝ build a sensitive attribute classifier; \footnotesize2⃝ produce $\Delta_{senatt}$ based on Eq. (5), $\Delta_{senatt}$ is the perturbation added to image to flip sensitive attribute; \footnotesize3⃝ generate $\Delta_{bias}$ based on NeuronFair, where $\Delta_{bias}$ is the bias perturbation added to an image to flip the detection result.

First, we need a sensitive attribute classifier $f_{sa}(x;\Theta_{sa})$ that can distinguish the face image’s $A_{s}$ (e.g., gender). We build the $A_{s}$ classifier by adding a new FCN module (i.e., Fig. 4 (iii)) to the face detector’s CNN module (i.e., Fig. 4 (i)). Then, we froze the weights of the CNN module, and train the weights of the newly added FCN module.

Next, we modify the face image’s $A_{s}$ based on the adversarial attack. A classic adversarial attack FGSM (Goodfellow et al., 2015) is adopted to flip the predicted result of the sensitive attribute by generating $\Delta_{senatt}$ as follows:

where $\epsilon$ is a hyper-parameter to determine perturbation size, $\rm sign(\cdot)$ is a signum function return “-1”, “0”, or “1”, $x$ is an input image, $y_{sa}$ is the ground-truth of $x$ about sensitive attributes, $\Theta_{sa}$ is the weights of the $A_{s}$ classifier.

Finally, we leverage NeuronFair to generate $\Delta_{bias}$, and then determine whether the instance pair $<\!x\!+\!\Delta_{bias},x\!+\!\Delta_{bias}\!+\!\Delta_{senatt}\!>$ satisfy Definition 1. We determine the discrimination at each layer of the detector at first. For the CNN module, the activation output of the convolutional layer is flattened. Then, in the process of image IDI generation, only the global generation is employed, which is due to the different data forms between image and structured data. Taking the input image in Fig. 4 as an example, its attributes can be regarded as $A\in\mathbb{R}^{{\rm 64\times 64\times 3}}$. Based on a seed image IDI generated in the global phase, numerous image IDIs will evolve in the local phase. However, these image IDIs are similar, with only a few pixel differences, which have little effect on fairness improvement of the face detector. Besides, cancel the signum function ${\rm sign}(\cdot)$ at line 15 of Algorithm 2 for image data.

We evaluate NeuronFair on 7 datasets of which five are structured datasets and two are image datasets. Each dataset is divided into three parts, i.e., 70%, 10%, 20% as training, validation, and testing, respectively.

The 5 open-source structured datasets include Adult, German credit (GerCre), bank marketing (BanMar), COMPAS, and medical expenditure panel survey (MEPS). The details of these datasets are shown in Tab. 1. All datasets can be downloaded from GitHub ¹¹1https://github.com/Trusted-AI/AIF360/tree/master/aif360/data and preprocessed by AI Fairness 360 toolkit (AIF360) (Bellamy et al., 2018).

The 2 image datasets (i.e., ClbA-IN and LFW-IN) are constructed by ourselves for face detection. ClbA-IN dataset consists of 60,000 face images from CelebA (Liu et al., 2015) and 60,000 non-face images from ImageNet (Deng et al., 2009). LFW-IN dataset consists of 10,000 face images from LFW (Huang et al., 2008) and 10,000 non-face images from ImageNet (Deng et al., 2009). The pixel value of each image is normalized to [0,1].

We implement 5 FCN-based classifiers (Zhang et al., 2020; Bellamy et al., 2018) for structured datasets and 2 CNN-based face detectors (Simonyan and Zisserman, 2015; He et al., 2016) for image datasets since FCN and CNN are the most widely used basic structures in real-world classification tasks.

The 5 FCN-based classifiers can be divided into two types. The one is composed of 5 hidden layers for processing low-dimensional data (i.e., Adult, GerCre, BanMar), denoted as LFCN. The another is composed of 8 hidden layers for processing high-dimensional data (i.e., COMPAS and MEPS), denoted as HFCN. The activation functions in hidden layers and the output layer are ReLU and Softmax, respectively. The hidden layer structures of LFCN and HFCN are [64, 32, 16, 8, 4] and [256, 256, 64, 64, 32, 32, 16, 8], respectively.

The 2 CNN-based face detectors serve for face detection, which are variants from two pre-trained models (i.e., VGG16 (Simonyan and Zisserman, 2015) and ResNet50 (He et al., 2016)) of keras.applications. We use the CNN module of VGG16 and ResNet50 as Fig. 4 (i), and design the FCN module of Fig. 4 (ii) and (iii) as [512, 256, 128, 64, 16].

We implement and compare 4 state-of-the-art (SOTA) methods with NeuronFair to evaluate their performance, including Aequitas (Udeshi et al., 2018), SymbGen (Aggarwal et al., 2018), ADF (Zhang et al., 2020), and EIDIG (Zhang et al., 2021). Note that Themis (Galhotra et al., 2017) has been shown to be significantly less effective for DNN and thus is omitted (Zhang et al., 2020; Aggarwal et al., 2018). We obtained the implementation of these baselines from GitHub ²²2https://github.com/pxzhang94/ADF ³³3https://github.com/LingfengZhang98/EIDIG. All baselines are configured according to the best performance setting reported in the respective papers.

Five aspects of NeuronFair are evaluated, including generation effectiveness, efficiency, interpretability, the utility of AUC metric, and generalization of NeuronFair.

To fairly study the performance of the baselines and NeuronFair, our experiments have the following settings: (1) the hyperparameters of each method are set according to Tab. 2, where ‘Glo.’ and ‘Loc.’ represent the global and local phases, respectively; (2) for the FCN-based classifier, we set the learning rate to 0.001, and choose Adam as the optimizer; for the CNN-based face detector, we set the learning rate to 0.01, and choose SGD as the optimizer; the training results are shown in Tab. 3, where “99.83%/92.80%/94.30%” represents the accuracy of face detector, gender classifier, and race classifier, respectively.

We conduct all the experiments on a server with one Intel i7-7700K CPU running at 4.20GHz, 64 GB DDR4 memory, 4 TB HDD and one TITAN Xp 12 GB GPU card.

How effective is NeuronFair in generating IDIs?

When reporting the results, we focus on the following aspects: generation quantity and quality.

Generation Quantity. The evaluation results are shown in Tabs. 4, 5, and 6, including three scenarios: the total number of IDIs, the IDIs number in global phase, and the IDIs number in local phase.

Implementation details for quantity evaluation: (1) SymbGen works differently from other baselines, thus we follow the comparison strategy of Zhang et al. (Zhang et al., 2020), i.e., evaluating the generation quantity of NeuronFair and SymbGen within the same time (i.e., 500 sec) limit, as shown in Tab. 5; (2) for a fair global phase comparison, we generate 1,000 non-duplicate instances without constrained by $num_{g}$, then count IDIs number and record it in Tab. 6, where the seeds used are consistent for different methods; (3) for a fair local phase comparison, we mix IDIs generated globally by different methods, and randomly sample 100 as the seeds in local phase; then generate 1,000 non-duplicate instances for each seed without constrained by $max_{-}iter_{l}$, count the IDIs number on average for each seed and record it in Tab. 6.

• •

  NeuronFair generates more IDIs than baselines, especially for densely coded structured data. For instance, in Tab. 4, on Adult dataset with different attributes, the IDIs number of NeuronFair is 217,855 on average, which is 16.5 times and 1.6 times that of Aequitas and EIDIG, respectively. In addition, in Tab. 5, NeuronFair generates much more IDIs than SymbGen on all datasets. The outstanding performance of NeuronFair is mainly because the optimization object of NeuronFair takes into account the whole DNNs’ discrimination information through the biased neurons while Aequitas and EIDIG only depend on the output layer. However, the IDIs number on COMPAS dataset with race gender is 11,232, which is slightly lower than that of EIDIG. Since the COMPAS is encoded as one-hot in AIF360 (Bellamy et al., 2018), we speculate the reason is that too sparse data coding reduces the derivation efficiency from biased neurons.

• •

  As shown in Tab. 6, NeuronFair generates much more IDIs than all baselines in the global phase, which is beneficial to increase the diversity of NeuronFair’s IDIs in the subsequent local phase. For instance, on all datasets, the IDIs number of NeuronFair is 866 on average, which is 9.45 times and 1.42 times that of Aequitas and EIDIG, respectively. This is mainly because the optimization object of NeuronFair takes into account the dynamics through the dynamic combination of biased neurons. Thus, NeuronFair searches a larger space to generate more global IDIs.

• •

  In the local phase, NeuronFair is much more efficient than baselines in general. For instance, in Tab. 6, on average, NeuronFair returns 78.97%, 45.35%, 10.81%, and 2.90% more IDIs than Aequitas, SymbGen, ADF, and EIDIG, respectively. Recall that Aequitas, ADF, EIDIG, and NeuronFair all guide local phase through a probability distribution, which is the likelihood of IDIs by modifying several certain attributes (i.e., the loop from lines 11 to 16 of Algorithm 3). The probability determination of NeuronFair takes into account the momentum and SoftMax activation (i.e., at line 10 of Algorithm 3) while the baselines do not. Hence, NeuronFair generates more local IDIs.

Generation Quality. The evaluation results are shown in Tabs. 4, 5, 7, and Fig. 5, including the generation success rate (GSR), generation diversity (GD), and fairness improvement (DM-RS).

Implementation details for quality evaluation: (1) for a fair diversity comparison, we seed each method with the same set of 10 global IDIs and apply them to generate 100 local IDIs for each seed without considering $max_{-}iter_{l}$, as shown in Fig. 5; (2) we randomly select 10% IDIs of each method to retrain DNNs, then compute their fairness improvement results; to avoid contingency, we repeat 5 times and record the average DM-RS value in Tab. 7.

• •

  As shown in Tabs. 4 and 5, the GSR values of NeuronFair are higher than that of baselines on almost all datasets, i.e., NeuronFair can search for a larger valid input space, where the input space is calculated by ‘#IDIs/GSR’. For instance, in Tab. 4, on all datasets, Aequitas has a GSR of 17.62% on average, whereas NeuronFair achieves a GSR of 36.12%, which is $\sim\!\!\times$2.1 more than that of Aequitas. The outstanding performance of NeuronFair is mainly because it not only considers the whole DNN’s discrimination through biased neurons, but also takes into account the dynamics of the optimization object through the combination of biased neurons. Thus, NeuronFair searches a larger valid input space than Aequitas.

  Meanwhile, the GSR value of NeuronFair on different sensitive attributes is more robust than ADF and EIDIG. For instance, in Tab. 4, on the GerCre dataset with gender and age attributes, the GSR values of NeuronFair are 36.57% and 63.35%, whereas that of EIDIG are 17.23% and 59.38%. We speculate the reason is that the discrimination about the gender attribute in the output layer is not obvious, but NeuronFair can find potential fairness violations through the internal discrimination information of biased neurons. Therefore, we can realize stable testing for different sensitive attributes.

  In addition, the valid input space of NeuronFair is larger than baselines in general, i.e., a larger input space supports more diverse IDI generation. For instance, in Tab. 5, the average input space of NeuronFair is 3.30 times that of SymbGen. It is mainly because the momentum acceleration strategy employs historical gradient as auxiliary guidance, which reduces the number of invalid searches. Hence, NeuronFair generates more IDIs in a large input space.

• •

  In all cases, NeuronFair can generate more diverse IDIs, which is beneficial to discover more potential discrimination and then improve fairness through retraining. For instance, in Fig. 5, compare to Aequitas and ADF, the $\rm GD_{NF}$ values are all greater than ‘1’ under different radius values $\rho_{\rm cons}$, and as the radius increases, the value of $\rm GD_{NF}$ gradually converges to ‘1’. It demonstrates that the IDIs generated by NeuronFair can always cover that of baselines. We speculate the reason is that the dynamic loss function expands the valid input space by combining different biased neurons as the optimization object.

  Besides, a close investigation shows that there is a similar trend in the generation diversity for the same sensitive attribute. For instance, when $\rho_{\rm cons}<$0.1 in Fig. 5 (a) or $\rho_{\rm cons}<$0.02 in Fig. 5 (b), the line ‘L2’ with race is always the highest, the line ‘L4’ with age is always the lowest, while lines ‘L1’ and ‘L3’ with gender are in the middle. Since both datasets Adult and GerCre are related to money (i.e., salary and loans), we speculate that there is similar discrimination for gender in classifiers LFC-A and LFC-G for similar tasks, thus $\rm GD_{NF}$ shows similar trends in gender attribute.

• •

  In all cases, NeuronFair can obtain larger DM-RS values, i.e., the IDIs generated by NeuronFair contribute more to the DNNs’ fairness improvement. For instance, in Tab. 7, measured by DM-RS, NeuronFair realizes fairness improvement of 87.24% on average, versus baselines, i.e., 81.18% for Aequitas, 80.78% for SymbGen, 83.30% for ADF, and 84.49% for EIDIG. It is because the IDIs of NeuronFair are more diverse than those of baselines, so it can discover more potential fairness violations and implement higher fairness improvement through retraining.

How efficient is NeuronFair in generating IDIs?

When answering this question, we refer to the generation speed. The evaluation results are shown in Tab. 8, where the time cost of SymbGen includes generating the explainer and constraint solving. Here we have the following observation.

• •

  NeuronFair generates IDIs more efficiently, which meets the rapidity requirements of software engineering testing. For instance, in Tab. 8, on average, NeuronFair takes only 26.07%, 5.47%, 49.47%, and 79.32% of the time required by Aequitas, SymbGen, ADF, and EIDIG, respectively. The outstanding performance of NeuronFair is mainly because it uses a momentum acceleration strategy and shortens the derivation path to reduce computational complexity. Hence, it takes less time than baselines.

How to interpret NeuronFair’s utility by biased neurons?

When interpreting the utility, we refer to the biased neuron coverage. The evaluation results are shown in Fig. 6.

Implementation details for interpretation: (1) we conduct experiments on the Adult dataset with gender attribute for the LFC-A classifier; (2) we compare the interpretation results of NeuronFair with ADF and EIDIG; (3) for a fair interpretation, we randomly select 10% IDIs and 10% non-IDIs (i.e., the generated failure instances) for each method, and then compute the coverage of biased neurons, as shown in Fig. 6.

• •

  Biased neurons can be adopted to interpret the utility of IDIs and NeuronFair. First, IDIs trigger discrimination by activating biased neurons. For instance, the neurons activated by IDIs can cover most of the biased neurons in Fig. 6 (a), while the coverage of the biased neurons by non-IDIs of different methods is 0 in Fig. 6 (b). We can further interpret the utility of testing methods is related to the coverage of biased neurons, i.e., NeuronFair is more effective than ADF and EIDIG because they miss some discrimination contained in biased neurons while NeuronFair does not. For instance, in Fig. 6 (a), the NeuronFair’s IDIs activate all 24 biased neurons in the 2-nd layer of LFC-A classifier, while the neurons activated by other IDIs cannot cover all (15 for ADF and 18 for EIDIG).

When answering this question, we refer to the following aspects: the consistency, significance, and complexity between AUC and DM-RS. The evaluation results on Adult and GerCre datasets with multiple sensitive attributes are shown in Tab. 9. From the results, we have the following observations.

• •

  In all cases, AUC can correctly distinguish DNNs’ fairness violations, i.e., AUC can serve the discrimination measurement of DNN. For instance, in Tab. 9, all of the $\rho_{s}$ values are 1.00, indicating that the discrimination ranking results of different DNNs based on AUC are completely consistent with those based on DM-RS. Since DNN’s decision results are determined by the neurons’ activation, we speculate that the biased decisions are also caused by the neurons’ activation, i.e., neurons contain discrimination information. Therefore, we can leverage the discrimination information in neurons to determine DNNs’ fairness.

• •

  In all cases, AUC can distinguish the different DNN’s discrimination more significantly than DM-RS, which is beneficial for a more accurate evaluation of IDIs of different testing methods. For instance, in Tab. 9, all $\sigma$ values of AUC are higher than those of DM-RS, and the average $\sigma$ value of AUC is 8.46 times that of DM-RS. The outstanding performance of AUC is mainly because we use the neurons’ ActDiff to measure the discrimination, which extracts more bias-related information from the whole DNN; while MD-RS only uses the bias-related information from the output layer.

• •

  The computational complexity of AUC is much lower than that of MD-RS, which is beneficial to quickly distinguish DNNs’ discrimination or the IDIs’ effect. The time frequency of AUC is ${\rm T}(N_{l})=(7+count)\times N_{l}+1$ based on Algorithm 1. Thus the time complexity of AUC is $\mathcal{O}(N_{l})$, while that of DM-RS is $\mathcal{O}(n\log n)$, where $n$ is the instance number, $N_{l}$ is the layer number, $n>>N_{l}$. It is mainly because AUC only conducts matrix operations, while DM-RS requires iterative operations until convergence.

Answer to RQ4: The AUC is useful for discrimination measurement. Compared to the results in Tab. 9, AUC is (1) 100% consistent with DM-RS, (2) $\times$8.46 more significant than DM-RS, (3) low computational complexity with $\mathcal{O}(N_{l})$.

How generic is NeuronFair for the task of image IDI generation?

When reporting the results, we focus on two aspects: generation quantity and quality.

Implementation details for NeuronFair generalized on image data: (1) we only perform comparisons with ADF and EIDIG at global phase, because the effect of ADF and EIDIG on DNNs is much better than that of Aequitas and SymbGen; (2) we remove the $\rm KMeans(\cdot)$ operation, set $step_{-}size_{g}$=0.15 for image, and all face images are used as input; (3) we retrain the face detector with all image IDIs of each method and measure its fairness improvement by AUC, (4) we measure the bias perturbation $\Delta_{bias}$ and sensitive attribute perturbation $\Delta_{senatt}$ by $L_{2}$-norm.

Generation Quantity. The evaluation results are shown in Tab. 10 measured by the IDIs number in global phase. From the results, we have the following observation.

• •

  In all cases, NeuronFair can obtain more IDIs than ADF and EIDIG, especially for the discrimination against race attribute. For instance, in Tab. 10, on average, NeuronFair generates 4.34 times and 2.07 times IDIs of ADF and EIDIG, respectively. The outstanding performance of NeuronFair is because it adopts dynamic loss to expand the valid input space while ADF and EIDIG do not consider the dynamics of search. Meanwhile, the number of IDIs generated by NeuronFair for race is 3.91 times that for gender. We speculate the reason is that the pixel information related to race is mainly skin color (i.e., light & dark, or black & white), while the pixel information related to gender is more diverse (such as hair, makeup, face shape, etc.). Therefore, the image IDIs generation for race is easier through manipulating skin color.

Generation Quality. The evaluation results are shown in Tabs. 10 and 11, including three scenarios: the generation success rate (GSR), the fairness improvement (AUC), and the detection rate (DR).

• •

  Among image data, image IDIs of NeuronFair are of higher quality than those of ADF and EIDIG, which can be applied to retrain face detectors and contribute to their fairness improvement in face detection scenarios. For instance, in Tab 10, on average, the GSR value of NeuronFair is 2.60 times and 1.63 times that of ADF and EIDIG, respectively. It is because NeuronFair reduces the probability of gradient vanishing, which in turn improves the probability of non-duplicate IDIs generation guided by the gradient. Hence, all GSR values of NeuronFair are higher than those of baselines. Meanwhile, the valid input space of NeuronFair is 1.67 times and 1.27 times that of ADF and EIDIG, respectively. Since the probability of falling into a local optimum is reduced by dynamically combining biased neurons, we can perform valid searches in limited instance space.

• •

  NeuronFair contributes more to the fairness improvement of the face detector, i.e., its generalization on image data is better than that of ADF and EIDIG. For instance, in Tab. 11, on average, the discrimination of detectors retrained with IDIs of NeuronFair dropped by 53.77%, while the AUC values of ADF and EIDIG only dropped by 8.06% and 10.90%, respectively. We speculate that the valid input space of NeuronFair is larger, so its IDIs can find potential discrimination that other methods’ IDIs cannot. Then improve the detector’s fairness through retraining.

• •

  NeuronFair hardly affects the detector’s DR values while improving its fairness. For instance, in Tab. 11, on average, the DR value of detectors retrained with NeuronFair’s IDIs only dropped by 0.87%, while that of ADF and EIDIG dropped by 2.92% and 4.80%, respectively. We compare the $L_{2}$ norm of $\Delta_{bias}$ and $\Delta_{senatt}$ generated by different methods, and find that $\Delta_{bias}$ of NeuronFair is much lower than that of ADF and EIDIG. Therefore, NeuronFair can not only improve the detector’s fairness but also maintain its detection performance.

Answer to RQ5: The generalization performance of NeuronFair on the image dataset is better than the SOTA methods (i.e., ADF and EIDIG) in two aspects: (1) quantity - it generates $\sim\!\!\times$4.34 and $\sim\!\!\times$2.07 image IDIs on average compared to ADF and EIDIG, respectively; (2) quality - it searches $\sim\!\!\times$1.47 input space with more than $\sim\!\times$2.11 GSR on average, it is beneficial to detectors’ fairness improvement of 53.77% on average but hardly affects their detection performance. Thus, NeuronFair shows better generalization performance than ADF and EIDIG.