Visual Analysis of Discrimination in Machine Learning

Existing studies mainly investigate algorithmic discrimination from two aspects: the data and the model training process [23].

Training data may include human discrimination and then influence the trained model [37]. Various approaches have been proposed to discover discrimination existing in the training data [10, 29, 44, 50, 73]. A pragmatic solution is to examine different treatments among similar items [29, 50]. For example, Luong et al. [43] used the k-nearest neighbor algorithm to group similar items and identify the discrimination among them. Pedreshi et al. [52] employed classification rules to reveal itemsets that may lead to unequal treatment. Recently, to provide an interpretable notation of discrimination, several studies considered the relationships among item attributes and examined discrimination from a causal modeling perspective [37, 69].

Even though human discrimination can be identified and eliminated from the training data, a model can still make discriminatory predictions due to the data distribution and the model learning mechanism (e.g., over-fitting) [42, 51, 67, 71, 73]. To remove discrimination introduced in the training process, researchers developed regularizers that penalize discriminatory predictions [31, 68] and methods that directly modify the model predictions [25, 71]. Many studies have defined different criteria for model discrimination to provide meaningful [51, 71] and interpretable [25] notations. Meanwhile, to support users in measuring and comparing discrimination in different models, initial efforts have been made to quantify model discrimination [58].

These studies shed light on discrimination analysis and form the foundation of this study. However, the discrimination mined by these methods can be complex and thus hard to interpret (e.g., a large number of rule lists), requiring the support of effective visual presentation. More importantly, the analysis of discrimination is, by nature, a subjective task and varies based on the application domains. This paper provides interactive analysis to help incorporate human domain knowledge in discrimination analysis.

Visualization, an effective tool for information communication, has been employed to present and analyze discrimination in ML [1, 8, 9, 27, 59, 38]. For example, Google Big Picture developed an interactive visualization demo [8] to illustrate the discrimination in ML decisions. Other tools, such as IBM AI Fairness 360 [27], Tensorflow Fairness Indicators [59], and FairLearn, enable the easy computation of fairness metrics and support visualizations of these fairness metrics. However, most tools only support segmenting groups based on one protective attribute and only provide basic visualizations (e.g., bar charts). Therefore, they cannot be directly applied to investigate and present the potentially complicated discrimination between different groups.

Recently, advanced visualization tools have been developed to enables more comprehensive analysis of discrimination, including FairSight [1] and FairVis [9]. FairSight represents a workflow that supports the four fairness-aware actions (i.e., understand, measure, identify, and mitigate) required in decision making. FairSight shares the same high-level goal as DiscriLens and also supports the analysis of individual fairness. However, FairSight either a) identifies discriminated instances based on an KNN algorithm or b) provides a global-level measure by aggregating over all instances. As a result, FariSight fails to uncover the discriminatory itemsets and to reveal “when and where will a model yield discriminatory predictions”. FairVis is more related to DiscriLens due to its focus on the analysis of discriminatory itemsets. This tool helps users to audit the fairness among different itemsets. However, FairVis focuses on suggesting similar discriminatory itemsets during analysis. The possible complex definition of the itemsets are presented using tables, which is not an optimal representation method. No special visualizations are designed to reveal the intertwining relationships among the subgroups.

In DiscriLens, we propose novel visualizations for better understanding and analysing ML discrimination.

The core of discrimination analysis is to understand how a set of items, grouped by attribute values, are treated unequally. Due to the ubiquity of set-based analysis, a large body of literature has been developed. Below we review some visualizations that are germane to our work. A comprehensive review of set visualizations can be found in the work of Alsallakh et al. [4].

The most common set visualizations are Euler and Venn diagrams [21, 39], which represent sets as convex or concave shapes and show set intersections as overlapping shapes. Previous work has proposed many extensions of Euler diagram with varying design goals [13, 17, 55, 57]. For example, Riche et al. [55] simplified a sophisticated collection of intersecting sets into a strict hierarchy, thereby untangling Euler diagrams and improving its readability. Bubble Sets [13] delineate set memberships and minimize cluster overlap while retaining the layout that reflects the semantic spatial organization. However, methods based on Euler and Venn diagrams often impose severe limitations on the number of sets and the complexity of set intersections [3].

Considering the inherent limits of Euler and Venn diagrams, researchers have developed many alternative presentations of set data, including curve-based presentation [2], treemap-based presentation [5], parallel coordinate-based presentation [36], matrix-based presentation [35, 39], and chord diagram-based presentation [3]. The most relevant ones to our work are Upset [5] and Conset [35], which employed matrix to communicate the properties of the set aggregations and intersections. However, Upset and Conset are specified for visualizing binary set data and cannot be directly applied to the categorical set data in discrimination analysis. Meanwhile, in existing work, the comparison of two collections of set data has not been adequately explored and requires further investigation.

In this work, to facilitate the analysis and comparison of categorical set data, we propose a novel set visualization that combines an extended Euler diagram, RippleSet, with a matrix-based visualization.

The goal of DiscriLens is to enable model users to flexibly interpret and analyze the discrimination in ML. The users have a certain level of expertise in ML and wish to apply ML models to a real-world application. They want to analyze discrimination to better avoid the potential ethical or legal issues. Based on the previous literature on algorithmic discrimination and discussions with two ML experts, we identify the following design goals.

1. G1

   Customize the definition of discrimination.

   Discrimination is a complex concept and can have different representations at different domains [47]. For example, a decision based on age is discrimination in certain domains but not in others. Also, the resolving attributes to a decision significantly vary from domain to domain. Therefore, the domain knowledge of the users is essential for the analysis of discrimination. Users should be allowed to customize the definition of discrimination (i.e., protected attribute, proxy attributes, resolving attributes, the threshold $\tau$) based on their domain knowledge as well as the results of discrimination discovery algorithms.

2. G2

   Measure the degree of discrimination.

   For the analysis of discrimination, model users need to measure the degree of algorithmic discrimination [51, 58], which can be evaluated by two indicators: the risk difference and size. The risk difference answers the question: “To which extent are the protected group ($A=1$) and non-protected group ($A=0$) treated unequally?” The size answers the question: “How many people are influenced by algorithmic discrimination?” High risk difference and large size indicate severe discrimination.

3. G3

   Identify the condition of discrimination.

   Apart from measuring the degree of discrimination, the condition of discrimination should be identified to reveal when a model will yield discriminatory predictions. This information can be conveyed by a set of attribute values that specify the discriminatory itemsets, i.e., a group of individuals who are similar to the task but are treated differently. The condition of discrimination is crucial since it can provide detailed guidance on applying models (e.g., identify the failure mode [52]), improving the training data (e.g., modify inappropriate data labels [69], collect new data [72]), and adjusting the training process (e.g., add regularization to penalize discriminatory predictions [68, 44]).

4. G4

   Depict the distribution of discrimination. The degree and the condition of discrimination can vary a lot among itemsets [52, 73]. For example, in income prediction, the degree of discrimination towards blacks may vary with other attribute values, such as working hours, education level, and work class. As a result, algorithmic discrimination can hardly be comprehensively described by summary statistics or easily interpreted by humans [58]. The difficulty is significantly increased when the intersections between itemsets are complex. To fully capture the discriminatory behavior of a machine learning model, the distribution of discrimination should be offered to help users understand how discrimination varies among and inside different itemsets.

5. G5

   Compare discrimination. Summary statistics can be ineffective for the analysis of ML models [67]. Models with the same accuracy can assign conflicting predictions for the same input data [45], leading to discriminatory predictions against different groups of data items. For example, one model may discriminate equally against all female students, while another model discriminates severely against females with low scores but mildly against other females. Therefore, to choose an appropriate model, it is important to compare models and analyze conflicting predictions.

   Apart from model-level comparison, it is also important to compare discrimination among different itemsets. Several studies [51, 14] have observed that model discrimination cannot be completely removed. Therefore, different itemsets should be compared to prioritize the most severe discrimination, whose identification is objective and requires human domain knowledge.

DiscriLens consists of two main modules: a discovery module and a visualization module (Figure 3). The discovery module takes the training data, the model, and the user-defined protected group as input. It then goes through a three-stage pipeline and produces a collection of potentially discriminatory itemsets. The visualization module serves as an interface that helps understand the discrimination, as well as a tool that provides guidance on applying and improving the model. Details of the discovery and the visualization module are introduced in the following two sections.

In this step, we mine resolving attributes following the causality-based method proposed in [69]. Causal modeling is a complicated task and requires strong prior knowledge. Fortunately, when analyzing discrimination, we only need to mine local causality and identify the parents of $Y$ (i.e., $(Pa(Y)$) without building the complete causal graph. We refer the readers to [69, 34] for more details and the complete mathematical proof. The resolving attributes can be then represented as $Par(Y)\setminus(A\cup P)$, where $P$ indicates the proxy attributes. The identification of proxy and resolving attributes requires users’ domain knowledge. Therefore, we allow users to interactively modify resolving attributes, remove proxy attributes, and analyze the results in the visualization module. In this implementation, we use the Fast Greedy Equivalence Search [12] for local causal discovery.

In this stage, we can extract a list of classification rules to explain the outcome $Y$. Take a support vector machine trained on Adult dataset [18] as an example. The task is to predict whether the annual income of a person is over 50k USD. One of the classification rules is shown in Figure 4(a). The support of this rule $supp(s\rightarrow\hat{Y}=1)=supp(s,\hat{Y}=1)=16$ and the confidence of the rule $conf(s\rightarrow\hat{Y}=1)=P(\hat{Y}=1|s)=supp(s,\hat{Y}=1)/supp(s)=0.9$. In this implementation, we use minimum description length [20] to discretize continuous attributes and FP-Growth [24] to mine the classification rules. Alternative methods can also be used, and we refer the reader to [66] for a detailed discussion.

Given two classification rules: $(s,A=0)\rightarrow\hat{Y}=1,(s,A=1)\rightarrow\hat{Y}=1$, we can treat $s$ as a discriminatory itemset if $|P(\hat{Y}=1|A=0,s)-P(\hat{Y}=1|A=1,s)|>\tau$ and $s$ includes all resolving attributes and no proxy attributes. We do not specify the sign due to reverse discrimination, i.e., discrimination against the non-protected group.

As shown in Figure 4, if [workclass, education, hours per week] include all resolving attributes and no proxy attributes, the confidence difference between rule (b) and (c) can be treated as discrimination towards females. In other words, in the itemset $s=$[workclass=feder-gov, education<8, hours per week>65], males are less likely to be predicted to have low incomes compared with their female counterparts (0.8 vs. 1.0). Therefore, this itemset is called discriminatory itemset. To enable interactive modification, we do not specify resolving and proxy attributes when mining classification rules. Note that $s$ may contain attributes which are neither proxy nor resolving attributes. We call these attributes as context attributes. In real-world, a discriminatory itemset can be defined by a long list of attribute values, and the itemsets are usually intertwined together. As a result, effective visualization is needed to assist users in the analysis of discrimination.

When analyzing model discrimination, it is important to effectively represent discrimination among itemsets that have complex intersection and inclusion relationships (G4). Existing set visualization methods often impose severe limitations on the complexity of set intersections [3, 4], and thus can hardly satisfy the design goals of DiscriLens. To better support the analysis in DiscriLens, we extend the widely-used Euler diagram and design RippleSet.

The attribute matrix (Figure 5(A)) helps users investigate each discriminatory itemset in detail. The attribute matrix employs a similar matrix-based layout as RuleMatrix [46]. This design has been proven effective in presenting classification rules, which is used to identify discriminatory itemsets. In the attribute matrix, each row stands for a group of individuals and each column stands for an attribute. The head (A1) represents the distribution of data items on different attributes, a root row (A2) represents a collection of discriminatory itemsets with the same resolving attribute values, and a row (A3) represents one discriminatory itemset.

The head (Figure 5(A1)) of the attribute matrix consists of a row of charts. Histograms are used for visualizing the distribution on continuous attributes, and bar charts are used for discrete attributes. In each chart, the x-axis represents the attribute value, and the y-axis represents the number of items. We use dark blue to indicate the items classified as the beneficial class (e.g., a loan approval) and light blue for the non-beneficial class (e.g., a loan decline). To distinguish resolving attributes, the names of resolving attributes are highlighted. Users can modify the resolving attributes based on their domain knowledge as well as their observation of the data distribution (G1). Discriminatory itemsets will dynamically update according to the modified resolving attributes.

To facilitate the interpretation and exploration, we group and aggregate discriminatory itemsets to offer a clear summary. We first group the discriminatory itemsets based on their resolving attribute values. All itemsets sharing the same resolving attribute values are put in one collection, which is represented as a bordered rectangle called root row (Figure 5(A2)). The number of items in a collection is labeled at the left end of the bordered rectangle. Then, in each collection, we organize itemsets, which are represented as rows (Figure 5(A3)), according to their inclusion relations. If an itemset $A$ is included in itemset $B$ (i.e., $A\in B$), $A$ can be opened by clicking the expand icon on $B$.

For each discriminatory itemset, a simple glyph (Figure 5(A5)) is put at the left end to indicate the degree of discrimination (G2). The radius of the glyph encodes the size of the itemset. The angle of the outer arc is determined by the probability that the non-protected group is labeled as beneficial class (i.e., $P(\hat{Y}=1|A=0,s)$) and the angle of the inner arc is determined by the probability that the protected group is labeled as beneficial class (i.e., $P(\hat{Y}=1|A=1,s)$).

The condition of discrimination (i.e., a series of attribute values) is represented by an array of rectangles, whose solid parts represent the specified ranges/values of the aligned attributes (G3). Since we discretize continuous attributes for mining classification rules, all attributes here are actually categorical and are visualized using similar encodings with continuous attributes. The color saturation of these rectangles is determined by the absolute value of the risk difference and the color hue is determined by the sign of the risk difference (G2). To better compare the condition of discrimination (G5), itemsets are vertically arranged in an order that maximizes the Jaccard similarity of adjacent itemsets.

In the head of the attribute matrix (Figure 5(A1)), users can modify resolving attributes (G1) through drag and drop. Such modification changes the mined discriminatory itemsets. Other visual components, i.e., the scatter plot, the RippleSet, and the rows in the attribute matrix, will be updated dynamically.

Three modes of analysis are supported: the compact mode, the parallel mode, and the comparison mode. In the compact mode (Figure 5(a)), rows in the attribute matrix are compact, and each RippleSet is placed to minimize the overall L1 distance between the RippleSets and the corresponding root rows. The compact mode provides space efficiency and can offer an overview of a relatively large number of itemsets. A “focus+context” technique is employed in the compact mode. The focused RippleSet will be horizontally aligned with the corresponding rows (Figure 5(D)). In the parallel mode (Figure 5(b)), RippleSets are placed vertically and each root row is horizontally aligned with the corresponding RippleSet. The parallel mode provides clear alignment and thus can guide users on the detailed exploration of a relatively small number of itemsets. The compact mode and the parallel mode help users smoothly switch between the overview and the details they are interested in. In the comparison mode (Figure 5(c)), users can conduct a side-by-side comparison of the discriminatory itemsets of two machine learning models (G5). As pointed out by Marx et al. [45], real-world datasets can admit ML models that have equal accuracy but assign conflicting predictions to the same input data. To help users identify these conflicting predictions and compare the fairness of two models, two RippleSets and their corresponding root rows are horizontally aligned if they have the same resolving attribute values.

We recruited 16 participants (12 males, 4 females, age 21-30) through university mailing lists. All participants are postgraduate students in the school of engineering and have experiences in machine learning.

A within-subject design was employed for the user study. We designed four tasks to assess participants’ ability to conduct discrimination analysis (G1-G5), as shown in Table 1. Each task contains 2-4 different questions in the same format. We trained models and mined discriminatory itemsets for two datasets: an adult income dataset [18] with a small number of discriminatory itemsets and a student performance dataset [6] with a large number of discriminatory itemsets. For the user study baseline, we first considered existing fairness visualization tools, such as FairVis [9], FairSight [1], FairLearn [38], and TensorFlow Fairness Indicator [59]. However, it is unfair to compare these tools with DiscriLens since they focus on different aspects and have different design goals, as discussed in subsection 2.2. On the contrary, we compare DiscriLens with other visualizations that can illustrate set intersections based on the survey results in [32]. The result is shown in Table 2. DiscriLens, as a combination of Euler-based and Matrix-based set visualization, supports a more detailed analysis of discrimination. Since no existing set visualization can meet all the requirements of discrimination analysis, we used Euler diagram + table (with search, sort, and filter functions) as the baseline method (Figure 8).

For each participant, two datasets were randomly associated with the two conditions (DiscriLens and baseline) and were presented in a counter-balanced order. Before the formal study, each participant received a 20-minutes tutorial to learn the tool, complete the trial tasks, and freely ask questions. In each condition, participants completed the four tasks. Participants are randomly ordered in this user study. Finally, each participant completed a post-study questionnaire and received a short informal interview.

We set a threshold at 0.05 for p value and hypothesized that, compared with the baseline, H1) DiscriLens is harder to understand; H2) DiscriLens doesn’t decrease the accuracy of tasks; H3) DiscriLens decreases the completion time of tasks.

We summarized the results of the performed tasks in Figure 9 and the results of the post-study questionnaire in Figure 10.

Contrary to our expectation, participants thought DiscriLens was easier to understand than the baseline (Figure 10), rejecting H1. Although most (10/16) participants expressed that “the tool seems complicated at first glance”, they stated that “the visual encodings are actually easy to understand and remember once you explained them” and “the tool offers better support to the analysis than the baseline”. This highlighted the importance of designing visualizations suitable for the analysis tasks. Complicated visual designs are sometimes inevitable for the analysis of complicated data. A simple visualization can be hard to understand when it fails to support the analysis.

As shown in Figure 9(a), except T2 in small size ($p=0.09$), participant gave significantly more accurate answers ($p<.05$) when using DiscriLens. This result supports H2. Note that the overlap between accuracy confidence intervals decreased with the increase of data size. This phenomenon might indicate that DiscriLens has a more significant advantage in accuracy when analyzing a large volume of discrimination.

As shown in Figure 9(b), the time cost of DiscriLens is significantly less than that of the baseline ($p<.05$), which supports H3. Unlike accuracy, the overlap between time confidence intervals increased with the increase of data size. This was because participants tended to give up some tasks when they found these tasks were difficult to complete using the baseline, which led to short completion time.

Meanwhile, we observed that participants employed different strategies with the two tools. When using the baseline to answer a question, most participants (12/16) switched back and forth between different question options and compared them carefully using the baseline tool. When using DiscriLens, most participants (10/16) first read the question, then examined the visualization to conclude an answer, and finally chose an option directly in the questionnaire.

For the discrimination discovery, it takes about six minutes to run the four-stage pipeline on 4,000 samples with 14 features on a PC (2.3GHz dual-core, Intel Core i5 processor). The major bottlenecks lie in the FEGS algorithm (two minutes), the FP-Growth algorithm (one minute), and the discriminatory rules mining (three minutes).

The scalability of Rippleset is mainly limited by the number of sets and the number of items. According to our user study, Rippleset effectively reduces the visual clutter in traditional Euler diagrams, which has difficulties to handle more than three sets. However, the readability of RippleSet hasn’t been validated with more than seven intertwining sets. For a large number of items, RippleSet supports the aggregation of items, as shown in Figure 5(D1). In the future, we plan to support more grouping strategies, such as those discussed in Squares [54], and provide more statistical summaries in the aggregation view of RippleSet.

We acknowledge that the novel and complicated visualization designs in DiscriLens can pose challenges to users, especially those with no prior knowledge in visual analytics. Even though we have provided interactive tutorials and legends to alleviate this issue, the steep learning curve still exists and can lead to low accessibility to novices. For example, RippleSet and the attribute matrix can be overwhelming when there are too many itemsets. Users may stop at the scatter plot, blindly trying to optimize RD without conducting detailed visual analysis of discrimination. On the other hand, complicated visual designs are sometimes inevitable for the analysis of complicated data [62, 11]. Even with easy-to-understand encodings, a simple visualization can be hard to use when it fails to support the required analysis. How to strike a good balance between designing intuitive visualizations and accomplishing complex analysis tasks is still an open question in the field of visual analytics and requires further investigation.

In this paper, we proposed a novel set visualization, RippleSet, to assist the analysis of discrimination. RippleSet supplements the attribute matrix by illustrating the discrimination distribution among data items, especially these belong to intricately intertwined sets. While the current evaluation shows the utility of DiscriLens as a whole, it fails to demonstrate the contribution of RippleSet to the effectiveness of DiscriLens. Apart from discrimination analysis, RippleSet also shows the potential to stand alone and be applied for more general tasks. Unfortunately, due to page limits and our focus on discrimination analysis, we are unable to provide a stand-alone evaluation of RippleSet in this paper.

In DiscriLens, we allow users to customize the definition of discrimination and support the integration of human domain knowledge. While this feature is regarded as a strength by the interviewed experts, we also admit that user customization can act as a double-edged sword in discrimination analysis.

On the one hand, user customization enables the integration of domain knowledge. Since the definition of discrimination differs across domains, the involvement of domain knowledge enables a more comprehensive analysis. On the other hand, additional bias can be potentially imposed by the subjective choices of the analysts. Different customization may lead to different analysis results. One possible solution is to support what-if analysis and cross-validate the analysis results of different customization. Analysts can test and compare different customization before drawing the final conclusion. Similar methods are commonly used in verifying subjective analysis.

In this work, we do not consider the hidden attributes in the causal graph(i.e., attributes that cannot be explicitly observed). We assume the key attributes can capture the main model decisions and set a threshold of the risk difference to allow oscillations caused by hidden attributes. However, the protected or resolving attributes may not be included in the dataset, and this assumption sometimes can fail. In future work, we plan to take the hidden attributes into consideration and offer more comprehensive explanations of the model predictions.

Meanwhile, the current version of DiscriLens only supports the analysis of one protected attribute and requires users to define the protected group as input. When there are many groups (e.g., females, blacks, LGBTs) qualified for special protection by a law, users cannot analyze multiple protected attributes at the same time and need to examine different attributes separately one by one.

Compared with other explainable models (e.g., decision tree), our study provides limited support in explaining the identified discrimination. Therefore, an interesting future direction is to integrate discrimination analysis with model interpretation and diagnosis techniques, which can help us track the origin of algorithmic discrimination and better understand model behaviors.