We first identify input tokens that carry sensitive information. To be more specific, for an input text $\mathbf{x}=\{x_{1},x_{2},x_{3},\cdots,x_{n}\}$ with $n$ tokens (e.g., biography of a person), we predict the bias label $y_{b}$ (e.g. gender of the person, having $K_{b}$ categories) based on $\mathbf{x}$ with model $f_{b}(\mathbf{x};\theta_{b})$ parameterized by $\theta_{b}$, so that the predicted bias label $\hat{y_{b}}$ is close to ground truth $y_{b}$

$$\hat{y_{b}}=\operatorname*{arg\,max}_{k_{b}\in K_{b}}f_{b}(\hat{y_{b}}=k_{b}|\mathbf{x};\theta_{b}),$$

which is optimized by minimizing the cross-entropy error $\mathcal{L}_{bias}(f(\mathbf{x}),y_{b};\theta_{b})$. We are interested in identifying the tokens that are most predictive for $\hat{y_{b}}$, i.e. bias rationales.

Rationale is defined as a short yet sufficient snippet of an input responsible for the prediction Bastings et al. (2019). Here, we obtain the bias rationale using an extractive framework that includes two modules – an extractor that identifies parts of input as the rationale, and an encoder that makes a prediction only based on the rationale. The extractor and encoder together compose the rationale extraction framework (REF). The proposed rationale comes in the form of a sequence of binary variables, indicating if a particular input token is informative to the task. The extractor and the encoder are jointly trained to minimize the prediction error.

Therefore, to extract bias rationale, we augment $f_{b}$ with the sequence of latent binary variables $\mathbf{z^{b}}=\{z^{b}_{1},z^{b}_{2},z^{b}_{3},\cdots,z^{b}_{n}\}$, $z^{b}_{i}\in\{0,1\}$ Lei et al. (2016), which is optimized to maximize the predictive probability of the correct bias label by regulating the contribution of each token:

	$\displaystyle\mathbf{z^{b}}$	$\displaystyle\sim g_{b}(\mathbf{x}|\phi_{b})$
	$\displaystyle\hat{y_{b}}$	$\displaystyle=\operatorname*{arg\,max}_{k_{b}\in K_{b}}f_{b}(y_{b}=k_{t}|\mathbf{x}\odot\mathbf{z^{b}};\theta_{b})$

where $g_{b}$ is a bias rationale extractor parameterized by $\phi_{b}$, that predicts the probability of how much each token contributes to predict the bias label. We sample the binary vector $\mathbf{z^{b}}$ from $g_{b}$ and $\mathbf{x}\odot\mathbf{z^{b}}$ is treated as the bias rationale. We model $g_{b}$ such that the output of $g_{b}$ satisfies Kuma distribution Bastings et al. (2019) to avoid $\mathbf{z^{b}}$ being non-differentiable.

Bias REF is trained with the following objective and important tokens for predicting bias are selected as bias rationales:

$\displaystyle\mathcal{C}_{b}=\mathcal{L}_{b}(f_{b}(\mathbf{x}\odot\mathbf{z^{b}});\theta_{b})+\lambda_{b}\Omega_{b}(\phi_{b})$

where $\lambda_{b}$ is hyperparameter and $\Omega_{b}$ is a sparsity constraint penalizing the number of selections and translations, making learned rationale concise and sufficient.

Based on the bias rationale obtained so far, we want to influence a predictive model to use input tokens in a debiased way. Elaborately, we want the contribution of the biased tokens to be as minimal as possible for the predictive task. To achieve this, we encourage the predictive model for a task (e.g., profession classification with $K_{t}$ classes) to use informative tokens (task rationales) with minimal bias.

Similar to bias rationale extraction, we train a task REF consists of an extractor $g_{t}$ that generates $\mathbf{z_{t}}=[z_{1}^{t},z_{2}^{t},\cdots,z_{3}^{t}]$, and an encoder $f_{t}$ that makes prediction with extracted rationale $\mathbf{x}\odot\mathbf{z}^{t}$

	$\displaystyle\mathbf{z^{t}}$	$\displaystyle\sim g_{t}(\mathbf{x}|\phi_{t})$
	$\displaystyle\hat{y}_{t}$	$\displaystyle=\operatorname*{arg\,max}_{k\in K_{t}}f_{t}(\hat{y}_{t}=k_{t}|\mathbf{x}\odot\mathbf{z}^{t};\theta_{t})$

where $\hat{y}_{t}$ is the task prediction and $y_{t}$ is the ground truth label ($y_{t}\in C_{t}$). Task rationale is extracted by minimizing the task cross-entropy loss $\mathcal{L}_{t}$ and maintaining the sparsity $\Omega_{t}$, as

$\displaystyle\mathcal{C}_{t}=\mathcal{L}_{t}$

$\displaystyle(\mathcal{F}(\mathbf{x}\odot\mathbf{z^{t}});\theta_{t})+\lambda_{t}\Omega_{task}(\phi_{t})$

However, we would like to modify the task REF to consider bias rationale, and optimize task rationale in such a way that they contain minimal bias. For this, we introduce a debiasing constraint that adds a penalty if a biased token is used as the part of the task rationale, and optimize the task rationale to incur minimal penalty.

Our debiasing constraint should regulate the importance of the biased tokens towards the predictive task. We capture the importance of each token for being biased and being important for the predictive task, using energy scores¹¹1We did not use direct probabilities from REFs since they produce unstable performance as $p(z^{b}_{i}=0)$ and $p(z^{t}_{i}=0)$ may not be independent and may not be summable. See Sectio n 4 for the experimental evidences. . Energy is defined as the negative log-likelihood of the non-selection probability of each token (LeCun et al., 2006). Higher energy indicates stronger importance.

We obtain the task energy for the $i$-th token as:

	$\displaystyle e_{i}^{t}$	$\displaystyle=-\operatorname{log-likelihood}(p(z^{t}_{i}=0))$
		$\displaystyle=-\operatorname{log-likelihood}(1-g_{t}(x_{i}|\phi_{t})),$

where $g_{t}(x_{i}|\phi_{t})$ is the probability for selecting the $i$-th token $x_{i}$ for the task prediction. Similarly, the bias energy for the $i$-th token would be:

$\displaystyle e_{i}^{b}$

$\displaystyle=-\operatorname{log-likelihood}(1-g_{b}(x_{i}|\phi_{b}))$

We construct the debiasing constraint using both task and bias energy for a token. For an $i$-th token that has a high bias energy, we will penalize its importance for the predictive task by decreasing its task energy. In contrast, for tokens with low bias energy, we keep their task energy as it is. This is realized by a debiasing constraint as:

$\displaystyle D(i)=\left\{\begin{aligned} e_{i}^{t}+(e_{i}^{b}&-A)&\text{ if }e_{i}^{b}>A,\\ &0&\text{otherwise}\end{aligned}\right.$

where $A$ is a hyperparameter indicating the bias tolerance threshold ²²2Setting the threshold to the minimum of bias energy values will result in removing all biased tokens, prohibiting using any sensitive information.. This constraint will eventually get rid of highly biased token for being important to the task and use low-bias energy replacements instead, in order to boost the task performance. This modifies our task objective as:

$$\mathcal{C}=\mathcal{C}_{t}+\gamma\sum_{i}^{|\mathbf{x}|}D(i)$$

where $\gamma$ is the hyperparameter.

The pipeline of our algorithm is shown in Figure 2. We first pretrain a bias REF $f_{b}$ by minimizing $\mathcal{C}_{b}$. During the debiasing process, this model is served as a fixed reference model. During debiasing, we then train the task model $f_{t}$ by minimizing $\mathcal{C}$. For classification tasks, $\mathcal{L}_{t}$ is a cross-entropy loss and for generation task, $\mathcal{L}_{t}$ is a language-modeling loss. Hyperparameters and more details on training are provided in Appendix B.

We evaluate our debiasing algorithm on two text classification tasks influenced by gender bias –toxicity detection and profession classification, and an open-ended text generation task influenced by racial bias. We use the Jigsaw Toxicity dataset ³³3https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification for toxicity detection, BioBias dataset De-Arteaga et al. (2019) for profession classification, and BOLD dataset Dhamala et al. (2021) for open-ended generation.

To ensure the optimal trade-off between bias removal and task performance we evaluate our model based on three desiderata: (1) task performance, (2) bias mitigation, and (3) rationale faithfulness.

We present the comparative performances of the baselines and our method for the open-ended generation task in Table 6. While we see that debiasing in generation task is challenging as perplexity (PPL) for all methods are far from that of the ground-truth human-written answers, our method achieves the best bias mitigation as well as best perplexity and BertScore as compared to other debiasing methods. While PPLM is fluent with a good perplexity and mitigates bias reasonably, it has low BertScore indicating low generation quality. We achieve better generation results by using sparser rationales as compared to GPT2 and Probability baselines. While Rerank selects fewest input words as rationales it eventually have poor generation quality showing lack of control on bias exposure to maintain task performance. While the Probability model acted as a strong baseline for classification tasks, for generation task, it performs worse than the GPT2 baseline. We attribute this to the lack of independence assumption between $p(z^{b}_{i}=0)$ and $p(z^{t}_{i}=0)$, as task labels and bias labels appears to be closely related and hence directly minimizing their sum in $D$ might suffer from confounding in some cases. We also notice that both PPLM and our method achieve best faithfulness in terms of sufficiency but we achieve that using sparser rationales and better generation quality.

We compare extracted rationales with two different inputs across different rationale-based debiasing methods for toxicity detection task in Table 7. More examples are provided in the Appendix D.

In the first example, ‘mother’ appears to be in the task rationales for toxicity as often offensive expressions and slangs include the word ‘mother’. On the other hand, ‘mother’ is also highly predictive of gender (female). However, in the current context, ‘mother’ is not indicative of toxicity but only acts as a sensitive token, hence our method penalizes its importance and does not use it for the task prediction after debiasing.

In the second example, ‘lip’ (frequently appears as a part of lipstick) and ‘homosexuals’ appear as indicator for gender as well as predicting toxicity. It is understandable that ‘homosexuals’ strongly indicates toxicity as it regularly appears in homophobic comments. While removing both them will decrease gender bias greatly, something that happens for Rerank baseline, it is not fair to not include ‘homosexuals’ in task rationales. While our method drops ‘lip’ from task rationales after debiasing it still keeps (and fairly so) ‘homosexuals’ in its task rationales thus controlling the bias exposure for a fair and interpretable toxicity prediction.