Language Model Detoxification in Dialogue with
Contextualized Stance Control

Large pretrained Language Models, such as GPT2 Radford et al. (2019), can produce coherent, almost human-like texts, but they are prone to generating offensive language, which hinders their safe deployment Gehman et al. (2020). An extensive body of work has focused on detoxifying pretrained LMs Dathathri et al. (2020); Krause et al. (2020); Qian et al. (2022). However, it can be more complicated when the LMs are applied to downstream Natural Language Generation (NLG) tasks, such as dialogue response generation. When applied in dialogue, the uncontrolled models tend to generate toxic content and in addition to explicitly offensive utterances, Baheti et al. (2021) suggest that these models can also implicitly insult a group or individual by aligning themselves with an offensive statement, as shown in Figure 1.

Therefore, to detoxify a pretrained LM applied in dialogue, the stance of the generated response needs to be taken into consideration. In a normal dialogue, we do not need to control the stance, but if the user inputs offensive language, the model should not respond with a positive stance. In other words, the eligible stance is context-dependent and we need to consider the dialogue context.

One straightforward solution is to design a control flow with a binary offensive language classifier, where the dialogue context is taken as input for the classifier. If the context contains offensive language, an NLG model with both toxicity control and stance control is used for response generation. We would like the self-toxicity to be low and the stance not to be supportive. On the other hand, if the context does not contain offensive language, the stance does not need to be controlled, so another NLG model with only toxicity control is used for response generation. However, this Classify-then-Generate framework has several limitations. First, it requires training a classifier and controlled NLG models separately, introducing additional model parameters. Second, its performance relies heavily on the classifier, so the performance of this classifier can be a bottleneck.

To address these limitations, we propose a novel method to do context-dependent control, where the offensive language classification is learned implicitly together with the stance control, instead of being learned explicitly by a classifier. Following Li and Liang (2021) and Qian et al. (2022), we use prefix, a small continuous vector prepended to the LM, to achieve controllability, and we further introduce hierarchical prefixes for contextualized control. More specifically, meta prefixes are introduced to control the underlying LM to generate the desired stance prefix according to the dialogue context, which is then combined with the toxicity prefix to guide the response generation. Therefore, the model can be trained end to end, without the bottleneck of a classifier. Besides the Language Modeling loss, we introduce two novel training loss terms to push the model to learn about the context-dependent control strategy. Experimental results show that our method effectively controls the stance according to the offensiveness of the user utterance while keeping the self-toxicity at a low level. Compared with the baselines, our controlling method has significantly less effect on the stance of the generations when the input user utterance is not offensive and when the input user utterance is offensive, our method achieves a lower support stance score.

To conclude, our main contributions are:

• •

  We propose a novel control framework that combines context-dependent and context-independent control utilizing hierarchical prefixes.

• •

  We introduce novel contrastive training objectives to guide the meta prefixes to learn the control strategy implicitly.

• •

  Experiments show that our proposed method can effectively learn the contextualized stance control while keeping a low self-toxicity of the NLG model.

To reduce the offensive content generated by the LMs, previous research on offensive language detection can be utilized to filter out undesired generations.

Given a user utterance $c$, our goal is to guide the generation model to deliver a contextually safe response, which includes a context-independent attribute: self-toxicity, and a context-dependent attribute: stance. Each example in the training dataset $X$ is a tuple of ($c$, $r$, $t_{c}$, $t_{r}$, $s_{r}$), where $c$ is the user utterance text, $r$ is the response to the user utterance, $t_{c}$ and $t_{r}$ are the offensiveness annotations of $c$ and $r$ respectively, and $s_{r}$ is the stance annotation of the response $r$. Following Li and Liang (2021) and Qian et al. (2022), we use prefix, a small continuous vector prepended to the LM’s hidden representations, to control the generation. Note that during the training or the application of the prefixes, the parameters of the underlying LM are kept frozen, so only the prefixes need to be optimized and stored. Since the self-toxicity control is context-independent and offensiveness annotations are available for training, we first train the toxicity control prefixes, denoted as $H_{\beta}$, following the supervised method in Qian et al. (2022). $H_{\beta}$ consists of two prefixes: $h_{\beta}^{0}=H_{\beta}[0,:,:]$ and $h_{\beta}^{1}=H_{\beta}[1,:,:]$. $h_{\beta}^{0}$ corresponds to non-offensive text and $h_{\beta}^{1}$ is the opposite. Both $h_{\beta}^{0}$ and $h_{\beta}^{1}$ are vectors of dimensions $M\times D$, where M is the length of a prefix and $D$ is the size of the hidden dimension (see Appendix A for detailed explanation).

However, the controlled generation model should avoid not only generating a response that is offensive by itself, but should also avoid generating a response that supports an offensive user utterance. More specifically, there are four cases of training examples in contextualized stance control:

• •

  Case 1 $t_{c}=0$, $s_{r}=0$: The user utterance is not offensive. The response $r$ does not support the user utterance. It satisfies our stance requirement.

• •

  Case 2 $t_{c}=0$, $s_{r}=1$: The user utterance is not offensive. The response supports the user utterance. It satisfies our stance requirement.

• •

  Case 3 $t_{c}=1$, $s_{r}=0$: The user utterance is offensive, but the response does not support it. Our stance requirement is still satisfied.

• •

  Case 4 $t_{c}=1$, $s_{r}=1$: The user utterance is offensive, and the stance of the response is supportive. Our stance requirement is violated.

Note that the offensiveness annotations of the user utterances are not available during evaluation, so the model needs to learn it along with the controllability. One way to learn offensive language detection is to train a binary classifier in an explicit way. However, the errors from the detector will propagate to the generated responses (Section 4.2). Instead of learning offensive language detection explicitly, we propose to learn it in an implicit way along with stance control.

We introduce another set of prefixes, meta prefixes $H_{\alpha}$, to achieve contextualized controllability. Meta prefixes are trained to generate the stance control prefix according to the user utterance, which is then combined with the toxicity control prefix mentioned above to guide the generation, as shown in the upper part of Figure 2. Same as $H_{\beta}$, $H_{\alpha}$ is of dimension $2\times M\times D$. $h_{\alpha}^{0}=H_{\alpha}[0,:,:]$ indicates that the stance of the response meets our requirement (Case 1, 2, 3 above), while $h_{\alpha}^{1}=H_{\alpha}[1,:,:]$ means that the stance of the response violates our requirement (Case 4). More formally, given the annotations $t_{c}$ and $s_{r}$, we define a binary variable meta prefix index $m_{r}$ as follows: $m_{r}=1$ if and only if $t_{c}=1$ and $s_{r}=1$. In all other cases, $m_{r}=0$.

Given a training example, we first infuse the corresponding prefixes with the stance and offensiveness attributes by encouraging them to reconstruct the response $r$. As illustrated in Figure 2, according to $m_{r}$, we select the corresponding meta-prefix $h_{\alpha}^{m_{r}}$ and prepend it to the user utterance $c$ as input for the LM. The output of the LM is a generated prefix of dimension $M\times D$. Then the generated prefix is combined with the toxicity prefix $h_{\alpha}^{t_{r}}$ by element-wise addition. The resultant prefix is appended to the user utterance $c$ to guide the LM to generate the response $r$. Therefore, the first part of the training loss is the Language Modeling loss $\mathcal{L}_{LM}$.

$$\mathcal{L}_{LM}=-\sum_{t=1}^{T}\log p(r_{t}|r_{<t},c,t_{r},m_{r})$$

(1)

The computation of $\log p(r_{t}|r_{<t},c,t_{r},m_{r})$ is parameterized as $\log p_{\alpha,\beta,\gamma}(r_{t}|r_{<t},c,h_{\alpha}^{m_{r}},h_{\beta}^{t_{r}})$, where $\gamma$ is the set of fixed LM parameters, and $\alpha,\beta$ represent learnable prefix parameters.

Although $\mathcal{L}_{LM}$ infuses the corresponding prefixes with the stance and offensiveness attributes, the offensiveness annotation of the user utterance $t_{c}$ is ignored in $\mathcal{L}_{LM}$ and thus the meta prefixes are not pushed to learn the offensiveness and rely on it to control the stance. To address this problem, we introduce two additional contrastive loss terms utilizing the annotation $t_{c}$.

In order to push the meta prefixes to learn about the stance requirement when the user utterance is offensive, we add a stance contrastive loss $\mathcal{L}_{s}$ to differentiate between Case 3 and Case 4 stated above. As shown in Figure 2, each offensive user utterance in the training dataset is combined with the two meta prefixes separately, and the distance between two generated prefixes is used to calculate the stance contrastive loss $\mathcal{L}_{s}$.

$$\mathcal{L}_{s}=\mathbbm{1}_{t_{c}=1}\max(m-d_{s},0)^{2}$$

(2)

where $m$ is a pre-set margin, and $\mathbbm{1}$ is the indicator function. $\mathbbm{1}_{t_{r}=1}=1$ if $t_{r}=1$ and $\mathbbm{1}_{t_{r}=1}=0$ if $t_{r}=0$. $\mathcal{L}_{s}$ is only calculated when the input example consists of an offensive user utterance ($t_{c}=1$). $d_{s}$ is the distance between the generated prefixes as in the equation below.

$$d_{s}=\lVert f_{\alpha,\gamma}(h_{\alpha}^{m_{r}},c)-f_{\alpha,\gamma}(h_{\alpha}^{\neg{m_{r}}},c)\rVert_{2}$$

(3)

where $f_{\alpha,\gamma}$ is the function corresponding to the underlying LM controlled by the meta prefixes and $f_{\alpha,\gamma}(h_{\alpha}^{m_{r}},c)$ is the generated prefix given the meta prefix $h_{\alpha}^{m_{r}}$ and the user utterance $c$. $\neg{m_{r}}=1-{m_{r}}$ is the opposite of $m_{r}$. Optimizing $\mathcal{L}_{s}$ pushes the prefix generated given $h_{\alpha}^{m_{r}},c$ and that generated give $h_{\alpha}^{\neg{m_{r}}},c$ to be away from each other by a margin $m$. In other words, it encourages the meta prefixes to learn that when the user utterance is offensive, the two meta prefixes should generate opposite stance prefixes. By combining $\mathcal{L}_{LM}$ and $\mathcal{L}_{s}$, the meta prefixes are pushed to learn that when the user utterance is offensive, $h_{\alpha}^{0}$ is supposed to generate a non-supportive stance prefix, while $h_{\alpha}^{1}$ is supposed to generate a supportive stance prefix.

Since the stance requirement is different when the user utterance is offensive and inoffensive, the meta prefixes also need to learn about the context-dependency and about what stance to achieve when the user utterance is not offensive. We achieve this by introducing another context contrastive loss $\mathcal{L}_{c}$ to differentiate between the aforementioned Case 3 and the union of Case 1, 2. As illustrated in Figure 3, the meta prefix $h_{\alpha}^{0}$ is combined with offensive user utterances and non-offensive user utterances respectively, and the distance between the generated prefixes in these two cases is used to calculate the context contrastive loss.

	$\displaystyle\mathcal{L}_{c}$	$\displaystyle=\max(m-d_{c},0)^{2}$		(4)
	$\displaystyle d_{c}$	$\displaystyle=\lVert\overline{e_{0}}-\overline{e_{1}}\rVert_{2}$		(5)
	$\displaystyle\overline{e_{0}}$	$\displaystyle=\frac{\sum_{X}{\mathbbm{1}_{t_{c}=0}f_{\alpha,\gamma}(h_{\alpha}^{0},c)}}{\sum_{X}\mathbbm{1}_{t_{c}=0}}$		(6)
	$\displaystyle\overline{e_{1}}$	$\displaystyle=\frac{\sum_{X}{\mathbbm{1}_{t_{c}=1}f_{\alpha,\gamma}(h_{\alpha}^{0},c)}}{\sum_{X}\mathbbm{1}_{t_{c}=1}}$		(7)

$\overline{e_{0}}$ is the average of the generated prefix given the meta prefix $h_{\alpha}^{0}$ and a non-offensive user utterance while $\overline{e_{1}}$ is the average of the generated prefix given the meta prefix $h_{\alpha}^{0}$ and an offensive user utterance. Since $h_{\alpha}^{0}$ corresponds to the acceptable stances, $\mathcal{L}_{c}$ teaches the model that using $h_{\alpha}^{0}$ to guide generation, the generated stance prefix should be different when the user utterance is offensive ($t_{c}=1$) and when it is not offensive ($t_{c}=0$), so this loss term pushes the meta prefixes to consider the user utterance and to differentiate between offensive user utterance and inoffensive user utterance implicitly.

The final training loss $\mathcal{L}$ is a weighted sum of the three loss terms described above.

$$\mathcal{L}=\omega_{1}\mathcal{L}_{LM}+\omega_{2}\mathcal{L}_{s}+\omega_{3}\mathcal{L}_{c}$$

(8)

After training, the meta prefix $h_{\alpha}^{1}$ and the toxicity prefix $h_{\beta}^{1}$ do not need to be saved. Only $h_{\alpha}^{0}$ and $h_{\beta}^{0}$ are used to guide generation during evaluation.

In this work, we propose a novel method for contextual detoxification, where a context-dependent attribute: stance, and a context-independent attribute: toxicity, are controlled within a unified hierarchical prefix framework. Experimental results show that our proposed method can successfully guide an NLG model to generate safer responses with the stance taken into consideration. Besides the dialogue detoxification task we experimented with, our proposed framework can be extended to other combinations of the context-dependent and the context-independent control.

Context is important for identifying offensive language, especially for implicit offensive language. In this work, we consider one category of contextual offensive language, where a response supports a previous offensive utterance in the context. Other categories of contextual offensive language, such as sarcasm and circumlocution, are not covered in this work. Future work in this area may cover more types of contextual offensive language. Although experimental results show that our methods can effectively lower the support stance score of the generations given an offensive input, it is not guaranteed that the model with our controlling method will produce a generation with a safe stance.

Our proposed method is intended for context-dependent detoxification with stance control. It can be extended to other combinations of the context-dependent and the context-independent control. However, it is not intended for hallucination or factuality control. After training, the prefixes $h_{\alpha}^{1}$ and $h_{\beta}^{1}$ should be discarded and only $h_{\alpha}^{0}$ and $h_{\beta}^{0}$ should be used for evaluation or application. $h_{\alpha}^{1}$ and $h_{\beta}^{1}$ should not be used to generate offensive language or the responses supporting offensive language. Due to the sensitive nature of this work, examples in Figure 1 and Table 3 contain offensive language. We would like to clarify that the examples shown in this paper do not represent any opinion of the authors.