Improving Factual Consistency Between a Response and Persona Facts

By using the persona-consistency sub-reward as the only reward function, the policy learns only to generate responses that state information about the persona facts and ignores the topical coherence with its former utterance.

Table 1 illustrates an example dialogue that is conducted with an agent trained with only the persona consistency sub-reward ($R=R_{1}$). The agent always repeats, “i fix airplanes. i fix them.”, no matter what the input message is about. This problem not only produces topically irrelevant responses but also makes the agent look nagging and self-centered in a conversation.

Table 2 illustrates another example dialogue with the agent where it is trained only by persona-consistency sub-reward. The agent keeps repeating “hunting” from the persona to maximize its reward. The NLI model used for $R_{1}$ evaluates the inference relation between a response and a persona and does not capture the topical coherence of the response with its former utterance and language fluency of the response. It is therefore necessary to use $R_{1}$ in combination with topical coherence ($R_{2}$) and language fluency sub-rewards ($R_{3}$ and $R_{4}$), as we propose in our reward function.

We examine various weight sets $(\gamma_{1},\gamma_{2},\gamma_{3},\gamma_{4})$ to balance the contribution of sub-rewards in the complete reward function on the held out set (10% of the PersonaChat training set). Table 3 shows those weights.

The balanced weights give the highest F1 score, suggesting that a combination of sub-rewards leads to responses that are more similar to the human responses.

We also evaluate the use of each sub-reward in isolation, and show the results in Table 4, in comparison with our chosen balanced weights in the bottom line.

For the other metrics, we can see that $\gamma_{1}=1$ maximizes the number of entailments from the persona facts, $\gamma_{3}=1$ minimizes perplexity, and $\gamma_{4}=1$ gives lowest repetition. Besides F1 score, the balanced weights give good performance across perplexity, repetition, and persona consistency. The setups with fewer neutral responses also tend to have more responses that contradict the persona facts, e.g., for $\gamma_{1}=1$. Neutral responses are a trivial way to avoid contradictory responses and the setup with the least contradictions, $\gamma_{3}=1$, has almost no responses that are consistent with the persona facts. The better overall persona consistency is reflected in the highest PC score for $\gamma_{1}=1$ and next highest for the balanced weights, which trades of PC for less repetition, lower perplexity and a higher F1 score.

Figures 1 and  2 show the trend of changes in our reward function during training by REINFORCE and Actor-Critic, respectively. All parameters are the same for the two experiments. We observe that the actor-critic approach converges faster \MMF[as well as]and also is less noisy (has a lower variance) than REINFORCE.

For each sample, we show to each participant a set of persona facts, a dialogue history, and the response generated by one of TransferTransfo-SL and TransferTransfo-RL. We instruct our participants to assess semantic plausibility according to the following objective definition: “grammatical correctness, lowest repetitiveness, and coherence”. The plausibility rates are integer values between 1 and 5, where 5 is most plausible.

To measure persona consistency, we instruct participants as follows:

An answer is considered consistent if:

• •

  It contradicts with neither the dialogue history nor the persona facts;

• •

  It is relevant to any of the given persona facts.

An answer is considered neutral if:

• •

  It contradicts with neither the dialogue history nor the persona facts;

• •

  It is not relevant to any of the given persona facts.

Table 5 presents the distributions of consistency labels for TransferTransfo-RL’s responses given the consistency labels for TransferTransfo-SL’s responses. For the majority of cases whose TransferTransfo-SL’s responses are contradictory or neutral, TransferTransfo-RL generates consistent responses, showing improved factual consistency with persona facts. However, TransferTransfo-RL generates contradictory responses for some cases whose TransferTransfo-SL responses are consistent with their personas. This may be due to errors in the NLI model’s predictions of entailment, hence a more accurate NLI model may improve the quality of the reward function and consequently the consistency of responses. Alternatively, these contradictory responses may receive high rewards from the topic consistency and fluency sub-rewards, which could override $R_{1}$.