A Brief Study on the Effects of Training Generative Dialogue Models with a Semantic loss

Data for language generation tasks in goal-oriented dialogue has semantically diverse samples, where the diversity can be observed from the dialogue topics to the utterances used for getting information on specific slot-values from the user. But, in many niche domains, collecting a large high-quality annotated data set is costly, and often a small data set focused on specific tasks Wei et al. (2018); Asri et al. (2017) is used for training. This restricts the model to only learn task-specific frequent contexts and seldom learn semantically similar context due to the lack of sufficient samples Vinyals and Le (2015); Serban et al. (2015); Li et al. (2017); Parthasarathi and Pineau (2018).

Optimizing only on objectives like negative log-likelihood (NLL), and Cross-Entropy (CE) losses foster learning by making the models mimic targets at the token level Dušek et al. (2020). The models, hence, mostly generate only the observable patterns in the targets in training set Huang et al. (2017). This can be attributed to the training procedure being uninformative about the semantic similarity of responses. To better understand, consider Target: Would you like to travel to Paris ?, R1: How about Paris as your destination ?, R2: Would she like to read to me ? . R2 has 4 tokens in the same position as in the target but R1 is semantically similar to the target. However, the NLL/CE loss for predicting R2 will be lower than predicting R1. This is a common occurrence when training a language generation model, and training on a small data set can exacerbate this issue even further.

Word embeddings from large language models like GloVe Pennington et al. (2014) , BERT Devlin et al. (2018) or fastText Bojanowski et al. (2017) have been shown to have nice properties that preserve some of the linguistic structures Sinha et al. (2020) that help in understanding semantic and temporal structures in dialogue. We make use of the semantics in the large word embeddings by computing a distance heuristic between the sampled text from model distribution and the target during training. This auxiliary semantic loss ¹¹1https://github.com/ppartha03/Semantic-Loss-Dialogue-Generation encourages the model in generating sentences that are similar to the target and thereby potentially diversifying the model responses. Although the results are on dialogue generation tasks, the results are comparable to any broad conditional language generation tasks like caption generation Vinyals et al. (2015), text summarization Luhn (1958) and others Gatt and Krahmer (2018).

Our contributions in the paper are:

• •

  Comprehensively evaluate the proposed semantic loss on two differently sized data sets.

• •

  Show that minimizing a semantic loss on the sampled responses as a training objective improves text generation diversity in limited data setting.

• •

  Show that language model embeddings are useful as semantic loss than word embedding initialization.

In an encoder-decoder architecture, the encoder neural network Lang et al. (1990) encodes a textual summary of previous utterance exchanges between a user and an agent, $H_{i-1}$, and the current user utterance $u_{i}$. The encoded summary is used by a decoder network to generate the corresponding agent response ($a^{*}_{i}=(w_{1}^{i},w_{2}^{i},\ldots,w_{T}^{i})$).

Language generation models are mostly trained with NLL objective as defined in Equation 1,

where $T$ is the number of tokens generated in the response ($a^{*}_{i}$), $w^{i}_{t}$ is the $t$-th word in the $i$-th utterance, and $w^{i}_{<t}$ denote tokens generated till step $t$.

We introduce training with a semantic loss computed with word embeddings from any trained language model. The semantic loss to be minimized is computed in three steps: (1) $a_{i}^{sampled}=(w_{1}^{i},w_{2}^{i},\ldots,w_{T^{\prime}}^{i})$ is generated by sampling tokens from decoder’s distribution over the vocabulary at every step. (2) Average the word vectors of the sampled ($\hat{b}^{a^{sampled}_{i}}$) and ground truth responses ($\hat{b}^{a_{i}}$) with the embeddings from large language models like BERT, GloVe or fastText. Then, compute L2 distance between the two as shown in Equation 2.

(3) Minimize $d^{i}_{SEM}$ calculated with the non-differentiable sampling operation, we use REINFORCE Williams (1992) to compute $\mathbb{L}_{SEM}$ (Equation 3).

where $T^{\prime}$ is the number of tokens in $a_{i}^{sampled}$ and $r(b)$ is the reward baseline computed with average over a moving window of previous rewards to reduce the variance. The model minimizes $\mathbb{L}_{Train}$ as shown in Equation 4.

where $\alpha\in\mathbb{R}^{+}$ is a hyperparameter to specify the strength of the regularization by $\mathbb{L}_{SEM}$, the optimal value for $\alpha$ depends on the data set. Note: $\mathbb{L}_{Train}$ prefers $R1$ over $R2$ from the example in Section 1, unlike $\mathbb{L}_{MLE}$.

We experiment on two differently sized data sets – Frames Asri et al. (2017) and MultiWoZ 2.0 Budzianowski et al. (2018) – which are relatively small and large. We compute $\mathbb{L}_{SEM}$ using the commonly used language model embeddings BERT-Base Devlin et al. (2018), GloVe Pennington et al. (2014) and fastText Bojanowski et al. (2017) to compare the benefit of using different embeddings.

Evaluation Metrics: We measure the performance on overlap based metric BLEU Papineni et al. (2002); and diversity in the generated text by computing the fraction of distinct-1 and distinct-2 grams, similar to Welleck et al. (2019); Li et al. (2015), on validation set. Also, as a proxy to evaluate generalization to generating n-grams that the decoder was never trained to, we measure the fraction of bigrams generated by the model during validation that were not in the training targets, as % Unseen. Also, to measure the effects of minimizing the semantic loss on language quality, we perform human evaluation for comparing the different training techniques. Further we compare the improvements in diversity between using BERT for initialization of word embeddings and using it in a semantic loss objective.

Training with a semantic loss has a positive effect in a smaller data set and that reflects on the model’s improvement in diversity measuring metrics. But, the semantic loss was not very effective in a large data set due to the lack of diversity within and a hard bias dictated by the samples in the data set. The results obtained in the paper shows that training with semantic loss can be effective in low data setting.

We would like to acknowledge Compute Canada and Calcul Quebec for providing computing resources used in this work. The authors would also like to thank members of Chandar Research Lab, Mila for helping with the code reviews and reviewing the manuscripts. Sarath Chandar and Joelle Pineau are supported by Canada CIFAR AI Chair, and Sarath Chandar is also supported by an NSERC Discovery Grant.