Transforming Multi-Conditioned Generation from Meaning Representation

Our system uses GPT2-small model as a backbone to generate utterances and is illustrated in Figure 1. GPT2 is an unsupervised pre-trained model with large-scale open-domain corpora of unlabeled text. GPT2 uses only the Transformer decoder and generates sentences from left to right autoregressively without an encoder. GPT2-small has 768-dimensional embedding size, 12 heads, and 12 layers, so the total number of parameters is 117M. Our system has the advantage of starting from knowing the token distribution by using the pre-trained model as the initial state.

As conditions of an utterance generation, $(s_{i},v_{i})$ pairs of a flat MR are given, slots are treated as special tokens. Specifically, the tokens for slots and start are added to vocabulary, and the regular tokens in vocabulary are used for values and end. In order to distinguish between special tokens and regular tokens, special tokens are changed to <SPECIAL TOKEN> to form vocabulary. Inputs are given by concatenating the given $(s_{i},v_{i})$ pairs in order (e.g. <name>, Giraffe, <eatType>, pub, …). We use <name>, <eatType>, <food>, <priceRange>, <customer rating>, <area>, <familyFriendly>, <near> in the fixed order of slots as formed in the E2E dataset, and do not put slots not in the given MR as input. Our system generates utterances by considering the MR as multi-conditions of generation. Through training end-to-end Multi-Conditioned generation, we hope that the model find out the role of a MR.

Transformer Decoder is autoregressively unidirectional from left to right, so the model only sees the previous tokens:

$$\bm{o_{i}}=\textrm{TransformerDecoder}(u_{i}^{1:k},\textit{{\textless{}START\textgreater{}}},s_{i},v_{i})$$

(1)

where $u_{i}^{1:k}$ denotes tokens up to $k^{th}$ tokens of $u_{i}$. $\bm{o_{i}}$ is the output of Transformer Decoder, $k\times h$ tensor, and $h$ is the hidden embedding dimension of the decoder.

To predict the token of the next step $(k+1)$, multiply the $k^{th}$-vector $o_{i}^{k}$ by matrix $M$ as follows:

$$u_{i}^{k+1}=argmax(M(o_{i}^{k}))$$

(2)

where $M$ is a randomly initialized matrix.

Because our system is a one-stage framework that generates utterance directly from flat MRs, sentence planning and surface realization are not considered separately. Our approach shows strong results in 4.2 without additional techniques such as delexicalization, data augmentation, and extra datasets. When testing, it is possible to deal with unseen values by delexicalization, which is described in detail in Section 4.3. In addition, Section 4.4 introduces utterance generation for $(s_{i},v_{i})$ pairs that exist in the real world even if they are not in the training dataset.

We experiment using one V100 16GB GPU in Linux environment on an AWS server. Our system is end-to-end trained with the AdamW (Loshchilov and Hutter, 2019) optimizer for 5 epochs. The initial value of the learning rate is 2e-5 and is adjusted with a linear scheduler. The model is trained so that the output at the current step $(k)$ predicts the token of the next step $(k+1)$ and the loss of the objective function is calculated as:

$$\begin{split}\mathcal{L}(\theta)&=-\sum_{(\bm{M}_{i},v_{i})\in D}(\mathrm{log}\,p(u^{1}|\textit{{\textless{}S\textgreater{}}},s_{i},v_{i})\\ &+\sum_{k=1}\mathrm{log}\,p(u_{i}^{k+1}|u_{i}^{1:k},\textit{{\textless{}S\textgreater{}}},s_{i},v_{i})\\ &+\mathrm{log}\,p(\textit{{\textless{} E \textgreater{}}}|u_{i},\textit{{\textless{}S\textgreater{}}},s_{i},v_{i}))\end{split}$$

(3)

where $u_{i}^{k}$ is the $k^{th}$ tokens of $u_{i}$ and <S>, <E> are start, end token respectively. Since the ground truth given in the dataset is only utterance, the outputs before <S> is entered cannot be used for loss calculation.

During training time, tokens are generated by applying teacher-forcing, and tokens are generated by self-feeding during testing time.

We use the five automatic evaluation metrics used in the E2E Challenge, BLEU (Papineni et al., 2002), NIST (Lin and Och, 2004), METEOR (Denkowski and Lavie, 2014), ROUGE (Lin, 2004) and CIDEr (Vedantam et al., 2015), equally as the basis. Evaluation scripts are provided by challenge organizers ²²2https://github.com/tuetschek/e2e-metrics. We additionally calculate the similarity F1 scores of the two sentences using the BERTscore of RoBERTa model (Liu et al., 2020) provided by the library ³³3https://pypi.org/project/bert-score/. The two sentences entered in the BERTscore library are the generated utterances and human references. The BERTScore metric is task agnostic and, unlike previous metrics, uses importance weighting between contextual embedding. Therefore, it is a common metric that calculates a better correlation by solving the disadvantages of the previous metric. BERTScore is measured only for the systems that provided the output for the test dataset.

Table 5 shows the experimental results of our system and comparison systems. The first section is our model Multi-Conditioned Transformer, which consists of the Transformer decoder.

Our model is not trained on unseen values, so it can have weaknesses in real applications. Therefore, we introduce a zero-shot generation method through Sim-Delexicalization. Table 8 shows an example of this experiment. The value of <familyFriendly> is not treated as unseen value because it only has (yes or no). Our system generates proper utterance through the following two steps for zero-shot generation.

(1) Sim-Delexicalization: The given unseen values are replaced with similar values among the lists of value corresponding to the same slot. In the first example of Table 8, ”Blue Man” is replaced by ”Green Man” and ”2.1 out of 5” is replaced by ”3 out of 5”. In the second example we observe that expensive is replaced by high. Also, taking into account the grammatical aspect, if an unseen value containing ”the” in the <name> and <near> slots are given, the value list containing ”the” is limited as a candidate (and vice versa). There can be several ways to find similar tokens, but we use BERTscore to select the value with the highest score.

(2) Relexicalization: Replaced values are changed back to unseen values in generated utterances. ”Green Man” is deciphered as ”Blue Man” and other values proceed as well.

The generated utterances are of appropriate quality from a human perspective. In the previous study, unlike delexicalization of unseen values to one placeholder, we have the difference of converting to similar values. Changing to one placeholder in the test also has the same risk as in training above, so we used the existing list of values to change it to an appropriate value each time. In other words, our system can generate utterances that are suitably customized for a given $(s_{i},v_{i})$. Rather than using only BERTscore, it may be helpful to find similar words using word embedding techniques such as Glove (Pennington et al., 2014) and FastText (Bojanowski et al., 2017) but this will be left for further study.

The E2E training dataset and test dataset have an average number of 5.52 and 6.91 slots, respectively, and at least 3 slots are given. In other words, the model is not trained when only one or two slots are given and is not considered in the evaluation of the test dataset. However, in the real world, the ability to generate utterances for one or two $(s_{i},v_{i})$ pairs is necessary, because the conditions and conditions on the generation may be insufficient. In general, methods that need to determine the structure of a sentence in advance are difficult to cope with small $(s_{i},v_{i})$ pairs because each value and the entire utterance do not correspond. However, our approach is end-to-end training that does not take into account the structure of the sentence, so Table 9 shows the possibilities for generating small pairs.

Our paper focuses on the E2E dataset, but for the possibility of scaling, we do a simple experiment in the WebNLG challenge task (Colin et al., 2016) similar to the E2E dataset with the same approach. The WebNLG dataset is collected from DBpedia, and the train, development, and test datasets are 6940, 872, and 1862, respectively, and examples are shown in Table 10. It is significantly smaller than the E2E dataset, and MRs consisting of values corresponding to four (category, subject, property, object) slots are given as a condition. In other words, unlike E2E, the WebNLG dataset has 4 fixed slots, but multiple values can be given.

Table 11 shows the experimental results, and the three comparison systems that participated in the challenge (Gardent et al., 2017) are as follows: (1) The baseline is a neural system trained with OpenNMT ⁵⁵5https://opennmt.net/. (2) Melbourne shows the best score for all automatic evaluations in the challenge with an end-to-end LSTM with attention model. Performance is improved by preprocessing entity tagging by collecting information from DBPedia. (3) UPF-FORGe (Mille et al., 2017) is a grammer-based NLG system and has the highest score in human evaluation through rule-based graph-transducers for syntacticization.

The systems are evaluated in the same way as the E2E dataset and our system is better than the baseline but slightly worse than the best systems in many metrics. However, our system shows meaningful results in BERTscore and is a simple method that utilizes a pre-trained model without any rule definition, preprocessing, or other datasets. If we preprocess the data like the comparison systems above, our model can expect better performance.

Section 4.3 and  4.4 experimentally demonstrates that our approach is capable of zero-shot generation, a situation not found in the training. Our system can cope with small $(s_{i},v_{i})$ pairs without considering the structure of sentences, such as the two-stage framework, sentence planning and surface realization. The system is trained to generate appropriate utterances for $(s_{i},v_{i})$ without the delexicalization. Therefore, there is no need to take the ambiguous risk of replacing unseen values with a single delexicalization of placeholders. Instead we introduce sim-delexicalization, which allows the system to reflect unseen values.

The two-stage framework needs to reveal the structure of the sentence, so it is difficult to solve as the number of $(s_{i},v_{i})$ pairs increases. However, our approach is easily extensible for more pairs. In the WebNLG dataset, we show that it is possible to extend multiple values as well. Since only slots are replaced with special tokens and values are used as regular tokens, our system can be trained to learn the utterance corresponding to MRs without limiting the number of pairs.

We also conducted experiments using a larger backbone model, GPT2-large, but the change in performance is small. However, if the model is trained on various and massive data with a complicated sentence structure, it is expected that the experiment results using GPT2-large as a backbone would be good.