May the Force Be with Your Copy Mechanism: Enhanced Supervised-Copy Method for Natural Language Generation

Natural language generation is a task in any setting in which we generate new text, such as machine translation, summarization, and dialogue systems (Gatt and Krahmer, 2018). Approaches using neural networks known as sequence-to-sequence (seq2seq) have achieved reliable results on such tasks (Sutskever et al., 2014). However, basic seq2seq models exhibit weaknesses in dealing with rare and out-of-vocabulary (OOV) words. Vinyals et al. (2015) resolved this problem by copying words from the source sequence and directly inserting them into the output generation. This idea has been successfully applied to abstractive summarization tasks (Gulcehre et al., 2016; Gu et al., 2016; See et al., 2017; Xu et al., 2020) and has been further adopted by Wiseman et al. (2017) and Puduppully et al. (2019a) to help the model copy correct values in data-to-text generation tasks.

Unfortunately, the generation results obtained from copying mechanisms can be suboptimal. Zhou et al. (2018) observed that some unrelated words appear unexpectedly in the middle of the phrase, or the phrase is not copied completely and some words are missing. See et al. (2017) and Gehrmann et al. (2018b) reported the lack of abstractness in the excessive copying of source words, resulting in insufficient novel expressions of generated summarization. For example, the CNN/DailyMail dataset (Nallapati et al., 2016) contains approximately 17% of novel words on gold summaries, but most copy models create less than 1%. Moreover, most data-to-text models based on copy mechanisms still suffer from poor factual inconsistencies. Accurately conveying the facts is especially important in informational communication, such as news, and low levels of veracity make these models unreliable and useless in practice (Kryscinski et al., 2020). Wiseman et al. (2017) indicated that there is a significant gap between neural models and template systems in terms of generating text containing factual (i.e., correct) records. Despite the efforts of the dedicated research community (Puduppully et al., 2019a; Rebuffel et al., 2020, 2021) there are still many challenges to narrow the gap.

In our experiments, we found that the probability of the copying words crucially affects the decoding process, resulting in fallacious generation when calculated inaccurately. Therefore, we argue that deciding between copying or generating can benefit from distinct guiding. Accordingly, we propose a force-copy method that forces the model to copy every word when it appears in source and target sequences simultaneously so as to increase copy precision. Furthermore, to relieve the lacking abstractness of the generated text, we present the force-copy-unk method, which forces model to copy a word only if it does not exist in the target vocabulary even if it appears in both the source and target sequences.

Our contributions are three-fold:

• •

  We analyzed and compared the characteristics of prior copy models. On the basis of the analysis, we present force-copy and force-copy-unk methods that promote accurate copy of source sequence.

• •

  The force-copy and force-copy-unk methods improve the RG Precision score in data-to-text, which indicates that the text generation based on the source data is conducted more correctly.

• •

  The force-copy-unk achieves an improvement in copy precision, which not only increases the ROUGE score but also shows better abstraction in abstractive summarization task.

As our work builds on the prior copy mechanism, we introduce Pointer-Generator Network (See et al., 2017) as a baseline. In this study, the source text $x$ is fed into a bidirectional LSTM (BiLSTM) encoder. However Klein et al. (2020) reported that there is no significant difference in the copying performances of Transformer (Vaswani et al., 2017) encoder compared to that of BiLSTM; thus we replaced it with a Transformer encoder producing a sequence of encoded hidden states $h_{i}$. At each timestep $t$, the decoder receives the representation of the previously generated word to produce the decoder hidden states $s_{t}$. From the hidden states, a context vector $c_{t}$ is calculated based on the attention distribution (Bahdanau et al., 2015) as follows:

$$e_{t,i}=v^{T}\mathrm{tanh}(W_{h}h_{i}+W_{s}s_{t})$$

(1)

$$\alpha_{t}=\mathrm{softmax}(e_{t})$$

(2)

$$c_{t}=\sum\mathop{}_{\mkern-5.0mui}\alpha_{t,i}h_{i}$$

(3)

where $v$, $W_{h}$ and $W_{s}$ are learnable parameters.

The vocabulary distribution $P_{vocab}$ over all words in the target vocabulary is computed from $s_{t}$, $c_{t}$, and the learnable parameters $W_{v}^{{}^{\prime}}$, $W_{v}$, $b$ and $b^{\prime}$:

$$P_{vocab}=\mathrm{softmax}(W^{\prime}_{v}(W_{v}[s_{t};c_{t}]+b)+b^{\prime})$$

(4)

The generation probability $p_{gen}$ is used as a soft switch to either generate from vocabulary or copy from the source words by sampling from the attention distribution $\alpha_{t}$. Additionally, $p_{gen}$ for timestep $t$ is calculated from the context vector $c_{t}$, decoder state $s_{t}$, and decoder input $y_{t}$:

$$p_{gen}=\mathrm{sigmoid}(w_{h}^{T}c_{t}+w_{s}^{T}s_{t}+w_{y}^{T}y_{t}+b_{ptr})$$

(5)

where vectors $w_{h}^{T}$, $w_{s}^{T}$, $w_{y}^{T}$ and the scalar bias $b_{ptr}$ are learnable parameters. Let a word $w$ be an element in the set of extended vocabulary, which refers to the union of the target vocabulary and all words appearing in the input sequence; then, the final probability distribution $P(w)$ is computed as:

$$P(w)=p_{gen}P_{vocab}(w)+(1-p_{gen})\textstyle{\sum_{i:w_{i}=w}\alpha_{t,i}}$$

(6)

If $w$ is an OOV word, then $P_{vocab}(w)$ is zero. Similarly, if $w$ does not exist in the input sequence, then $\sum_{i:w_{i}=w}\alpha_{t,i}$ is zero. During training, the loss at timestep $t$ is the negative log-likelihood of the target word $w_{t}^{*}$ for that timestep:

$$loss^{t}=-\mathrm{log}P(w_{t}^{*})$$

(7)

We next consider techniques for incorporating supervised learning of the copying decision into the model. To identify the number of occasions at which every copy and generation occurs, Eq. (6) is split into three cases: (i) No word is copied from the source sequence, i.e., words are sampled from the vocabulary probability distribution. In this case, $p_{gen}$ is guided to take higher value during the training time; (ii) The word copied from the source sequence does not exist in the target vocabulary. In this case, $p_{gen}$ is guided to take lower value during the training time; (iii) The word copied from the source sequence exists in the target vocabulary. In this case, $p_{gen}$ could have a scalar range in $[0,1]$ and is learned implicitly.

We augment the decoder by supervising a soft switch $p_{gen}$ that determines whether the model generates or copies. First, we re-define the loss function as

$$loss^{t}=loss_{vocab}^{t}+loss_{attn}^{t}+loss_{p_{gen}}^{t}$$

(8)

where $loss_{vocab}^{t}$ is the maximum likelihood estimation (MLE), which is used as a standard training criterion in the sequence-to-sequence model:

$$loss_{vocab}^{t}=-\mathrm{log}(P_{vocab}(w_{t}^{*}))$$

(9)

It should be noted that $P_{vocab}(w_{t}^{*})$ indicates the probability of generating the unknown token if $w_{t}^{*}$ is OOV. Similarly to the pointer network (Vinyals et al., 2015), we use attention distribution as a guide for copying, and train it to minimize the negative log-likelihood:

$$loss_{attn}^{t}=-\mathrm{log}(\textstyle{\sum_{i:w_{i}=w}\alpha_{t,i}})$$

(10)

Finally, we adopt $loss_{p_{gen}}^{t}$ to train $p_{gen}$ explicitly. In fact, the optimal way is to feed the gold copy answer for every target word; However, there rarely exist supervised data for this task. Hence, we suggest two methods leveraging the source sequences and target vocabularies.

Force-copy Given a source sequence $X=(x_{1},x_{2},...,x_{T_{x}})$, if a target word $w_{t}^{*}$ appears in $X$, $w_{t}^{*}$ can be a copy-candidate. For example, the word ’the’ and ’COVID-19’ are both copy-candidates in Figure 2. In the force-copy model, we assume that every copy-candidate word is copied from the source sequence. Therefore, $p_{gen}$ can be derived from the loss function:

$$loss_{p_{gen}}^{t}=\left\{\begin{array}[]{l@{}l@{\qquad}l}-\mathrm{log}(1-p_{gen})\;if\;w_{t}^{*}\in X\\[3.0pt] -\mathrm{log}(p_{gen})\;otherwise\end{array}\right.$$

(11)

This forces the copy switch $p_{gen}$ to perform a copy all copy-candidate words. However, we do not penalize the generating ability even on copy circumstances. We always train $loss_{vocab}^{t}$ on the loss function regardless of the copy process.
Force-copy-unk Whereas the force-copy model tries to copy every copy-candidate word, it is also possible to generate words from the vocabulary distribution instead of copying if the word exists in the target vocabulary. For example, the copy-candidate word ’the’ could be generated instead of being copied in Figure 2. In this way, we restrict the scope of copy to unknown words in the force-copy-unk model. Therefore, $p_{gen}$ can be derived from the loss function:

$$loss_{p_{gen}}^{t}=\left\{\begin{array}[]{l@{}l@{\qquad}l}-\mathrm{log}(1-p_{gen})\;if\;w_{t}^{*}\in X,\notin V\\[3.0pt] -\mathrm{log}(p_{gen})\;otherwise\end{array}\right.$$

(12)

where $V$ is a set of target vocabularies. This loss function forces the copy switch $p_{gen}$ to copy the word only if it is an unknown token. For other copy-candidate words that are not OOV, we found that it is effective to utilize the copy information as well (see Section 6.1). Inspired by work in guided alignment training for machine translation (Chen et al., 2016), we retain the $loss_{attn}^{t}$ on the loss function to inform the decoder that the word is a copy-candidate, thus inducing copy-like generation.

Abstractive Summarization
Abstractive summarization aims to generate accurate and concise summaries that contain novel words in contrast to extractive summarization that extracts almost entire sentences from the input. Most prior works on abstractive summarization that employed neural networks achieved inspiring results (Rush et al., 2015; Nallapati et al., 2016; See et al., 2017). Especially after Vinyals et al. (2015), Gulcehre et al. (2016) showed dramatic improvement by applying copy mechanisms, it has become a primary module for abstractive summarization. Recent work leveraging such pre-training models (Dong et al., 2019; Song et al., 2019; Lewis et al., 2020; Zhang et al., 2020) has presented impressive advancements in performance when fine-tuned for text generation tasks. Applying the copy mechanism to these pre-training models has extended the success (Xu et al., 2020; Bi et al., 2020).
Data-to-text Generation
Data-to-text generation tasks aim to produce texts from non-linguistic input data (Reiter, 2007), including the generation of weather forecasts (Liang et al., 2009) and biographical content from Wikipedia (Lebret et al., 2016). Specifically, Wiseman et al. (2017) showed that a neural encoder-decoder model powered with the copy mechanism (Gu et al., 2016; Gulcehre et al., 2016) can generate fluent multi-sentence summaries from game statistics without explicit templates or rules. Based on their approach, various promising neural approaches have been proposed for ROTOWIRE tasks (Puduppully et al., 2019a, b; Rebuffel et al., 2020; Iso et al., 2019). Regarding the copy mechanism, Puduppully et al. (2019a, b) and Iso et al. (2019) adopted the same method as Wiseman et al. (2017), and Rebuffel et al. (2020) utilized Pointer-Generator Network (PGNet) (See et al., 2017)¹¹1We analyzed released code (if available) to identify which copy mechanism they employed in case it was not mentioned in their paper..
Pointer / Copy Mechanism
The fundamental structure of the copy mechanism was first introduced in the form of the pointer network (Vinyals et al., 2015). Expanding their work, various approaches combining attention probability and generation probability have been proposed (Miao and Blunsom, 2016; Gu et al., 2016; Gulcehre et al., 2016; Nallapati et al., 2016). In particular, See et al. (2017) proposed PGNet, where two distributions are weighted and merged into a single mixture distribution with the aid of a soft switch mechanism. PGNet has become de facto standard for copy mechanisms in various tasks, including summarization (Paulus et al., 2018; Gehrmann et al., 2018b), data-to-text (Gehrmann et al., 2018a; Rebuffel et al., 2020), and question answering (McCann et al., 2018).

Table 1 shows the characteristics of the copy models. While our models bear a resemblance to models that trains the switch probability, ours are considerably different from theirs in two aspects: (i) Compared to Gulcehre et al. (2016); Nallapati et al. (2016); Chen et al. (2020); Wu et al. (2020), those works train their copy components to activate only for certain words (i.e., OOV words, named entites, keywords, or values of structured data) whereas we activate it for all copy-candidates. Our force-copy-unk model also seems like it trains to copy only for unknown words, but it works for all copy-candidates by maintaining attention loss (see Eq. 10). (ii) We retain the vocabulary loss (see Eq. 9) even if the model copies a word whereas the other models induce a trade-off between copy and generation. Models that has a trade-off between copy and generation definitely weaken the effect of vocabulary loss when they activate the copy process. However, we believe it’s necessary to keep vocabulary loss to prevent mistaken prediction from the vocabulary distribution because the switch probability $p_{gen}$ takes value in scalar range of $[0,1]$, not a binary value.

In this work we analyzed the prior copy models in detail. Based on the analysis, we presented a novel copy mechanisms that leverages Pointer-Generator network. We showed that our models conduct more accurate copy than baselines via data-to-text experiment. In addition, our force-copy-unk model outperformed the baselines in both ROUGE score and the novel $n$-gram score in abstractive summarization task.