Neural Language Generation: Formulation, Methods, and Evaluation

The field of machine translation is focusing on the automatic translation of textual content from one language into another language. The field has undergone major changes in recent years, with end-to-end learning approaches for automated translation based on neural networks replacing conventional phrase-based statistical methods Bahdanau et al. (2014), Wu et al. (2016a). In contrast to statistical models which consist of several sub-components trained and tuned separately, neural machine translation models build and train a single, large neural network end-to-end by feeding it as input textual content in the source language and retrieving its corresponding translation in the target language. Neural machine translation is a typical example of conditional text generation, where the condition encapsulated by the conditional attribute code $c$ is represented by the input sentence in the source language and the goal task is to generate its corresponding translation in the target language. In addition, neural machine translation is also an instance of constrained text generation given that it imposes the constraint to generate text in the target language. Additional constraints can be placed on the inclusion in the target sentence of named entities already present in the source sentence. In what follows we formally define the problem of neural machine translation.

We denote with $V_{s}$ the vocabulary of the source language and with $V_{t}$ the vocabulary of the target language, with $|V_{t}|\approx|V_{s}|$ and $V_{t}\cap V_{s}=\phi$. Let us also denote with with $V_{s}^{*}$ and $V_{t}^{*}$ all possible sentences under $V_{s}$, respectively $V_{t}$. Given a source sentence $X=(x_{1},x_{2},\ldots,x_{l}),X\in V_{s}^{*},x_{i}\in V_{s}$, where $x_{i}$ is the $i^{th}$ word in $X$, $\forall i=1,\ldots,l$, the goal is to generate the distribution over the possible output sentences $Y=(y_{1},y_{2},\ldots,y_{l^{{}^{\prime}}}),Y\in V_{t}^{*},y_{j}\in V_{t}$, where $y_{j}$ is the $j^{th}$ word in $Y$, $\forall j=1,\ldots,l^{{}^{\prime}}$ by factoring $Y$ into a chain of conditional probabilities with left-to-right causal structure using a neural network with parameters $\theta$:

$$p(Y|X;\theta)=\prod_{t=1}^{l^{{}^{\prime}}+1}p(y_{t}|y_{0:t-1},x_{1:l};\theta)$$

(5)

Special sentence delimiters $y_{0}(<$S$>)$ and $y_{l^{{}^{\prime}}+1}(<$E$>)$ are commonly added to the vocabulary to mark the beginning and end of target sentence $Y$. Typically in machine translation the source and target vocabularies consist of the most frequent words used in a language (for eg., top 15,000 words), while the remaining words are replaced with a special $<$UNK$>$ token. Every source sentence $X$ is usually mapped to exactly one target sentence $Y$, and there is no sharing of words between the source sentence $X$ and the target sentence $Y$.

Although neural network based approaches to machine translation have resulted in superior performance compared to statistical models, they are computationally expensive both in training and in translation inference time. The output of machine translation models is evaluated by asking human annotators to rate the generated translations on various dimensions of textual quality, or by comparisons with human-written reference texts using automated evaluation metrics.

Text summarization is designed to facilitate a quick grasp of the essence of an input document by producing a condensed summary of its content. This can be achieved in two ways, either by means of extractive summarization or through abstractive/generative summarization. While extractive summarization Nallapati et al. (2017) methods produce summaries by copy-pasting the relevant portions from the input document, abstractive summarization Rush et al. (2015), Nallapati et al. (2016), See et al. (2017) algorithms can generate novel content that is not present in the input document. Hybrid approaches combining extractive summarization techniques with a a neural abstractive summary generation serve to identify salient information in a document and generate distilled Wikipedia articles Liu et al. (2018b). Characteristics of a good summary include brevity, fluency, non-redundancy, coverage and logical entailment of the most salient pieces of information from the input document(s).

Text summarization is a conditional text generation task where the condition is represented by the given document(s) to be summarized. Additional control codes are used in remainder summarization offering flexibility to define which parts of the document(s) are of interest, for eg., remaining paragraphs the user has not read yet, or in source-specific summarization to condition summaries on the source type of input documents, for eg., newspapers, books or news articles. Besides being a conditional text generation task, text summarization is also a typical example of constrained text generation where the condition is set such that the length of the resulting summary is strictly less than the length of the original document. Unlike machine translation where output length varies depending on the source content, in text summarization the length of the output is fixed and pre-determined. Controlling the length of the generated summary allows to digest information at different levels of granularity and define the level of detail desired accounting for particular user needs and time budgets; for eg., a document can be summarized into a headline, a single sentence or a multi-sentence paragraph. In addition, explicit constraints can be placed on specific concepts desired for inclusion in the summary. Most frequently, named entities are used as constraints in text summarization to ensure the generated summary is specifically focused on topics and events describing them. In addition, in the particular case of extractive summarization, there is the additional constraint that sentences need to be picked explicitly from the original document. In what follows we formally define the task of text summarization.

We consider the input consisting of a sequence of $M$ words $x=(x_{1},x_{2},\ldots,x_{M}),x_{i}\in\mathcal{V_{\mathcal{X}}},i=1,\ldots,M$, where $\mathcal{V_{\mathcal{X}}}$ is a fixed vocabulary of size $|\mathcal{V}_{\mathcal{X}}|$. Each word $x_{i}$ is represented as an indicator vector $x_{i}\in\{0,1\}^{\mathcal{V}_{\mathcal{X}}}$, sentences are represented as sequences of indicators and $\mathcal{X}$ denotes the set of all possible inputs. A summarization model takes $x$ as input and yields a shorter version of it in the form of output sequence $y=(y_{1},y_{2},\ldots,y_{N})$, with $N<M$ and $y_{j}\in\{0,1\}^{\mathcal{V_{\mathcal{Y}}}},\forall j=1,\ldots,N$.

Abstractive / Generative Summarization We define $\mathcal{Y}\subset(\{0,1\}^{\mathcal{V}_{\mathcal{Y}}},\ldots,\{0,1\}^{\mathcal{V}_{\mathcal{Y}}})$ as the set of all possible generated summaries of length $N$, with $y\in\mathcal{Y}$. The summarization system is abstractive if it tries to find the optimal sequence $y^{*},y^{*}\subset\mathcal{Y}$, under the scoring function $s:\mathcal{X}\times\mathcal{Y}\rightarrow\mathcal{R}$, which can be expressed as:

$$y^{*}=\arg\max_{y\in\mathcal{Y}}s(x,y)$$

(6)

Extractive Summarization As opposed to abstractive approaches which generate novel sentences, extractive approaches transfer parts from the input document $x$ to the output $y$:

$$y^{*}=\underset{m\in\{1,\ldots,M\}^{N}}{\arg\max}s(x,x_{[m_{1},\dots,m_{N}]})$$

(7)

Abstractive summarization is notably more challenging than extractive summarization, and allows to incorporate real-world knowledge, paraphrasing and generalization, all crucial components of high-quality summaries See et al. (2017). In addition, abstractive summarization does not impose any hard constraints on the system output other than shorter length and gives the system a lot of freedom to generate suitable content, which in turn results in the system’s ability to fit a wide range of training data. Approaches for neural abstractive summarization build upon advances in machine translation. Attention mechanisms Bahdanau et al. (2014) and pointer networks Vinyals et al. (2015b) are used to focus on specific parts of the source document and copy input entities. Nevertheless, a common limitation of current abstractive summarization models is their tendency to copy long passages from the source document as opposed to generating novel content. Consequently, the word overlap between the source document and the generated abstractive summary is generally high See et al. (2017), Kryściński et al. (2018).

Very related in nature to the task of text summarization is the problem of text compression, which takes as input a text document and aims to produce a short summary of it by deleting the least critical information, while retaining the most important ideas and preserving sentence fluency. Sentence compression is referred in the literature as a “scaled down version of the text summarization problem”Knight and Marcu (2002), and is of practical importance in many downstream applications including text summarization, subtitle generation and displaying text on small screens Mallinson et al. (2018). Similar to text summarization, text compression is both a conditional text generation and constrained text generation task. The condition is represented by the input document for which the text compression system needs to output a condensed version. The task is also constrained text generation given the system needs to produce a compressed version of the input strictly shorter lengthwise. In addition, there can be further constraints specified when the text compression output is desired to be entity-centric.

We denote with $C_{i}=\{c_{i1},c_{i2},\ldots,c_{il}\}$ the set of possible compression spans and with $y_{y,c}$ a binary variable which equals 1 if the $c^{th}$ token of the $i^{th}$ sentence $\hat{s_{i}}$ in document $D$ is deleted, we are interested in modeling the probability $p(y_{i,c}|D,\hat{s_{i}})$. Following the same definitions from section 2.4.2, we can formally define the optimal compressed text sequence under scoring function $s$ as:

$$y^{*}=\underset{m\in\{1,\ldots,M\}^{N},m_{i-1}<m_{i}}{\arg\max}s(x,x_{[m_{1},\dots,m_{N}]})$$

(8)

Shorter paraphrases are generated through end-to-end neural networks in Filippova et al. (2015). Unsupervised models based on denoising autoencoders wihout the need for paired corpora are used in Fevry and Phang (2018). The level of compression is computed as the character length ratio between the source sentence and the target sentence Martin et al. (2019b). Sentence compressions are identified by constituency parsing and scored by neural models for text summarization Xu and Durrett (2019). Factors directly impacting the generated summary complexity where are the compression rate, the summarization technique and the nature of the summarized corpus Vodolazova and Lloret (2019).

Current datasets, models and evaluation metrics for text summarization are considered not robust enough Kryscinski et al. (2019). Shortcomings include uncurated automatically collected datasets, models that overfit to biases in the data and produce outputs with little diversity, as well as non-informative evaluation metrics weakly correlated with human judgements.

Text simplification is designed to reduce the lexical and syntactic complexity of text, while preserving the main idea and approximating the original meaning. The goal of text simplification systems is to make highly specialized textual content accessible to readers who lack adequate literacy skills, such as children, people with low education, people who have reading disorders or dyslexia, and non-native speakers of the language. In the literature text simplification has been addressed at multiple levels: i) lexical simplification Devlin (1999) is concerned with replacing complex words or phrases with simpler alternatives; ii) syntactic simplification Siddharthan (2006) alters the syntactic structure of the sentence; iii) semantic simplification Kandula et al. (2010), sometimes also known as explanation generation, paraphrases portions of the text into simpler and clearer variants. More recently, end-to-end models for text simplification attempt to address all these steps at once.

Text simplification is an instance of conditional text generation given we are conditioning on the input text to produce a simpler and more readable version of a complex document, as well as an instance of constrained text generation since there are constraints on generating simplified text that is shorter in length compared to the source document and with higher readability level. To this end, it is mandatory to use words of lower complexity from a much simpler target vocabulary than the source vocabulary. We formally introduce the text simplification task below.

Let us denote with $V_{s}$ the vocabulary of the source language and with $V_{t}$ the vocabulary of the target language, with $|V_{t}|\ll|V_{s}|$ and $V_{t}\subseteq V_{s}$. Let us also denote with with $V_{s}^{*}$ and $V_{t}^{*}$ all possible sentences under $V_{s}$, respectively $V_{t}$. Given source sentence $X=(x_{1},x_{2},\ldots,x_{l}),X\in V_{s}^{*},x_{i}\in V_{s}$, where $x_{i}$ is the $i^{th}$ word in $X$, $\forall i=1,\ldots,l$, the goal is to produce the simplified sentence $Y=(y_{1},y_{2},\ldots,y_{l^{{}^{\prime}}}),Y\in V_{t}^{*},y_{j}\in V_{t}$, where $y_{j}$ is the $j^{th}$ word in $Y$, $\forall j=1,\ldots,l^{{}^{\prime}}$ by modeling the conditional probability $p(Y|X)$. In the context of neural text simplification, a neural network with parameters $\theta$ is used to maximize the probability $p(Y|X;\theta)$.

Next we highlight differences between machine translation and text simplification. Unlike machine translation where the output sentence $Y$ does not share any common terms with the input sentence $X$, in text simplification some or all of the words in $Y$ might remain identical with the words in $X$ in cases when the terms in $X$ are already simple. In addition, unlike machine translation where the mapping between the source sentence and the target sentence is usually one-to-one, in text simplification the relation between the source sentence and the target sentence can be one-to-many or many-to-one, as simplification involves splitting and merging operations Surya et al. (2018). Furthermore, infrequent words in the vocabulary cannot be simply dropped out and replaced with an unknown token as it is typically done in machine translation, but they need to be simplified appropriately corresponding to their level of complexity Wang et al. (2016a). Lexical simplification and content reduction is simultaneously approached with neural machine translation models in Nisioi et al. (2017), Sulem et al. (2018c). Nevertheless, text simplification presents particular challenges compared to machine translation. First, simplifications need to be adapted to particular user needs, and ideally personalized to the educational background of the target audience Bingel (2018), Mayfield et al. (2019). Second, text simplification has the potential to bridge the communication gap between specialists and laypersons in many scenarios. For example, in the medical domain it can help improve the understandability of clinical records Shardlow and Nawaz (2019), address disabilities and inequity in educational environments Mayfield et al. (2019), and assist with providing accessible and timely information to the affected population in crisis management Temnikova (2012).

Style transfer is a newly emerging task designed to preserve the information content of a source sentence while delivering it to meet desired presentation constraints. To this end, it is important to disentangle the content itself from the style in which it is presented and be able to manipulate the style so as to easily change it from one attribute into another attribute of different or opposite polarity. This is often achieved without the need for parallel data for source and target styles, but accounting for the constraint that the transferred sentences should match in style example sentences from the target style. To this end, text style transfer is an instance of constrained text generation. In addition, it is also a typical scenario of conditional text generation where we are conditioning on the given source text. Style transfer has been originally used in computer vision applications for image-to-image translation Gatys et al. (2016), Liu and Tuzel (2016), Zhu et al. (2017), and more recently has been used in natural natural language processing applications for machine translation, sentiment modification to change the sentiment of a sentence from positive to negative and vice versa, word substitution decipherment and word order recovery Hu et al. (2017).

The problem of style transfer in language generation can be formally defined as follows. Given two datasets $X_{1}=\{x_{1}^{(1)},x_{1}^{(2)},\ldots,x_{1}^{(n)}\}$ and $X_{2}=\{x_{2}^{(1)},x_{2}^{(2)},\ldots,x_{2}^{(n)}\}$ with the same content distribution but different unknown styles $y_{1}$ and $y_{2}$, where the samples in dataset $X_{1}$ are drawn from the distribution $p(x_{1}|y_{1})$ and the samples in dataset $X_{2}$ are drawn from the distribution $p(x_{2}|y_{2})$, the goal is to estimate the style transfer functions between them $p(x_{1}|x_{2};y_{1},y_{2})$ and $p(x_{2}|x_{1};y_{1},y_{2})$. According to the formulation of the problem we can only observe the marginal distributions $p(x_{1}|y_{1})$ and $p(x_{2}|y_{2})$, and the goal is to recover the joint distribution $p(x_{1},x_{2}|y_{1},y_{2})$, which can be expressed as follows assuming the existence of latent content variable $z$ generated from distribution $p(z)$:

$$p(x_{1},x_{2}|y_{1},y_{2})=\int_{z}p(z)p(x_{1}|y_{1},z)p(x_{2}|y_{2},z)dz$$

(9)

Given that $x_{1}$ and $x_{2}$ are independent from each other given $z$, the conditional distribution corresponding to the style transfer function is defined:

$$\begin{split}p(x_{1}|x_{2};y_{1},y_{2})&=\int_{z}p(x_{1},z|x_{2};y_{1},y_{2})dz\\ &=\int_{z}p(x_{1}|y_{1},z)p(x_{2}|y_{2},z)dz\\ &=\mathbb{E}_{z\sim p(z|x_{2},y_{2})}[p(x_{1}|y_{1},z)]\end{split}$$

(10)

Models proposed in the literature for style transfer rely on encoder-decoder models. Given encoder $E:X\times Y\rightarrow Z$ with paramters $\theta_{E}$ which infers the content $z$ and style $y$ for a given sentence $x$, and generator $G:Y\times Z\rightarrow X$ with parameters $\theta_{G}$ which given content $z$ and style $y$ generates sentence $x$, the reconstruction loss can be defined as follows:

$$\begin{split}\mathcal{L}_{\text{rec}}=&\mathbb{E}_{x_{1}\sim X_{1}}[-\log p_{G}(x_{1}|y_{1},E(x_{1},y_{1}))]+\\ &\mathbb{E}_{x_{2}\sim X_{2}}[-\log p_{G}(x_{2}|y_{2},E(x_{2},y_{2}))]\end{split}$$

(11)

Latent VAE representations are manipulated to generate textual output with specific attributes, for eg. contemporary text written in Shakespeare style or improving the positivity sentiment of a sentence Mueller et al. (2017). Style-independent content representations are learnt via disentangled latent representations for generating sentences with controllable style attributes Shen et al. (2017), Hu et al. (2017). Language models are employed as style discriminators to learn disentangled representations for unsupervised text style transfer tasks such as sentiment modification Yang et al. (2018d).

A dialogue system, also known as a conversational agent, is a computer system designed to converse with humans using natural language. To be able to carry a meaningful conversation with a human user, the system needs to first understand the message of the user, represent it internally, decide how to respond to it and issue the target response using natural language surface utterances Chen et al. (2017a). Dialogue generation is an instance of conditional text generation where the system response is conditioned on the previous user utterance and frequently on the overall conversational context. Dialogue generation can also be an instance of constrained text generation when the conversation is carried on a topic which explicitly involves entities such as locations, persons, institutions, etc. From an application point of view, dialogue systems can be categorized into Keselj (2009):

• •

  task-oriented dialogue agents: are designed to have short conversations with a human user to help him/ her complete a particular task. For example, dialogue agents embedded into digital assistants and home controllers assist with finding products, booking accommodations, provide travel directions, make restaurant reservations and phone calls on behalf of their users. Therefore, task-oriented dialogue generation is an instance of both conditional and constrained text generation.

• •

  non-task oriented dialogue agents or chat-bots: are designed for carrying extended conversations with their users on a wide range of open domains. They are set up to mimic human-to-human interaction and unstructured human dialogues in an entertaining way. Therefore, non-task oriented dialogue is an instance of conditional text generation.

We formally define the task of dialogue generation. Generative dialogue models take as input a dialogue context $c$ and generate the next response $x$. The training data consists of a set of samples of the form $\{c^{n},x^{n},d^{n}\}\sim p_{\text{source}}(c,x,d)$, where $d$ denotes the source domain. At testing time, the model is given the dialog context $c$ and the target domain, and must generate the correct response $x$. The goal of a generative dialogue model is to learn the function $\mathcal{F}:C\times D\rightarrow X$ which performs well on unseen examples from the target domain after seeing the training examples on the source domain. The source domain and the target domain can be identical; when they differ the problem is defined as zero-shot dialogue generation Zhao and Eskenazi (2018). The dialogue generation problem can be summarized as:

$$\begin{split}\text{Training data}&:\{c^{n},x^{n},d^{n}\}\sim p_{\text{source}}(c,x,d)\\ \text{Testing data}&:\{c,x,d\}\sim p_{\text{target}}(c,x,d)\\ \text{Goal}&:\mathcal{F}:C\times D\rightarrow X\end{split}$$

(12)

A common limitation of neural networks for dialogue generation is that they tend to generate safe, universally relevant responses that carry little meaning Serban et al. (2016), Li et al. (2016a), Mou et al. (2016); for example universal replies such as “I don’t know” or “something” frequently occur in the training set are likely to have high estimated probabilities at decoding time. Additional factors that impact the conversational flow in generative models of dialogue are identified as repetitions and contradictions of previous statements, failing to balance specificity with genericness of the output, and not taking turns in asking questions See et al. (2019). Furthermore, it is desirable for generated dialogues to incorporate explicit personality traits Zheng et al. (2019) and control the sentiment Kong et al. (2019a) of the generated response to resemble human-to-human conversations.

Question answering systems are designed to find and integrate information from various sources to provide responses to user questions Fu and Feng (2018). While traditionally candidate answers consist of words, phrases or sentence snippets retrieved and ranked appropriately from knowledge bases and textual documents Kratzwald et al. (2019), answer generation aims to produce more natural answers by using neural models to generate the answer sentence. Question answering can be considered as both a conditional text generation and constrained text generation task. A question answering system needs to be conditioned on the question that was asked, while simultaneously ensuring that concepts needed to answer the question are found in the generated output.

A question answering system can be formally defined as follows. Given a context paragraph $C=\{c_{1},c_{2},\ldots,c_{n}\}$ consisting of $n$ words from word vocabulary $\mathcal{V}$ and the query $Q=\{q_{1},q_{2},\ldots,q_{m}\}$ of $m$ words in length, the goal of a question answering system is to either: i) output a span $S=\{c_{i},c_{i+1},\ldots,c_{i+j}\},\forall i=1,\ldots,n$ and $\forall j=0,\ldots,n-i$ from the original context paragraph $C$, or ii) generate a sequence of words $A=\{a_{1},a_{2},\ldots,a_{l}\},a_{k}\in\mathcal{V},\forall k=1,\ldots,l$ as the output answer. Below we differentiate between multiple types of question answering tasks:

• •

  Factoid Question Answering: given a description of an entity (person, place or item) formulated as a query and a text document, the task is to identify the entity referenced in the given piece of text. This is an instance of both conditional and constrained text generation, given conditioning on the input question and constraining the generation task to be entity-centric. Factoid question answering methods combine word and phrase-level representations across sentences to reason about entities Iyyer et al. (2014), Yin et al. (2015).

• •

  Reasoning-based Question Answering: given a collection of documents and a query, the task is to reason, gather, and synthesize disjoint pieces of information spread within documents and across multiple documents to generate an answer De Cao et al. (2019). The task involves multi-step reasoning and understanding of implicit relations for which humans typically rely on their background commonsense knowledge Bauer et al. (2018). The task is conditional given that the system generates an answer conditioned on the input question, and may be constrained when the information across documents is focused on entities or specific concepts that need to be incorporated in the generated answer.

• •

  Visual Question Answering: given an image and a natural language question about the image, the goal is to provide an accurate natural language answer to the question posed about the image Antol et al. (2015). By its nature the task is conditional, and can be constraint when specific objects or entities in the image need to be included in the generated answer.

Question answering systems that meet various information needs are proposed in the literature, for eg., for answering mathematical questions Schubotz et al. (2018), medical information needs Wiese et al. (2017), Bhandwaldar and Zadrozny (2018), quiz bowl questions Iyyer et al. (2014), cross-lingual and multi-lingual questions Loginova et al. (2018). In practical applications of question answering, users are typically not only interested in learning the exact answer word, but also in how this is related to other important background information and to previously asked questions and answers Fu and Feng (2018).

Image captioning is designed to generate captions in the form of textual descriptions for an image. This involves the recognition of the important objects present in the image, as well as object properties and interactions between objects to be able to generate syntactically and semantically correct natural language sentences Hossain et al. (2019). In the literature the image captioning task has been framed from either a natural language generation perspective Kulkarni et al. (2013), Chen et al. (2017b) where each system produces a novel sentence, or from a ranking perspective where existing captions are ranked and the top one is selected Hodosh et al. (2013). Image/ video captioning is a conditional text generation task where the caption is conditioned on the input image or video. In addition, it can be a constrained text generation task when specific concepts describing the input need to be present in the generated output.

Formally, the task of image/ video captioning takes as input an image or video $I$ and generates a sequence of words $y=(y_{1},y_{2},\ldots,y_{N}),y\in V^{*}\text{ and }y_{i}\in V,\forall i=1,\ldots,N$, where $V$ denotes the vocabulary of output words and includes special tokens to mark the beginning $<$S$>$ and end $<$E$>$ of a sentence, as well as the unknown token $<$UNK$>$ used for all words not present in the vocabulary $V$, and $V^{*}$ denotes all possible sentences over $V$. Given training set $\mathcal{D}=\{(I,y^{*})\}$ containing $m$ pairs of the form $(I_{j},y_{j}^{*}),\forall j=1,\ldots,m$ consisting of input image $I_{j}$ and its corresponding ground-truth caption $y_{j}^{*}=(y_{j_{1}}^{*},y_{j_{2}}^{*},\ldots,y_{j_{M}}^{*}),y_{j}^{*}\in V^{*}\text{ and }y_{j_{k}}^{*}\in V,\forall k=1,\ldots,M$, we want to maximize the probabilistic model $p(y|I;\theta)$ with respect to model parameters $\theta$.

Neural narrative generation aims to produce coherent stories automatically and is regarded as an important step towards computational creativity Gervás (2009). Unlike machine translation which produces a complete transduction of an input sentence which fully defines the target semantics, story telling is a long-form open-ended text generation task which simultaneously addresses two separate challenges: the selection of appropriate content (“what to say”) and the surface realization of the generation (“how to say it”)Wiseman et al. (2017). In addition, the most difficult aspect of neural story generation is producing a a coherent and fluent story which is much longer than the short input specified by the user as the story title. To this end, many neural story generation models assume the existence of a high-level plot (commonly specified as a one-sentence outline) which serves the role of a bridge between titles and stories Chen et al. (2019a), Fan et al. (2018b), Xu et al. (2018b), Drissi et al. (2018), Yao et al. (2019). Therefore, narrative generation is a constrained text generation task since explicit constraints are placed on which concepts to include in the narrative so as to steer the generation in particular topic directions. In addition, another constraint is that the output length needs to be strictly greater than the input length. We formally define the task of narrative generation below.

Assuming as input to the neural story generation system the title $x=x_{1},x_{2},\ldots,x_{I}$ consisting of $I$ words, the goal is to produce a comprehensible and logical story $y=y_{1},y_{2},\ldots,y_{J}$ of $J$ words in length. Assuming the existence of a one sentence outline $z=z_{1},z_{2},\ldots,z_{K}$ that contains $K$ words for the entire story, the latent variable model for neural story generation can be formally expressed:

$$P(y|x;\theta,\gamma)=\sum_{z}P(z|x;\theta)P(y|x,z;\gamma)$$

(13)

where $P(z|x;\theta)$ defines a planning model parameterized by $\theta$ and $P(y|x,z;\gamma)$ defines a generation model parameterized by $\gamma$.

The planning model $P(z|x;\theta)$ receives an input the one sentence title $z$ for the narrative and generates the narrative outline given the title:

$$P(z|x;\theta)=\prod_{k=1}^{K}P(z_{k}|x,z_{<k};\theta)$$

(14)

where $z_{<k}=z_{1},z_{2},\ldots,z_{k-1}$ denotes a partial outline. The generation model is used to produce a narrative given a title and an outline:

$$P(y|x,z;\gamma)=\prod_{j=1}^{J}P(y_{j}|x,z,y_{<j};\gamma)$$

(15)

where $y_{<j}$ denotes a partially generated story.

Hierarchical models for story generation break down the generation process into multiple steps: first modelling the action sequence, then the story narrative, and finally entities such as story characters Fan et al. (2019). While existing models can generate stories with good local coherence, generating long stories is challenging. Difficulties in coalescing individual phrases into coherent plots and in maintaining character consistency throughout the story lead to a rapid decrease in coherence as the output length increases van Stegeren and Theune (2019). Neural narrative generation combining story-writing with human collaboration in an interactive way improves both story quality and human engagement Goldfarb-Tarrant et al. (2019).

Automatic poetry generation is an important step towards computational creativity. In the poem generation literature, the generator operates in an interactive context where the user initially supplies the model with a set of keywords representing the concepts which outline the main writing intents, as well as their ordering. The user is also in charge of selecting a particular format for the generated poem. For example, common formats are quatrain, consisting of 4 lines of sentences, or regulated verse in which the poem is made up of 8 lines of sentences. The process is interactive and the author can keep modifying terms to reflect his writing intent. Poetry generation is a constrained text generation problem since user defined concepts need to be included in the generated poem. At the same time, it can also be a conditional text generation problem given explicit conditioning on the stylistic features of the poem. We define the petry generation task below.

Given as input a set of keywords that summarize an author’s writing intent $K=\{k_{1},k_{2},\ldots,k_{|K|}\},\text{ where each }k_{i}\in V,i=1,\ldots,|K|$ is a keyword term from vocabulary $V$, the goal is to generate a poem $\mathcal{P}=\{w|w\in\Omega\}$ where each term $w$ is selected from the candidate term set $\Omega=\{w|w\in\{K\cup\{V-K\}\}\}$, and $K\subseteq P,P\subseteq\Omega$ to fit the user specified constraints of the poetry format. The generative model computes the probability of line $S_{i+1}=w_{1},w_{2},\ldots,w_{m}$ given all previously generated poem lines $S_{1:i},i\geq 1$ (or alternatively, only the previous generated line or lexical n-grams) as follows:

$$P(S_{i+1}|S_{1:i})=\prod_{j=1}^{m-1}P(w_{j+1}|w_{1:j},S_{1:i})$$

(16)

Poetry composition is formulated as a constrained optimization problem in a generative summarization framework in iPOET Yan et al. (2013). Candidate terms from a large human-written poem corpus are retrieved to match the user intent, and then clustered to fit the poetry format, tone, rhythm, etc. Each cluster generates one line of the poem in a multi-pass generative summarization framework by conducting iterative term substitutions so that the generated poem matches the initial user constraints and poetic preference, and the relevance and coherence of the output is maximized. Generative models that jointly perform content selection and surface realization are proposed in Zhang and Lapata (2014). Generated poems are revised and polished through multiple style configurations in Ghazvininejad et al. (2017). Neural poetry generation models based on maximum likelihood estimation (MLE) only learn the most common patterns in the poetry corpus and generate outputs with little diversity Zhang et al. (2017a). In addition, these MLE based models suffer from loss-evaluation mismatch Wiseman and Rush (2016) manifested through incompatibility at evaluation time between the word-level loss function optimized by MLE and humans focusing on whole sequences of poem lines and assessing fine-grained criteria of the generated text such as fluency, coherence, meaningfulness and overall quality. These human evaluation criteria are modeled and incorporated into the reward function of a mutual reinforcement learning framework for poem generation Yi et al. (2018). For a detailed overview of poetry generation we point the reader to Oliveira (2017).

Product reviews allow users to express opinions for different aspects of products or services received, and are popular on many online review websites such as Amazon, Yelp, Ebay, etc. These online reviews encompass a wide variety of writing styles and polarity strengths. The task of review generation is similar in nature to sentiment analysis and a lot of past work has focused on identifying and extracting subjective content in review data Liu (2015), Zhao et al. (2016). Automatically generating reviews given contextual information focused on product attributes, ratings, sentiment, time and location is a meaningful conditional text generation task. Common product attributes used in the literature are the user ID, the product ID, the product rating or the user sentiment for the generated review Dong et al. (2017), Tang et al. (2016). The task can also be constrained text generation when topical and syntactic characteristics of natural languages are explicitly specified as constraints to incorporate in the generation process. We formally define the review generation task below.

Given as input a set of product attributes $a=(a_{1},a_{2},\ldots,a_{|a|})$ of fixed length $|a|$, the goal is to generate a product review $r=(y_{1},y_{2},\ldots,y_{|r|})$ of variable length $|r|$ by maximizing the conditional probability $p(r|a)$:

$$p(r|a)=\prod_{t=1}^{|r|}p(y_{t}|y_{<t},a)$$

(17)

where $y_{<t}=(y_{1},y_{2},\ldots,y_{t-1})$. The training data consists of pairs $(a,r)$ of attributes $a$ with their corresponding reviews $r$, and the model learns to maximize the likelihood of the generated reviews given the input attributes for the training data $\mathcal{D}$. The optimization problem can therefore be expressed as:

$$\max\sum_{(a,r)\in\mathcal{D}}\log p(r|a)$$

(18)

Generating long, well-structured and informative reviews requires considerable effort when written by human users and is a similarly challenging task to do automatically Li et al. (2019a).

Handwriting synthesis aims to automatically generate data that resembles natural handwriting and is a key component in the development of intelligent systems that can provide personalized experiences to humans Zong and Zhu (2014). The task of handwritten text generation is very much analogous to sequence generation. Given as input a user defined sequence of words $x=(x_{1},x_{2},\ldots,x_{T})$ which can be either typed into the computer system or fed as an input image $I$ to capture the user’s writing style, the goal of handwriting generation is to train a neural network model which can produce a cursive handwritten version of the input text to display under the form of output image $O$ Graves (2013). Handwriting generation is a conditional generation task when the system is conditioning on the input text. In addition, it is also a constrained text generation task since the task is constrained on generating text in the user’s own writing style. While advances in deep learning have given computers the ability to see and recognize printed text from input images, generating cursive handwriting is a considerably more challenging problem Alonso et al. (2019). Character boundaries are not always well-defined, which makes it hard to segment handwritten text into individual pieces or characters. In addition, handwriting evaluation is ambiguous and not well defined given the multitude of existent human handwriting style profiles Mohammed et al. (2018).

Other related tasks where natural language generation plays an important role are generating questions, arguments, counter-arguments and opinions, news headlines and digests, reports, financial statements, stock market reports, sports reports, slides and entire presentations, error corrections, generating creative and entertaining texts, composing music, lyrics and tweets, data-to-text generation, paraphrasing, speech synthesis, generating proteins as sequences of aminoacids, code in a programming language of choice, etc. All these tasks illustrate the widespread importance of having robust models for natural language generation.

Recurrent Neural Networks (RNNs) Rumelhart et al. (1986), Mikolov et al. (2010) are able to model long-term dependencies in sequential data and have shown promising results in a variety of natural language processing tasks, from language modeling Mikolov (2012) to speech recognition Graves et al. (2013) and machine translation Kalchbrenner and Blunsom (2013). An important property of RNNs is the ability of learning to map an input sequence of variable length into a fixed dimensional vector representation.

At each timestep, the RNN receives an input, updates its hidden state, and makes a prediction. Given an input sequence $x=(x_{1},x_{2},\ldots,x_{T})$, a standard RNN computes the hidden vector sequence $h=(h_{1},h_{2},\ldots,h_{T})$ and the output vector sequence $y=(y_{1},y_{2},\ldots,y_{T})$, where each datapoint $x_{t},h_{t},y_{t},\forall\text{ }t\in\{1,\ldots,T\}$ is a real valued vector, in the following way:

$$\begin{split}h_{t}&=\mathcal{H}(W_{xh}x_{t}+W_{hh}h_{t-1}+b_{h})\\ y_{t}&=W_{hy}h_{t}+b_{y}\end{split}$$

(21)

In Equation 21 terms $W$ denote weight matrices, in particular $W_{xh}$ is the input-hidden weight matrix and $W_{hh}$ is the hidden-hidden weight matrix. The $b$ terms denote bias vectors, where $b_{h}$ is the hidden bias vector and $b_{y}$ is the output bias vector. $\mathcal{H}$ is the function that computes the hidden layer representation. Gradients in an RNN are computed via backpropagation through time Rumelhart et al. (1986), Werbos (1989). By definition, RNNs are inherently deep in time considering that the hidden state at each timestep is computed as a function of all previous timesteps. While in theory RNNs can make use of information in arbitrarily long sequences, in practice they fail to consider context beyond the few previous timesteps due to the vanishing and exploding gradients Bengio et al. (1994) which cause gradient descent to not be able to learn long-range temporal structure in a standard RNN. Moreover, RNN-based models contain millions of parameters and have traditionally been very difficult to train, limiting their widespread use Sutskever et al. (2011). Improvements in network architectures, optimization techniques and parallel computation have resulted in recurrent models learning better at large-scale Lipton et al. (2015).

Long Short Term Memory (LSTM) Hochreiter and Schmidhuber (1997) networks are introduced to overcome the limitations posed by vanishing gradients in RNNs and allow gradient descent to learn long-term temporal structure. The LSTM architecture largely resembles the standard RNN architecture with one hidden layer, and each hidden layer node is modified to include a memory cell with a self-connected recurrent edge of fixed weight which stores information over long time periods. A memory cell $c_{t}$ consists of a node with an internal hidden state $h_{t}$ and a series of gates, namely an input gate $i_{t}$ which controls how much each LSTM unit is updated, a forget gate $f_{t}$ which controls the extent to which the previous memory cell is forgotten, and an output gate $o_{t}$ which controls the exposure of the internal memory state. The LSTM transition equations at timestep $t$ are:

$$\begin{split}i_{t}&=\sigma(W^{(i)}x_{t}+U^{(i)}h_{t-1}+b^{(i)})\\ f_{t}&=\sigma(W^{(f)}x_{t}+U^{(f)}h_{t-1}+b^{(f)})\\ o_{t}&=\sigma(W^{(o)}x_{t}+U^{(o)}h_{t-1}+b^{(o)})\\ u_{t}&=\sigma(W^{(u)}x_{t}+U^{(t)}h_{t-1}+b^{(t)})\\ c_{t}&=i_{t}\odot u_{t}+f_{t}\odot c_{t-1}\\ h_{t}&=o_{t}\odot\text{tanh}(c_{t})\end{split}$$

(22)

In Equation 22, $x_{t}$ is the input at the current timestep $t$, $\sigma$ denotes the logistic sigmoid function and $\odot$ denotes elementwise multiplication. $U$ and $W$ are learned weight matrices. LSTMs can represent information over multiple time steps by adjusting the values of the gating variables for each vector element, therefore allowing the gradient to pass without vanishing or exploding. In both RNNs and LSTMs the data is modeled via a fully-observed directed graphical model, where the distribution over a discrete output sequence $y=(y_{1},y_{2},\dots,y_{T})$ is decomposed into an ordered product of conditional distributions over tokens:

$$P(y_{1},y_{2},\dots,y_{T})=P(y_{1})\prod_{t=1}^{T}P(y_{t}|y_{1},\dots,y_{t-1})$$

(23)

Similar to LSTMs, Gated Recurrent Units (GRUs) Cho et al. (2014) learn semantically and syntactically meaningful representations of natural language and have gating units to modulate the flow of information. Unlike LSTMs, GRU units do not have a separate memory cell and present a simpler design with fewer gates. The activation $h_{t}^{j}$ at timestep $t$ linearly interpolates between the activation at the previous timestep $h_{t}^{j-1}$ and the candidate activation $\widetilde{h}_{t}^{j}$. The update gate $z_{t}^{j}$ decides how much the current unit updates its content, while the reset gate $r_{t}^{j}$ allows it to forget the previously computed state. The GRU update equations at each timestep $t$ are:

$$\begin{split}h_{t}^{j}&=(1-z_{t}^{j})h_{t-1}^{j}+z_{t}^{j}\widetilde{h}_{t}^{j}\\ z_{t}^{j}&=\sigma(W_{z}x_{t}+U_{z}h_{t-1})^{j}\\ \widetilde{h}_{t}^{j}&=\tanh(Wx_{t}+U(r_{t}\odot h_{t-1}))^{j}\\ r_{t}^{j}&=\sigma(W_{r}x_{t}+U_{r}h_{t-1})^{j}\end{split}$$

(24)

Models with recurrent connections are trained with teacher forcing Williams and Zipser (1989), a strategy emerging from the maximum likelihood criterion designed to keep the recurrent model predictions close to the ground-truth sequence. At each training step the model generated token $\hat{y}_{t}$ is replaced with its ground-truth equivalent token $y_{t}$, while at inference time each token is generated by the model itself (i.e. sampled from its conditional distribution over the sequence given the previously generated samples). The discrepancy between training and inference stages leads to exposure bias, causing errors in the model predictions that accumulate and amplify quickly over the generated sequence, Lamb et al. (2016). As a remedy, Scheduled Sampling Bengio et al. (2015) mixes inputs from the ground-truth sequence with inputs generated by the model at training time, gradually adjusting the training process from fully guided (i.e. using the true previous token) to less guided (i.e. using mostly the generated token) based on curriculum learning Bengio et al. (2009). While the model generated distribution can still diverge from the ground truth distribution as the model generates several consecutive tokens, possible solutions are: i) make the self-generated sequences short, and ii) anneal the probability of using self-generated vs. ground-truth samples to 0, according to some schedule. Still, models trained with scheduled sampling are shown to memorize the distribution of symbols conditioned on their position in the sequence instead of the actual prefix of preceding symbols Huszár (2015).

Many extensions of vanilla RNN and LSTM architectures are proposed in the literature aiming to improve generalization and sample quality Yu et al. (2019). Bidirectional RNNs Schuster and Paliwal (1997), Berglund et al. (2015) augment unidirectional recurrent models by introducing a second hidden layer with connections flowing in opposite temporal order to exploit both past and future information in a sequence. Multiplicative RNNs Sutskever et al. (2011) allow flexible input-dependent transitions, however many complex transition functions hard to bypass. Gated feedback RNNs and LSTMs Chung et al. (2014) rely on gated-feedback connections to enable the flow of control signals from the upper to lower recurrent layers in the network. Similarly, depth gated LSTMs Yao et al. (2015) introduce dependencies between lower and upper recurrent units by using a depth gate which connects memory cells of adjacent layers. Stacked LSTMs stack multiple layers at each time-step to increase the capacity of the network, while nested LSTMs Moniz and Krueger (2018) selectively access LSTM memory cells with inner memory. Convolutional LSTMs Sainath et al. (2015), Xingjian et al. (2015) are designed for jointly modeling spatio-temporal sequences. Tree-structured LSTMs Zhu et al. (2015), Tai et al. (2015) extend the LSTM structure beyond a linear chain to tree-structured network topologies, and are useful at semantic similarity and sentiment classification tasks. Multiplicative LSTMs Krause et al. (2016) combine vanilla LSTM networks of fixed weights with multiplicative RNNs to allow for flexible input-dependent weight matrices in the network architecture. Multiplicative Integration Wu et al. (2016b) RNNs achieve better performance than vanilla RNNs by using the Hadamard product in the computational additive building block of RNNs. Mogrifier LSTMs Melis et al. (2019) capture interactions between inputs and their context by mutually gating the current input and the previous output of the network. For a comprehensive review of RNN and LSTM-based network architectures we point the reader to Yu et al. (2019).

A recurrent free-text generation model becomes a conditional recurrent text generation model when the distribution over training sentences is conditioned on another modality. For example in machine translation the distribution is conditioned on another language, in image caption generation the condition is the input image, in video description generation we condition on the input video, while in speech recognition we condition on the input speech.

Content and stylistic properties (such as sentiment, topic, style and length) of generated movie reviews are controlled in a conditional LSTM language model by conditioning on context vectors that reflect the presence of these properties Ficler and Goldberg (2017). Affective dialogue responses are generated by conditioning on affect categories in an LSTM language model Ghosh et al. (2017). A RNN-based language model equipped with dynamic memory outperforms more complex memory-based models for dialogue generation Mei et al. (2017). Participant roles and conversational topics are represented as context vectors and incorporated into a LSTM-based response generation model Luan et al. (2016).

Metropolis-Hastings sampling Miao et al. (2019) is proposed for both soft and hard constrained sentence generation from models based on recurrent neural networks. The method is based on Markov Chain Monte Carlo (MCMC) sampling and performs local operations such as insertion, deletion and replacement in the sentence space for any randomly selected word in the sentence.

Hard constraints on the generation of scientific paper titles are imposed by the use of a forward- backward recurrent language model which generates both previous and future words in a sentence conditioned on a given topic word Mou et al. (2015). While the topic word can occur at any arbitrary position in the sentence, the approach can only generate sentences constrained precisely on one keyword. Multiple constraints are incorporated in sentences generated by a backward-forward LSTM language model by lexically substituting constrained tokens with their closest matching neighbour in the embedding space Latif et al. (2020). Guiding the conversation towards a designated topic while integrating specific vocabulary words is achieved by combining discourse-level rules with neural next keywords prediction Tang et al. (2019). A recurrent network based sequence classifier is used for extractive summarization in Nallapati et al. (2017). Poetry generation which obeys hard rhythmic, rhyme and topic constraints is proposed in Ghazvininejad et al. (2016).

Sequence to sequence Sutskever et al. (2014) models can be conditioned on specific attributes at training time so as to control their output at inference time. In the literature the main ways in which generative encoder-decoder models are conditioned are categorized Sennrich et al. (2016) as follows: i) adding special tokens at the beginning Johnson et al. (2017) or end Sennrich et al. (2016) of the source text , ii) incorporating additional conditions into the decoder hidden states, therefore bypassing attention Logeswaran et al. (2018), and iii) connecting the conditions directly to the decoder output layer. In addition, when categorical attributes are used in conjuction with end-to-end neural text classification models, incorporating these attributes in the attention mechanism is the least effective method Amplayo (2019).

Attribute-conditioned review generation uses an attention-enhanced attribute-to-sequence model to generate reviews conditioned on specific product attributes Dong et al. (2017), Tang et al. (2016). Natural language descriptions of database events are generated via encoder-decoder models for concept-to-text generation Mei et al. (2016) and table-to-text generation Liu et al. (2018c). Generating questions from long documents is achieved by combining a sequence-to-sequence model with multi-stage attention to represent the broad document context Tuan et al. (2019). A hierarchical sequence-to-sequence architecture is proposed for generating dialogue responses by first sampling a continuous variable in the latent space and then generating the response conditioned on that latent variable Serban et al. (2017). A dual attention sequence-to-sequence framework which conditions on both the source text and factual information is used in abstractive summarization for encouraging faithfulness of the generated content to the source document Cao et al. (2018).

Hierarchical story generation systems Fan et al. (2018b) use a two-step approach to text generation: first generate a prompt describing the topic of the story, and then conditioned on the given prompt generate the content of the story. To this end, fusion mechanisms Sriram et al. (2018) used on top of sequence-to-sequence models encourage conditioning on the story outline and are found useful at building dependencies between the given inputs and the generated outputs. Entity-focused story generation is proposed in Clark et al. (2018a).

Concise summaries of a specific desired length are obtained by controlling the output sequence length for neural encoder-decoder models through either learning or decoding-based methods Kikuchi et al. (2016). The most salient sentences in a document are identified using extractive summarization and then paraphrased using an encoder-decoder based sentence abstractor model Nikolov and Hahnloser (2020). Nevertheless, in abstractive summarization conventional sequence-to-sequence models often suffer from repetition and semantic irrelevance Lin et al. (2018), and the attention-based encoder outputs are noisy when there is no obvious alignment relationship between the source text and the target summary Zhou et al. (2017). To alleviate the problem, global encoding of the source context is used to filter encoder outputs and refine represnetations learnt at each timestep based on the global context Lin et al. (2018). An encoder-decoder framework for extractive summarization which incorporates both a sentence encoder and a document encoder to capture context surrounding a sentence, as well as a document decoder to predict sentence labels for inclusion in the summary based on representations learned by the document encoder is proposed in Zhang et al. (2018b). A hierarchical document encoder is combined with an attention-based extractor for selecting sentences and words in extractive summarization Cheng and Lapata (2016). Other approaches to hierarchical abstractive multi-document summarization combine maximal marginal relevance which serves to extract sentences from the original document based on their relevance and redundancy with the pointer-generator network See et al. (2017) used to alternate between copying words from the source documents with outputting other vocabulary words Fabbri et al. (2019). In text simplification appending special tokens to input sentences at training time is found to improve performance Nishihara et al. (2019). Similarly, sequence to sequence models parameterized on specific attributes of the target simplification, such as length, paraphrasing, lexical complexity and syntactic complexity are proposed in Martin et al. (2019b). An attentional encoder-decoder framework with side constraints for politeness is used to control for the level of courtesy in machine translation Sennrich et al. (2016). Textual attributes of sentences such as sentiment, tense, voice, mood and negation are modified by incorporating conditioning information into neural encoder-decoder models Logeswaran et al. (2018). Generating emotional responses in neural conversational systems is achieved by feeding the emotion category embedding to a sequence-to-sequence decoder Zhou et al. (2018), Asghar et al. (2018). Furthermore, informative and on-topic dialogue responses are generated via sentence control functions Ke et al. (2018). Many encoder-decoder models treat the entire dialogue history as one single sequence Serban et al. (2016), while others treat each conversational turn as a separate sequence Vinyals and Le (2015), Shang et al. (2015),Dušek and Jurcicek (2016). A topic aware sequence-to-sequence model is used to generate on-topic conversational responses Xing et al. (2016). Topic words are extracted using Latent Dirichlet Allocation and the decoder produces each word conditioned on both the input message and the topics through a joint attention mechanism. Abstractive summaries are produced relying on a pointer-generator network See et al. (2017), a hybrid model which combines pointer networks Vinyals et al. (2015b) to accurately reproduce source side information with a sequence-to-sequence attentional model to generate new words.

Synthesizing sentences containing specific keywords is done in a sequence-to-sequence backward and forward Mou et al. (2016) model by first generating the sentence fragment to the left of the given keyword (in reverse order and conditioned on that keyword), then encoding the sentence fragment generated so far, followed by decoding the sentence fragment to the right of the keyword conditioned on the already generated first part. An encoder-decoder framework for factoid question answering Yin et al. (2016) is able to query and generate answers containing terms retrieved from a knowledge base. Constrained modification of factual Wikipedia sentences according to given claims is performed via a two-encoder sequence-to-sequence model with copy attention Shah et al. (2020). Style transfer between scientific paper titles and newspaper titles is approached using multiple decoders for each style, or by passing encoded representations combined with style embeddings to a single decoder Fu et al. (2018). Generating memorable headlines constrained on stylistic attributes such as humour, romance and clickbait is performed in a sequence-to-sequence Transformer-based model for stylistic headline generation Jin et al. (2020). A denoising autoencoding approach is adopted for style transfer which replces learning disentangled latent representations with back-translation Lample et al. (2018). Neural machine translation models with attention are used for style transfer via backtranslation in the absence of parallel aligned data Zhang et al. (2018d) Neural paraphrase generation is performed through a syntactically constrained encoder-decoder model Iyyer et al. (2018) or via a sequence-to-sequence model with pivoting over multiple sentences from multiple languages Mallinson et al. (2017). Neural poetry translation with a sequence-to-sequence model is proposed in Ghazvininejad et al. (2018).

Generative Adversarial Networks (GANs) Goodfellow et al. (2014) train generative models through an adversarial process which consists of two competing models trained simultaneously: a generative model $G$ that captures the data distribution and whose objective is to generate fake data which is indistinguishable from real data, and a discriminative model $D$ which estimates the probability that a sample came from the training data rather from the generator $G$. The generator $G(z,\theta_{G})$ learns the distribution $p_{g}$ over real data x by mapping function the prior noise distribution $p_{z}(z)$ to real data, while the discriminator $D(x,\theta_{D})$ outputs a single scalar value representing the probability that a sample came from the training data instead of $p_{g}$. The optimization objective for the two-player mini-max game between the $G$ and $D$ can be formally expressed as:

$$\begin{split}\min_{G}\max_{D}V(D,G)&=\mathbb{E}_{\text{x}\sim p_{\text{data}}(\text{x})}[\log D(\text{x})]\\ &+\mathbb{E}_{\text{z}\sim p_{z}(\text{z})}[\log(1-D(G(\text{z}))]\end{split}$$

(36)

The GAN framework relies on an implicit probability model, as opposed to incorporating an explicit formulation of the probability density. Unlike autoregressive modeling in which exposure bias is a common issue, in GANs this issue is avoided by sampling synthetic examples at training time from the generator and providing them as input to the discriminator for comparison with real sentences (i.e. sentence-level comparison instead of word-level comparison). The latent representations extracted from real data are distributed according to the specified prior $p_{z}(z)$ (for eg. Gaussian or uniform). Gradients are backpropagated from the discriminator $D$ through the generated samples to the generator $G$; note that this is possible only when the generated samples are differentiable w.r.t the $\theta_{G}$ generator parameters. When the discriminator is trained to optimality before each generator parameter update, the GAN adversarial game is equivalent to minimizing the Jenson-Shannon divergence between the real data distribution $p_{x}(.)$ and the synthetic data distribution $p(G(z)),z\sim p_{z}(.)$ Arjovsky and Bottou (2017). Nevertheless, in such cases gradients can vanish as the discriminator saturates, learns to reject all samples and gives meaningless gradients to the generator. In practice, the second term in Equation (36) is replaced with $\mathbb{E}_{\text{z}\sim p_{z}(\text{z})}[\log(D(G(\text{z}))]$ which helps circumvent the vanishing gradient problem to a certain extent Goodfellow et al. (2014).