Human or Machine: Automating Human Likeliness Evaluation of NLG Texts^††thanks: Supported by the project no. 19-26934X (NEUREM3) of the Czech Science Foundation and ELITR (H2020-ICT-2018-2-825460) of the EU.

NLG (Natural Language Generation) is a set of techniques and practices for automatically transforming structured data into natural language. Among the typical applications of NLG, we can mention weather predictions, financial reporting, customer support, etc. Same as other related disciplines such as MT (Machine Translation) or TS (Text Summarization), it has surged in the last decade, greatly pushed by the significant advances in text applications of deep neural networks [27, 3] as well as the creation of large datasets [16, 8, 17]. In parallel with the data processing, researchers are also finding ways to automate the evaluation of the intelligent text-related systems they propose. An evaluation practice that is gaining popularity compares method output texts against human-written references of a standard corpus using automatic metrics. Some of the most popular metrics are BLEU of Papineni et al. [19], ROUGE of Lin [15], and METEOR of Banerjee and Lavie [4]. This automatic evaluation practice is tempting for NLP (and NLG) researchers since it is fast and cheap to perform, does not require domain expertise, and usually produces repeatable results.

When comparing two methods A and B, besides accuracy, we usually want to also observe other quality aspects (e.g., coherence, readability, naturalness, fluency, adequacy or grammaticality) of the text they produce. BLEU and the automatic evaluation process have yet been criticized by several authors [25, 5]. As pointed out by Reiter [22], BLEU is not appropriate for evaluating all quality criteria of the NLG systems. Furthermore, the results of the automatic evaluation process do not always correlate well with those of human campaigns [23]. Even adding the data efficiency scores in the process would not tell us anything about the text quality aspects of A and B outputs [6].

As pointed out by Novikova et al. [18], there is a need for new evaluation metrics to objectively assess specific text quality characteristics such as human likeliness (or naturalness), coherence, etc. In this paper, we present the proof of concepts for a procedure that can be used to automatically evaluate human likeliness of NLG outputs. We propose to use a discriminator model that can label each test set output as being human-written (natural) or machine-generated (synthetic). This model will use existing pretrained language models such as BERT or GPT-2 and assume that synthetic texts do mostly contain high-rank words sampled from probability distributions, in contrast to natural texts that contain more low-rank words. This way, it will compute the fraction probability (the ratio between the probability of the actual word in a position and the highest rank word for that position) of each word and the average value for the entire text sample. The latter will be discretized to get the sample class (h or m).

We start with a survey of evaluation procedures in the last five INLG conference proceedings and notice an increasing trend of automatic evaluations against human-based ones. Later on, we discuss the possibility of automating human likeliness evaluation and propose the h score as an objective metric to estimate it. In Section 4, the discrimination model is described, together with possible shortcomings and a few alternative approaches. In the end, we conclude with the follow-up steps of the near future.

To have a quantitative picture of the NLG evaluation trends, we carefully examined the papers published in the last five INLG conference proceedings (2014–2019). We focused on the evaluation practices of these papers, sorting out those that used automatic metrics from those using human studies or those using both. The results are presented in Table 1. As we can see, the number of studies performing automatic evaluation only (Auto column) has been rising, especially in the last year. Their portion goes up from 12 % in 2014 to 30.1 % in 2019. Among the evaluation metrics, BLEU was the most popular, followed by METEOR and ROUGE. The studies using both human and automatic evaluation (Both column) have also risen during the last five years. These findings are in line with earlier observations such as those of Gkatzia and Mahamood [12] who considered a wider range of NLP publications and reported an increase in automatic metric evaluations from 44 % in the period 2005–2008 to 60 % in 2012–2015.

The opposite is true for the studies that perform human evaluation only (Human column). They have been following a decreasing trend, falling from 32 % to 11 %. A similar trend is reported in other recent surveys [2, 7]. We also observed that for the human evaluation process, existing studies usually involve a few human experts or some dozens of students. Moreover, a considerable number of studies report to have used crowdsourced reviewers from Amazon Mechanical Turk or similar platforms. Accuracy, coherence, readability, and human likeliness are the most considered text quality criteria. Finally, there is also a category of studies (None column) that do not perform any evaluation or that do not assess the text generation quality. They were mostly proposals for shared tasks, findings from challenges, surveys or reviews, etc.

The results of our survey (and similar ones) suggest that it is highly desirable for researchers to benchmark the methods or techniques they propose using automatic metrics. As also pointed out by Reiter [22] and Novikova et al. [18], new evaluation metrics are required. There is thus a strong incentive for formulating new quantitative measures and methodologies. Some attempts have developed objective measures like average word length, mean parse tree height, and the average number of nouns [1, 26]. These measures are combined in formulas to obtain a score for automatically assessing text readability.

In this paper, we present a similar attempt regarding the human likeliness of the generated outputs. The h score we propose reveals the capability of an NLG model to produce texts that are human-like or written by humans. This characteristic is important and highly desired in NLG applications as well as in various surging domains such as MT, TS, question answering and others that go beyond NLG. The most common approach is to assess human likeliness by means of human studies that ask participants to rate the texts using point-based schemes. Likert 5-points scale of is one of the most popular methods in the literature [13, 14].

Our idea is to automate the human likeliness evaluation of any text generation method M by considering the task as a binary discrimination problem and computing a metric that we call the h score. Let’s assume we are using a test set of $n$ reference samples for the evaluation. We can expect to have $n=n_{h}+n_{m}$, where $n_{h}$ is the number of texts that are perceived as human-written and $n_{m}$ is the number of those which are perceived as machine-generated. We can consider the h score and m score of M as the fraction of its outputs being perceived as human-written or machine-generated. They can be computed using Equation 1.

In this paper, we first presented the results of a survey which reveals the trend towards the automatic evaluation of text quality criteria. We further described a method for automatically evaluating the naturalness of the NLG output texts by computing the human-likeliness score as the average of human-labeled test samples. Instead of relying on human participants to label those test samples, we propose to use a discrimination approach based on large pretrained language models like BERT or GPT-2 and the computation of the fraction probability of each word and the entire text. We also presented an alternative scheme that can be used to discretize the fraction probability values in case of high information loss from the discriminator. Several empirical observations using synthetic and natural samples will be conducted to find the optimal setup of our proposal. The test set of the popular CNN/Dailymail news dataset created by Nallapati et al. [16] is probably a good source of texts. Furthermore, we plan to conduct a comprehensive validation of the scheme by involving human participants who will judge the naturalness of the text samples. This will check the agreement between the automatic predictions and the human evaluations.