Three Ways of Using Large Language Models to Evaluate Chat

This paper describes the systems submitted by team6 for ChatEval, the DSTC 11 Track 4 competition aimed at evaluating open-domain chat.¹¹1 Results & task description at chateval.org/dstc11. Our experimental code is available at github.com/oplatek/chateval-llm. We participated in Task 2, which focuses on evaluating multiple criteria on the level of individual dialogue turns. The task of evaluating responses in a chat is challenging because it requires an understanding of the interlocutor’s roles (pragmatics), the conversation’s context, and the response’s meaning (semantics). At the same time, the conversations are often ungrammatical Rodríguez-Cantelar et al. (2023) and vary in style Zhang et al. (2018). The commonly used metrics, such as BLEU Papineni et al. (2002), METEOR Banerjee and Lavie (2005), or BERTScore Zhang et al. (2019), are based on comparison to human references and thus correlate poorly with human judgments on the turn-level, as they penalize many correct responses for a given chat context Zhao et al. (2017). At the same time, human evaluation is expensive and time-consuming. Previous referenceless metrics based on neural networks and language models still do not reach sufficient correlations with human judgements Zhang et al. (2020); Lowe et al. (2017).

In our work, we followed up on the recent development of pretrained Large Language Models (LLMs) with instruction finetuning Brown et al. (2020); Raffel et al. (2020), which have been found to be capable evaluators in machine translation, summarization as well as dialogue Kocmi and Federmann (2023); Liu et al. (2023). Therefore, we applied LLMs and specific prompting to elicit ratings for the multiple qualities evaluated in DSTC11 Track 4 Task 2: appropriateness, content richness, grammatical correctness, and relevance. We present three different systems used for our three submissions, all of which are based on LLMs and few-shot prompting: (1) We evaluate a straightforward approach with manually designed fixed prompts for off-the-shelf open LLMs checkpoints. (2) We train a simple feed-forward regression neural network (FNN) on top of frozen LLM embeddings to predict the turn-level metrics scores. (3) We used the ChatGPT API and few-shot examples retrieved dynamically from the development set to improve the prompting performance. As no data annotated with the target metrics were available for the challenge, we heuristically mapped existing annotations from the development set to the target metrics, and we manually annotated a small rehearsal dataset for hyperparameter search.

Based on the human annotations released after the challenge finished, our team6 achieved second place thanks to our third method, dynamically prompted chatGPT with few-shot examples. This approach showed that LLM prompting is a viable option for prototyping chat evaluation. However, the two other methods we explored scored worse: open LLMs with fixed prompts generally showed poor performance, and the regression FFN worked well on the development set but did not generalize well to the test set.

The goal of the DSTC11 Track 4 Task 2 was to predict several turn-level metrics automatically on the test set. For each dialogue turn, considering the preceding dialogue history, the participants were to submit a system to predict the score of the target metrics, defined by the organizers as:

• •

  Appropriateness – The response is appropriate given the preceding dialogue.

• •

  Content Richnes – The response is informative, with long sentences including multiple entities and conceptual or emotional words.

• •

  Grammatical Correctness – Responses are free of grammatical and semantic errors.

• •

  Relevance – Responses are on-topic with the immediate dialogue history.

Table 1 shows chat conversations from the rehearsal dataset with the turn-level metric annotations.

The organizers provided the participants with training, development, and test sets Rodríguez-Cantelar et al. (2023), each coming from different domains and annotated with different metrics:

• •

  Training set – consists of 390k dialogues, annotated with sentiment and toxicity labels. This set was not used in our experiments at all since our goal was to fine-tune or select LLMs that are already well-performing with no finetuning.

• •

  Development set – consists of 24 datasets, some annotated with dataset-specific metrics. For our experiments, we created a heuristic mapping to the target metrics on a subset of the development set (see Section 3).

• •

  Test set – consists of 3,470 dialogues and 130k turns, annotated with the target metrics. The data was only published in an anonymized form and at the end of the challenge, with no annotations or metadata, so that challenge participant could produce their model outputs. The annotations were published after the challenge was finished.

• •

  Rehearsal set – this is a set of 156 turns collected in the same way as the test set, released earlier than the test set. We manually annotated this set with the target metrics (see Section 3) and used the result for hyperparameter search.

The submitted systems were benchmarked for the quality of their ranking using the Spearman correlation coefficients (SCC) Zar (2005) computed between the predicted scores and the human judgments. As a secondary measure, the Pearson correlation coefficients (PCC) Freedman et al. (2007) were used to evaluate the correlation. The measures were computed for each of the target metrics separately. The overall submissions’ ranking was determined using the average of the four SCCs.

Since no information was provided on how the individual development dataset metrics relate to the target dialogue metrics, we built a heuristic to obtain target metric scores. The heuristic uses a linear combination of one or more dataset-specific metrics to the target metrics, chosen based on individual descriptions from the literature.²²2 See the line 354 for the turn metric mapping for different datasets. Using the development set and the heuristic, we created a supervised dataset and split it into training and development splits. We used this development dataset for model selection or supervised training, and we use this dataset to develop the three systems described in Section 4.

During our experiments, we struggled to find representative labels and input data which could be used as a development set. Therefore, we decided to annotate the additional 156 turns from the rehearsal set with the target metrics described in Section 2. We used this data to find our submitted systems’ optimal hyperparameters. We assumed that this data came from the same distribution as the test set, but this later proved clearly not to be the case, as seen in Figure 2.

Note that we did not use the training set at all.

Inspired by Kocmi and Federmann (2023), we used pre-trained LLMs with prompts for predicting the individual metrics. We started with the simplest approach possible and manually designed the prompts.

We experimented with the three methods described in Section 4. First, we we experimented with the Simple Prompting method using the open-source LLMs (Section 4.1). Based on the results, we started two independent experiments. Section 5.2 describes the FFN training and Section 5.3 describes the development of vector storage which we used with ChatGPT API. For all three methods, we used the rehearsal set to select the best-performing model-template combination and hyperparameters.

Following is a dialogue context and the response to it.
Express how the response is appropriate given the context
with a continuous number between 1 and 5.
The higher the score, the more appropriate the sentences are.
Here are a few examples:
--------
{examples}
--------
Now complete the following with just a single float number:
Context: {dialogue_context}
Response: {response}
Appropriateness Score:

We report positive findings related to Method 3 (Section 4.3), but we also report lessons learned from implementing the other two methods and, in general, using the data provided for the challenge. First, we summarize observations from our use of the data (Section 6.1). Then we report negative results from the simple prompting and FFN fine-tuning (Sections 6.2 and 6.3, respectively). We also report our best results from the vector store (Section 6.4) and discuss what our best model in the challenge is capable of evaluating. Finally, we add an ablation study in Section 6.5 performed after the challenge was complete, comparing few-shot capabilities of ChatGPT with the newly released Llama 2 model.

We are aware that LLMs are trained on large datasets, some of which (e.g., ChatGPT) are not public. However, due to the novelty of the test set Rodríguez-Cantelar et al. (2023), we believe that the test set has not leaked to their training set.

Recent works in chat evaluation focus on referenceless approaches, as these do not suffer from penalizing appropriate responses based on surface dissimilarity to a single human-written reference response Liu et al. (2017); Lowe et al. (2017). Here, Lowe et al. (2017) trained a neural network from scratch on relatively large annotated data to predict a single score, but this approach was later found to generalize poorly, even to basic data perturbations, let alone other datasets Sai et al. (2019); Lowe (2019).

Later works leveraged pretrained language models for better generalization abilities, such as BERT Zhang et al. (2020); Gao et al. (2020), RoBERTa Mehri and Eskenazi (2020c), GPT-2 Sinha et al. (2020) or DialoGPT Mehri and Eskenazi (2020a). These metrics are trained on human-labeled sets of system outputs based on popular open-domain datasets, similar to the ChatEval development data. Some of them use additional data augmentation techniques, such as self-training Zhang et al. (2022a). While they do achieve good correlations on some datasets, generalization with respect to unseen datasets is still not guaranteed Yeh et al. (2021).

Sai et al. (2021) stressed the importance of predicting multiple qualities, such as, fluency and appropriateness, in dialogue evaluation. At the same time, they asserted that metrics should be sensitive enough to distinguish between similar responses. Using simple text perturbations targeting the individual qualities, they show that most existing metrics are not robust enough.

Two very recent works, closely related to ours, propose the usage of instruction-tuned LLMs to evaluate generated text in various tasks like summarization and dialogue response generation  Liu et al. (2023), or machine translation Kocmi and Federmann (2023). Both approaches use in-context learning and multiple prompting techniques to obtain scalar metric predictions or candidate rankings. They achieved good results and correlations with human judgments. However, they used only closed models for the evaluation and did not experiment with few-shot prompting using relevant examples.

We presented three simple approaches to using LLMs for turn-level chat evaluation. We achieved promising results using ChatGPT prompting with few-shot example retrieval from a vector score, and ranked as the second-best team. Based on the results of our best system, we argue that chat turn evaluation systems based on current state-of-the-art LLMs are usable only for system-level evaluation but not for segment-level evaluation, i.e., they cannot distinguish between the quality of individual turns, especially for outputs of high-quality latest systems based on LLMs such as ChatGPT and GPT3.

We observed that LLMs are fragile to the prompts, few-shot examples and cannot be used out-of-the-box for chat evaluation. We also report implementing a simple regressor on top of embeddings obtained from the prompted LLM decoder. We attribute its poor performance to our incorrect implementation of data preparation.

We also presented an ablation study that investigated the influence of the few-shot examples on the performance of LLMs. We found that few-shot examples help the LLMs to generalize better to unseen data, especially with respect to fitting the desired output format. However, using examples dynamically obtained from the vector store instead of hand-picked fixed examples did not bring any additional improvements.

We reached a new best Spearman correlation coefficient of 0.6136 for appropriateness with ChatGPT and fixed few-shot examples in our ablation study. In addition, the Llama 2 open model used in our ablation showed significant improvements over the challenge baseline.

This research was supported by Charles University projects GAUK 40222 and SVV 260575 and by the European Research Council (Grant agreement No. 101039303 NG-NLG). It used resources provided by the LINDAT/CLARIAH-CZ Research Infrastructure (Czech Ministry of Education, Youth, and Sports project No. LM2018101). The authors thank the anonymous reviewers for their valuable feedback, Milan Fučík and Mateusz Krubiński for their suggestions and technical support.