Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks

There has been a number of studies in recent years comparing encoder models to decoder models. Zhong et al. (2023) compared GPT-3.5-turbo (January 2023 version) to (finetuned versions of) the base- and large versions of BERT Devlin et al. (2018) and RoBERTa Liu et al. (2019) on the English GLUE benchmark Wang et al. (2018). They found that GPT-3.5-turbo was on average on par with the base-sized encoder models, but fell short of the large-sized ones. They also note that despite being on par with the base-sized models, there was a big discrepancy between the models on individual tasks, with GPT-3.5-turbo for instance being better on the inference tasks while being worse on the paraphrase tasks. We note however that they only evaluate the decoder model in a zero-shot setting, and furthermore they only evaluate the models on 25 samples for each class in the development split, leading to a potential lack of robustness in their evaluation.

Wang et al. (2023) compared GPT-3.5-turbo (January 2023 version) to a finetuned version of the base-sized BERT model on 18 English benchmark datasets related to sentiment analysis. Like Zhong et al. (2023), they found that the zero-shot performance of GPT-3.5-turbo was on par with the base-sized BERT model, and that the few-shot performance of GPT-3.5-turbo (with 27 few-shot examples) was slightly better than BERT, on average. Their test sets contained, on average, 538 samples, which is a significant improvement over Zhong et al. (2023). However, the narrow focus on the evaluation tasks as well as only benchmarking a single encoder and decoder model makes it hard to generalise the results to other tasks and models.

Kocoń et al. (2023) built a benchmark suite of 25 tasks, where 21 of these tasks were classification tasks (binary, multi-class and multi-label), 3 being question answering tasks and the last one being a token classification task. Two of the classification tasks were in Polish, the rest being English. They compared the zero-shot and few-shot performance of GPT-3.5-turbo (January 2023 version) to the state-of-the-art encoder performance on each task. GPT-3.5-turbo was generally found to be worse than state-of-the-art encoder models. They also evaluated GPT-4 on five of the tasks (inference, question-answering and emotion datasets), and only found GPT-4 to be marginally better than GPT-3.5-turbo, still far off the encoder models.

Qiu and Jin (2024) compared GPT-3.5-turbo (January 2023 version) to a finetuned version of the base-sized BERT model on three manually curated English multi-class classification datasets with 19, 12 and 7 test samples, respectively, where they found that the BERT model performed marginally better than GPT-3.5-turbo in a few-shot setting (and that the zero-shot performance was significantly worse). The tiny test sets make it hard to generalise the results, however.

In recent times, several benchmarks of generative language models have been introduced. The major ones are EleutherAI’s Evaluation Harness Gao et al. (2023), Hugging Face’s Open LLM Leaderboard hug which uses the Evaluation Harness as evaluation engine, and Stanford University’s HELM Bommasani et al. (2023). These are firstly all English-only benchmarks, making it hard to generalise the results to other languages, and they only include point estimates of the dataset performance, and thus do not necessarily provide a robust assessment of the models. Further, these benchmarks are exclusively for decoder models, and thus does not provide a way to compare encoders with decoders.

There has been several language-specific benchmarks introduced as well. NorBench Samuel et al. (2023) is a collection of Norwegian evaluation datasets moreso than a dedicated evaluation framework. Further, several datasets in this collection (NorQuAD, NoReC and NorNE) are already part of ScandEval. SuperLim Berdičevskis et al. (2023) falls into the same category for Swedish. DUMB de Vries et al. (2023) is a Dutch benchmarking framework, which is only focused on encoder models. Danoliterate Holm (2024) is a Danish benchmarking framework, which has been based largely on the present ScandEval benchmark, and solely focused on evaluating decoder models. Aside from language modelling performance, the Danoliterate benchmark also measures calibration, efficiency, toxicity and fairness. While the development of language-specific benchmarks is important, it leads to too little overview of trends across benchmarks and languages and incentivises model development focused on monolingual models ignoring a potential broader appeal. ScandEval provides a unified and robust approach for comparison across model categories and Germanic languages.

Benchmarking is not the only way to evaluate language models. A new “arena approach” has been popularised by the LMSYS Arena Chiang et al. , where users can submit a prompt and get two responses from two anonymised models at random, and have to evaluate the responses. The Arena is predominantly used for English, but also currently supports six other languages. This approach is a promising way to evaluate language models, but we fear that it is not as suitable for low-resource languages due to the need of many volunteers to evaluate the responses.

Lastly, the Scandinavian Embedding Benchmark Enevoldsen et al. (2024b) complements ScandEval and focuses on evaluating embedding models on a wide range of tasks in the Scandinavian languages.

For Norwegian, Swedish and Icelandic we use the NorNE Jørgensen et al. (2020), SUC 3.0 Gustafson-Capková and Hartmann (2006), MIM-GOLD-NER Ingólfsdóttir et al. (2020) datasets, which were already included in the ScandEval framework. For Faroese we replace the previous WikiANN-fo dataset Rahimi et al. (2019) with the new human annotated FoNE dataset Snæbjarnarson et al. (2023). We also replace the previous DaNE dataset Hvingelby et al. (2020) with the new DANSK dataset Enevoldsen et al. (2024a) covering a wider variety of domains. For German, Dutch and English we add the established NER datasets GermEval Benikova et al. (2014), the Dutch part of CoNLL-2002 Sang (2002), and the English CoNLL-2003 Sang and De Meulder (2003).

We re-use the sentiment classification datasets AngryTweets Pauli et al. (2021), NoReC Velldal et al. (2018) and SweReC Svensson (2017), for Danish, Norwegian and Swedish, respectively. For German, Dutch and English we add the existing datasets SB10k Cieliebak et al. (2017), Dutch Social Gupta (2022) and SST5 Socher et al. (2013). We convert SST5 to the standardised trinary (negative, neutral, positive) format by converting the “very negative” and “very positive” labels to “negative” and “positive”, respectively.

For linguistic acceptability we re-use the ScaLA datasets for all the Scandinavian languages, and extend the ScaLA datasets by applying the ScaLA method from Nielsen (2023) to German, Dutch and English by using the German McDonald et al. (2013), Dutch van der Beek et al. (2002) and English Zeldes (2017) dependency treebanks.

Here we use the ScandiQA dataset Nielsen (2023) for Danish and Swedish, but replace the manually translated Norwegian ScandiQA dataset with the new curated NorQuAD dataset Ivanova et al. (2023). We further add the new Natural Questions in Icelandic dataset Snæbjarnarson and Einarsson (2022) for Icelandic. For German and English we add the existing extractive question-answering datasets GermanQuAD Möller et al. (2021) and SQuAD Rajpurkar et al. (2016), respectively. For Dutch we add the machine translated version of SQuAD to Dutch Havinga (2023). We acknowledge that a machine translated dataset is not ideal, and will return to this point when analysing the evaluation results.

In this section we describe how we rephrase the NLU tasks as text-to-text tasks, which makes it possible to evaluate generative models on the tasks. We formulate all the tasks as few-shot tasks, generally formatted as follows:

[prefix prompt]

[document prefix]: [document]

[label prefix]: [label]

[document prefix]: [document]

[label prefix]: [label]

(…)

[document prefix]: [document]

[label prefix]:

We found that the separation of the few-shot examples with double newlines makes it easier to know when to stop the generation - for the same reason, we ensure that there are no double newlines in any of the documents. See the prompts used for the English datasets in Table 2; a full table of the prompts used for all the tasks in all the languages can be found in the appendix.

For the sentiment classification task, we simply have the models generate translations of the three labels (positive, negative and neutral). For the linguistic acceptability task, also a text classification task, we use the translations of “yes” and “no” as the two labels, corresponding to whether the document is grammatically correct or not. For the extractive question answering task, we have the model output the answer directly. For this task we found that changing the label prefix from “Answer” to “Answer in max 3 words” resulted in a drastic improvement, due to many of the answers of instruction tuned models starting with unnecessary text akin to “The answer is”. Lastly, for the named entity recognition task, we require the output to be a JSON dictionary ISO/IEC 21778: (2017), with keys being the translated named entity tags, and values being lists of named entities of that category. To ensure that we are not biasing the evaluation toward models knowing the JSON format, we employ structured generation using the outlines package Louf (2023), which modifies the logits outputted by the model to ensure that the output is always a valid JSON dictionary in the aforementioned format.

We keep the evaluation methodology for the generative models to be as close to the methodology for encoder models in Nielsen (2023). We think of the few-shot examples as analogous to training examples for encoder models. Indeed, as von Oswald et al. (2023) shows, this assumption is theoretically grounded. We thus evaluate the models 10 times, where on each iteration we sample few-shot examples at random from the training split, and we evaluate the model on a bootstrapped version of the test split. As with the encoder models, this allows us to take into account more noise in evaluation process, resulting in more robust evaluation scores.

From the raw scores of the 10 evaluations per dataset, we need to aggregate the model scores into a single score. We want an aggregation method that satisfies the following criteria:

1. 1.

   Task Fairness: Each task should be weighted equally.

2. 2.

   Comparison: If we evaluate models in multiple languages, then it should be possible to meaningfully compare the language scores of these models with each other.

3. 3.

   Robustness: If two models do not have a significantly different score on a dataset, then the aggregated score should reflect this.

4. 4.

   Magnitude Preservation: The magnitude of the difference between the dataset score of two models should be reflected in the aggregated score.

5. 5.

   Minimal Change: Adding a new model should minimally affect the aggregated scores of the other models.

Before we introduce our chosen aggregation method, we will briefly discuss some common aggregation methods and how they do not satisfy the criteria.

The mean score is the most common aggregation method, which would simply be the mean of the 10 scores for each dataset, and then the mean of the dataset scores for each task. This method does not satisfy the Task Fairness criterion, as it does not take into account that metrics have different ranges and variances. The Comparison criterion is also not satisfied, as datasets vary from language to language, with some datasets being more difficult than others. It does, however, satisfy the Robustness, Magnitude Preservation and Minimal Change criteria.

The mean rank is another common aggregation method, where we compute the rank of each model on each dataset, and then take the mean of the ranks. This method satisfies the Task Fairness criterion, as it re-casts the scores into a common comparable framework, which therefore weights each task equally. For the same reason, it also satisfies the Comparison criterion (it is important here that we evaluate all the models on all the languages for this to be satisfied). It does not satisfy the Robustness and Magnitude Preservation criteria, by definition of rank. It partially satisfies the Minimal Change criterion, since it only affects the scores of the models which are worse than the new model.

We thus see that the mean score and mean rank methods satisfy a disjoint set of the criteria, but that they together satisfy all the criteria. Based on this observation, we introduce the mean rank score method, defined as follows. For each dataset, we start by sorting the models by their mean score on the dataset. As with a rank, we assign the best model with rank score 1. For the next best model, we conduct a one-tailed Welch’s t-test to see if the next best model is significantly worse than the first model ($p<0.05$). If so, we compute the absolute difference between the mean score of the two models, and divide that by the standard deviation of all the mean scores of the models on the dataset.

We then add this to the rank score of the first model. We continue this process for all the models to get the rank scores for the dataset, and to compute the overall score for the model, we take the mean of the rank scores for the datasets. An overview of this aggregation method can be found in Algorithm 1.

We note that the mean rank score has an intuitive interpretation: it is the average number of standard deviations from the best scoring model (+1).

This metric satisfies Task Fairness since we normalise all the scores by dividing by the standard deviation of the dataset scores. The Robustness criterion is satisfied due to our use of a one-tailed Welch’s t-test. The Magnitude Preservation criterion is also satisfied, as the magnitude of the difference between the dataset score of two models is reflected in the rank score. It also satisfies Comparison, as we compare the models on a common scale (same argument as the mean rank method). Finally, the Minimal Change criterion is partially satisfied, as adding new models only minimally changes the score of existing models. Concretely, adding new scores will affect the standard deviation normalising factor (this effect tends to zero as the number of models grows, however), and if the model beats all the other models then all the scores will be affected, due to the relative nature of the metric.

Excerpts of the English, Danish and Icelandic leaderboards can be found in Table 3, Table 4 and Table 5, respectively. We found that these three represent three main categories of languages with respect to the open-closed source divide. Similar excerpts for the remaining languages (Swedish, Norwegian, Faroese, German and Dutch) can be found in the appendix. The full leaderboards for all the languages can be found at https://scandeval.com.

From the English results we see that the state-of-the-art decoder model GPT-4-0613 Achiam et al. (2023) is still outperformed by the DeBERTa-v3-large and DeBERTa-v3-base models He et al. (2020) as well as the ELECTRA-base model Clark et al. (2020). Here GPT-4-0613 is, on average, 0.44 standard deviations worse than the best model. The same pattern is seen for Norwegian, Dutch, German and Faroese; see the appendix for the corresponding leaderboard excerpts.

In contrast, on the Danish leaderboard, the top-3 models are all decoder models, with GPT-4-0613 and GPT-4-1106-preview OpenAI (2023b) in the lead, followed by the closed-source DanskGPT-Chat-Llama3-70B model from Syv.AI¹¹1https://www.syv.ai/, being a continuation of the Llama-3-70B model AI@Meta (2024). The GPT-4-0613 model is, on average, 0.24 standard deviations from the best model. Similar results were found with Swedish; see the appendix for the corresponding leaderboard excerpt.

Lastly, for Icelandic, we see that the encoders and decoders are tied in performance, with the mDeBERTa-v3-base model and the GPT-4-1106-preview model being the top models. The GPT-4-1106-preview model is, on average, 0.24 standard deviations from the best model. We note that Icelandic is the only language where the switch from GPT-4 (gpt-4-0613) to GPT-4-turbo (gpt-4-1106-preview) resulted in a significant increase in performance. We speculate that this is due to the collaboration between OpenAI and Iceland OpenAI (2023a).

We can thus give an affirmative answer to research question (Q1), showing that decoder models can achieve significantly better NLU performance than encoder models, albeit with an order of magnitude more model parameters. For (Q3), we see that this varies between languages, but without being correlated to the language resource spectrum.

In this section we analyse our research question (Q2), asking whether the NLU performance results from the previous section is dependent on the type of NLU task.

Firstly, we analyse whether the score distribution across the four NLU tasks is different for the encoder and decoder models. This is done by applying a UMAP McInnes et al. (2018) to the results of a given leaderboard, stratified by both the mean rank score and whether the model is generative. UMAP plots for the English, Danish, Swedish, Norwegian, German and Dutch leaderboards can be found in Figure 1.

We see that the worst and best performing models have similar distributions, irrespective of whether they are generative or not. However, we also note that the rest of encoder and decoder models follow different “paths” in the UMAP space, leading to our hypothesis that they learn different things during their development.

In Figure 2 we show the correlation between a model being generative and its performance on the four NLU tasks. We see that being generative is a strong predictor for good question answering performance, as well as poor named entity recognition and linguistic acceptability performance. The correlation is weaker for sentiment classification and varies across languages. We also see that these findings seem to generalise across languages, both high- and low-resource. The large question answering performance persists for non-instruction-tuned decoder models (see the leaderboards at https://scandeval.com), showing a likely side-effect of the pre-training algorithm or the architecture of decoder models making them better at this task.