Multilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluation

When word embeddings were introduced [21], it was then shown that two or more monolingual embeddings could be aligned effectively [22]. State-of-the-art alignment methods for word embeddings give impressive results on bilingual lexicon induction (BLI), a task where the translation of a word is retrieved with nearest-neighbor search [23, 24, 25]. In our experiments, a specific version of those aligned word embeddings will be used as a baseline: monolingual FastText embeddings [26] aligned with RCSLS [24].

A multilingual word embedding provides a static representation of each word, meaning that a word will always be represented by the same vector regardless of the context, whereas the Transformer-based models which are the focus of our paper provide a contextualized representation of a word. The representation attributed to a word by such a deeper model varies according to the surrounding words.

Deeper multilingual language models were first proposed as a way to perform unsupervised machine translation [27]. Besides unsupervised translation models, LASER [28] is a multilingual sentence embedding based on RNNs and trained on several parallel corpora. MultiFiT [29] is a training procedure involving LASER and a monolingual model to perform a cross-lingual transfer. However, we will focus here on Transformer-based models which are pre-trained with fewer parallel texts and are not built specifically for sentence-level downstream tasks like LASER and MultiFit.

The Transformer architecture was initially proposed as an encoder-decoder architecture [1], for sequence-to-sequence tasks like translation. But BERT [6] uses only the encoder part of the architecture to set a new standard for tasks that require encoding the full sentence like sentence classification, named entity recognition, or extractive question answering. BERT is typically pre-trained to predict randomly masked-out words across a wide English corpus. A multilingual version of BERT, called mBERT, was proposed and is the same architecture pre-trained for the same task but on 104 monolingual corpora of distinct languages.

Despite not being pre-trained explicitly on parallel text, mBERT was shown to have surprisingly good cross-lingual transfer abilities [2, 19]. For example, when mBERT is fine-tuned on a classification task in English, it gives competitive results when evaluated in French. This has also been shown to work between typologically different languages, although to a lesser extent than similar languages.

Multilingual language models (MLLMs) are often either framed as a way to perform unsupervised machine translation or either as a ground for generalizing the learning of downstream tasks across languages. In this paper, we wonder whether MLLMs can hold “universal” patterns across languages, or at least patterns that are common to a wide range of different languages, and produce aligned representations.

Despite the surprisingly efficient cross-lingual transfer of models like mBERT, there is no consensus on whether those multilingual models learn universal multilingual patterns, as a recent literature review states [20]. With some probing tasks, it is shown that we can learn syntactic trees, which represent the grammatical structure of a sentence, that are consistent across languages with heuristics applied to attention activation values [30]. But other probing tasks show that the same models retain language-specific information at every layer [19, 31].

Another approach for studying multilingual patterns learned by a model is to perform inferences on parallel text. Different approaches lead to opposite results [2, 3]. As shown in Fig. 1, [2] compares sentence representations obtained by averaging sub-word representations, whereas [3] uses the representation of the initial token [CLS] which is supposed to hold a representation of the sentence. The former found aligned representations, while the latter concluded that mBERT “is not an interlangua”. We propose to compare word representations instead similarly to [5].

Indeed, even if the sentence-level alignment was guaranteed, does it necessarily mean that there is a word-level multilingual alignment in the representations produced by mBERT and others? the CLS representation is independent of the token representation. And if the same information was repeated in each token embedding, one could have an alignment of average sentence representation, but not of the single tokens.

It was shown that mBERT builds representations that enable the extraction of corresponding words in a pair of translated sentences with fair accuracy, and AWESOME [32] was proposed to fine-tune mBERT and improve this word-level matching. In another work [5], translated pairs of words were extracted from translated sentences with FastAlign [4]. the similarity between the contextualized representations built by mBERT for those words was compared with the similarity between representations of random words. They observed that the similarity for translated words was close to that of random pairs. We show that tools like FastAlign make too many alignment mistakes to be able to conclude about the quality of the multilingual alignment produced by models like mBERT. Our method relies on the extraction of translated words from translated sentences in a manner that is less prone to errors for evaluating multilingual models.

In order to extract translated words from translated sentences with a minimized number of errors, we start from a bilingual dictionary. And instead of trying to extract all possible word pairs with a word-alignment tool like FastAlign, we extract only those that we are certain of.

Our method requires two datasets: a translation dataset containing gold standard translated pairs of sentences and a bilingual dictionary. The bilingual dictionary contains pairs of translated words, like the pair ”rapide”-”fast” in our example Fig. 2. In a pair of translated sentences from the translation dataset, for each word from one sentence, every potential translated word indicated by the bilingual dictionary is collected from the other sentence. A pair of translated words is kept with its context if there is only one candidate for the translation.

The pairs obtained can be seen as a contextualized bilingual dictionary, where translated words go along with their context. We argue that we could not use pairs of words from the original bilingual dictionary directly because the models we want to evaluate are pre-trained on whole sentences and, since they have not been pre-trained on single words, they might not give relevant representations of words without context.

Working at the word level instead of the sentence level like [3] and [2] is chosen for several reasons. A good alignment of sentence representations does not necessarily guarantee a good alignment of word representations. Also, working at the word level allows a more direct comparison with multilingual word embeddings, which can then be used as a baseline.

Once we have extracted translated-in-context pairs of words, they are passed through the model we want to evaluate to produce contextualized representations of the translated words.

To build the representation of each contextualized word, the whole sentences are passed to the evaluated model (typically mBERT) and the contextualized representations of the words from the extracted word pairs are kept. We can extract such representations for each stacked Transformer block (or layer). By convention, the representation of index 0 will be the one from the input un-contextualized embedding layer. For example, mBERT, which uses 12 Transformer blocks, will produce 13 representations for a word, the 0th one for the initial embedding and the 1st to 12th for the output of each Transformer block.

For each layer of a Transformer-based model, a nearest-neighbor retrieval task is then performed on the produced representations. In a similar manner as [2], a fixed number $N$ of representations of translated-in-context pairs is randomly sampled. Then, for each of those sampled pairs of representations $(u_{i},v_{i})$, the aim is to check that $v_{i}$ is the nearest neighbor of $u_{i}$ with respect to all other $v_{j}$. The final evaluation score is then the proportion of pairs for which the translation is the nearest neighbor. More formally our score is given by:

Where $U$ and $V$ are the sets of sampled $u_{i}$ and $v_{i}$ respectively, $s$ is a similarity function or retrieval criterion and $N$ the number of sampled pairs.

The retrieval criterion used is the cross-domain similarity local scaling (CSLS) [23]. It is a modified cosine similarity commonly used for retrieval in word embeddings [23, 24, 25] which takes into account the density around the compared representations by deducing from the initial similarity the averaged cosine similarity of the $k$ nearest neighbor of each representation in the pair:

$c$ is the cosine similarity and $\mathcal{N}_{V}^{k}(u_{i})$ the set of $k$ nearest neighbors of $u_{i}$ in $V$.

In our reported experiments, we used a similarity based on the cosine similarity and not on the l2-distance for two reasons: (1) the CSLS criterion is commonly used for evaluating multilingual word embeddings which is our baseline and (2) we noticed that l2-distance gives similar results for the evaluated models which can be explained by the Transformer architecture using a layer normalization at the end of each Transformer block. If the representations have roughly constant norms, then l2-distance is approximately proportional to cosine distance.

With the retrieval score $S_{\text{weak}}$ described in Equation 1, we are only evaluating ”weak” alignment as described by [33]. Weak alignment is defined as the fact that the nearest neighbor in language $L_{1}$ of any item in a language $L_{2}$ is the most relevant item (i.e. the translation in our case). For example, as shown in Fig. 3, ”fast” will be closer to ”rapide” than ”slow” or ”cucumber” but ”lent” or ”concombre” can be even closer.

Strong alignment is defined as the fact that translated pairs are closer than irrelevant pairs, regardless of the language. In this case, ”fast” should be closer to ”rapide” than ”slow” and ”cucumber”, but also than ”lent” and ”concombre”. Strong alignment is useful for document retrieval. Indeed, if a document retrieval system is expected to answer a query with documents from different languages, it should be able to rank documents across languages.

We propose another retrieval score like in Eq. 1 which evaluates strong alignment by checking that for each pair of representations $(u_{i},v_{i})$ the translation $v_{i}$ is closer to $u_{i}$ than any element of $U$ instead of $V$:

The only change is in bold: $v_{j}$ has been replaced by $u_{j}$. For instance, we want the representation of the French word ”rapide” to be closer to its translation ”fast” than any other English word (instead of French for $S_{\text{weak}}$).

Before reporting our results on word-level alignment, we investigate the contradiction between [2] and [3] on sentence-level alignment for the mBERT model.

On the one hand, [2] performed a nearest-neighbor search similar to ours on sentence representations. Each sentence was represented by the average of the embeddings of its tokens and those vectors were centered for each language. High retrieval accuracy is observed with this method for typologically similar languages.

On the other hand, [3] reported a Canonical Correlation Analysis (CCA) across layers of the representation in various contexts of the initial token [CLS] which is expected to encode the meaning of the sentence. This method shows that those representations are dissimilar, and the dissimilarity grows stronger towards the deeper layers.

On Fig. 4 we observe the same decrease across layers, as [3], for the similarity between [CLS] tokens of translated sentences with the cosine similarity instead of the CCA. But the similarity decreases even more for an equal number of random pairs of sentences drawn from the same dataset.

The decrease of the similarity between translated pairs can be deduced from the fact that it reaches exactly 1 at the 0th layer, corresponding to non-contextualized embeddings which will have the same value if the token is the same, here ”[CLS]”. The similarity can do nothing but decrease when information from the context is injected into the contextualized representation of the [CLS] token.

However, to fairly compare representations based on the CLS token and averaged representations of the sub-words, they can both be used in the same nearest-neighbor task as proposed in our method for word representations. We apply Equation 1 to sentence representations. The sentence representations are not centered as in [2] as we want to evaluate directly the quality of alignment, and not to artificially improve it.

Results reported in Table I show that whatever the chosen layer is, averaged representations of the sentences provide better multilingual alignment than CLS representations. And for deeper layers, both give better alignment than aligned word embeddings averaged over the sentence. It must also be noted that alignment of the averaged mBERT representation suffers less from the typological distance between languages than the CLS representation or aligned FastText.

Sentence representations can give different results according to the chosen method, but it seems that whatever it is, sentence representations produced by mBERT are relatively aligned across languages. However, the CLS representation seems to be less relevant. Furthermore, the CLS token does not exist in all Transformer-based models, hence we recommend relying on averaged representations instead.

Fair multilingual alignment of sentence representations does not guarantee a good word-level multilingual alignment. In Section III, we argued that the proposed method is a way to extract translated words which is less prone to errors. Table II shows the proportion of accurate pairs extracted by our method and FastAlign [4]. It demonstrates that our method extracts proportionally more accurate pairs than FastAlign, although it provides fewer pairs in quantity to be fair. For 10 000 sentences from WMT19 for the English-German pair: our method extracts 50 590 word pairs and 190 665 for FastAlign.

[5] used FastAlign to compare the similarity of translated pairs of words and random ones with mBERT. When measuring that similarity on the last layer and plotting the sampled distribution, they observe that the pairs obtained with FastAlign give a very broad distribution that overlaps a lot with random pairs, which leads them to conclude that word-level representations built by mBERT are not well aligned across languages and it motivates them to propose a method to re-align representation after pre-training.

But because they use FastAlign, they are considering many unrelated pairs as translations. Fig. 5 shows those distributions for the eighth layer of mBERT. With its right and wrong extracted pairs, the distribution of FastAlign pairs (in blue) overlaps more with the distribution of random pairs (in purple) than the distribution of pairs extracted with our methods (in yellow). The distribution of random pairs from the same sentence pair (in light purple) is very close to the distribution of random pairs in the whole dataset (in dark purple), which justifies that any error in the extraction of a pair might increase the overlap between extracted pairs and random pairs.

This confirms that FastAlign generates too many mistakes to make an accurate evaluation of the multilingual alignment produced by a model.

Having advocated for our method, we perform the nearest-neighbor search described in Equation 1 for evaluating the multilingual alignment. Results for mBERT and five language pairs are shown in Fig. 6. We observe that for a few deep layers, mBERT produces representations that are better aligned than multilingual word embeddings.

Lower layers give worse alignment. This implies that multilingual patterns are high-level features. It also goes against the hypothesis made by several papers [3, 2, 19], that shared vocabulary is what allows models like mBERT to align representations without having been exposed to parallel texts.

The very last layers also give worse results than layers 8 to 10. [19] have shown that mBERT representations hold language-specific information at each layer. These language-specific components might take more importance in the last layers as the pre-training objective is to predict masked words, a language-specific task. Indeed, the model must learn not to replace a masked-out word with its translation.

This type of curve for $S_{\text{weak}}$ is observed for all the multilingual models we evaluated. Table III show retrieval scores for the first, best and last layer of models described in Section IV.

The best layer of mBERT gives the best results with respect to all other models. It might be due to the fact that the next sentence prediction task helps, an objective on which it is the only pre-trained model. However, it could also be explained by the fact that mBERT is trained solely on Wikipedia whereas models like XLM-R are trained on the CommonCrawl corpus which might contain texts that are less comparable across languages. It is also to be noted that the TLM objective on parallel texts proposed by the XLM model seems to make the alignment worse but it is also used in AWESOME.

As the different evaluated models have many differences and obtain somewhat similar results, one cannot isolate a single parameter that makes the multilingual alignment better or worse. Nevertheless, it seems that most of those multilingual models build multilingual representations that are competitive with word embeddings that have been explicitly aligned. Further research is needed for identifying the factors that make the quality of such alignment.

Finally, the same models are evaluated for the strong alignment retrieval criterion $S_{\text{strong}}$ defined in Equation 3. Results for mBERT are reported on Fig. 7 and results for all models are reported on Table IV.

The multilingual alignment of most of the models seems to be robust. Retrieval accuracy is significantly greater for those multilingual models than for multilingual word embeddings. There is yet again no way to identify what makes one Transformer-base model perform better than another. Nevertheless, we have demonstrated that there is a word-level strong alignment in most multilingual Transformer-based language models, even for those like mBERT and XLM-R which have no explicit information about the language in input and haven’t been pre-trained on parallel texts.