LAWDR: Language-Agnostic Weighted Document Representations from Pre-trained Models

Language representations map words, sentences or documents to dense vectors, capturing their semantic meaning Al-Rfou (2013). Cross-lingual representations project different languages into the same vector space. Recently there has been a surge of research on pre-training cross-lingual language models with large-scale corpora. These studies include LASER Artetxe and Schwenk (2018b), multilingual BERT Devlin et al. (2019), XLM Lample and Conneau (2019) and XLM-RoBERTa Conneau et al. (2019). Multilingual sentence representations produced by these models have led to strong improvements in downstream sentence-level tasks.

Despite the extensive study of cross-lingual sentence representations, there has been few works on cross-lingual document embeddings. Transformer based models such as BERT, XLM and XLM-RoBERTa have strict length limits on inputs and therefore cannot process long documents. While complex, neural-based models have been proposed, averaging sentence embeddings has remained a hard-to-beat baseline Guo et al. (2019). Here, we propose a document-level model based on weighted sentence embeddings, and the sentence weights are inversely related with its density in the corpus, akin to an inverse corpus frequency.¹¹1The intuition behind sentence weighting is that a sentence with higher density in a corpus is less informative and unique to its document, and thus should be assigned with a lower weight.

Cross-lingual representations enable the use of the same model for downstream tasks, regardless of their languages. In this work, we reveal that pre-trained cross-lingual embeddings are not language-agnostic, and that the language bias hurts their semantic representation power. We propose to remove the language bias efficiently and derive language-agnostic sentence representations. We show that debiased representations improve performance on cross-lingual tasks without the need for fine-tuning.

In this work, we propose Language-Agnostic Weighted Document Representation (LAWDR). Our document representations are built upon sentence embeddings obtained from off-the-shelf pre-trained language models. We first obtain language-agnostic sentence embedding by removing their language bias. Then we derive document representations from a weighted combination of these sentence embeddings. LAWDR is an unsupervised approach without model fine-tuning and task-specific supervision.

Cross-lingual document representations can be naturally applied to align multi-lingual document alignment and mine parallel data for machine translation Buck and Koehn (2016). To measure document similarity, we study and compare two similarity metrics, cosine similarity Mikolov et al. (2013) and the margin function Artetxe and Schwenk (2018a). Empirically, the margin metric is a better measure of cross-lingual document similarity. The learned document representation is evaluated on two public datasets from WMT16 and WMT19 shared tasks, and achieves strong performance in cross-lingual alignment.

We summarize our contributions below:
(1) We reveal that state-of-the-art pre-trained models carry language bias which hurts their semantic representation power. This work proposes an efficient approach to derive language-agnostic sentence embeddings.
(2) We propose an unsupervised approach LAWDR to derive language-agnostic document representations from sentence embeddings. It achieves strong empirical performance in cross-lingual document alignment.

Recently state-of-the-art performance has been achieved in cross-lingual tasks by pre-trained models such as BERT Devlin et al. (2019), XLM Lample and Conneau (2019) and XLM-RoBERTa (abbreviated as XLM-R in the following discussions) Conneau et al. (2019). These models learn contextualized representations for tokens in given input sentences. There are two commonly used approaches to obtain sentence representations from the token embeddings. One approach is to concatenate special tokens such as [CLS] in BERT, $\langle\backslash s\rangle$ in XLM and $\langle s\rangle$ in XLM-R with a sentence as the input sequence to the pre-trained models. Embeddings of these special tokens are taken as the sentence embedding Devlin et al. (2019); Lample and Conneau (2019). We refer to this method as special token (ST) approach for simplicity, and it demonstrates good performance on sentence-level tasks when fine-tuned on external labeled datasets.

The other way is mean pooling (MP), deriving sentence embeddings as the average of token embeddings. It is a simple yet efficient approach to encode compositional word semantics into sentences representations Le and Mikolov (2014). In this work, we study and compare sentence embeddings derived using these two approaches.

Cross-lingual representations are expected to be language-agnostic. However, we argue that this is not the case, as we will show both qualitatively and quantitatively that the pre-trained sentence embedding carries language signals.

We have shown that the sentence embeddings from pre-trained models are biased towards languages. Hence the cross-lingual representations are actually a mixture of language and semantic signals. Suppose that the sentence $s$ has an embedding $\mathbf{v}_{s}$ in the $r$-dimension space. We assume that the embedding space can be decomposed into two orthogonal subspaces: a language subspace $L$ and a semantic subspace $S$. Suppose that the basis vectors of space $L$ is $\{\mathbf{u}^{\text{lang}}_{i}\}_{i=1}^{m}$, and the basis of space $S$ is $\{\mathbf{u}^{\text{sem}}_{j}\}_{j=1}^{n}$. The basis vectors are orthogonal to each other within a subspace. Moreover, basis vectors from the two subspaces are also orthogonal, i.e., $\langle\mathbf{u}^{\text{lang}}_{i},\mathbf{u}^{\text{sem}}_{j}\rangle=0$, considering that subspace $L$ is orthogonal to $S$. Therefore, the sentence embedding $\mathbf{v}_{s}$ can be expressed as a sum of its language component $\mathbf{v}_{s}^{\text{lang}}$ and semantic component $\mathbf{v}_{s}^{\text{sem}}$.

The language component can be represented as a linear combination of language basis vectors, i.e., $\mathbf{v}^{\text{lang}}_{s}=\sum_{i=1}^{m}\alpha_{s,i}\mathbf{u}^{\text{lang}}_{i}$, where the coefficient $\alpha_{s,i}$ is its projection onto the basis $\mathbf{u}^{\text{lang}}_{i}$. Similarly, the semantic component consists of semantic basis vectors $\mathbf{v}^{\text{sem}}_{s}=\sum_{j=1}^{n}\beta_{s,j}\mathbf{u}^{\text{sem}}_{j}$, and $\beta_{s,j}$ is the projection onto $\mathbf{u}^{\text{sem}}_{j}$.

In this section, we will discuss how to derive language-agnostic document representation from the debiased sentence embeddings.

Document alignment is an important application of cross-lingual document representations, which also serves as an evaluation task for the semantic representation quality. The task is to measure the the semantic similarity of documents in a source and a target language. In this section, we study the geometry of the learned document representations and discuss the metric to measure document similarity from the embeddings.

Geometry of document representations. Cosine similarity is the most commonly used metric for language representations Mikolov et al. (2013); Guo et al. (2018). Given a document in a source language, we use the cosine similarity to find its nearest neighbors from a target language in the vector space. We observe that its parallel document is usually among its (top four) nearest neighbors but not the closest one.

A possible explanation is that the document representation carries more than semantic information Mu and Viswanath (2018). Language is one kind of non-semantic signal, which has been removed in debiasing procedure. Domain, syntax and text style are examples of non-semantic signals that have been found in language representations Mikolov et al. (2013); Nooralahzadeh et al. (2018). The non-semantic component would change the local geometry of document embeddings. Hence cosine similarity may not accurately reflect the semantic closeness.

Similarity metric. A margin function has been recently proposed to evaluate multilingual sentence similarity which outperforms cosine function Artetxe and Schwenk (2018a). Suppose that there two corpora $D_{l}$ and $D_{l^{\prime}}$ in language $l$ and $l^{\prime}$ respectively. A margin function $\text{margin}(d,d^{\prime})$ is defined for documents $d$ and $d^{\prime}$ from corpora $D_{l}$ and $D_{l^{\prime}}$ respectively:

where $\text{knn}(\mathbf{v}_{d},l^{\prime})$ is the embeddings of $k$ nearest neighbors in language $l^{\prime}$ of document $d$ in terms of cosine similarity.

Given a source document, Artetxe and Schwenk proposed to find parallel candidates using cosine similarity, and re-rank these candidates using margin function. The document with the maximal margin value is identified as the parallel document Artetxe and Schwenk (2018a).

We evaluate the document representations on a semantic similarity task – cross-lingual document alignment. It identifies parallel documents, and mines massive data for machine translation El-Kishky et al. (2019).

In our study, the pre-trained models are applied to generate representations for single sentences. It is natural to consider encoding multiple sentence so that these model could capture the inter-sentence information Ethayarajh (2019). Now we take as many sentences as possible under the length limit of the models, and concatenate them as one input sequence. Again we use MP for sentence embeddings, LAWDR for weighted document representations and margin function for the similarity metric.

Table 5 compares WMT19 recalls with single and multiple sentences as model inputs. With biased embedding, multi-sentence inputs outperform single-sentence inputs for all models. With debiased embedding, multi-sentence inputs yield gains of $6.8\%$ for BERT and $26.6\%$ XLM-R, while falls behind single-sentence inputs by $2.6\%$ for XLM. The complete results of all models on both datasets are provided in the supplementary material.

There has been a long research venue in multilingual sentence representations learning. Recently NLP has seen a surge in the study of multilingual pre-trained models. One of the strong models is Language-Agnostic SEntence Representations (LASER) Artetxe and Schwenk (2018b), which is built on LSTM network. Its representation quality degrades with the increase of text length. Other cross-lingual models are based on Transformer Vaswani et al. (2017) such as BERT Devlin et al. (2019), XLM Lample and Conneau (2019) and XLM-RoBERTa Conneau et al. (2019). These models are not able to encode long documents due to strict limits on inputs.

Despite a large body of research on monolingual document representations, there have been very few works in cross-lingual setting. The most recent work designs hierarchical multilingual document encoder (HiDE), which is trained on a large corpus of parallel documents Guo et al. (2019). It only experiments with English-French and English-Spanish language pairs. Considering the large cost of training the model on other languages, we do not include it in our experiments.

In this paper, we proposed LAWDR to learn language-agnostic document representation on top of pre-trained cross-lingual models. We first apply debiasing to mitigate language signals and derive language-agnostic sentence embeddings. The debiased sentence embeddings are then weighted with their inverse density as document representations. We further reveal that margin function is a more accurate measure of document similarity than cosine similarity with empirical evaluation. The learned document embeddings have achieved state-of-the-art performance on cross-lingual document alignment.