Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction

Bilingual Lexicon Induction (BLI) aims to find bilingual translation lexicons from monolingual corpora in two languages (Haghighi et al., 2008; Xing et al., 2015; Zhang et al., 2017a; Artetxe et al., 2017; Conneau et al., 2017), and is applied on numerous NLP tasks such as POS tagging (Zhang et al., 2016), parsing Xiao and Guo (2014), and machine translation Irvine and Callison-Burch (2013); Qi et al. (2018).

Most work on BLI learns a mapping between two static word embedding spaces, which are pre-trained on large monolingual corpora Ruder et al. (2019). Both linear mapping Mikolov et al. (2013); Xing et al. (2015); Artetxe et al. (2016); Smith et al. (2017) and non-linear mapping Mohiuddin et al. (2020) methods have been studied to align the two spaces. Recently, other than the static word embeddings, contextual representations are used for BLI due to the significant progresses on cross-lingual applications Aldarmaki and Diab (2019); Schuster et al. (2019). Although the static word embeddings and the contextual representations exhibit properties suited for alignment, there is no works to combine the two paradigms.

On one hand, the static word embeddings have been widely used for BLI, but one specific embedding mapping function does not ensure that in all conditions, words in a translation pair are nearest neighbors in the mapped common space. On the other hand, the contextual representations contain rich semantic information beneficial for alignment, but the dynamic contexts of word tokens pose a challenge for aligning word types.

In this paper, we propose a combination mechanism to utilize the static word embeddings and the contextual representations simultaneously. The combination mechanism consists of two parts. The first part is the unified word representations, in which a spring network is proposed to use the contextual representations to pull the static word embeddings to better positions in the unified space for easy alignment. The spring network and the unified word representations are trained via a contrastive loss that encourages words of a translation pair to become closer in the unified space, and penalizes words of a non-translation pair to be farther. The second part is the weighted interpolation between the words similarity in the unified word representation space and the words similarity in the contextual representation space.

We test the proposed combination mechanism in both the supervised BLI setting which can utilize a bilingual dictionary as the training set, and the unsupervised BLI setting which does not allow using any parallel resources as supervision signal. On BLI benchmark sets of multiple language pairs, our combination mechanism performs significantly better than systems using only the static word embeddings and systems using only the contextual representations. Our mechanism improves over robust BLI baselines on all language pairs, achieving average 3.2 points improvement in the supervised setting, and average 3.1 points improvement in the unsupervised setting.

The early works on Bilingual Lexicon Induction (BLI) date back to several decades ago, including feature-based retrieval (Fung and Yee, 1998), distributional hypothesis (Rapp, 1999; Vulić and Moens, 2013), and decipherment (Ravi and Knight, 2011). Following Mikolov et al. (2013), which pioneered the embedding based BLI method, word representation based method becomes the dominant approach, and can be categorized into two classes: static word embedding based method, and contextual representation based method.

• •

  Static Word Embedding Based Method

  Word embeddings of different languages are pre-trained in large monolingual corpora independently. Then a mapping function is applied to align the embedding spaces of the two languages (Mikolov et al., 2013; Xing et al., 2015; Artetxe et al., 2016; Smith et al., 2017).

  We follow one robust BLI system VecMap (Artetxe et al., 2018a, b), which maps both source space and target space into a third common space. Let $E_{x}$ and $E_{y}$ be the word embedding matrices in two languages for a given bilingual dictionary such that their $i$th rows are the embeddings of words of the $i$th translation pair in the dictionary. The training objective is to find mapping functions $W_{x}$ and $W_{y}$ such that

  $$W_{x}^{\star},W_{y}^{\star}=\underset{W_{x},W_{y}\in M_{d}(\mathbb{R})}{\mathop{\arg\max}}cos(E_{x}W_{x},E_{y}W_{y})$$

  (1)

  where $d$ is the dimension of the embeddings, $M_{d}(\mathbb{R})$ is the space of $d\times d$ matrices of real numbers. The optimal $W_{x}$ and $W_{y}$ maximizes the cosine similarity between words of each translation pair in the mapped common space. In the unsupervised version where no bilingual dictionary is given, an artificial dictionary is initialized and iteratively updated through training $W_{x}$ and $W_{y}$ according to equation (1) (Artetxe et al., 2018b).

  Both mapping functions are constrained to be orthogonal during training by setting $W_{x}=U$ and $W_{y}=V$, where $U\Sigma V^{T}=X^{T}Y$ is the singular value decomposition of $X^{T}Y$. Such orthogonal constraint is based on the assumption that the source embedding space and the target embedding space are isometric, which is a particularly strong assumption that does not hold in all conditions (Zhang et al., 2017b; Søgaard et al., 2018). To depart from the isometry assumption, Patra et al. (2019) uses a semi-supervised technique that leverages both seed dictionary and a larger set of unaligned word embeddings, Mohiuddin et al. (2020) uses a non-linear mapping function that is not constrained to be orthogonal.

  We propose another method to relax the isometry assumption by combining the contextual representations with the word embeddings to compensate the shortage of the overly strong assumption.

• •

  Contextual Representation Based Method

  Contextual representations can be obtained through multilingual pre-training, which encodes whole sentence and outputs contextual representation for each word (Devlin et al., 2019; Lample and Conneau, 2019). Due to the rich context information contained in the contextual representations, there are endeavors to align them in different languages Schuster et al. (2019); Aldarmaki and Diab (2019); Wang et al. (2020); Kulshreshtha et al. (2020); Cao et al. (2020).

  Since a word may appear in different sentences with different contexts, Schuster et al. (2019) use an average anchor to summarize multiple contexts for a word type and align the anchors of different languages, while other works aim to align each individual context representation based on parallel corpora, including learning alignment on sentence level representations and applying the learned mapping on word level contextual representations Aldarmaki and Diab (2019), using word alignments in a parallel corpora to learn the mapping for word contextual representations Wang et al. (2020), and directly minimizing the distance between two contextual representations of an aligned word pair in parallel corpora without mapping Cao et al. (2020).

  We adopt the average anchor method for the contextual representations Schuster et al. (2019), which does not depend on parallel corpora. Let the contextual representation of a source word $x$ in context $c_{i}$ be denoted as $r_{x,c_{i}}$. If $x$ appears a total of $p$ times in the source corpus, the average anchor for $x$ across all contexts is:

  $$a_{x}=\frac{\sum_{i=1}^{p}r_{x,c_{i}}}{p}$$

  (2)

  Similar to the mapping for the static word embeddings, we conduct mapping for the average anchors. Let $A_{x}$ and $A_{y}$ be the matrices of average anchors in two languages with correspondence to word pairs from a given bilingual dictionary. The mapping functions $V_{x}$ and $V_{y}$ are optimized by maximizing $cos(A_{x}V_{x},A_{y}V_{y})$, where $A_{x}$ and $A_{y}$ are fixed, $V_{x}$ and $V_{y}\in M_{d^{\prime}}(\mathbb{R})$, and $d^{\prime}$ is the dimension of the contextual representations.

Besides the above methods, there is another direction that extracts word alignments in pseudo parallel corpora for BLI. The pseudo parallel corpora are built by either the unsupervised machine translation (Artetxe et al., 2019) or the unsupervised bitext mining Shi et al. (2021). Both methods need significant computation overload or use monolingual corpora that are magnitudes larger than ours, and are beyond the scope of this paper that focuses on representation based methods.

Since there is no work to combine both the static word embeddings and the contextual representations, we propose a combination mechanism illustrated in Figure LABEL:fig:framework. The mechanism first builds a unified word representation space that unifies the static word embeddings and the contextual representations, then performs similarity interpolation between the unified space and the contextual space.

We test our combination mechanism in supervised and unsupervised BLI tasks on English-Espanish (EN-ES), English-Arabic (EN-AR), English-Chinese (EN-ZH), English-German (EN-DE), and English-French (EN-FR).

The static word embeddings in our paper are trained by skip-gram or CBOW, while the word embeddings from XLM/mBART are trained by the pre-training objectives. Different training objectives result in quite different word embeddings. Actually, they show remarkably different behavior for BLI. By using VecMap, the word embeddings from XLM/mBART perform averagely around 30 points lower than the static word embeddings used in our paper. We also test fastText with 1024 dimension and word2vec with 300 dimension for fair comparison. They all perform remarkably better than the word embeddings from XLM/mBART. We plug in the word embeddings from XLM/mBART in place of the fastText static embeddings in our combination approach, and obtain much worse performance. This indicates that the static word embeddings trained by skip-gram or CBOW are more suitable for BLI and our combination approach.

In addition, regarding the asymmetry in Figure LABEL:fig:framework, we used to try the spring function that takes the static word embeddings as input, but got much worse results. This indicates that the spring function conditioned on the static space may be not helpful for BLI. Such observation may also explain that symmetrizing Figure LABEL:fig:framework by yielding two unified spaces to combine performs slightly worse than the asymmetry version of Figure 1 of this paper. This is because that the spring function conditioned on the static space is introduced to maintain the symmetry, while this introduced spring function is not helpful for the combination.

Most BLI systems use either the static word embeddings or the contextual representations, but there is no works to combine both. In this paper, we propose a combination mechanism, which consists of the unified word representations and the similarity interpolation. The unified word representations use a spring network to pull the static word embeddings with offsets produced by the contextual representations, and compose a unified space such that parallel words are nearest neighbors to each other. The similarity interpolation is applied afterward to interpolate the similarities in the unified space and the contextual representation space. BLI experiments on multiple language pairs show that our combination mechanism can utilize the merits of both the static word embeddings and the contextual representations, achieving significant improvements over robust baseline systems in both the supervised and the unsupervised BLI tasks.

The authors would like to thank the anonymous reviewers for the helpful comments. This work was supported by National Natural Science Foundation of China (Grant No. 62036004, 61673289)