Cross-lingual Word Embeddings in Hyperbolic Space

Cross-lingual word representations have been a subject of extensive research Upadhyay et al. (2016); Ruder et al. (2019). Recent advances in the field can be grouped into unsupervised, supervised, and joint learning algorithms. Unsupervised models Lample et al. (2017); Artetxe and Schwenk (2019); Chen et al. (2018) exploit existing monolingual word embeddings, followed by various cross-lingual alignment procedures. Supervised models Mikolov et al. (2013); Smith et al. (2017); Grave et al. (2018) learn a mapping function from a source embedding space to the target embedding space based on different objective criteria. Joint learning models Coulmance et al. (2015); Josifoski et al. (2019); Sabet et al. (2019); Lachraf et al. (2019) use parallel corpora to train bilingual embeddings in the same space jointly. This work adopts the settings similar to the joint learning model for embedding alignments by Lachraf et al. (2019).

Hyperbolic spaces offer a continuous representation for embedding tree-like structures with arbitrarily low distortion Sala et al. (2018); Chami et al. (2020). Word embeddings in hyperbolic spaces have been applied to diverse NLP applications such as text classification Zhu et al. (2020), learning taxonomy Astefanoaei and Collignon (2020), and concept hierarchy Le et al. (2019). By using hyperbolic space these applications were able to outperform their euclidean counterparts by exploiting the benefits of hierarchical structure of the text data with high quality embedding which capture similarity and generality of concept together enforce transitivity of the is-a-relations in a smaller embedding space Le et al. (2019). Some recent work use supervised models Nickel and Kiela (2017, 2018); Ganea et al. (2018) that require external information on word relations such as WordNet or ConceptNet in addition to free text corpora to learn word and sentence embeddings in the hyperbolic space. Nickel and Kiela (2017) consider a non-parametric method to learn hierarchical representation from a lookup table for symbolic data. Ganea et al. (2018) propose a supervised method to learn embeddings for an acyclic graph structure of words. Unsupervised word embedding models Leimeister and Wilson (2018); Dhingra et al. (2018); Tifrea et al. (2018) which can directly learn from text corpora have been recently applied in the hyperbolic spaces. Leimeister and Wilson (2018) employ the skip-gram with negative sampling architecture of the Word2Vec model for learning word embeddings from free text. Dhingra et al. (2018) present a two-step model to embed a co-occurrence graph of words and map the output of the encoder to the Poincaré ball using the algorithm from Nickel and Kiela (2017). Tifrea et al. (2018) remodel the GloVe algorithm to learn unsupervised word representation in hyperbolic spaces.

Hyperbolic space in Riemannian geometry is a homogeneous space of constant negative curvature with special geometric properties. Hyperbolic space can endow infinite trees to have nearly isometric embeddings. We embed words using the Poincaré ball model of the hyperbolic space.
The Poincaré Ball. The Poincaré ball model $\mathcal{B}^{n}$ of $n$-dimensional hyperbolic geometry is a manifold equipped with a Riemannian metric $g^{B}$. Formally, an $n$-dimensional Poincaré unit ball is defined as $(\mathcal{B}^{n},g^{B})$ and the metric $g^{B}$ is conformal to the Euclidean metric $g^{E}$ as $g^{B}={\lambda_{x}}^{2}.g^{E}$. Where $\lambda_{x}=\frac{2}{1-||x||^{2}}$, $x\in\mathcal{B}^{n}$, and $||.||$ stands for the Euclidean norm. Notably, the hyperbolic distance $d_{\mathcal{B}^{n}}$ between $n$-dimensional points $(x,y)\in\mathcal{B}^{n}$ in the Poincaré ball is defined as:

$$d_{\mathcal{B}^{n}}(x,y)=\operatorname{arcosh}\left(1+2\frac{||x-y||^{2}}{(1-||x||^{2})(1-||y||^{2})}\right)$$

(1)

where $\operatorname{arcosh}(w)=\ln(w+\sqrt{w^{2}-1})$ is the inverse of hyperbolic cosine function. Using ambient Euclidean geometry, the geodesic distance between points $(x,y)$ can be induced using Equation (1) as $d_{\mathcal{B}^{n}}(x,y)=\operatorname{arcosh}\left(1+\frac{1}{2}{\lambda_{x}}{\lambda_{y}||x-y||^{2}}\right)$. This indicates that the distance changes evenly w.r.t. $||x||$ and $||y||$, which is a key point to learning continuous representation for hierarchical structures Chen et al. (2020); Saxena et al. (2020).

We first adopt the mono-lingual hyperbolic word embedding from a model defined in the work by Tifrea et al. (2018). We extend it to cross-lingual hyperbolic word embedding by using parallel text corpora input to capture word relationsships through bilingual word co-occurrence statistics. Tifrea et al. (2018) added a hyperparameter function $h$ on the distance between word and context pairs in the hyperbolic Word2Vec’s objective function. Hence, the effective distance function in the objective function becomes $h(d_{\mathcal{B}^{n}}(x,y))$.

Hyperbolic word embeddings have shown to embed general words near the origin and specific words towards the edges – we attempt to exploit this property to identify latent hierarchies and in hypernym evaluation task by using the Poincaré norms of the words to determine their hierarchy as words with higher norm will be more specific, i.e., lower in hierarchy Nickel and Kiela (2017); Dhingra et al. (2018); Linzhuo et al. (2020). We evaluate the hyperbolic model on the cross-lingual analogy task to compare it with its Euclidean counterpart.

To train the cross-lingual Word2Vec model in the hyperbolic space, we perform a pre-processing step of word-to-word alignment as defined by Lachraf et al. (2019) using parallel sentences from a bilingual parallel corpus. We generate word-to-word alignment by matching the indices of tokens from both languages in parallel sentences.

Hypernymy Evaluation. We perform hypernymy evaluation to assess performance of the proposed model based on learning the latent hierarchical structure from free text. In the hypernymy evaluation task, given a word pair $(u,v)$, we evaluate $is$-$a(u,v)$ i.e., to what degree $u$ is of type $v$.

For English, German and cross-lingual German-English hypernymy evaluation, we use the HyperLex benchmark Vulić et al. (2017, 2019), which contains word pairs $(u,v)$ and a corresponding degree to which $u$ is of type $v$ i.e. the $is$-$a$ score. This score has been obtained by human annotators, scored by the degree of typicality and semantic category membership Vulić et al. (2017). For example, in the HyperLex dataset, $is$-$a(chemistry,science)=6.00$ and $is$-$a(chemistry,knife)=0.50$ as chemistry is a type of science but not a type of knife.

To generate the $is$-$a$ score we follow the same approach as used by Nickel and Kiela (2017):

$$\textit{$is$-$a$}(u,v)=-(1+\alpha(||v||-||u||))d_{\mathcal{B}^{n}}(u,v)$$

(2)

The evaluation is performed by calculating the Spearman correlation between the ground-truth score and the predicted score. Note that our model is not trained on any hypernymy detection task but tries to learn latent hierarchy from free text.

Cross-lingual Analogy Evaluation. The analogy evaluation task is one of the standard intrinsic evaluations for word embeddings. In cross-lingual analogy evaluation task, given a word pair $(w_{1},w_{2})$ in one language, and a word $w_{3}$ in the other language, the goal is to predict the word $w_{4}^{*}$ such that $w_{4}^{*}$ is related to $w_{3}$ same way $w_{2}$ is related to $w_{1}$. For example, as prince ($w_{1}$) is to princess ($w_{2}$), prinz ($w_{3}$; German equivalent for prince) is to prinzessin ($w_{4}^{*}$; German equivalent for princess). For evaluating cross-lingual analogy for the German and English language, we use the cross-lingual analogy dataset provided by Brychcín et al. (2018).

This paper uses the Wikipedia corpus of parallel sentences extracted by Wołk and Marasek (2014) to train the model. The dataset is accessed through OPUS Tiedemann (2012). The corpus consists of ~2.5 million parallel aligned German-English sentence pairs with 43.5 million German tokens and 58.4 million English tokens.

We reference Tifrea et al. (2018)’s Poincaré Word2Vec implementation³³3https://github.com/alex-tifrea/poincare_glove and extended it to learn cross-lingual word embeddings. We set the minimum frequency of words in the vocabulary to 100, and a window size of 5. The models use Negative-Log-Likelihood loss. The non-hyperbolic vanilla Word2Vec uses Stochastic Gradient Descent optimizer, whereas hyperbolic Word2Vec uses Weighted Full Riemannian Stochastic Gradient Descent optimizer Bonnabel (2013). For hyperbolic embeddings, the hyperparameter $h$ is set to $cosh^{2}(x)$. During the analogy evaluation, we use the cosine distance instead of Poincaré distance for hyperbolic models. We use the hypernymysuite⁴⁴4https://github.com/facebookresearch/hypernymysuite for hypernymy evaluation Roller et al. (2018).

Hypernymy Evaluation. We present the top closest children of selected words in Table 1. As described in Section 3.2, the closest children are calculated by finding the target word’s $(t)$ nearest neighbours $(N)$ and extracting the neighbour $n\in N$ such that $||n||_{p}>||t||_{p}$, where $||.||_{p}$ is the Poincaré norm. We observe that the model is able to find the hyponyms of the words using the closest children across languages. For example, the children of ‘Physics’ are its subtypes – ‘astrophysik’ (astrophysics), ‘astrophysics’, ‘mechanik’ (mechanics), and ‘biophysics’.

Table 2 reports the results on the hypernymy evaluation task. Although the models were not trained on hypernymy tasks, we observe that they could still learn some latent hierarchies from the free text across languages. Word pairs with out-of-vocabulary words were ignored during evaluation.

Table 3 shows lists of related words in order of increasing hyperbolic norm and specificity, similar to Dhingra et al. (2018)’s evaluation. We show counts of these words in the corpus. Higher the count, more generic the word, and has a smaller hyperbolic norm. The Spearman correlation between 1/$f$, where $f$ is the frequency of a word in the corpus, and its embedding’s hyperbolic norm is $0.747$ using a 300D w/bias Poincaré model.

Cross-lingual Analogy Evaluation. Table 4 reports the results on the cross-lingual analogy task. We observe that for 20D models, hyperbolic model outperformed the vanilla model. For higher dimension models, hyperbolic Word2Vec performed on par with its Euclidean counterpart. Similar to hypernymy evaluation, analogy pairs with out-of-vocabulary words were ignored during evaluation.