Linguistic Entity Masking to Improve Cross-Lingual Representation of Multilingual Language Models for Low-Resource Languages

Encoder-based multiPLMs such as mBERT and XLM-R were trained on monolingual data using the Masked Language Modeling (MLM) objective. These models have significantly enhanced the performance of various downstream tasks [14, 15, 16].

In BERT (and its multiPLM variant, mBERT), which was trained with MLM¹¹1BERT was also trained using the next sentence prediction task., 15% of the input tokens were randomly selected for corrupting, following a uniform distribution. Out of these, 80% of the time the tokens were replaced with a [MASK] token, 10% of the time the tokens were replaced with a random token and 10% of the time they were left unchanged. Here the MLM objective predicts the corrupted (both masked and replaced) tokens. Successor models such as XLM-R [2] adopt the same 15% masking percentage and 80%-10%-10% corruption rule during pre-training. The contextualized representations produced by these pre-trained models are then used to obtain sentence embeddings for downstream NLP tasks. However, these models have been reported to be suboptimal for cross-lingual tasks such as bitext mining, due to the lack of an explicit objective for improving cross-lingual representations [6].

To address this limitation, Conneau and Lample [7] introduced Translation Language Modeling (TLM), which extended the MLM objective using parallel data. TLM accepts a concatenated pair of parallel sentences as input, and tokens were masked from both sentences. The rationale was to utilize the context of its translation counterpart to accurately predict the masked token, there by strengthening the cross-lingual capability. TLM was applied in a continual pre-training step, on top of the MLM pre-trained model. In this setting, the MLM step was still required to learn the linguistic information inherent to the languages, while the TLM step strengthened the cross-lingual signal across the language pairs.

In BERT’s MLM strategy, the sub-words were masked. Subsequent work experimented by varying the type of tokens for masking, as summarized in Table 1. The follow-up work masked consecutive sub-words in text spans [12] and correlated text spans with Point-wise Mutual Information (PMI) masking [13]. Zhuang [17] proposed a heuristic-masking strategy where they considered the unmasked token prediction in addition to the masked token prediction during language model pre-training. Golchin et al. [18] utilized an in-domain keyword masking strategy for domain adaptation of the PLM. Most of these techniques have primarily relied on monolingual data and have been evaluated predominantly on high-resource languages.

Closest to our work is Entity/Phrase masking [11]. This contrasts with our work in three ways. Entity/Phrase masking, selected NEs, noun phrases and verb phrases for masking. As per analysis in Fig 1, we considered only verb and noun words, in addition to NEs for masking. Secondly, while Entity/Phrase masking masked all consecutive tokens within NEs or Noun/Verb phrases identified by a chunking tool, LEM takes a more targeted approach by limiting masking to a single token within the selected linguistic entity span. Finally, they followed a multi-staged pre-training approach. The pre-training stages were, sub-word masking similar to BERT, followed by Phrase masking and Named Entity masking. In comparison, our approach’s two continual pre-training stages apply the same LEM strategy with monolingual data and parallel data. They have evaluated the strategy only on High Resource Languages English and Chinese. Further, this strategy has not been extended with parallel data for cross-lingual improvement.

Wettig et al. [19] conducted an empirical study on what tokens should be masked and in what percentages. However, their study was limited to the English language and focused only on downstream tasks such as classification and question-answering. To date, no empirical study has explored these alternative masking strategies specifically for sentence retrieval tasks, particularly in the context of LRL pairs.

The theoretical framework of LEM in a monolingual setting ($LEM_{mono}$) can be described as follows:
Let the monolingual sequence $X$ be defined as $X=x_{1}~{}x_{2}~{}x_{3}~{}...x_{i}...~{}x_{n}$ where $x_{i}$ is a word and $n$ is the number of words in the sequence. After tokenization, sequence $X$ can be represented as $\bar{X}$ as in Eq. 1.

During training, the cross-entropy loss ($\mathcal{L}_{LEM_{mono}}$) for masked token prediction, as in Eq. 3 is minimized. In the equation, N is the total number of tokens for prediction and $y_{j}$ is the expected true label.

Finally, we extend the TLM objective with LEM into the parallel data setting (LEM_para). Here we concatenate the source sentence ($X=x_{1}~{}x_{2}~{}x_{3}~{}......~{}x_{m}$) and target sentence ($Y=y_{1}~{}y_{2}~{}y_{3}~{}......~{}y_{n}$) from the parallel sentence-pair as a single input example and obtain the tokenized output as represented by $\bar{Z}$ in Eq. 4. $\bar{X}=\bar{x}_{1}~{}\bar{x}_{2}~{}\bar{x}_{3}.......~{}\bar{x}_{k}$ and $\bar{Y}=\bar{y}_{1}~{}\bar{y}_{2}~{}\bar{y}_{3}.......~{}\bar{y}_{l}$ are the tokenized source and target sentences, respectively. $k$ and $l$ are the number of tokens (sub-words) in the source and target sentences (respectively) after tokenization.

Similar to LEM_mono, in this step, a single token from each linguistic entity (NE, verb or noun) from both languages are selected for corruption according to the 80%-10%-10% rule. If 15% of linguistic units were not found in the sequence, the balance is sampled from the remaining tokens. During training, the corrupted token prediction cross-entropy loss, ($\mathcal{L}_{LEM_{para}}$) (Eq. 5) is minimized. S and T correspond to the total number of tokens masked from the source and target side sentences respectively. $z_{s}$ and $z_{t}$ are the true tokens to be predicted.

Languages such as Sinhala and Tamil exhibit morphological richness, requiring words to be inflected based on attributes such as number, gender, and case category. Table 2 shows examples for such word inflections. Additionally, the presence of out-of-vocabulary words in LRLs often leads to an increased number of sub-words after tokenization. Therefore, approaches like whole-word masking, span masking, or entity/phrase masking tend to mask longer spans. This reduction in context weakens the ability to accurately predict the masked tokens, ultimately hindering representation learning. In contrast, LEM mitigates this issue by masking a single token from a linguistic entity, which we empirically prove in Section 7.1.

We experiment with independent and dependent monolingual data to observe its impact on the continual pre-training step $LEM_{mono}$. We sample 60,000 sentences from the MADLAD-400 and all the available 60,000 sentences from SiTa-Trilingual dataset for each language. Next, we continually pre-train XLM-R with those datasets separately with $LEM_{mono}$ and evaluate on the bitext evaluation dataset.

To assess whether increasing the independent training data would yield in any improvements, we repeat the experiment with a sample size of 100,000 from MADLAD-400. Finally, as an extreme case, we conduct a third experiment using 500,000 sentences for the Sinhala-Tamil language pair. Here, 60,000, 100,000, and 500,000 correspond to the training data size per language, with the full training set size being double the amount specified. We conduct this evaluation for all three language pairs.

We empirically evaluate various masking strategies and assess their performance on the bitext mining task. The masking strategies explored in this study are as follows:

This section describes the ablation experiments we conduct to determine the most contributing linguistic entity or their combination in the LEM strategy. We use the baselines as described in Section 5.3.

We identify the NEs in English, Sinhala, and Tamil sentences, using an in-house fine-tuned multilingual NER model [21]. To identify nouns and verbs in the sequences, we employ Flair POS tagger [22] for English, the Sinhala TnT POS Tagger [23, 24] for Sinhala, and ThamizhiUDp [25] for Tamil. Flair reported an F1 score of 98.19% and is the best model for English POS Tagging. The Sinhala POS Tagger had been trained using SVM and has an overall accuracy of 84.68% with a 59.86% accuracy for tagging unknown words. The Tamil POS Tagger is a neural-based model, with a F1 score of 93.27%. For Sinhala and Tamil, these are the models that returned optimal results.

The initial ablation experiment masks a single linguistic entity type, such as only NEs, only verbs, or only nouns. Subsequently, combinations of these linguistic entity types are examined.

In our experiments, 100%NE+15%MLM means, priority is given for sampling from NEs. If it does not produce enough tokens for masking, then the balance is sampled from the remaining tokens. When combining several linguistic entities, e.g. 100%NE+100%VB+15%MLM means, priority is given to sample the tokens for masking from both NEs and verbs.

We evaluate the success of our LEM masking strategy on three downstream tasks - bitext mining, parallel data curation and code-mixed sentiment classification.

We consider English, Sinhala and Tamil for our experiments. Sinhala and Tamil are the official languages of Sri Lanka and English is used as a link language. They belong to three distinct language families; Indo-European, Indo-Aryan and Dravidian language families, respectively. Further, Sinhala and Tamil are low-resource and medium-resource languages respectively [40, 41]. They are also morphologically rich languages. Sinhala, in particular, is only used in the island nation of Sri Lanka and has seen slow progress in language technologies [41, 42].

We preprocess the MADLAD-400 data to extract clean sentences for each language as follows: First, the document-level data is segmented into sentences using the nltk³³3https://www.nltk.org/index.html sentence tokenizer. We then filter these sentences using the LID (Language IDentification) model⁴⁴4https://github.com/gordicaleksa/Open-NLLB. Subsequently, we remove noisy data, including HTML tags, URLs, and sentences with less than 60% textual content. Finally, religious texts were excluded through a keyword filter.⁵⁵5Keywords being bible book names along with common words from the bible. However, for SiTa-Trilingual data, no preprocessing was applied as the data was of high quality.

We select XLM-R as the base multiPLM for our experiments. Other popular multiPLMs, XLM and mBERT do not cover Sinhala language. XLM-R has already demonstrated promising performance in downstream tasks for the low-resource languages considered in this study [4, 47, 49, 21]. It is a 278M parameter model, pre-trained for 100 languages. However, the amount of Sinhala and Tamil data used during XLM-R pre-training is much lower than that for English (English 55B, Sinhala 243M, Tamil 595M).

In our evaluation of downstream tasks, we set up two baseline experiments.

• •

  XLM-R - Obtain embeddings from the out-of-the-box XLM-R pre-trained model.

• •

  XLM-R_MLM+TLM [7] - We continually pre-train the XLM-R with MLM+TLM objectives and use the representation improved encoder to obtain embeddings.

Figure 4 shows the bitext mining results for the $LEM_{mono}$ step using independent and dependent monolingual data. The detailed results are available in Table 12 in Appendix  A.

The highest performance was observed for the dependent monolingual data, a trend consistent across all three language pairs. Surprisingly, when the independent set was increased to 100,000, this increase did not surpass the scores obtained using the dependent monolingual 50,000 training dataset. This was evident in all three language pairs. When the dataset size was further increased to 500,000, the results further deteriorated.¹⁶¹⁶16Due to resource constraints and the reduced performance for the Sinhala-Tamil direction, the experiment using 500,000 was not conducted for the other two language pairs. Therefore, it is evident that utilizing dependent monolingual data is advantageous during the LEM_mono step to enhance cross-lingual representations.

Table 6 shows the experimental results for the bitext mining task using XLM-R continually pre-trained with different MLM strategies (Section 4.2). Based on the averaged recall scores, the baseline XLM-R consistently delivered the highest performance in bitext mining tasks. An exception was observed for the Sinhala-Tamil BW criterion, where the sub-word masking strategy outperformed the baseline. However, a key observation was that continual pre-training with the existing masking strategies in general deteriorated the already learnt cross-lingual representations in the XLM-R model.

As outlined in Section 3.1, morphologically rich, low-resource languages tend to generate a higher number of sub-word tokens during the tokenization process due to the presence of infrequent words in the sequences. Consequently, masking strategies like whole-word masking and span masking result in longer masked spans, which reduce the available context for accurate predictions. We hypothesize that this reduced context reduces the prediction accuracy, thus weakening the already learnt representations in the XLM-R model. As a result, the existing masking strategies return degraded results.

The ablation study results for considering each linguistic entity along with combinations of them, on the LEM strategy are available in Tables 14, 15, 16 in the Appendix C. In Table 7, we summarise the final gains.

For the Sinhala-Tamil language pair, compared to the scores obtained with XLM-R raw embeddings, XLM-R_LEM returned the highest gain of +3.1 Recall points. Compared with XLM-R_MLM+TLM, this was a gain of +1.4 points. For the English-Tamil language pair, the LEM_mono step produced a reduced score compared to the XLM-R baseline. However, after the second continual pre-training step XLM-R_LEM scores surpassed the XLM-R baseline by +1.2. Further, the highest gain produced by our method compared to XLM-R_MLM+TLM was for the English-Tamil language pair, which was +2.4 points. With the English-Sinhala language pair, a similar behaviour was observed. Here XLM-R_LEM compared to baseline XLM-R has a gain of +0.4 points while the improvement compared to XLM-R_MLM+TLM was +1.7 scores.

In all these language pairs, the best scores were produced when the linguistic entity NEs were included in the LEM strategy. Due to the consistent gains across the three language pairs, we can safely conclude that LEM is favourable to improving the cross-lingual representations in existing multiPLMs.

Table 8 shows the NMT results for training the top 50,000 sentence pairs obtained by ranking the parallel sentences with the baseline and XLM-R_LEM models for the NLLB and CCAligned corpora. It can be observed consistently that both XLM-R_MLM+TLM and XLM-R_LEM improved models outperform the baseline XLM-R scores.

The NMT results show that the XLM-R_LEM model produce superior results compared to XLM-R_MLM+TLM, for all three language pairs across the two corpora. We believe the magnitude of the gain is dependent on the characteristics of the parallel corpus and the size of the training data sample. For the English-Tamil language pair, the CCAligned corpus produce a significant gain for XLM-R_LEM compared to XLM-R_MLM+TLM. This justifies the effectiveness of the LEM strategy which was not evident with random masking followed in MLM+TLM. The rest of the gains vary from +0.3 to +0.8 ChrF points. According to metric analysis by Kocmi et al. [51], these gains are equivalent to +0.48 to +1.12 BLEU points with a human accuracy of 54.2% to 66% respectively. This means the improvement in the translation quality in the NMT systems is almost in line with a minimum human accuracy rating of 54.2% to 66%.

The observations are consistent with other metrics, as shown in Table 13 in the Appendix B as well. The results further prove that the scoring from the XLM-R_LEM model has managed to identify quality sentence pairs more than the other models. Therefore, improvement in the cross-lingual representations with the LEM strategy benefits the parallel data curation task as well.

Table 9 summarizes the evaluation scores for the code-mixed sentiment analysis task on the English-Sinhala dataset. The XLM-R_LEM model consistently outperformed both XLM-R and XLM-R_MLM+TLM when fine-tuned directly on the code-mixed dataset.

In the zero-shot evaluation (fine-tuned on English-only data), all models showed reduced F1 scores. We suspect that the English monolingual fine-tuning step adversely impacted the cross-lingual alignment between the two languages.

When further fine-tuned on the code-mixed dataset, XLM-R_LEM achieved the highest scores, indicating that the cross-lingual enhancement benefits the code-mixed sentiment classification task. These results highlight the promise of LEM in improving multilingual models, particularly for low-resource and code-mixed language applications.

To evaluate the impact of the masked token count within linguistic entities, we conducted an ablation study by varying the number of masked tokens. We conducted this for the Sinhala-Tamil language pair. As reported in Table 16, the best result was returned for the 100% NE+15% MLM and 100% NE+15% TLM combinations in the $LEM_{mono}$ and $LEM_{para}$ steps respectively. Therefore, we used this setting and the number of tokens for masking was varied. Results on the bitext mining task are reported in Table 10. It reveals a clear trend of decreasing performance.

When masked only one token per linguistic entity, the average performance across tasks was the highest. This outcome suggested that minimal masking preserved more contextual information, allowing the model to better capture dependencies critical for downstream tasks. As the masked token count increased to two or more, the average performance dropped. This drop was significant when increasing the token count to 3 and 4. This indicated that excessive masking had disrupted the contextual integrity of linguistic entities, which lead to suboptimal representations.

Interestingly, the performance drop became less pronounced when the number of masked tokens increased from 3 tokens to 4 tokens (a decrease of only 0.02). This suggested a potential saturation point where further masking within an entity had diminishing negative effects, as the model might already struggle to leverage the remaining context effectively.

We investigated the impact of applying the LEM strategy to noisy data. This analysis provides critical insights into the robustness and adaptability of LEM when faced with real-world noisy data. We specifically focus on the English-Sinhala language pair and use the ParaCrawl¹⁷¹⁷17https://opus.nlpl.eu/ dataset.

As per Table 14, the current ablation revealed that the best results for English-Sinhala were achieved with the combination of 100%VB+15%MLM and 100%VB+15%TLM during the $LEM_{mono}$ and $LEM_{para}$ steps, respectively. We ran the LEM experiments with ParaCrawl data for the same combinations.

Table 11 presents the final scores. We observed that the results were comparable to those derived from the cleaner SiTa-Trilingual dataset. Further, in BW criteria the scores slightly surpass and in FW criteria the scores are the same as that obtained when using high-quality data. This equivalence underscores the resilience of the LEM strategy to noise in the training data.

The ability of LEM to maintain high performance in noisy settings highlights its practical applicability in low-resource scenarios, where parallel data is often noisy or inconsistent. This robustness not only complements our findings but also demonstrates that LEM can effectively mitigate the challenges associated with data quality, a common issue in low-resource language processing.