Enhancing Cross-lingual Transfer via Phonemic Transcription Integration

Despite recent advances in cross-lingual pre-trained language models (PLM) such as mBERT Devlin et al. (2019), XLM-R Conneau et al. (2020), PLMs remain heavily biased towards high-resourced languages due to the skewed amount of available pre-training data under parameter capacity constraints. This heavily affects the downstream task performance of less-represented languages during pre-training. In addition, as most high-resourced languages share the similar scripts (i.e. mostly Latin scripts), PLMs tend to perform well on languages sharing similar scripts with those languages Pires et al. (2019); Muller et al. (2021); Fujinuma et al. (2022). As most challenging low-resourced languages do not share similar scripts with common high-resourced languages, this limitation leaves them significantly behind.

To alleviate the challenges on low-resource or zero-resource target languages, recent works attempt to transfer knowledge from high-resourced languages (typically English (EN)) to low-resourced target languages via augmentations from bilingual dictionaries and parallel corpora. However, these approaches are restricted to English source language and results in less significant performance gain on languages that are more distant from English Yang et al. (2022); Fujinuma et al. (2022). Languages can be considered distant due to the differences in orthography, phonology, morphology and grammatical structures. The fact that performance drop occurs on distant languages when transferring from English indicates that additional works are needed to exploit connectivity between closely related languages, especially under extreme parameter constraints.

Besides purely relying on the orthographic representation in the format of written scripts, additional modality of languages such as articulatory signals can provide essential information beyond the written scripts to enhance textual representations (Bharadwaj et al., 2016; Chaudhary et al., 2018). Phonemic transcriptions which capture linguistic articulatory signals are beneficial to understanding non-Latin-based languages when integrated with PLMs Sun et al. (2021). They can also facilitate for knowledge transfer between languages sharing lexical similarities in phonology but possessing different writing scripts. As demonstrated in Table 1, despite differences in orthographic representations, the terms “电子 ” (ZH) and “Điện tử” (VI) possess significant phonemic similarities when encoded into International Phonetic Alphabet (IPA). Similarly, although “ベトナム ” (JA) and “베트남 ” (KO) are different, their phonemic representations (“\tipaencodingbIt@nAmu” and “\tipaencodingbEtunæm” respectively) are almost identical in terms of articulatory features.

Motivated by the inherent lexical similarities in terms of phonology among CJKV languages, we propose a novel cross-lingual transfer framework to integrate and synthesize two specific linguistic modalities (1) textual orthographic input scripts, (2) phonemic transcription, represented in International Phonetic Alphabet (IPA) format. Our unified cross-lingual transfer framework aims to effectively (1) align both orthographic and phonemic transcriptions via multi-modality learning, (2) capture additional alignment between the two modalities via contextual information, (3) enhance cross-lingual transfer of the two modalities with additional bilingual dictionary. Our work specifically targets Chinese-Vietnamese-Japanese-Korean languages which are not well-studied in cross-lingual transfer and possess lexical similarities with one another in terms of phonology. Our contributions can be summarized as follows:

• •

  We provide the first pre-processed orthographic-phonemic transcription alignment dataset for token-level tasks (i.e. Part-of-Speech Tagging (POS) and Named Entity Recognition (NER)) among CJKV languages (Chinese-Japanese-Korean-Vietnamese).

• •

  We propose a multi-modality learning paradigm with unsupervised alignment objectives to fuse the knowledge obtained from both modalities/ transcriptions to enhance cross-lingual transfer.

• •

  Our proposed framework yields consistent improvements over the orthographic-based multilingual PLMs (mBERT and XLM-R) on both POS and NER tasks.

In this work, we study the problem of Cross-lingual Transfer in a bilingual setting where there exists annotated data collection of high-resource language S, namely $D_{S}^{train}=\{(X_{i}^{(S)},Y_{i}^{(S)})\}_{i=1}^{N_{s}}$, and unlabeled data collection of low-resource target language T, denoted as $D_{T}^{test}=\{(X_{j}^{(T)})\}_{j=1}^{N_{t}}$. $N_{s},N_{t}$ denote the sample size of source language training data and target language inference dataset respectively.

Formally, given an i-th input source language utterance with the length of $M$ orthographic tokens $x_{i}^{(S)}=[x_{i,1}^{(S)},x_{i,2}^{(S)}...,x_{i,M}^{(S)}]$ and the corresponding phonemic transcriptions $z_{i}^{(S)}=[z_{i,1}^{(S)},z_{i,2}^{(S)}...,z_{i,M}^{(S)}]$ and token-level labels $y_{i}^{(S)}=[y_{i,1}^{(S)},y_{i,2}^{(S)}...,y_{i,M}^{(S)}]$, the overall training objective is summarized as:

$$\theta_{S}=\underset{\theta}{\mathrm{argmin}}\frac{1}{N_{s}}\sum_{i=1}^{N_{s}}l(F(x^{(S)}_{i},z^{(S)}_{i};\theta),y_{i}^{(S)})$$

(1)

where $F(\cdot)$ denotes the transformation function that takes an input of both $x_{i}^{(S)},z_{i}^{(S)}$ to output probability prediction of label $y_{i}^{(S)}$ for individual tokens. $\theta$ denotes the parameters of the transformation framework and $l(\cdot)$ is the token-level cross-entropy loss.

The overall trained framework is then evaluated in a zero-shot setting on target language $T$ as follows:

$\displaystyle p(y^{(T)}|x^{(T)},z^{(T)})=\underset{k}{argmax}F(x^{(T)},z^{(T)};\theta_{S})$

(2)

where $k$ denotes the label space of token-level tasks. In our cross-lingual transfer settings, no labeled target data or parallel data between source and target language is leveraged during training.

In this section, we introduce our proposed Cross-lingual Transfer Framework, namely PhoneXL, with 3 major learning objectives: (1) Orthographic-Phonemic Alignment ($\mathcal{L}_{align}$), (2) Contextual Cross-modality Alignment ($\mathcal{L}_{MLM}$) , (3) Contextual Cross-lingual Alignment ($\mathcal{L}_{XMLM}$). The overall learning objective is summarized as follows:

$$\mathcal{L}=\mathcal{L}_{task}+\alpha\mathcal{L}_{align}+\beta\mathcal{L}_{MLM}+\gamma\mathcal{L}_{XMLM}$$

(3)

where $\mathcal{L}_{task}$ denotes the corresponding downstream token-level task and $\lambda,\beta,\gamma$ correspond to weights of the respective losses for balanced loss integration. For $\mathcal{L}_{task}$ is computed based on the training objective in Equation 1 as we leverage the generic CRF layer on top of the sequence output from PLM to generate the probability distribution of each token over the token-level class labels.

Our experimental results for NER and POS tasks are summarized in Table 3 and 4. Based on the empirical study, our proposed PhoneXL framework consistently outperforms by the backbone PLM architecture of mBERT and XLM-R in both evaluated token-level tasks for low-resourced target languages VI and KO. For instance, with ZH-VI pair, we observe the target language’s F1 evaluation metric improvement of 2.34 points and 2.01 points for NER and POS tasks respectively as compared to the fine-tuned backbone mBERT architecture. This improvement implies the phonemic information provides essential information beyond orthographic representation to further bridge the gap between the source and target languages.

In NER task, the larger and state-of-the-art multilingual PLM XLM-R yields worse performance than mBERT in both of our language pair performance on source and target languages. On the other hand, for POS task, XLM-R based architecture only results in marginal performance gains when compared with mBERT. Interestingly, our mBERT-based framework achieves competitive performance with XLM-R backbone architecture on POS task despite leveraging smaller vocabulary size and less pre-training data. We hypothesize this might be due to the fact that XLM-R has been trained with more languages, leading to biases towards certain types of languages during pre-training that might not share common properties with CJKV languages.

Despite the performance gain over XLM-R based architecture observed in Table 3, our PhoneXL framework does not consistently outperform the previous baselines such as FILTER, xTune and XMIXUP in Table 4. However, these baselines require extra parallel corpora obtained from machine translation which might not always be readily available for all languages, especially for low-resourced ones. On the other hand, our proposed method achieves state-of-the-art performance among methods leveraging only bilingual dictionary. In addition, despite its impressive performance, xTune requires two-stage training procedures, four different exhaustive augmentation methods as well as the knowledge of both machine translation and bilingual dictionary. Therefore, it is deemed more time-consuming and resource-intensive than our approach.

In our work, we propose PhoneXL, a novel mechanism to integrate phonemic transcription with orthographic transcription to further enhance representation capability of language models in cross-lingual transfer settings. By encouraging the alignment between the two linguistic modalities via direct one-to-one alignment, indirect contextual alignment and additional code-switching via bilingual dictionaries, our proposed PhoneXL yields consistent performance improvements over the backbone orthographic-based PLM architecture in downstream cross-lingual token-level task among the CJKV languages. We also release the first aligned phonemic-orthographic datasets for CJKV languages for two popular token-level tasks (NER and POS). In future work, we plan to train our proposed unsupervised objectives with larger CJKV corpora as pre-training mechanisms to evaluate effectiveness of the representations in multi-granularity downstream tasks (i.e. sentence-level classification tasks to Question-Answering tasks). Further extensions towards few-shot learning settings Nguyen et al. (2020); Xia et al. (2020) where a small number of target language examples can be leveraged to exploit orthographic-phonemic similarity between source and target languages is a promising direction for our future work.

Our approach is heavily dependent on the quality of the pre-processed orthographic-phonemic transcription data as it provides the ground-truth for unsupervised alignment objectives. Generating phonemic transcriptions and aligning them correctly with orthographic representations can be costly. Despite our significant efforts, the alignment is still far from perfect optimality.

Secondly, our approach might not be effective in improving performance on randomly chosen language pairs. As our framework aims to exploit phonemic similarities of languages with different orthographic representations, the methods are only effective in cross-lingual transfer between lexically similar languages in terms of phonology such as CJKV languages. Languages that do not fall into this category might observer little to no performance gains with our proposed framework.