FreeTransfer-X: Safe and Label-Free Cross-Lingual Transfer from Off-the-Shelf Models

For a specific natural language processing (NLP) task, given a model $\theta_{src}$ and the unlabeled data $\mathcal{D}_{src}$ in the source language $src$ and the unlabeled data $\mathcal{D}_{tgt}$ in the target language $tgt$. As shown in Figure 1, we propose to train a model $\theta_{tgt}$ in the target language $tgt$ via 2 KD steps:

1. 1.

   Leverage the NLP capability of the off-the-shelf model $\theta_{src}$, e.g. an English sentence classifier $\theta_{en,cls}$. We distill knowledge from the teacher $\theta_{src}$ to the student mPLM $\theta_{src}^{m}$ on data $\mathcal{D}_{src}$.

2. 2.

   Due to the zero-shot cross-lingual transfer capability of the mPLMs, $\theta_{src}^{m}$ implicitly achieve the NLP capability on the target language $\theta_{tgt}^{m}$. Then similar to the step 1, we distill knowledge from the teacher $\theta_{tgt}^{m}$ to the student $\theta_{tgt}$ in the target language $tgt$ on data $\mathcal{D}_{tgt}$.

The proposed framework works for arbitrary network including but not limited to Transformers (Vaswani et al., 2017), BiLSTM (Schuster and Paliwal, 1997) and CNN (Kim, 2014).

The training is purely based on KD that no other training objectives is included. We only apply KD between the classification distribution $P_{T}(\cdot)$ and $P_{S}(\cdot)$ of the teacher and the student respectively, which is compatible to arbitrary model architecture. Freezing the parameters of the teacher, we train the student by minimizing the Kullback-Leibler Divergence ($Div_{KL}$, Joyce, 2011) between them. Denote the prediction category as $\mathcal{C}=[c_{0},c_{1},...,c_{k}]$, then the $Div_{KL}$ can be formalized as,

However, KD objectives of different NLU tasks varies a lot. We classify NLU tasks into two categories: 1) sentence-level tasks like sentence classification, 2) word-level tasks like sequence tagging. Given an input example $\mathcal{X}\in\mathcal{D}$ as a sequence of words $\mathcal{X}=[x_{0},x_{1},...,x_{n}]$. For sentence-level tasks, $\mathcal{X}$’s sentence-level category is $\mathcal{C_{\mathcal{X}}}$. The teacher model and student model respectively output sentence-level prediction distribution $P_{T}(\mathcal{C_{X}}|\mathcal{X})$ and $P_{S}(\mathcal{C_{X}}|\mathcal{X})$. For word-level tasks, $\mathcal{X}$’s word-level category is $\mathcal{C}_{x_{i}},i\in[0,n]$. Then the KD objective can be written as,

It’s worth noting that word-level $Div_{KL}$ cannot be directly applied for heterogeneous teacher and student models since their tokenizations are different. In order to align the predictions of teacher and student, we only adopt the prediction on the first sub-word of each word.

During the first KD step that training the mPLM from an English (i.e. the source language) classifier, to leverage the cross-lingual transferarability of mPLM, the conventional method is to train the mPLM only on the English corpus. However, in our preliminary experiments, we notice that the mPLM’s accuracy gap between English and the target languages are very huge. It’s over 5% between the English target model (94.0) and the average of all target models (88.4), as reported by 2-step KD in Table 7, Appendix A.

Hence, we propose to translate the unlabeled English corpus $\mathcal{D}_{en}$ to target languages $\mathcal{D}^{trans}_{tgt}$, as depicted by Figure 2. Since $\mathcal{D}_{en}$ and $\mathcal{D}^{trans}_{tgt}$ are aligned, source English model’s predicted distribution $P_{en}(\mathcal{C}|\mathcal{X})$ of $\mathcal{D}_{en}$ can be directly applied to $\mathcal{D}^{trans}_{tgt}$. In this way, KD is able to be performed on not only the source language but also the target languages.

As shown in the lower left of Figure 1, the translated $\mathcal{D}^{trans}_{tgt}$ is incorporated in the training of KD step one.

Inspired by data augmentation for KD (Jiao et al., 2020) and multilingual paraphrase generation (Guo et al., 2019), we augment the unlabeled target corpus $\mathcal{D}_{tgt}$ via paraphrasing.

We experiment with three mainstream NLU model architectures to verify the universality of the proposed framework. They are used as the backbones of the off-the-shelf models $\theta_{src}$ and the output models $\theta_{tgt}$ in target language.

English is regarded as the source language in all the experiments. Off-the-shelf English models are trained on the hold-out annotated English corpus as described in Section 3.1. All the experimented models are controlled in comparable model scale. AdamW (Loshchilov and Hutter, 2019) is adopted as the optimizer with $\epsilon=1e-8$. We train the models for 50 epochs and take the checkpoint of the best validation accuracy as the final model. Table 2 reports the hyper-parameters of the model architectures.

Initial learning rate is decided based on a gradient-based searching heuristics proposed by Smith (2015), since in our preliminary experiments Smith (2015) stably finds better learning rates than manual searching. We build vocabularies of 10k words for each language via Byte Pair Encoding (BPE) Sennrich et al. (2016). Experiments are implemented with PyTorch (Paszke et al., 2019) and conducted on a single Nvidia V100 32GB GPU.

As shown in Table 1, the proposed 2-step KD framework significantly outperforms the MT baselines on most model architectures, except for the CNN of Translate-test. Although Translate-test is strong in a very few cases, it requires 2-pass inference (MT and classification) that results in a high latency. On the contrary, the proposed framework directly produces classification models in the target languages, which is more efficient. In addition, the language-balanced distillation and language-specific data augmentation further enhance our model to a large extent, +2.3% and +1.1% respectively. Language-wise results in Table 7 demonstrate the robustness of our method across various languages.

To our surprise, the naive 2-step KD model even performs on par with the Gold-supervised reference on average. We guess it’s due to the regularization effects of knowledge distillation that brings a good generalizability to the proposed model. It implies the proposed framework may be a annotation-free alternative to current zero-shot cross-lingual transfer framework.

However, comparing the results of the English source model and those of the target models in Table 7, the cross-lingual transferred models still lag far behind the original English models. There is a great potential of the proposed framework.

On the sequence tagging task, the proposed model beats the baselines by a wide margin. The MT-based baselines perform very poor on this task due to the error from word-level annotation alignment. Also because of the alignment error, we do not apply language balanced distillation and language-specific data augmentation on this task.

As to the comparison with the Gold-supervised reference, our model performs slightly worse than it. It may due to the insufficient knowledge distillation from the teacher to the student, which comes from the discrepancy between teacher’s and student’s tokenizations. Although, as described in Section 2.2.2, we perfectly align their prediction at word-level, only the first subword of each word is used for distillation. More informative subword-level aligning and distillation methods can be explored. We leave this problem for the future research. Besides, similar to sentence classification, gap between the English source model and the transferred target models is huge, as shown in Table 3.

In sum, both experimental results on sentence classification and sequence tagging demonstrate that the proposed model is significantly stronger than MT-based cross-lingual transfer methods. Furthermore, the proposed model only slightly lags behind or even performs on par with the strong Gold-supervised reference, which is not able to address the FreeTransfer-X problem.