Samsung R&D Institute Philippines at WMT 2023

Given that a significant portion of the training dataset is synthetically-aligned, we need to use a comprehensive data preprocessing pipeline to ensure good translation quality. In particular, we use a combination of heuristic-based, ratio-based, and embedding-based methods to filter our data.

We experiment with two model sizes for each language pair: a Base model with 65M parameters and a Large model with 200M parameters. Both models use the standard Transformer Vaswani et al. (2017) sequence-to-sequence architecture and are trained using Fairseq Ott et al. (2019) with the hyperparameters listed in Table 2.

We parallelize with 8 NVIDIA Tesla P100 GPUs and initially train for a total of 100K steps for experimentation. For the submitted systems trained with backtranslated data, we train for a total of 1M steps.

We use backtranslation Sennrich et al. (2015a) as a form of data augmentation to improve our initial models. We generate synthetic data via combined top-k and nucleus sampling:

$$\begin{split}\sum_{i=0}^{\delta_{k}}P(\hat{y}^{(T)}_{i}|x;\hat{y}^{(T-1)})*\delta_{temp}\leq\delta_{p}\end{split}$$

(1)

where $\delta_{k}$ is the top values considered for top-k sampling, $\delta_{temp}$ is the temperature hyperparameter, and $\delta_{p}$ is the maximum total probability for nucleus sampling.

Backtranslation is only performed once using the provided monolingual data. We produce a total of 10,000,000 synthetic sentences for the en$\rightarrow$he direction and 73,278,018 synthetic sentences for the he$\rightarrow$en direction. The same data preprocessing used on the original parallel corpus is then applied to the synthetic corpus. We produce backtranslations using Large 100K models with the sampling hyperparameters listed in Table 3.

Statistics on generated synthetic data before and after filtering can be found on Table 1.

We further improve translations by using Noisy Channel Reranking Yee et al. (2019), which reranks every candidate translation token $\hat{y}^{(T)}_{i}$ using Bayes’ Rule, as follows:

$$\begin{split}P(\hat{y}^{(T)}_{i}|x;\hat{y}^{(T-1)})=\\ \frac{P(x|\hat{y}^{(T-1)})P(\hat{y}^{(T-1)})}{P(x)}\end{split}$$

(2)

where $P(\hat{y}^{(T)}_{i})$ refers to the probability of the $i$th candidate token at timestep $T$ given source sentence $x$ and current translated tokens $\hat{y}^{(T-1)}$.

All probabilities are parameterized as standard encode-decoder Transformer neural networks: the Direct Model $f_{\phi_{D}}(x,\hat{y}^{(T-1)})$ models $P(\hat{y}^{(T)}_{i}|x;\hat{y}^{(T-1)})$ or translation between source to target language; the Channel Model $f_{\phi_{C}}(x|\hat{y}^{(T-1)})$ models $P(x|\hat{y}^{(T-1)})$, or the probability of the target translating back into the predicted translation, and; the Language Model $f_{\phi_{L}}(\hat{y}^{(T-1)})$ models $P(\hat{y}^{(T-1)})$ or the probability of the translated sentence to exist. $P(x)$ is generally not modeled since it is constant for all $y$. This allows us to leverage a strong language model to guide the outputs of the direct model, while using a channel model to constrain the preferred outputs of the language model (which may be unrelated to the source sentence).

During beam search decoding, we rescore the top candidates using the following linear combination of all three models:

$$\begin{split}P(\hat{y}^{(T)}_{i}|x;\hat{y}^{(T-1)})^{{}^{\prime}}=\frac{1}{t}log(P(x|\hat{y}^{(T-1)})\\ +\frac{1}{s}[\delta_{ch}log(P(x|\hat{y}^{(T-1)})\\ +\delta_{lm}log(P(\hat{y}^{(T-1)}))]\end{split}$$

(3)

where $s$ and $t$ are source / target debiasing terms, $\delta_{ch}$ refers to the weight of the channel model, and $\delta_{lm}$ refers to the weight of the language model.

For Noisy Channel Reranking, our direct and channel models use the same size and setup at all times (i.e. if the direct model is a Large model trained for 100K steps, then the channel model is also a Large model trained for 100K steps in the opposite translation direction).

For the language model, we train one Base-sized decoder-only Transformer language model for English and one for Hebrew. We concatenate the cleaned data from the parallel corpus with the provided monolingual data for each language to train the LM. We use the same training setup as with translation models, except we use a weight decay of 0.01 and a learning rate of 5e-4.

Hyperparameters used for decoding with Noisy Channel Reranking can be found in Table 4.

We evaluate our models using two metrics: BLEU Papineni et al. (2002) and ChrF++ Popović (2015), both scored via SacreBLEU⁴⁴4SacreBLEU outputs the following signature for evaluation: nrefs:1—case:mixed—eff:no—tok:spm-flores—smooth:exp—version:2.2.1 Post (2018). We develop our models using both the FLORES 200 Costa-jussà et al. (2022) and NTREX 128 Federmann et al. (2022) datasets, using the validation sets during training and reporting scores on the test sets.

To benchmark our models’ performance, we mainly compare BLEU and ChrF++ against two (unconstrained) models: mBART 50 M2M Tang et al. (2020), a  610M-parameter finetuned version of mBART for many-to-many translation, and NLLB 200 MoE Costa-jussà et al. (2022), the full 54.5B-parameter mixture-of-experts version of NLLB 200 for many-to-many translation.

To find the best values for $\delta_{ch}$ and $\delta_{lm}$, as well as to understand how these parameters affect performance, we use Bayesian Hyperparameter Search. We use the Large 1M + BT models and run 1000 iterations of search, keeping the length penalty static at 1.0, and sampling both $\delta_{ch}$ and $\delta_{lm}$ from a gaussian with minimum of 0.01 and maximum of 0.99.

We perform this for both en$\rightarrow$he and he$\rightarrow$en translation directions and use the results for the final submission model.

Our submission systems (Large 1M + BT + NC) exhibit strong performance on both translation directions. On FLORES-200, we achieve 44.24 BLEU for en$\rightarrow$he and 42.42 BLEU for he$\rightarrow$en. The same systems score 33.77 BLEU for en$\rightarrow$he and 36.89 BLEU for he$\rightarrow$en on NTREX-128.

We note that these systems perform strongly when compared against much larger, unconstrained baseline models. On FLORES-200, we significantly outperform mBART 50 M2M on en$\rightarrow$he by +24.75 BLEU and on he$\rightarrow$en by +11.92 BLEU despite having 67% less parameters (200M vs 610M). Notably, our system performs only slightly worse compared to NLLB 200 MoE despite having 96% less parameters compared to the mixture-of-experts model. On FLORES-200, we perform -2.56 BLEU worse on en$\rightarrow$he and -6.58 BLEU worse on he$\rightarrow$en compared to NLLB 200 MoE.

In order to find optimal hyperparameters for both $\delta_{ch}$ and $\delta_{lm}$, we ran bayesian hyperparameter search for both at the same time while keeping length penalty static. We plot the results of the hyperparameter search over 1000 iterations in Figure 1.

We observe that performance is optimal when both hyperparameters are set to 0.2$\sim$0.3, making performance increasingly worse as both hyperparameters approach closer to 1. We hypothesize that this signifies the model capturing the original distribution close enough that it does not need much correction or aid from the accompanying language model. Noisy channel reranking, however, is still empirically shown to be useful in this case as guidance from the language model produces better candidates in cases where the direct model may be searching a too-constrained space.

We explored multiple configurations of our submission systems in terms of model size, presence of synthetic data during training, and the use of reranking methods during online decoding. Our results show that each step improves performance directly:

• •

  The initial Base 100K performs at 39.88 BLEU for en$\rightarrow$he on FLORES-200.

• •

  Increasing the size to 200M parameters (Large 100K) improves performance by +1.38 BLEU.

• •

  Adding backtranslated data (Large 100K + BT) is by far the most beneficial, improving performance by +2.06 BLEU.

• •

  We then experiment with longer training times (1M iterations for Large 1M + BT) to adapt to the new dataset size, increasing the score by +0.44 BLEU.

• •

  Finally, using noisy channel reranking (Large 1M + BT + NC) improves the score by +0.48 BLEU.

Overall, all of our methods improve performance by a total of 4.36 BLEU for the en$\rightarrow$he direction on FLORES-200.

We note an interesting jump in performance from Base 100K to Large 1M + BT + NC on the FLORES-200 he$\rightarrow$en direction at +30.36 BLEU. Base 100K underperforms at 12.06 BLEU, and we hypothesize that this is due to the model not having enough capacity to embed information from Hebrew, which causes it to greatly benefit from the guidance of a language model during noisy channel reranking.