An Effective Method using Phrase Mechanism in
Neural Machine Translation

In the NLP area, Machine Translation is the primary task that has a long time history of development, especially in the approaches using Neural Networks (Sutskever et al., 2014; Bahdanau et al., 2015). Besides, there are numerous proposed architectures in this domain that have got a significant effect on the NLP community in many domains (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017; Devlin et al., 2019). Therefore, in this work, we focus on Machine Translation task in The Ninth International Workshop on Vietnamese Language and Speech Processing (VLSP 2022)¹¹1https://vlsp.org.vn/vlsp2022.

VLSP 2022 is a competition of one of the biggest NLP communities in Vietnam, Association for Vietnamese Language and Speech Processing, hosted by the VNU University of Science, Hanoi. This is a good chance for researchers in many domains to introduce their awesome research results and catch the state-of-the-art machine learning models in different domains.

This year, the machine translation task is designed to translate from Vietnamese to Chinese and on the contrary direction. In this task, given the input is a natural sentence in Chinese (or Vietnamese), the machine translation system is required to generate a new sentence in Vietnamese (or Chinese) that has the same meaning as the input sentence (Table 1).

Recently, the self-attention mechanism has occupied a lot of attention, especially the Transformer architecture (Vaswani et al., 2017; Devlin et al., 2019). This architecture leads many state-of-the-art results in various NLP domains (Devlin et al., 2019; Lewis et al., 2020; Raffel et al., 2020). In this work, we applied a new approach based on the Transformer architecture, PhraseTransformer (Nguyen et al., 2023), which leverages a phrase attention mechanism to solve the machine translation task. The main idea of this approach is to enhance the word representation by its local contexts (or phrases), and apply the self-attention mechanism to model the dependencies between phrases in a sentences. The experimental results on the machine translation dataset of the VLSP 2022 workshop show that our PhraseTransformer clearly beat the original Transformer model in both two directions from Vietnamese to Chinese and from Chinese to Vietnamese. Furthermore, the PhraseTransformer does not require any external syntax tree information as the previous works (Yang et al., 2018; Wang et al., 2019; Nguyen et al., 2020; Bugliarello and Okazaki, 2020) and is more lightweight compared with other phrase-level attention models (Xu et al., 2020). To this end, our PhraseTransformer improved the original Transformer by 1.1 BLEU scores on the setting from Vietnamese to Chinese and 1.3 BLEU scores on the setting from Chinese to Vietnamese.

The remainder of this paper, we detail our work in four sections. Section 2, we show the related works in this domain. And describe the details of our PhraseTransformer architecture in section 3. Then, the experiments and result analysis are shown in section 4. Finally, in section 5, we conclude this paper.

In the success of Transformer Vaswani et al. (2017) in the machine translation task, there are a numerous works proposed new directions to improve this architecture. Among them, the approach considering of utilize the phrase linguistic structure representation is promising. The works presented in Yang et al. (2020); Shang et al. (2021); Nguyen et al. (2022) demonstrate that the template prediction can guide an NMT system in the decoding process to improve performance. The works (Wu et al., 2018; Nguyen et al., 2020; Wang et al., 2019) indicate that the phrase information extracted from the syntax tree can improve the meaning representation of sentence. However, the performance of these systems is affected by the quality of the syntax tree extraction step which is usually lower performance in low-resource languages. Close to our PhraseTransformer, Xu et al. (2020) also presents a different method to model the phrase in a source sentence, however, this architecture uses a huge number of parameters to learn the attention scores between source phrases with target words. Compared with this work, by using LSTM architecture (Hochreiter and Schmidhuber, 1997) in the Multi-head layers to model various local context information, our PhraseTransformer increases the model size with a small margin but works effectively.

In this section, we describe our PhraseTransformer model (Nguyen et al., 2023), which is used to submit the machine translation task in VLSP 2022 workshop (Figure 1). We define the input vector ($\mathbf{x}$) is synthesized from the word embedding and positional encoding $\mathbf{x}=[\mathbf{x}_{1},...,\mathbf{x}_{|S|}]$ ($|S|$ is sentence length) similar to Vaswani et al. (2017). This vector is processed by a Dense layer to get multi-views of input data corresponding to different heads.

$\displaystyle\mathbf{q}_{i},\mathbf{k}_{i},\mathbf{v}_{i}$

$\displaystyle=\mathbf{x}\mathbf{W}_{i}^{q},\mathbf{x}\mathbf{W}_{i}^{k},\mathbf{x}\mathbf{W}_{i}^{v}$

(1)

where $i$ is the head index, $1\leq i\leq{H}$ where $H$ is the number of heads in Multi-Head layer. We define a $\operatorname{Phrase}$ function to extract local context:

(2)

where $m_{t}$ is the hyper-parameter gram size, $\mathbf{s}_{i}$ is the sequence of words vector, $n\mathrm{\_gramLSTM}_{k}$ is the function which capture local context using LSTM model:

	$\displaystyle m_{t}\mathrm{\_gram}_{k}(\mathbf{s}_{i})$	$\displaystyle=[\mathbf{H}_{k-n_{i}+1},\mathbf{H}_{k-n_{i}+2},...,\mathbf{H}_{k}]$
	$\displaystyle n\mathrm{\_gramLSTM}_{k}(\mathbf{s}_{i})$	$\displaystyle=\mathrm{LSTM}^{f}_{i}(m_{t}\mathrm{\_gram}_{k}(\mathbf{s}_{i}))+$
		$\displaystyle\quad\quad\mathrm{LSTM}^{b}_{i}(m_{t}\mathrm{\_gram}_{k}(\mathbf{s}_{i}))$

Then, the phrase information is integrated to the word representations:

	$\displaystyle\operatorname{zip}(A,B)$	$\displaystyle=[A^{\intercal};B^{\intercal}]^{\intercal}$		(3)
	$\displaystyle\mathbf{ph}^{k}_{i,m_{t}},\mathbf{ph}^{q}_{i,m_{t}}$	$\displaystyle=\operatorname{unzip}(\operatorname{Phrase}(\operatorname{zip}(\mathbf{q}_{i},\mathbf{k}_{i}),m_{t}))$		(4)
	$\displaystyle\mathbf{ph}^{q}_{i}$	$\displaystyle=\operatorname{zip}(\{\mathbf{ph}^{q}_{i,m_{t}}\mid m_{t}\in\mathbf{m}\})$		(5)
	$\displaystyle\mathbf{ph}^{k}_{i}$	$\displaystyle=\operatorname{zip}(\{\mathbf{ph}^{k}_{i,m_{t}}\mid m_{t}\in\mathbf{m}\})$		(6)

After that, these vectors are fed to the Self-Attention layer to learn the dependencies between words and phrases.

$\displaystyle\mathbf{head}_{i}$

$\displaystyle=\mathrm{softmax}(\frac{\mathbf{ph}^{q}_{i}\,\,{\mathbf{ph}^{k}_{i}}^{\intercal}}{\sqrt{d_{h}}})\,\mathbf{v}_{i}$

(7)

where $d_{h}$ is dimension of one head in Multi-Head layer. Finally, all head vector representation is combined to get the final sentence vector ($\mathbf{h}_{out}$) following Vaswani et al. (2017):

	$\displaystyle\mathbf{h}_{mh}$	$\displaystyle=[\mathbf{head}_{1};...;\mathbf{head}_{H}]\mathbf{W}^{o}$		(8)
	$\displaystyle\mathbf{h}_{no}$	$\displaystyle=\mathrm{LayerNorm}(\mathbf{h}_{mh}+\mathbf{x})$		(9)
	$\displaystyle\mathbf{h}_{out}$	$\displaystyle=\mathrm{LayerNorm}(\mathrm{FeedForward}(\mathbf{h}_{no})+\mathbf{h}_{no})$		(10)

This sentence vector is processed via a stack of $N$ Identical layers with the same architecture and forwarded to the Transformer Decoder layer (Vaswani et al., 2017) for decoding process.

In this section, we describe in detail about our experiments and the corresponding results.

In this paper, we first applied our PhraseTransformer model to the machine translation task for the pair of languages Vietnamese - Chinese. In this architecture, the original Transformer Encoder is enhanced by incorporating the phrase dependencies information into the Self-Attention mechanism. Our experimental results showed that our PhaseTransformer beat the original Transformer by a large margin in both translation directions. In future work, we would like to apply our model architecture to other NLP tasks and explore other effective phrase modeling methods to achieve better results.