Making Better Use of Bilingual Information for
Cross-Lingual AMR Parsing

Abstract Meaning Representation (AMR) Banarescu et al. (2013) is a rooted, labeled, acyclic graph representing sentence-level semantic of text. Nodes in the graph are concepts in the texts and edges in the graph are relations between concepts. Since AMR abstracts away from syntax and preserves only semantic information, it can be applied to many semantic related tasks such as summarization Liu et al. (2015); Liao et al. (2018), paraphrase detection Issa et al. (2018), machine translation Song et al. (2019) and so on.

Previous works on AMR parsing mainly focus on English, since AMR is designed for English texts and parallel corpus of non-English texts and AMRs are scarce. Early work of AMR announces that AMR is biased towards English and is not an interlingua Banarescu et al. (2013). Besides, some studies show that aligning AMR with non-English language is not always possible Xue et al. (2014); Hajic et al. (2014). However, recent studies Damonte and Cohen (2018); Blloshmi et al. (2020) show that AMR parsers are able to recover AMR structures when there are structural differences between languages, which demonstrate that it is capable to overcome many translation divergences. Therefore, it is possible for us to parse texts in target (non-English) languages into AMRs.

Another problem of cross-lingual AMR parsing is the scarcity of parallel corpus. Unlike machine translation or sentiment classification which have abundant resources on the Internet, we can only get non-English text and AMR pairs by human annotation. Damonte and Cohen (2018) align a non-English token with an AMR node if they can be mapped to the same English token to construct training set. They further train a transition-based parser using the synthetic training set. They also attempt to translate test set into English and apply an English AMR parser. Blloshmi et al. (2020) build training data in two ways. One of the approaches is that they use gold parallel sentences and generate synthetic AMR annotations with the help of an English AMR parser. Another approach is to use gold English-AMR pairs and get non-English texts by a pre-trained machine translation system. They further use a sequence-to-graph parser Zhang et al. (2019a) to train a cross-lingual AMR parser.

According to Blloshmi et al. (2020), a cross-lingual AMR parser may predict the concepts less specific and accurate than the gold concepts. Therefore, we propose a new model introducing machine translation to enable our parser to predict more accurate concepts. In particular, we first build our training data similar to Blloshmi et al. (2020), translating English texts into target languages. Our basic model is a sequence-to-sequence model, rather than the sequence-to-graph model used in Blloshmi et al. (2020), since in English AMR parsing, sequence-to-sequence models can achieve state-of-the-art result with enough data for pre-training Xu et al. (2020). While training, we introduce bilingual input by concatenating translated target language texts and English texts as inputs. As for inference stage, the bilingual input is the concatenation of translated English texts and target language texts. We hope that our model can predict more accurate concepts with the help of the English tokens, while it can still preserve the meaning of the original texts if there are semantic shifts in the translated English texts. Besides, during training process, we also introduce an auxiliary task, requiring the decoder to restore English input tokens, which also aims at enhancing the ability of our parser to predict concepts. Our parser outperforms previous state-of-the-art parser XL-AMR Blloshmi et al. (2020) on LDC2020T07 dataset Cai and Knight (2013) by about 10.6 points of Smatch F1 score on average, which demonstrates the efficacy of our proposed cross-lingual AMR parser.

Our main contributions are summarized as follows:

• •

  We introduce bilingual inputs and an auxiliary task to a seq2seq cross-lingual AMR parser, aiming to enable the parser to make better use of bilingual information and predict more accurate concepts.

• •

  Our parser surpasses the best previously reported results of Smatch F1 score on LDC2020T07 by a large margin. The results demonstrate the effectiveness of our parser. Ablation studies show the usefulness of the model modules. Codes are public available ¹¹1https://github.com/headacheboy/cross-lingual-amr-parsing.

• •

  We further carry out experiments to investigate the influence of incorporating pre-training models into our cross-lingual AMR parser.

Abstract Meaning Representation (AMR) Banarescu et al. (2013) parsing is becoming popular recently. Some of previous works Flanigan et al. (2014); Lyu and Titov (2018); Zhang et al. (2019a) solve this problem with a two-stage approach. They first project words in sentences to AMR concepts, followed by relation identification. Transition-based parsing is applied by Wang et al. (2015b, a); Damonte et al. (2017); Liu et al. (2018); Guo and Lu (2018); Naseem et al. (2019); Lee et al. (2020). They align words with AMR concepts and then take different actions based on different processed words to link edges or insert new nodes. Due to the recent development in sequence-to-sequence model, several works employ it to parse texts into AMRs Konstas et al. (2017); van Noord and Bos (2017); Ge et al. (2019); Xu et al. (2020). They linearize AMR graphs and leverage character-level or word-level sequence-to-sequence model. Sequence-to-graph model is proposed to enable the decoder to better model the graph structure. Zhang et al. (2019a) first use a sequence-to-sequence model to predict concepts and use a biaffine classifier to predict edges. Zhang et al. (2019b) propose a one-stage sequence-to-graph model, predicting concepts and relations at the same time. Cai and Lam (2020) regard AMR parsing as dual decisions on input sequences and constructing graphs. They therefore propose a sequence-to-graph method by first mapping an input words to a concept and then linking an edge based on the generated concepts. Recently, pre-training models have been proved to perform well in AMR parsing Xu et al. (2020). Lee et al. (2020) employ a self-training method to enhance a transition-based parser, which achieves the state-of-the-art Smatch F1 score in English AMR parsing.

Vanderwende et al. (2015) first carry out research of cross-lingual AMR parsing. They parse texts in target language into logical forms as a pivot, which are then parsed into AMR graphs. Damonte and Cohen (2018) attempt to project non-English words to AMR concepts and use a transition-based parser to parse texts to AMR graphs. They also attempt to automatically translate non-English texts to English and exploit an English AMR parser. Blloshmi et al. (2020) try to generate synthetic training data by a machine translation system or an English AMR parser. They conduct experiments with a sequence-to-graph model in different settings, trying to find a best way to train with synthetic training data. Different from Blloshmi et al. (2020), we treat cross-lingual AMR parsing as a sequence-to-sequence transduction problem and improve seq2seq models with bilingual input and auxiliary task.

Cross-lingual AMR parsing is the task of parsing non-English texts into AMR graphs corresponding to their English translation. In this task, nodes in AMR graphs are still English words, PropBank framesets or AMR keywords, which are the same as the original design of AMR.

Figure 1 shows an example of cross-lingual AMR parsing. We define $X^{l}$ as an input sample in language l and $X^{l}_{i}$ is the $i$-th token of it. $y$ is the corresponding AMR sequence derived from the AMR graph, and $y_{i}$ is the $i$-th token. The model should predict the AMR sequence $y$ first and then transform the sequence into a graph.

Figure 2 shows the training and inference processes of our proposed model. The basic model we adopt is Transformer Vaswani et al. (2017) encoder-decoder model, since Xu et al. (2020) show that it can achieves state-of-the-art result in English AMR parsing. We introduce the bilingual input to our model. When training the model, the bilingual input contains original English text and translated text in non-English target language, which may not be very accurate. During inference, the bilingual input is composed of translated English text and original text in target language. With the help of bilingual input, our model can better understand and preserve the semantics of target language texts, and predict more accurate concepts according to the translated English texts. Apart from predicting AMR sequences, the model is also required to predict the English input texts as an auxiliary objective, which can further help the model learn the exact meaning of input tokens and predict their corresponding concepts more accurately.

We will first introduce the way we obtain training data and the pre-processing and post-processing process in Section 4.1 and Section 4.2, followed by introducing the basic sequence-to-sequence model, the bilingual input and the auxiliary task.

The coefficient of $Loss_{AMR}$ is 1, while the coefficient of $Loss_{Eng}$ is 0.5. We use Adam optimizer Kingma and Ba (2015) to optimize the final loss function. The number of transformer layers in both encoder and decoder is 6. The embedding size and hidden size are both 512 and the size of feed-forward network is 2048. The head number of multi-head attention is 8. We follow Vaswani et al. (2017) to tune the learning rate each step and the warmup step is 4000. The learning rate for decoder at each step is half of this learning rate. Following Blloshmi et al. (2020), we use machine translation system OPUS-MT Tiedemann and Thottingal (2020) to get our bilingual input. We use all data from different languages to train our model and the final model is able to parse sentences in different languages.

Figure 3 shows several AMRs parsed by models with and without auxiliary task. The AMR predicted by model without auxiliary task misses many concepts, while our full model predicts them correctly. What’s more, our full model can predict relations of concepts more accurately as well. For example, the full model adds ARG0 between hamper-01 and problem, retaining semantic information of the original sentence. The model trained without auxiliary task predicts the relation ARG2 instead, changing the meaning of the original sentence.

Another example in Figure 5 shows the efficacy of bilingual inputs. The AMR parsed by basic sequence-to-sequence model does not contain correct semantics. This model predicts many erroneous concepts such as launch-01, possible-01. Besides, the semantics of original sentence did not lose power is changed into have no electricity. The AMR produced by sequence-to-sequence model with bilingual input is almost correct except missing of concept silo. This example reveals that our bilingual input enables the parser to predict more accurate concepts and preserve the semantics of the sentence.

We also show an attention heatmap of an example in test set in Figure 4. This attention pattern shows that our parser can predict AMR tokens based on English translation (e.g. recommend-01) and based on both English and Spanish tokens (e.g. good-02).

In this paper, we focus on cross-lingual AMR parsing. Previous works have deficiency in predicting correct AMR concepts. We thus introduce bilingual inputs as well as an auxiliary task to predict more accurate concepts and their relations in AMR graphs. Empirical results on data in German, Italian, Spanish and Chinese demonstrate the efficacy of our proposed method. We also conduct ablation study to further verify the significance of the bilingual inputs and auxiliary task. For future work, we will attempt to adapt other methods used in English AMR parsing to cross-lingual AMR parsing, such as pre-training and self-training.

This work was supported by National Natural Science Foundation of China (61772036), Beijing Academy of Artificial Intelligence (BAAI) and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We appreciate the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.