A Two-Stage Method for Chinese AMR Parsing

Different from English AMR where nodes or concepts can have arbitrary variable names, a large portion of concepts of Chinese AMR have standard variable names which denote the alignment to input words. Moreover, there are different alignment patterns which make generating the right alignment a complex problem. In following sections, we’ll sequentially introduce the alignment rules we design for CAMRP-2022, the two-stage method and the model.

As shown in Figure 5, we design a RoBERTa-BiLSTM-Biaffine network to conduct relation prediction given the predicted concepts. All concepts are first split into characters before feeding into the RoBERTa model to extract hidden representations. After the RoBERTa model, all hidden states are fed into a one-layer BiLSTM network to better encode sequential information to the hidden states. At last, the first hidden states of every two concepts are fed into the biaffine network to get the relation between the two concepts. During training, we use Cross-Entropy as the loss function and use the average loss of $N\times N$ relations as the final loss, as described in Equation 2,

where $d$ denotes the hidden size, $c$ denotes the number of relations, $N$ denotes the number of input concepts, $\hat{r}$ denotes the one-hot vector for the gold relation and $\mathbf{h_{i}}$ denotes the $i^{th}$ output hidden state from the BiLSTM network.

Since our method has two stages, during inference the Relation-Prediction model takes the output of Concept-Prediction as input. To stabilize and prevent error propagation during training, we adopt the Teacher Forcing method, where we use the gold concepts and relations as the input of the relation prediction model. However, error propagation still exists in the inference phase. We will discuss the error propagation situation of our system in section 4.1.

The CAMRP-2022 evaluation uses the training, development, and test splits of the CAMR 2.0 dataset as its dataset and also involves an out-of-domain blind test set to measure the generalization performance of parsers. The statistics of the dataset are shown in Table 4. For concepts, there are 31941 different concepts in the training set. Among all concepts, there are 8443 different predicates that need to conduct word sense disambiguation and 184 Null-Aligned concepts. As for relations, there are 142 different relations and 841 relation alignment words. The top 5 most frequent relation alignment words are “的”,“是”,“和”,“在” and “对”.

Both the Concept Tagging and Relation Classification model adopt the HIT-roberta-large[Cui et al., 2020] pretrained model downloaded from HuggingFace model hub³³3https://huggingface.co/hfl/chinese-roberta-wwm-ext-large. For Concept Tagging models, the output size of the tag classifier is 8, 2, 185 for Surface Tagging, Normalization Alignment Tagging, and Null-Aligned Concept Tagging individually and the dropout rate of the classifiers is 0.1 in all experiments. For the Relation Classification model, there is one BiLSTM layer and the hidden size is 4096. the dimension of Biaffine matrix is $4097\times 142\times 4097$.

We use Adam as the optimizer and conduct hyper-parameter searches on batch-size (from 10 to 100) and learning rate (from 1e-5 to 1e-4 ) in all models. The optimal hyper-parameters for each model are listed in Appendix B. We train all models for 100 epochs with 1% warmup steps and select the one with the best result on the development set as the final model.

As shown in Table 2, we list the results of our trained 2-stage AMR Parser on the development, test, and blind-test set of CAMRP-2022. For the development set, we list the concrete results of all different sub-tasks in two stages along with the overall and fine-grained AlignSmatch scores.

As pointed out in Section 3.4, there exists error propagation in the two-stage model. The direct evidence is that while the relation prediction could reach 0.744 F1 with gold concepts, this score drops to 0.583 when giving it the model predicted concepts. Error propagation also exists in the Concept-Prediction stage since Normalization Alignment needs both the correct result from Surface Tagging and Normalization Alignment Tagging.

As pointed out in section 3.1, there exist severe class imbalance problems in both stages of the parsing task. As for the Concept-Prediction stage, the problem reflects the great differences in the distribution of different tags in the three tagging tasks, especially for the Null-Aligned Concept Tagging tasks. For the Relation-Prediction stage, a large portion of labels is “None Relation” as shown in Figure 6.

We have tried some techniques like using weighted loss that assigns greater weight to the minority classes to handle the class imbalance problem. While this can greatly reduce the time required for the model to converge, it does not improve the final performance when all epochs are finished.

In our model, we use a trigger-based method to predict concepts without alignment. This method could cover nearly 95% cases while the rest 5% is neglected because they are mostly triggered by another Null-Aligned concept. Though we design methods to overcome the drawback by tracing back to the first aligned concept, the overall result of Null-Aligned Concept Prediction is still the lowest in the Concept-Prediction stage, which could lead to great bias for the next stage. A more natural method to predict those concepts might greatly improve this task.