A Chinese Multi-type Complex Questions Answering Dataset over Wikidata

Earlier KBQA datasets, such as Free917 Cai and Yates (2013) that contains 917 pairs of question and formal query in FREEBASE, are too small to train neural networks models. The larger scale dataset WebQuestions Berant et al. (2013) consists of natural language question-answer pairs only, without formal queries. WebQuestionsSP Yih et al. (2016) extracts part of the WebQuestions dataset and supplements it with the corresponding formal query and improves the multi-relation questions. SimpleQuestions Bordes et al. (2015) builds a large-scale dataset, but only has one type of relation. Recently, ComplexWebQuestions Talmor and Berant (2018) presents a complex dataset that covers more components in the SPARQL grammar, but it only covers limited types of questions. LC-QUAD Trivedi et al. (2017) is the first complex KBQA dataset based on DBpedia. It starts by generating formal queries for DBpedia and then converts these template-based questions into natural language questions. QALD-9 Usbeck et al. (2018) is another well-known KBQA dataset based on DBpedia, which has more complex and colloquial questions than LC-QUAD. LC-QuAD 2.0 Dubey et al. (2019) is created in 2019, which expanded the number of data and increased formal query types.

In earlier work,  Bordes et al. (2014) starts to use neural networks to train question and answer vector representation, calculating the matching scores between question and candidate answers.  Dong et al. (2015); Hao et al. (2017) uses CNN and attention to learn the relation and type features and obtains some improvement. These methods are based on information retrieval provide a simple solution by strengthening the connection between the question and the answer. But their candidate answer space can be huge for complex question. By contrast, semantic parsing methods can achieve much better score in complex KBQA dataset.  Yih et al. (2015) puts forward a query graph structure, mapping the semantic parsing process to generate query graph process, which defines several operations to extend, including entity linking, attaching constraints, attribute recognition and so on. Following this idea,  Bao et al. (2016); Yu et al. (2017) adds new type constraints as well as dominant and recessive time constraints to solve more complex questions. Recently, Hu et al. (2018); Xu et al. (2019); Chen et al. (2020); Lan and Jiang (2020) continues this line of research by defining new abstract query graph and designing stronger query graph representation to further improve predicted query graph accuracy.

We show the statistics of three knowledge graphs in Table 2. Freebase Bollacker et al. (2008) is a collaboratively edited knowledge base designed to be a public repository of the world’s knowledge. DBpedia Bizer et al. (2009) is a knowledge graph mainly based on the English Wikipedia. Wikidata Vrandecic (2012) is a free, open, and massively linked knowledge base. Compared with above two knowledge graphs, we can find that Wikidata contains a larger number of entities and relations. More importantly, Wikidata defines prefixes to describe the IRIs of the RDF resources, that are suitable for variety of SQARQL queries, shown in Figure 1.

We compute the statistics of both LC-QuAD 2.0 and CLC-QuAD, and carry out a data analysis focusing on semantic coverage and question types. In this section, we will also compare them with other complex knowledge base question answering datasets.

Data statistics Table 3 summarizes the statistics of nine datasets. CLC-QuAD contains 28k+ pairs of question and SPARQL query in total, which is comparable to or bigger than most commonly used KBQA datasets. As for dataset variation, FREE917, WebQuestions and SimpleQuestion focus on simple questions without constraints, so most of data in these datasets even don’t have corresponding formal queries. LC-QuAD contains more question types and its SPARQL components cover SELECT, COUNT, ASK and DISTINCT. ComplexWebQuestions bulids a complex dataset by adding different and complicated constraints and its SPARQL components cover SELECT, DISTINCT, FILTER, LANG, DATATIME etc. In LC-QuAD 2.0 and CLC-QuAD, we ensure that corpus contains enough examples for all common SPARQL patterns in order to describe the question correctly. CLC-QuAD covers all the following SPARQL components: SELECT with one or multiple variable, COUNT, ASK, DISTINCT, FILTER, CONTAINS, YEAR, STRSTARTS, LIMIT, ORDER BY, LANG. Noticeably, most of datasets are built on Freebase and DBpedia, which makes datasets based on Wikidata more distinctive and challenging, as researchers need to explore the connection between questions and the structure of the Wikidata knowledge graph. In addition, CLC-QuAD provides Chinese questions that help study KBQA in a cross-lingual setting. Since the same question can be expressed quite differently in Chinese and in English, we also present statistics of characters for each language.

Semantic Coverage As shown in Table 4, we make a statistical comparison against previous KBQA datasets in this task. Obviously, CLC-QuAD and LC-QuAD 2.0 is vast in coverage of knowledge graph entities and relations, which can enable better generalization in model training. In addition, the SPARQL queries in CLC-QuAD and LC-QuAD 2.0 cover all common SPARQL keywords. As for vocabulary in datasets, we believe our translating work is much better than machine translation, as the translators make their focus on both questions and correct SPARQL queries so that they can use suitable word in different situations, as demonstrated by the vocabulary used in our dataset.

Question Distribution As shown in Figure 2, CLC-QuAD contains a fairly diverse set of question types over knowlegde graphs. Unsurprisingly, FACT and FACT_DEDUPLICATION questions are the two most commonly seen in KBQA systems. Among the rest of question types, approximately 35% are DUAL INTENTION questions, which pose huge challenges to semantic parsing. Another 20% of this subgroup is BOOLEAN question, which is also very common in real-world applications, but is often ignored by researchers. BOOLEAN questions can be hard to handle traditional algorithms, because it is more like a classification problem rather than mapping to existing knowledge graphs. Also, DATE, COUNTING, MAX/MIN questions strengthen the diversity of entire dataset and involve more corresponding SPARQL components.

As for the embedding layer, we consider two options as input to the next layer. First choice is pretrained word embedding Song et al. (2018). Alternatively, we use BERT Devlin et al. (2019) as the initial representations of the word. Formally, we concatenate question $Q$ and all entities $E$ and relations $R$ by this structure.

This sequence is fed into the pretrained BERT model and use the last hidden states as the initial representations.

To capture graph information across the question, we use relation-aware self attention layers Shaw et al. (2018) to compute new contextual representations of questions, entities and relations item. We define the input $X=\{x_{i}\}_{i=1}^{n}$ where $x_{i}\in\{q_{i}...,e_{i}...,r_{i}\}=Q\cup E\cup R$. This is the relation-aware self-attention process for each layer (consisting of $H$ heads):

where $W_{Q}^{(h)},W_{K}^{(h)},W_{V}^{(h)}\in\mathbb{R}^{d_{x}\times(d_{x}/H)}$, $r^{K}=r^{V}\in\mathbb{R}^{n\times n\times(d_{x}/H)}$. $r_{ij}$ is the vector which represent the relation type between the two item $x_{i}$ and $x_{j}$ in the input. And in our model, $r_{ij}$ is designed as a learned parameter for edge types in graph, such as ¡wdt-wd¿ and ¡wdt-p¿. After the attention procedure, we use fully connected feed-forward networks to transform the attention output and the ReLU activation is used between the two fully connected networks.

In sum, every relation-aware self attention layer use the corresponding graph $G_{Q}$ and compute a new contextual representations of question word, entities and relations.

Because the output SPARQL query consists of entities, relations and SPARQL keywords, we use pointer networks with three separate independent scaled dot-product attention for different components. During the decoding process, we use Long Short Term Memory (LSTM) with attention to generate SPARQL queries by incorporating the representation of entities, relations and SPARQL keywords.

Denote the decoding step as $t$, we provide the decoder input as a concatenation of the embedding of the SPARQL query token $S_{t}$ and the context vector $c_{t}$:

where $h_{t}$ is the hidden state of the decoder $\mathrm{LSTM}$ and the hidden state $h_{0}$ is initialized as random, as well as $S_{t},c_{t}$. And $S_{t}$ is generated by the types of the SPARQL query token.

where $h^{entity},h^{relation}$ are extracted from encoder result $h^{enc}$ and $h^{keyword}$ is a learnable embedding. In addition, we compute $c_{t}$ by multi-head attention with $combined$ components representation as follows:

where $h_{t}$ is equal to $h_{t}$ in Equation 2 and $d_{h}$ is the dimension of $h_{t}$. The context vector $c_{t}$ consists of attentions to both the question, entities, relations and SPARQL keywords for current step $k$.

As for output layer, our decoder is designed to generate an entity, a relation or a SPARQL keyword (eg. COUNT, ASK, FILTER, ORDER BY). In addition, entities and relations will be changeable based on different candidate graph so that we choose three separate independent layer for different components and then use softmax function to compute the output probability distribution:

As for loss computing, we compute the output character with ground truth SPARQL query character by the cross entropy loss function.

Because there is no model aiming at multi-type KBQA datasets, we modify two state-of-the-art models based on earlier datasets for our model to compare to. Due to limitation of their models, some types of questions cannot be handled, so we test them with specific types of questions.
AQG-net proposed by Chen et al. (2020) first uses a neural network-based generative model to generate an abstract query graphs that describes logical query structures, filling up it with all possible candidate permutations and then utilizes existing ranking model to get the most suitable query. In our experiment, we redesign the abstract query graphs due to the question types and relations are different and more complicated.
Multi-hop QGG proposed by Lan and Jiang (2020) explores a new strategy to expand the candidate query graph with both constraints and core paths. And it applies the REINFORCE algorithm to learn a policy function by using the F1 score of the predicted answers with respect to the ground truth answers as reward. In our experiment, we totally redefine the way of extending the current SPARQL path and adapt the model features to meet the our dataset.

We evaluate models from two aspects. First, we use the F1 scores metrics to calculate the accuracy between ground truth answer and answers obtained from predicted SPARQL queries. Second, as semantic parsing methods model aim to predict correct SPARQL queries by question analysis, the answer accuracy can not represent its analytical capability. Instead of simply taking string match, we decompose predicted SPARQL queries into different triples such as $(?ans_{1},wdt$:$P31,wd$:$Q22675015)$, $(?var_{1},order\ by,ascend)$ and compute scores for the triples set using exact set match, which measures whether the predicted query is entirely equivalent to the gold query. The predicted SPARQL query is correct only if all of the components are right. Because we employ a triples set comparison, this exact matching metric is invariant to the order of the components.

LC-QuAD 2.0 We report the overall results of our approach and others on LC-QuAD 2.0 in Table  5. Our method achieves the performance of 55.4% query exact match scores and 59.3% answer F1 scores, which is better than other methods. This demonstrates the effectiveness of our approach and that the text-to-SPARQL method can handle the semantics of multi-type questions to generate complex SPARQL queries. Furthermore, from the ablation study, we find that model without BERT has 7.0% decline but still achieves state-of-the-art. This shows that the effectiveness of our model is not simply because of the use of BERT. We also test two components of our method. Without the relation-aware self attention, the score is nearly 9% lower and it shows that the information of knowledge graph is very important to the model and the relation-aware self attention can effectively encode the items in the graph. Without the separate attention in pointer networks, the model has a little drop in results.
CLC-QuAD Table 5 also shows the results of models in our new dataset CLC-QuAD. Similar to LC-QuAD 2.0, our model with BERT achieves 51.8% answer F1 score and 45.4% query exact matching accuracy. In addition, we find that all models get pretty weak scores compared with results in LC-QuAD 2.0. In the results of our approach with BERT, there is 10% decline which is a huge gap. This demonstrates that the Chinese representation is still a big challenge for model semantic parsing.

In addition, we split our dataset into six question types. Table 6 shows the performance of various question types on CLC-QuAD. We find that there is a huge gap in accuracy between different types of questions. We find that Counting and Qualifier questions are harder due to more complex semantics, which brings challenges to semantic parsing. Our approach achieves 51.3%, 55.3% and 47.1% accuracy in Dual, Boolean and Max/Minimum questions relatively, but the accuracy for Qualifier and Counting questions is only 18.8% and 30.1%. As for other methods, Multi-hop QGG is designed for complex questions with constraints and it achieves best performance in Qualifier questions, but it cannot handle Dual, Boolean and Counting questions. AQG-net is designed to generate abstract query graphs, which can answer most of types questions, but the performance is inferior to our proposed approach. This also demonstrates our model is more competitive in answering different types of questions.