ShadowGNN: Graph Projection Neural Network for Text-to-SQL Parser

Before describing the details of R-GCN, we first give notations of relational directed graph. We denote this kind of graph as $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{R})$ with nodes (schema components) $v_{i}\in\mathcal{V}$ and directed labeled edge $(v_{i},r,v_{j})\in\mathcal{E}$, where $v_{i}$ is the source node, $v_{j}$ is the destination node and $r\in\mathcal{R}$ is the edge type from $v_{i}$ to $v_{j}$. $\mathcal{N}_{i}^{r}$ represents the set of the neighbor indices of node $v_{i}$ under relation $r$, where $v_{i}$ plays the role of the destination node.

Each node of the graph has an input feature $\mathbf{x}_{i}$, which can be regarded as the initial hidden state $\mathbf{h}_{i}^{(0)}$ of the R-GCN. The hidden state of each node in the graph is updated layer by layer with following step:

With the success of the large-scale language models, the transformer architecture has been widely used in natural language process (NLP) tasks to encode the sequence $X=[\mathbf{x}_{i}]_{i=1}^{n}$ with the self-attention mechanism. As introduced in Vaswani et al. Vaswani et al. (2017), a transformer is stacked by self-attention layers, where each layer transforms $\mathbf{x}_{i}$ to $\mathbf{y}_{i}$ with $H$ heads as follows:

where $h$ is the head index, $d_{z}$ is the hidden dimension of $\mathbf{z}_{i}^{(h)}$, $\alpha_{ij}^{(h)}$ is attention probability, $\mathop{\rm Concat}$ denotes the concatenation operation, $\mathop{\rm LayerNorm}$ is layer normalization Ba et al. (2016) and $\mathop{\rm FC}$ is a full connected layer. The transformer function can be simply denoted as:

where $\mathbf{Y}=\{\mathbf{y}_{i}\}_{i=1}^{|X|}$ and $\mathbf{X}=\{\mathbf{x}_{i}\}_{i=1}^{|X|}$ and $|X|$ is the sequence length.

Relation-aware transformer (RAT) Shaw et al. (2018) is an important extension of the traditional transformer, which regards the input sequence as a labeled, directed, fully-connected graph. The pairwise relations between input elements are considered in RAT. RAT incorporates the relation information in Equation 6 and Equation 8. The edge from element $\mathbf{x}_{i}$ to element $\mathbf{x}_{j}$ is represented by vectors $\mathbf{r}_{ij,K}$ and $\mathbf{r}_{ij,V}$, which are represented as biases incorporated in self-attention layer, as follows:

where $\mathbf{r}_{ij,K}$ and $\mathbf{r}_{ij,V}$ are shared in different attention heads. For each layer of RAT, the update process can be simply represented as:

where $\mathcal{R}=\{R\}_{i=1,j=1}^{|X|,|X|}$ is the relation matrix among the sequence tokens and $R_{ij}$ means the relation type between $i$-th token and $j$-th token.

Both R-GCN and RAT have been successfully applied into Text-to-SQL tasks. Bogin et al. (2019a) utilizes R-GCN to encode the structure of the semantic schema to get the global representations of the nodes. Wang et al. (2020) considers not only the schema structure but also the schema link between the schema and the NL question. They proposed a unified framework to model the representation of the schema and the question with RAT. However, they do not explicitly explore the impact of the domain information. In the next section, we will introduce our proposed GPNN and explain how to use GPNN to get the abstract representation of the schema and the question.

In this subsection, we introduce the structure of GPNN. As we discussed, the schema consists of database structure information and domain semantic information. GPNN looks at the schema from these two perspectives. Thus, GPNN has three kinds of inputs, abstract schema, semantic schema, and NL question. The input of the abstract schema is the type (table or column) of the schema nodes without any domain information, which can be regarded as a projection of semantic schema. Each node in the abstract schema is represented by a one-hot vector $\mathbf{a}_{j}^{(0)}$, which has two dimensions.

For semantic schema and NL question, we first use pretrained language model RoBERTa Liu et al. (2019) to initialize their representations. We directly concatenate NL question and semantic schema together, which formats as “ [CLS] question [SEP] tables columns [SEP]". Each node name in the semantic schema may be tokenized into several sub-tokens or sub-words. We add an average pooling layer behind the final layer of the RoBERTa to align the sub-tokens to the corresponding node. We indicate the initial representation of NL question and semantic schema as $\mathbf{q}_{i}^{(0)}$ and $\mathbf{s}_{j}^{(0)}$.

The main motivation of GPNN is to abstract the representations of question and schema. The abstract schema has been distilled from the semantic schema. The essential challenge lies on abstracting question representation. There are two separate operations in each GPNN layer: Projection Attention and Character Encoding. The projection attention of GPNN is to take the semantic schema as the bridge, where question updates its representation using abstract schema but attention information is calculated with the vectors of semantic schema. The character encoding is to augment the structure representation of the question sentence and the schema graph.

Before operating attention mechanism, inspired by Bogin et al. (2019a), we first calculate the maximum values $\mathbf{u}$ of attention probability,

where the physical meaning of $\mathbf{u}_{j}$ is the most probability that the $j$-th component of the schema is mentioned by the question. We distinct the initial representation of the abstract schema by multiplying $\mathbf{u}$ on $l$-th layer abstract schema representation $\mathbf{a}^{(l)}$ in element-wise way, $\hat{\mathbf{a}}^{(l)}=\mathbf{a}^{(l)}\cdot\mathbf{u}$.

When updating the question representation, we take the representation of augmented abstract schema $\hat{\mathbf{a}}^{(l)}$ as key value of attention at $l$-th layer of GPNN,

where $\mathop{\rm gate}(\cdot)=\mathop{\rm sigmoid}(\mathop{\rm Linear}(\cdot))$ and $\mathbf{W}_{V}^{(l)}$ is trainable weight. When updating semantic schema, we take the transpose of the above attention matrix as the attention from schema to question,

Similar to the update process of question from Equation 17- 21, the update process of semantic schema $\bar{\mathbf{s}}^{(l+1)}$ takes $\mathbf{\hat{e}}^{m\times n}$ as attention score and $\mathbf{q}^{(l)}$ as attention value. We can see that we only use the augmented abstract schema to update the question representation. In this way, the domain information contained in question representation will be removed. The update process of the abstract schema $\bar{\mathbf{a}}^{(l+1)}$ is the same as the semantic schema updating, where their attention weight $\mathbf{\hat{e}}^{m\times n}$ on the question $\mathbf{q}^{(l)}$ is shared. Noting that the input of attention operation for the abstract schema is the augmented abstract representation $\hat{\mathbf{a}}$.

The schema linking Guo et al. (2019); Lei et al. (2020) can be regarded as a kind of prior knowledge, where the related representation between question and schema will be tagged according to the matching degree. There are 7 tags in total: Table Exact Match, Table Partial Match, Column Exact Match, Column Partial Match, Column Value Exact Match, Column Value Partial Match, and No Match. The column values store in the databases. As the above description, the schema linking can be represented as $\mathcal{D}=\{d_{ij}\}_{i=1,j=1}^{n,m}$, which $d_{ij}$ means the match degree between $i$-th word of question and $j$-th node name of schema. To integrate the schema linking information into GPNN module, we calculate a prior attention score $\mathbf{p}^{n\times m}={\rm Linear}({\rm Embedding}(\mathbf{d}_{ij}))$, where $\mathbf{d}_{ij}$ is the one-hot representation of match type $d_{ij}$. The attention score in Equation 17 is updated as following:

where $p_{ij}$ is the prior score from $\mathbf{p}^{n\times m}$. The prior attention score is shared among all the GPNN layers.

If we split the schema into the tables and the columns, there are three kinds of inputs: question, table, column. RATSQL Wang et al. (2020) leverages the relation-aware transformer to unify the representation of the three inputs. RATSQL defines all the relations $\mathcal{R}=\{R_{ij}\}_{i=1,j=1}^{(n+m),(n+m)}$ among the three inputs and uses the $\mathop{\rm RAT}(\cdot)$ function to get unified representation of question and schema. The details of the defined relations among three components are introduced in RATSQL Wang et al. (2020). The schema linking relations are the subset of $\mathcal{R}$. In this paper, we leverage the RAT to further unify the abstract representation of question $\mathbf{q}^{(N)}$ and schema $\mathbf{a}^{(N)}$, which is generated by previous GPNN module. We concatenate sentence sequence $\mathbf{q}^{(N)}$ and schema sequence $\mathbf{a}^{(N)}$ together into a longer sequence representation, which is the initial input of RAT module. After RAT module, the final unified representation of question and schema is indicated as:

To effectively constrain the search space during synthesis, IRNet Guo et al. (2019) designed a context-free SemQL grammar as the intermediate representation between NL question and SQL, which is essentially an abstract syntax tree (AST). SemQL recovers the tree nature of SQL. To simplify the grammar tree, SemQL in IRNet did not cover all the keywords of SQL. For example, the columns contained in $\mathop{\rm GROUPBY}$ clause can be inferred from $\mathop{\rm SELECT}$ clause or the primary key of a table where an aggregate function is applied to one of its columns. In our system, we improve the SemQL grammar, where each keyword in SQL sentence is corresponded to a SemQL node. During the training process, the labeled SQL needs to be transferred into an AST. During the evaluation process, the AST needs to recovered as the corresponding SQL. The recover success rate means the rate that the recovered SQL totally equals to labeled SQL. Our improved grammar raises the recover success rate from 89.6% to 99.9% tested on dev set.

We leverage the coarse-to-fine approach Dong and Lapata (2018) to decompose the decoding process of a SemQL query into two stages, which is similar with IRNet. The first stage is to predict a skeleton of the SemQL query with skeleton decoder. Then, a detail decoder fills in the missing details in the skeleton by selecting columns and tables.

To fairly compared with our proposed ShadowGNN, we implement RATSQL Wang et al. (2020) with the same coarse-to-fine decoder and RoBERTa augmentation of ShadowGNN model. We also report the performance of GPNN encoder on test set. The detail implementations of these two baselines show as following:

• •

  RATSQL^♣ RATSQL model replaces the four projection layers with another four relation-aware self-attention layers. There are totally eight relation-aware self-attention layers in the encoder, which is consistent with orignal RAT-SQL setup Wang et al. (2020).

• •

  GPNN Compared with ShadowGNN, GPNN model directly removes the relation-aware transformer. There are only four projection layers in the encoder, which can get better performance than eight layers.

Table 1 presents the exact match accuracy of the novel models on development set and test set. Compared with the state-of-the-art RATSQL, our proposed ShadowGNN gets absolute 2.6% and 0.5% improvement on development set and test set with RoBERTa augmentation. Compared with our implemented RATSQL^♣, ShadowGNN can still stay ahead, which has absolute 2.1% and 2.1% improvement on development set and test set. ShadowGNN improved the encoder and SemQL grammar of IRNet obtains absolute 11.1% accuracy gain on test set. As shown in Table 1, our proposed pure GPNN model achieves comparable performance with state-of-the-art approach on test set. Compared with other GNN-based models (Global-GNN and R-GCN), GPNN gets over 10% improvement on development set and test set. To the best of our knowledge, our proposed GPNN gets the best performance on Spider dataset among all the GNN-based models.

We design an experiment to validate the effectiveness of the graph projection neural network (GPNN). Considering a question “What is name and capacity of stadium with most concert after year ?", which has been preprocessed, “name" and “capacity" are column names. We exchange their positions and calculate the cosine similarity with the representations of the final GPNN layer in ShadowGNN model. Interestingly, we find that “name" has the most similar with “capacity", as shown in Figure 3. The semantic meaning of the two column names seems to be removed that the representations of the two column names only dependent on the existed positions. It indicates the GPNN can get the abstract representation of the question.

To further validate the generalization ability of our proposed ShadowGNN, we conduct the experiments on the limited annotated training datasets. The limited training datasets are sampled from fully training dataset with 10%, 50% and 100% sampling rate. As shown in Figure 4, there is a large performance gap between RATSQL and ShadowGNN, when the annotated data is extremely limited only occupied 10% of the fully training dataset. ShadowGNN outperforms RATSQL and GPNN with over 5% accuracy rate on development set. Under this limited training data setup, we find an interesting phenomenon that the convergence speed of ShadowGNN is much faster than the other two models. As described in Section 3, the two-phase encoder of ShadowGNN simulates the inference process of a human when translating a question to a SQL query: abstracting and inferring. The experiments on limited annotated training datasets show these two phases are both necessary, which not only can improve the performance but also speed up the convergence.

We conduct ablation studies to analyze the contributions of well-designed graph projection neural network (GPNN). Except RATSQL and GPNN models, we implement other two ablation models: R-GCN and R-GCN+RAT. First, we introduce the implementations of the ablation models.

• •

  R-GCN^♣ We directly remove the projection part in the GPNN. When updating the question representation, we use the representation of semantic schema as attention value instead of abstract representation.

• •

  R-GCN+RAT In this model, there are four R-GCN layers and four relation-aware self-attention layers. To be comparable, the initial input of R-GCN is the sum of semantic schema and abstract schema.

The decoder parts of these four ablation models are the same as the decoder of ShadowGNN. We present the accuracy of the ablation models at the four hardness levels on the development set, which is defined in Yu et al. (2018). As shown in Table 2, ShadowGNN can get the best performance at three hardness levels. Compared with R-GCN Kelkar et al. (2020), our implemented R-GCN based on SemQL grammar gets higher performance. Compared with R-GCN+RAT model, ShadowGNN still gets the better performance, where the initial input information is absolutely the same. It denotes that it is necessary and effective to abstract the representation of question and schema explicitly.