GP: Context-free Grammar Pre-training for Text-to-SQL Parsers

RAT-SQL uses the Syntactic Neural Model (SNM) presented by (?) to create the SQL grammar. Yin et al. believed that the present methods treat code generation as a task of sequence generation not considering the grammar of the target programming language. Programming languages, especially SQL, have strict grammar rules, unlike natural languages. Based on these rules, SNM is essentially a method to improve the accuracy of the model by limiting the search space of the decoder.

Moreover, the basic framework of SQL grammar is context-free with the specific natural language description. For instance, regardless of the natural language description, the first clause of SQL is always $select$, and the next clause is always $from$. Based on the experiments, the loss value in the initial training stage of RAT-SQL is extremely large mainly coming from SQL grammar errors created by the decoder.

Regarding the above situation, we proposed a context-free Grammar Pre-training (GP) technique to pre-train the parameters on the decoder side. The encoder’s semantic information is replaced by zero vectors. The probability equation of RAT-SQL utilizing LSTM to output a sequence of $decoder$ actions is:

$$Pr(P|y)=\prod_{t}{}Pr(a_{t}|a_{<t},y)$$

(1)

where $y$ is always [$\bf{0}$] in the stage of GP and $a_{<t}$ are all previous actions. Correspondingly, the LSTM’s state updating strategy will be modified as:

$$m_{t},h_{t}=f_{LSTM}([\bf a_{t-1}||[0]||h_{p_{t}}||a_{p_{t}}||n_{f_{t}}],m_{t-1},h_{t-1})$$

(2)

where $m_{t}$ and $h_{t}$ are the LSTM cell state and output in step $t$, $a_{t-1}$ represents the embedding of the previous action, $p_{t}$ denotes the step equivalent to expanding the parent AST node of the current node, and $n_{f_{t}}$ is the current node type embedding. We used $\bf{[0]}$ to replace the former $z_{t}$ obtained through multi-head attention on $h_{t-1}$ over $y$.

Since GP no longer depends on semantic information, it cannot predict column’s or table’s names. It is assumed that each sample has only one column and one table in order to not change the framework of RAT-SQL, , thus

$$Pr(a_{t}=S{\scriptstyle ELECT}C{\scriptstyle OLUMN}[0]|a_{<t})=1$$

(3)

$$Pr(a_{t}=S{\scriptstyle ELECT}T{\scriptstyle ABLE}[0]|a_{<t})=1$$

(4)

To prevent overfitting, the number of decoder Grammar Pre-training steps was limited to 300.

Generally, the serialization technique of RAT-SQL is adopted. Since the utilized pre-trained semantic model is ${\bf G{\scriptstyle RAPPA}}$, the question tokens are preceded by $\langle s\rangle$ and ended up with $\langle/s\rangle$. Then, tables and columns are spliced in sequence based on the order of the schema presented by the Spider dataset. Moreover, we used $\langle/s\rangle$ as the separator.

As stated in (?), modeling with only table/field names and their relations is not always adequate for capturing the semantics of the schema and its dependencies with the question. Remarkably, we append values to mention columns only when they match the question exactly. For example, in Figure 2, the keyword $volvo$ in the question appears in both column $make$ and column $model$, respectively. Thus, there is a relationship between the token $volvo$ and a Column-Part-Match(CPM) with column $make$ as well as a Column-Exact-Match(CEM) relationship with column $model$. Intuitively, the exact match possesses a greater probability as the correct column. To strengthen this relationship, we put $volvo$ after the column $model$ during serializing while column $make$ not. The sequence can be converted as

$$S=\left\langle s\right\rangle,Q,\left\langle/s\right\rangle,C_{1},\left\langle/s\right\rangle,C_{2},V_{2},\left\langle/s\right\rangle,...,T_{1},\left\langle/s\right\rangle,T_{2},\left\langle/s\right\rangle,...,\left\langle/s\right\rangle$$

(5)

In RAT-SQL, the vector representation of a table or a column is the average of the last and first token. Research indicates that this encoding method may lose important information(?), hence, another technique is utilized by calculating the average of all tokens’ vector of the column or table. When a column is followed by a value, the column’s representation is determined by all column tokens and value tokens (Fig.3).

For deep learning, training loss keeps often decreasing while the validation loss suddenly starts to rise(?). (?) proposed a tricky and simple loss function $flooding$ to decrease validation loss continuously:

$$\tilde{J}(\theta)=|J(\theta)-b|+b$$

(6)

where $b>0$ represents the user-specified flooding level, and $\theta$ is the model parameter. It is assumed that the existence of parameter $b$ can prevent the model from falling into the local optimum to a certain extent, during the optimization process. Since spider dataset has various types of SQL grammar and databases sizes are usually inconsistent, it usually leads to overfitting and converges near a local extreme while training, here, this method was adopted to make final results well. Nevertheless, unsuitable $b$ usually result in gradient explosion.

The Adam optimizer(?) with default hyperparameters is adopted. In the stage of GP, the learning rate is set as $7.44\times 10^{-4}$. Owing to GPU memory limitation, we set $bs=3$ and $num\_batch\_accumulated=4$, in which, $bs$ and $num\_batch\_accumulated$ are the gradient accumulation parameters of RAT-SQL equivalent to batch size of 12. Considering GP and a smaller batch size, compared to RAT-SQL, we set the initial learning rate of ${\bf G{\scriptstyle RAPPA}}$ from the original $3\times 10^{-6}$ to $2\times 10^{-6}$, and the initial learning rate of other model parameters from $7.44\times 10^{-4}$ to $5.44\times 10^{-4}$. The rest of the setups are the same with RAT-SQL.

$\bf{Spider}$ (?) is a cross-domain Text-to-SQL dataset and large-scale complex. It includes both schema information and a corresponding SQL statement for each natural language problem. According to Table 1, it includes 10,181 questions and 5,693 unique complex SQL queries on 206 databases with multiple tables covering 138 different domains. Based on its hardness level, spider splits into 4 types of data sets as Easy, Medium, Hard, and Extra Hard. It is the only data set in the public data set of Text-to-SQL tasks containing both complex SQL statements and multi-table query. Here, the complex SQL denotes the nested query situation of $orderby$, $groupby$, and $where$ clauses in the statement.

The metric adopted to assess model performance is $\bf{Exact}$ $\bf{Match}$ $\bf{Accuracy}$ suggested by (?). This metric refers to utilize standardized definitions to process the prediction SQL and the true statement, and calculate matching degree between them, without considering the column names order.

While RAT-SQL and ${\bf G{\scriptstyle RAPPA}}$ are open-sourced, the offline result is worse compared to announced on the leaderboard in our experiments (Table 2). The reason can be explained by random seed or device differences. In this section, we mainly compared model performance based on offline results.

GP Figure 5 indicates that in the first 50 steps of GP, the training loss significantly drops, then, it remains at about 53. To prevent overfitting, the number of Grammar Pre-training steps is limited, even if the loss is still dropping at a tiny speed. Then, we used the pre-trained decoder to train our model, and the training loss is maintained at a lower level compared to the previous method without GP (Fig. 5). We computed the average and variance of loss before and after 1500 steps as stable values. From Table 3, the average loss with GP is 15.02, which is reduced by 78.9% compared to the former one. Furthermore, its variation rate is only 0.8% of the model without GP indicating that there is a smooth optimization during training. The final loss of less than 1.37 also proves that GP helps to find auxiliary information between SQL grammar and question words.

Flooding Equation 6 indicates that there is an extra parameter $b$ in loss function, and the model performance is extremely sensitive to $b$ and learning rate $lr$. Moreover, a slightly larger $b$ may lead to the model to gradient explosion during training. Table 4 indicates several examples about different parameter combination. $\emptyset$ denotes that the parameter combination will result in gradient explosion. It is worth mentioning that although $Flooding$ can enhance model performance, the results are not stable, in which, the best result may be as high as 72.1%, and the lowest result may be only 70.7% even if we used the same parameters.

Serialization with value By adding the equivalent value after the column, the recognition between columns is enhanced. It is indicated that a slight reduction exists in column selection errors. Table 5 represents the enhancements of Flooding(Fld.), Serialization with value(val.) and GP, respectively. The best result is 73.1% on Dev. offline.

The ultimate result on Spider is 72.8% on Dev. and 69.8% on Test. Compared to the result of RAT-SQL+${\bf G{\scriptstyle RAPPA}}$, the Dev. and Test. The results of RAT-SQL+${\bf G{\scriptstyle RAPPA}}$+GP are much closer indicating that our model is more robust, as shown in Table 6.