RYANSQL: Recursively Applying Sketch-based Slot Fillings
for Complex Text-to-SQL in Cross-Domain Databases

To get the embedding vector for a word $w$ in question, table names or column names, its word embedding and character embedding are concatenated. The word embedding is initialized with $d_{1}=300$-dimensional pre-trained GloVe (Pennington et al., 2014) word vectors, and is fixed during training. For character embedding, each character is represented as a trainable vector of dimension $d_{2}=50$, and we take maximum value of each dimension of component characters to get the fixed-length vector. The two vectors are then concatenated to get the embedding vector $e_{w}\in\mathbb{R}^{d_{1}+d_{2}}$. One layer highway network (Srivastava et al., 2015) is applied on top of this representation. For SPC $P$, each code element $c$ is represented as a trainable vector of dimension $d_{p}=100$.

One dimensional convolution layer with kernel size 3 is applied on SPC element embedding vectors $\{e^{P}_{1},...,e^{P}_{p}\}$. Max-pooling is applied on the output to get the SPC vector $v^{P}\in\mathbb{R}^{p\times d_{p}}$. $v^{P}$ is then concatenated to each question and column word embedding vector.

CNN with Dense Connection proposed in Yoon et al. (2018) is applied to encode each word of question and columns to capture local information. Parameters are shared across the question and columns. For each column, a max-pooling layer is followed; outputs are concatenated with their table name representations and projected to dimension $d$. Layer outputs are encoded question words $V^{Q}=\{v^{Q}_{1},v^{Q}_{2},...,v^{Q}_{n}\}\in\mathbb{R}^{n\times d}$, and hidden column vectors $H^{C}=\{h^{C}_{1},...,h^{C}_{m}\}\in\mathbb{R}^{m\times d}$.

In this layer, the column vectors are modeled with contextual information of the question by attending question tokens with their corresponding columns. Scaled dot-product attention (Vaswani et al., 2017) is used to align question tokens with column vectors:

Where each $i$-th row of $A_{QtoC}\in\mathbb{R}^{m\times d}$ is an attended question vector of the $i$-th column.

The heuristic fusion function $fusion(x,y)$, proposed in Hu et al. (2018), is applied to merge $A_{QtoC}$ with $H^{C}$:

Where $W_{r}$ and $W_{g}$ are trainable variables, $\sigma$ denotes the sigmoid function, $\circ$ denotes element-wise multiplication and $F^{C}\in\mathbb{R}^{m\times d}$ is fused column matrix. A transformer layer (Vaswani et al., 2017) is applied on top of $F^{C}$ to capture contextual column information. Layer outputs are the encoded column vectors $V^{C}=\{v^{C}_{1},...,v^{C}_{m}\}\in\mathbb{R}^{m\times d}$.

Column vectors belonging to each table are integrated to get the encoded table vector. For a matrix $M\in\mathbb{R}^{n\times d}$, self-attention function $f_{s}(M)\in\mathbb{R}^{1\times d}$ is defined as follows:

Where $W_{1}\in\mathbb{R}^{d\times d}$, $W_{2}\in\mathbb{R}^{1\times d}$ are trainable parameters. Then, for table $t$ with columns $\{C_{j},...,C_{k}\}$, the hidden table vector $h^{T}_{t}$ is calculated as follows:

Layer outputs are the hidden table vectors $H^{T}=\{h^{T}_{1},h^{T}_{2},...,h^{T}_{t}\}\in\mathbb{R}^{t\times d}$.

In this layer, the same network architecture as the Question-Column alignment layer is used to model the table vectors with contextual information of the question. Layer outputs are the encoded table vectors $V^{T}=\{v^{T}_{1},v^{T}_{2},...,v^{T}_{t}\}\in\mathbb{R}^{t\times d}$.

Final outputs of the input encoder are: (1) Encoded question words $V^{Q}=\{v^{Q}_{1},...,v^{Q}_{n}\}\in\mathbb{R}^{n\times d}$, (2) Encoded columns $V^{C}=\{v^{C}_{1},...,v^{C}_{m}\}\in\mathbb{R}^{m\times d}$, (3) Encoded tables $V^{T}=\{v^{T}_{1},...,v^{T}_{t}\}\in\mathbb{R}^{t\times d}$, and (4) Encoded SPC $v^{P}\in\mathbb{R}^{d_{p}}$. Additionally, (5) Encoded question $v^{Q}=f_{s}(V^{Q})$ and (6) Encoded DB schema $v^{D}=f_{s}(V^{C})\in\mathbb{R}^{d}$ are calculated for later use.

By the term base structure of a SELECT statement, we refer to the existence of its component clauses and the number of conditions for each clause. We first combine the encoded vectors $v^{Q}$, $v^{D}$ and $v^{P}$ to get the statement encoding vector $v^{S}$, as follows:

Where $W\in\mathbb{R}^{d\times(4d+d_{p})}$ is a trainable parameter, and function $hc(x,y)$ is the concatenation function for heuristic matching method proposed in Mou et al. (2016).

Eleven values $b_{g},b_{o},b_{l},b_{w},b_{h},n_{g},n_{o},n_{s},n_{w}$, $n_{h}$ and $c_{\texttt{IUEN}}$ are classified by applying two fully-connected layers on $v^{S}$. Binary values $b_{g},b_{o},b_{l},b_{w},b_{h}$ represent the existence of GROUPBY, ORDERBY, LIMIT, WHERE and HAVING, respectively. Note that FROM and SELECT clauses must exist to form a valid SELECT statement. $n_{g},n_{o},n_{s},n_{w},n_{h}$ represent the number of conditions in GROUPBY, ORDERBY, SELECT, WHERE and HAVING clause, respectively. The maximal number of conditions $N_{g}=3$, $N_{o}=3$, $N_{s}=6$, $N_{w}=4$, and $N_{h}=2$ are defined for GROUPBY, ORDERBY, SELECT, WHERE and HAVING clauses, to solve the problem as $n$-way classification problem. The values of maximal condition numbers are chosen to cover all the training cases.

Finally, $c_{\texttt{IUEN}}$ represents the existence of one of INTERSECT, UNION or EXCEPT, or NONE if no such clause exists. If the value of $c_{\texttt{IUEN}}$ is one of INTERSECT, UNION or EXCEPT, the corresponding SPC is created, and the SELECT statement for that SPC is generated recursively.

A list of $TBLs should be decided to predict the FROM clause. For each table $i$, $P_{\textbf{fromtbl}}(i|Q,D,P)$, the probability that table $i$ is included in the FROM clause, is calculated:

Where $W_{1}$, $W_{2}$ are trainable variables, $s=[s_{1},...,s_{t}]\in\mathbb{R}^{t}$ and $\sigma$ denotes the sigmoid function. From now on, we omit the notations $Q$, $D$ and $P$ for the sake of simplicity.

Top $n_{t}$ tables with the highest $P_{\textbf{fromtbl}}(i)$ values are chosen. We set an upper bound $N_{t}=6$ on possible number of tables. The formula to get $P_{\textbf{\#tbl}}(n_{t})$ for each possible $n_{t}$ is:

In the equation, $full_{2}$ means the application of two fully-connected layers, and table score vector $s$ is from equation 7.

The decoder first generates $N_{s}$ conditions to predict the SELECT. Since each condition depends on different parts of $Q$, we calculate attended question vector for each condition:

While $W_{1},W_{2}\in\mathbb{R}^{d\times d}$, $W_{3}\in\mathbb{R}^{N_{s}\times d}$ are trainable parameters, and $V_{\textbf{sel}}^{Q}\in\mathbb{R}^{N_{s}\times d}$ is the matrix of attended question vectors for $N_{s}$ conditions. $v^{P}$ is tiled to match the row of $V^{Q}$.

For $m$ columns and $N_{s}$ conditions, $P_{\textbf{sel\_col1}}\in\mathbb{R}^{N_{s}\times m}$, the probability matrix for each column to fill the slot $COL₁ of each condition, is calculated as follows:

Where $W_{4},W_{5}\in\mathbb{R}^{d\times d}$ and $W_{6}\in\mathbb{R}^{1\times d}$ are trainable parameters. The notation $M[i]$ is used to represent the $i$-th row of matrix $M$.

The attended question vectors are then updated with selected column information to get the updated question vector $U^{Q}_{\textbf{sel\_col1}}\in\mathbb{R}^{N_{s}\times d}$:

Where $W_{7}$ is a trainable variable, and $hc(x,y)$ is defined in equation 5. The probabilities for $DIST₁, $AGG₁, $ARI and $AGG are calculated by applying a fully connected layer on $U^{Q}_{\textbf{sel\_col1}}[i]$.

Equation 10 is reused to calculate $P_{\textbf{sel\_col2}}$, with $V^{Q}_{\textbf{sel}}[i]$ replaced by $U^{Q}_{\textbf{sel\_col1}}[i]$; then $U^{Q}_{\textbf{sel\_col2}}$ is retrieved in the same way as equation 11, and the probabilities of $DIST₂ and $AGG₂ are calculated in the same way as $DIST₁ and $AGG₁. Finally, the $DIST slot, DISTINCT marker for overall SELECT clause, is calculated by applying a fully-connected layer on $v^{S}$.

Once all the slots are filled for $N_{s}$ conditions, the decoder retrieves the first $n_{s}$ conditions to predict the SELECT clause. That is possible since the CNN with Dense Connection used for question encoding (Yoon et al., 2018) captures relative position information. In combine with the SQL consistency protocol of the Spider benchmark (Yu et al., 2018b), we expect the conditions are ordered in the same way as they are presented in $Q$. For the datasets without such consistency protocol, the proposed slot filling method could easily be changed to an LSTM-based model, as shown in Xu et al. (2017).

The same network structure as a SELECT clause is applied. The only difference is the prediction for $ORD slot; this could be done by applying a fully connected layer on $U^{Q}_{\textbf{ob\_col1}}$, which is the correspondence of $U^{Q}_{\textbf{sel\_col1}}$.

The same network structure as a SELECT clause is applied. For the GROUPBY case, retrieving only the values of $P_{\textbf{gb\_col1}}$ is enough to fill the necessary slots.

Questions do not contain the $NUM slot value for LIMIT clauses explicitly in many cases, if the questions are for the top-1 result (Example: “Show the name and the release year of the song by the youngest singer”). Thus, the LIMIT decoder first determines if the given $Q$ requests for the top-1 result. If so, the decoder sets the $NUM value to 1; otherwise, it tries to find the specific token for $NUM among the tokens of $Q$ using pointer network (Vinyals et al., 2015). LIMIT top-1 probability $P_{\textbf{limit\_top1}}$ is retrieved by applying a fully-connected layer on $v^{S}$. $P^{Q}_{\textbf{limit\_num}}[i]$, the probability of $i$-th question token for $NUM slot value, is calculated as:

$W_{1}$, $W_{2}\in\mathbb{R}^{d\times d}$, $W_{3}\in\mathbb{R}^{1\times d}$ are trainable parameters.

The same network structure as a SELECT clause is applied to get the attended question vectors $V^{Q}_{\textbf{wh}}\in\mathbb{R}^{N_{w}\times d}$, and probabilities for $COL₁, $COL₂, $DIST₁, $DIST₂, $AGG₁, $AGG₂ and $ARI. Besides, a fully-connected layer is applied on $U^{Q}_{\textbf{wh\_col1}}$ to get the probabilities for $CONJ, $NOT and $COND.

A fully-connected layer is applied on $U^{Q}_{\textbf{wh\_col1}}$ and $U^{Q}_{\textbf{wh\_col2}}$ to determine if the condition value for each column is another nested SELECT statement or not. If the value is determined as a nested SELECT statement, the corresponding SPC is generated, and the SELECT statement for the SPC is predicted recursively. If not, the pointer network is used to get the start and end position of the value span from question tokens.

The same network structure as a WHERE clause is applied.

Spider dataset (Yu et al., 2018b) is used to evaluate our proposed system. We use the same data split as Yu et al. (2018b); 206 databases are split into 146 train, 20 dev, and 40 test. All questions for the same database are in the same split; there are 8659 questions for train, 1034 for dev, and 2147 for test. The test set of Spider is not publicly available, so for testing our models are submitted to the data owner. For evaluation, we used exact matching accuracy, with the same definition as defined in Yu et al. (2018b).

The proposed system is implemented with Tensorflow (Abadi et al., 2015). Layernorm (Ba et al., 2016) and dropout (Srivastava et al., 2014) are applied between layers, with a dropout rate of 0.1. Exponential decay with decay rate 0.8 is applied for every 3 epochs. On each epoch, the trained classifier is evaluated against the validation dataset, and the training stops when the exact match score for the validation dataset is not improved for 20 consequent training epochs. Minibatch size is set to 16; learning rate is set to $4e^{-4}$. Loss is defined as the sum of all classification losses from the slot-filling decoder.

For BERT-based input encoding, we downloaded the publicly available pre-trained model of BERT, BERT-Large, Uncased (Whole Word Masking), and fine-tuned the model during training. The learning rate is set to $1e^{-5}$, and minibatch size is set to 4.