Prior Knowledge Driven Label Embedding for Slot Filling in Natural Language Understanding

The posterior probability is factorized:

where $\mathbf{o}(t_{i})$ is a one-hot vector with a dimension for tag $t_{i}$, so $\text{softmax}(\mathbf{z}_{i})^{\top}\mathbf{o}(t_{i})$ is precisely the $t_{i}$-th element of the distribution defined by the softmax function.

To model the time series dependence of output labels, several methods based on encoder-decoder architectures [9, 10, 6, 49] are proposed for slot filling. Besides that, CRF output layer can jointly decode the best chain of labels for a given input sentence [50, 51, 52]. The CRF output layer calculates the posterior probability at the sentence level:

The score of a sentence $\boldsymbol{w}$ along with a path of tags $\boldsymbol{t}$ is given by the sum of transition scores and network scores:

where $\mathbf{A}\in\mathbb{R}^{|\mathcal{T}_{d}|\times|\mathcal{T}_{d}|}$ is a transition matrix, and its element $[\mathbf{A}]_{m,n}$ modelling the transition from the $m$-th to the $n$-th label for a pair of consecutive time steps.

The maximum likelihood estimation is used for training, and the negative log-likelihood is given by

. Decoding is to search for the label sequence $\boldsymbol{t}^{*}$ with the highest posterior probability, i.e.

The searching is time independent for the softmax output function, while it can be solved efficiently by adopting the Viterbi algorithm for the CRF output layer.

We represent each slot $s$ of domain $d$ as a set of atomic concepts, i.e. $\text{atoms}(s)=\{a_{1},\cdots,a_{M_{s}}\}$, $a_{l}\in\mathcal{A}_{d}$, where $\mathcal{A}_{d}$ is the vocabulary of atomic concepts and $M_{s}$ is the number of atomic concepts belong to this slot. Thus, each slot can be viewed as a binary vector, $\mathbf{b}(\text{atoms}(s))$ with length of $|\mathcal{A}_{d}|$, while $\mathbf{b}(\text{atoms}(\text{SLOT}(\text{``O''})))=\mathbf{0}$. Due to the IOB schema, the label embedding consists of two parts, 3-dimensional IOB one-hot vector and a binary vector of atoms, as illustrated in Fig. 3 (a). The label embedding of tag $t$ is

where $\text{emb}(t)\in\mathbb{R}^{3+|\mathcal{A}_{d}|}$, $t\in\mathcal{T}_{d}$.

Finally, a label embedding matrix for domain $d$ is created, i.e., $\mathbf{B}_{d}\in\mathbb{R}^{(3+|\mathcal{A}_{d}|)\times|\mathcal{T}_{d}|}$, where $[\mathbf{B}_{d}^{\top}]_{t}=\text{emb}(t)$. Equation (1) can be rewritten as

where $\mathbf{C}_{d}\in\mathbb{R}^{(3+|\mathcal{A}_{d}|)\times 2n}$ denotes trainable parameters of the output layer, but $\mathbf{B}_{d}$ is fixed as a kind of prior knowledge.

Similar to subsection III-B, the domain adaptation with atomic concepts is also introduced as follows. For all the source domains, we have a union of atoms, $\mathcal{A}_{src}=\bigcup\limits_{d\in\mathcal{D}_{src}}\mathcal{A}_{d}$. Therefore, the output matrix is scaled up to $\mathbf{C}_{src}\in\mathbb{R}^{(3+|\mathcal{A}_{src}|)\times 2n}$, and $\mathbf{B}_{d}\in\mathbb{R}^{(3+|\mathcal{A}_{src}|)\times|\mathcal{T}_{d}|}$. For each domain $d\in\mathcal{D}_{src}$, there is mask vector $\mathbf{m}_{d}\in\mathbb{R}^{3+|\mathcal{A}_{src}|}$,

for each $a_{j}\in\mathcal{A}_{src}\cup\text{IOB}$. It is used to ignore the atomic concepts out of domain $d$ and replace Equation (6) with:

For fine-tuning in the target domain $td$, the word embedding matrix and BLSTM parameters are still retained. The initialization of $\mathbf{C}_{td}\in\mathbb{R}^{(3+|\mathcal{A}_{td}|)\times 2n}$ for $td$ is

for each $a_{j}\in\mathcal{A}_{td}\cup\text{IOB}$. Thus, we calculate output scores of the target domain as $\mathbf{z}_{i}=\mathbf{B}_{td}^{\top}\mathbf{C}_{td}\mathbf{h}_{i}$, where $\mathbf{B}_{td}\in\mathbb{R}^{(3+|\mathcal{A}_{td}|)\times|\mathcal{T}_{td}|}$.

Composition of atomic concepts for each slot provides an explicit interpretation about slot relations. However, it requires a large number of human efforts to design atomic concepts carefully, and it is hard to keep them universal for different domains.

Every description word is mapped to an $m$-dimensional vector via $\mathbf{x}_{l}=\mathbf{W}_{in}\mathbf{o}(x_{l})$, where $\mathbf{W}_{in}$ is the word embedding matrix and $\mathbf{o}(x_{l})$ a one-hot vector. The embedding of slot $s$ is simply the mean of word embeddings of all words in $\text{desc}(s)$, i.e.

where $unit(\mathbf{v})=\frac{\mathbf{v}}{||\mathbf{v}||}$ gets the unit $l_{2}$ norm of vector $\mathbf{v}$, which enforces slot embeddings in a limited scale.

The hidden vectors of BLSTM encoder are recursively computed at the $l$-th time step via:

where $\text{f}_{\text{{lstm}}}$ and $\text{b}_{\text{{lstm}}}$ are the LSTM functions for forward and backward passes respectively. We concatenate the final hidden vectors of bi-direction to be the slot embedding, i.e.

CNN is also a typical model to encode a sequence of words into a fixed-dimensional vector [53]. A linear transformation is applied on a window of $k$ words to produce a set of new features (we set $k=3$ in this work). To avoid that the window size is larger than the sequence length, zero-padding is used on both sides of the input sequence. After processing all possible windows of words, we get a sequence of new feature vectors. A max-over-time pooling operation [54] is applied over new features, which takes the maximum value of each row to form as the final features $\mathbf{c}$. Thus,

With one of these three encoders, we can encode the description of a slot as a vector. Thus, the label embedding of tag $t\in\mathcal{T}_{d}$ is

where ENCODER can be mean, BLSTM and CNN, $\text{emb}(t)\in\mathbb{R}^{K}$, $K$ is the embedding length. Note that $\text{emb}(\text{``O''})=\mathbf{0}$.

For any domain $d\in\mathcal{D}_{src}\cup\{td\}$, we get a label embedding matrix, $\mathbf{B}_{d}\in\mathbb{R}^{K\times|\mathcal{T}_{d}|}$, and $[\mathbf{B}_{d}^{\top}]_{t}=\text{emb}(t)$ for any $t\in\mathcal{T}_{d}$. Both Equation (1) and (6) can be rewritten as

where $\mathbf{z}_{i}\in\mathbb{R}^{|\mathcal{T}_{d}|}$, $\mathbf{C}\in\mathbb{R}^{K\times 2n}$ denotes trainable parameters of the output layer, but $\mathbf{B}_{d}$ is computed by the slot description encoder on domain $d$. Since the encoder is domain-independent, $\mathbf{C}$ and the encoder can also be fully shared across domains, as well as the sentence BLSTM which produces $\mathbf{h}_{i}$.

This method requires an unstructured short-text to describe each slot, which is easier to be obtained, while the involved label encoders may take more time for computation.

Every word in the exemplar sentence is mapped to a vector via $\mathbf{e}_{j}=\mathbf{W}_{e}\mathbf{o}(e_{j})$, where $\mathbf{W}_{e}$ is a word embedding matrix and $\mathbf{o}(e_{j})$ is a one-hot vector. To specify the value and context information of the slot, we split the exemplar sentence into three segmentations, left context, value and right context, as illustrated in Fig. 4 (a). Then, word embeddings in these three parts are respectively averaged and concatenated, composing the slot embedding

where $\text{ave}(\mathbf{e}_{<sp})$ and $\text{ave}(\mathbf{e}_{>ep})$ are zero vectors when $sp=1$ and $ep=|e|$ respectively.

Different from word embeddings, LSTM based bidirectional language models (biLMs) [55] directly encode context information into hidden states. The input sentence $\boldsymbol{e}=e_{1}\cdots e_{|e|}$ is converted into two sequences of hidden vectors ($\overrightarrow{\mathbf{u}}_{1:|e|},\overleftarrow{\mathbf{u}}_{1:|e|}$) by forward and backward LMs respectively, as shown in Fig. 4 (b). Since hidden states already contain information of history and current word, we use the hidden vectors at positions ranging from $sp$ to $ep$ to compose the slot embedding:

Now, we can encode the exemplars of any slot as a vector (if there are multiple exemplars for one slot, an average is taken). Thus, the label embedding of tag $t\in\mathcal{T}_{d}$ is

where $\text{emb}(t)\in\mathbb{R}^{K}$, $K$ is the embedding length, and $\text{emb}(\text{SLOT}(\text{``O''}))=\mathbf{0}$.

For any domain $d\in\mathcal{D}_{src}\cup\{td\}$, we get the label embedding matrix, $\mathbf{B}_{d}\in\mathbb{R}^{K\times|\mathcal{T}_{d}|}$, and $[\mathbf{B}_{d}^{\top}]_{t}=\text{emb}(t)$ for any $t\in\mathcal{T}_{d}$. Both Equation (1) and (6) can be rewritten as

where $\mathbf{z}_{i}\in\mathbb{R}^{|\mathcal{T}_{d}|}$, $\mathbf{C}\in\mathbb{R}^{K\times 2n}$ denotes trainable parameters of the output layer which can be shared across domains. However, $\mathbf{B}_{d}$ is computed by using pre-trained word embeddings or biLMs in this paper, which will be fixed.

This method requires minimal prior knowledge which is some kind of labeled data sample, while it may heavily rely on the quality of pre-trained word embeddings or biLMs.

Three different approaches to extracting label embeddings are investigated in this paper, via atomic concepts, slot descriptions and slot exemplars, which involve different degrees of human knowledge. Table III also provides the detailed statistics of label representations. For atomic concepts, we act as domain experts to manually split each slot into small units for all five domains, e.g., deny-starting_point is split into “deny”, “from_location” and “poi_name”. As an extension of DSTC2, DSTC3 contains 6 new atoms that constitute about $25\%$ of slot tags. For slot descriptions, we leverage tokenized slot names since each slot name is already a meaningful phrase in the datasets, e.g., deny-starting_point is described as “deny starting point”. The training data can be directly exploited as data exemplars for the corresponding slots.

The experimental setup was kept the same for all three datasets with minor adjustments in hyper-parameters. We adopt a 2-layer BLSTM model to encode sentences with $200$ hidden units. For training, the network parameters are randomly initialized under the uniform distribution $[-0.2,0.2]$. We use optimizer Adam [60] with learning rate $0.001$ for all experiments. The dropout with a probability of $0.5$ is applied to the non-recurrent connections during the training stage. The batch size is $20$ for domain DSTC2, MAP and WEATHER, $5$ for domain DSTC3 and FLIGHT, and $10$ for SNIPS. The maximum norm for gradient clipping is set to 5, and we use weight decay regularization on all weights with lambda=$1e\text{-}6$ to avoid over-fitting.

We exploit pre-trained word embeddings to initialize the input embedding layers. For AICar, we train an LSTM based bidirectional language models (biLMs) at Chinese-character level by using zhwiki²²2https://dumps.wikimedia.org/zhwiki/latest corpus. The embedding dimension is 200. For DSTC 2&3 and SNIPS, we directly adopt a pre-trained biLMs of ELMo [55] in English³³3https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md. The word embedding dimension is 1024.

The pre-trained input embeddings and BiLMs are also exploited to compute label embeddings based on slot exemplars, as shown in Section V-C. In this paper, we choose to fix the encoder of slot exemplars to reduce training cost. Therefore, the performance of slot filling based on slot exemplars may heavily rely on qualities of the pre-trained BiLMs.

To evaluate the efficiency of different methods in leveraging data, we set up a domain adaptation problem. In DSTC 2&3, DSTC2 is source domain, and DSTC3 is target domain. In AICar, we choose each domain as the target one and the rest two domains as source domains. Moreover, $1,2.5,5,10,20,40,60,80$ percent of the training data in the target domain are randomly selected to simulate low resource settings of the target domain. Each data selection is repeated 10 times by using different random seeds, and the final results of the evaluation are averaged. For SNIPS, we keep one intent as the target domain and the rest intents as the source domain. We pre-train the slot filling model on the source domains, then fine-tune and evaluate it on the target domain.

We randomly selected $80\%$ of the training data for model training and the remaining $20\%$ for validation. We keep the learning rate for 50 epochs and save the parameters that give the best performance on the validation set. Finally, we report the $\text{F}_{1}$-score of the semantic slots on the test set with parameters that have achieved the best $\text{F}_{1}$-score on the validation set. The $\text{F}_{1}$-score is calculated using CoNLL evaluation script⁴⁴4https://www.clips.uantwerpen.be/conll2000/chunking/output.html.

We use McNemar’s test to establish the statistical significance of a method over another ($\text{p}<0.05$).

Besides using pre-trained word embeddings, some advanced pre-trained language models (e.g. ELMo [55], BERT [61]) can also be used to get input embeddings. However, it is orthogonal to the investigation of label embeddings. Table VI shows results on DSTC3 test set using limited seed instances, indicating that our method can get improvements over different types of input embeddings simultaneously. For BERT, bert-base-cased model⁵⁵5https://github.com/google-research/bert#pre-trained-models is utilized, while it performs worse than ELMo. The reason may be that ELMo could get better generalization for the dataset by using char-based word embeddings.

Fig. 5 shows learning curves of different methods when the target domain is FLIGHT and 100% training data is used for fine-tuning. It shows slot $\text{F}_{1}$ scores on the validation set at increasing numbers of data iteration. We can see that our methods can converge much faster than the baselines without label embeddings. Especially, the “slot description (BLSTM)” method converges fastest.

Although we exploit all data samples of the training set as data exemplars for the corresponding slots, it is meaningful to investigate that how many exemplars for each slot are enough. Fig. 6 shows the number of slot exemplars used versus performance on FLIGHT domain. If there is no training data for fine-tuning, more exemplars usually give better performance. If there is an amount of labelled data for training, the performance is dominated by the size of training data in the target domain.

We also apply a CRF layer to model the time series dependence of output labels, which CT and ZAT cannot model. We want to know whether label embeddings can improve different slot filling models. Table VII shows the results of using different label embeddings with or without the CRF layer, in the case that FLIGHT is the target domain and $10\%$ training data is used for fine-tuning. From the table, we can see that CRF layers give improvements for the baseline and our methods. Compared to the traditional scalar transition model (as shown in Equation (4)), our proposed bilinear transition model (as shown in Section V-D) can improve significantly. The bilinear model calculates the transition score of two labels depending on their label embeddings, which shares transition patterns of related labels. It helps mitigate the data sparsity problem.

Different from atomic concepts or slot exemplars which can be in a united form, slot descriptions in natural language would be varied due to different domain experts or developers. Five participants were asked to describe each slot in the FLIGHT domain. The average length of slot descriptions from different participants varies from $6.2$ to $7.7$. If we use these slot descriptions in the “slot description (BLSTM)” method when FLIGHT is target domain and $10\%$ training data can be used for fine-tuning, slot $\text{F}_{1}$ scores range from $84.66\%$ to $86.43\%$. As shown in Table VIII, our method seems robust to varied slot descriptions.

We also report results on the SNIPS public dataset where one intent are kept as the target domain and the rest intents serve as the source domain. Only 50 train instances of the target domain can be utilized for fine-tuning, which are chosen in random. All 7 intents play a role of the target domain one by one, and the slot $\text{F}_{1}$ scores are averaged over seven target domains. As shown in Table IX, our methods can also outperform the baselines. Especially, the “slot exemplar (BiLMs)” method achieves the best slot $\text{F}_{1}$ score.