Modeling Token-level Uncertainty to Learn Unknown Concepts in SLU
via Calibrated Dirichlet Prior RNN

Slot filling is to extract semantic concepts from a natural language utterance. It has been modeled as a sequence labeling problem in literature. Given an input utterance as a sequence of words $\boldsymbol{\mathrm{x}}=(x_{1},\ldots,x_{n})$ of length $n$, slot filling maps $\boldsymbol{\mathrm{x}}$ to the corresponding label sequence $\boldsymbol{\mathrm{y}}$ of the same length $n$. Each word is labeled as one of the total $K$ labels. As shown in Table 1, each output label in $\boldsymbol{\mathrm{y}}$ is in the format of IOB, where “B” and “I” stand for the beginning and intermediate word of a slot and “O” means the word does not belong to any slot. As a sequence labeling problem, slot filling can be solved using traditional machine learning approaches McCallum et al. (2000); Raymond and Riccardi (2007) and recent mainstream recurrent neural network (RNN) based approaches which takes and tags each word in an utterance one by one Yao et al. (2014); Mesnil et al. (2015); Peng and Yao (2015); Liu and Lane (2015); Kurata et al. (2016). These RNN based approaches designed different models to estimate the maximum likelihood:

$$P(\theta|D)=\max_{\theta}\displaystyle E_{D}\Big{[}\prod_{t=1}^{n}P(y_{t}|y_{1},\ldots,y_{t-1},\boldsymbol{\mathrm{x}};\theta)\Big{]}$$

By optimizing the following loss function:

$$\mathcal{L}(\theta)\triangleq-{1\over n}\sum_{i=1}^{|S|}\sum_{t=1}^{n}y_{t}(i)\log P_{t}(i)$$

(1)

where $D=\{\boldsymbol{\mathrm{x}},\boldsymbol{\mathrm{y}}\}\sim p(\boldsymbol{\mathrm{X}},\boldsymbol{\mathrm{Y}})$ is the training set; $\theta$ are the model parameters. $P_{t}(i)$ is typically computed using softmax function. All models with such loss function are referred to as softmax RNN slot filling model.

We refer to the training data $D$ as in-distribution (IND) data and each utterance in $D$ is drawn from a fixed but unknown distribution $P_{IND}$. Our goal is to train a slot filling model only on IND data $D$ to achieve two objectives: (1) correctly label a test sequence $\boldsymbol{\mathrm{z}}$ drawn from the same distribution $P_{IND}$; (2) identify out-of-distribution elements in a test sequence $\boldsymbol{\mathrm{z}}$ drawn from a different distribution $P_{OOD}$, referred to as out-of-distribution (OOD) data. Note that the OOD sequence may only have a subset of out-of-distribution elements. In this case, our goal is to identify only these OOD elements and correctly label other IND elements.

Specifically, we take a pre-defined softmax RNN slot filling model architecture (described in previous subsection) as input. We train our calibrated prior RNN model only by enhancing its softmax layer. Our model outputs two sequences for each test sequence $\boldsymbol{\mathrm{z}}$ with an unseen concept $\boldsymbol{\mathrm{c}}^{*}$: (1) traditional label sequence $\boldsymbol{\mathrm{y}}$; (2) uncertainty sequence $\boldsymbol{\mathrm{u}}$ with an uncertainty score $u_{t}$ for each word $z_{t}$. For $\boldsymbol{\mathrm{z}}$ drawn from $P_{IND}$, all labels in $\boldsymbol{\mathrm{y}}$ are expected to be correct and all uncertainty scores in $\boldsymbol{\mathrm{u}}$ are expected to be low. For $\boldsymbol{\mathrm{z}}$ drawn from $P_{OOD}$, the model is expected to either label an element correctly or gives it a high uncertainty score. At last, the model tags all concepts with correct slot types or as unknown.

As discussed in Section 3, the existing approaches for slot filling are discriminative models which aim to find the best model parameters $\theta^{*}$ to fit training data. To estimate the uncertainty of each word, a naive solution is to use its softmax confidence at each step. However, since an unseen concept is out of distribution from the training data, such pretrained neural networks tend to be blindly overconfident on predictions of completely unrecognizable Nguyen et al. (2015) or irrelevant inputs Hendrycks and Gimpel (2017); Moosavi-Dezfooli et al. (2017).

Consider the IND training set $D=\{\boldsymbol{\mathrm{x}},\boldsymbol{\mathrm{y}}\}\sim p(\boldsymbol{\mathrm{X}},\boldsymbol{\mathrm{Y}})$. For an utterance $\boldsymbol{\mathrm{z}}$, we model its high-order distribution $p(\boldsymbol{\omega}|\boldsymbol{\mathrm{z}},D)$ using a Bayesian framework as follows:

$\displaystyle p(\boldsymbol{\omega}|\boldsymbol{\mathrm{z}},D)=\int\underbrace{p(\boldsymbol{\omega}|\boldsymbol{\mathrm{z}},\theta)}_{\text{Data}}\underbrace{p(\theta|D)}_{\text{Model}}d\theta$

(2)

where, similar as in Equation 1, we have:

$$p(\boldsymbol{\omega}|\boldsymbol{\mathrm{z}},\theta)=\prod_{t=1}^{n}p(\boldsymbol{\omega}_{t}|\boldsymbol{\omega}_{1},\ldots,\boldsymbol{\omega}_{t-1},\boldsymbol{\mathrm{z}};\theta)$$

where $\boldsymbol{\omega}_{i}$ is the predicted distribution of all possible $K$ labels for each word $i$ in $\boldsymbol{\mathrm{z}}$. The data uncertainty is described by label-level posterior $p(\boldsymbol{\omega}|\boldsymbol{\mathrm{z}},\theta)$ and model uncertainty is described by model-level posterior $p(\theta|D)$. However, the integral in Equation 2 is intractable in deep neural networks, thus Monte-Carlo Sampling algorithm is used to approximate it as follows:

$\displaystyle p(\boldsymbol{\omega}|\boldsymbol{\mathrm{z}},D)\approx\frac{1}{M}\sum_{i=1}^{M}p(\boldsymbol{\omega}|\boldsymbol{\mathrm{z}},\theta^{(i)});\theta^{(i)}\sim p(\theta|D)$

where each $p(\boldsymbol{\omega}_{t}|\boldsymbol{\omega}_{1},\ldots,\boldsymbol{\omega}_{t-1},\boldsymbol{\mathrm{z}},\theta^{(i)})$ is a categorical distribution $\boldsymbol{\mu}$ over the simplex with $\boldsymbol{\mu}=[\mu_{1},\cdots,\mu_{K}]=[p(y=\omega(1),\cdots,p(y=\omega(K))]^{T}$. This ensemble is a collection of points on the a simplex which can be viewed as an implicit distribution induced by the posterior over the model parameters $p(\theta|D)$.

In addition to the intractability of posteriori estimation, such ensemble of implicit distribution also leads to the challenge of measuring uncertainty. For example, a high entropy of $p(\boldsymbol{\omega}|\boldsymbol{\mathrm{z}},D)$ could indicate uncertainty in the prediction due to either an IND input in a region of class overlap or an OOD input far from the training data. When using other measures like mutual information Gal (2016) to determine the uncertainty source, it is very hard in practice due to the difficulty of selecting an appropriate prior distribution and the expensive computation needed for Monte-Carlo estimation in RNN.

While the proposed degenerated Dirichlet prior RNN is sufficient in some relatively simpler case, the uncertainty measure is still sensitive and erratic to noises that its detection accuracy can hardly provide accurate estimation for model-level uncertainty. This is mainly because that Dirichlet prior RNN is trained only on IND data. Specifically, it is caused by the known over-fitting issue in the pre-trained neural network, where the model becomes over-confident about its prediction though it mislabels it. In our problem of modeling uncertainty for each word in a sequence, this drawback is further amplified due to the overconfidence from both the current word and the utterance context. Moreover, the classification model greatly emphasizes certain dimensions in the concentration parameter regardless of the form of inputs, which causes both data sources to have indistinguishable high confidence.

Our goal is to further make the IND and OOD concepts more separable using only IND data. Previous works Kurakin et al. (2016); Liang et al. (2017b) have designed various approaches to perturb the original data to simulate OOD samples. Unfortunately, these approaches are not applicable onto natural language processing problems due to it discrete nature.

Thanks to the modeling of Dirichlet distribution $\mu$, we will use its entropy $H$ to computer the uncertainty score for each word in an utterance. For a Dirichlet distribution $Dir(\boldsymbol{\alpha})$, it has a close-form entropy form as follows:

$$\begin{split}H(\boldsymbol{\alpha})=\log B(\boldsymbol{\alpha})+(\alpha(0)-K)\psi(\alpha(0))-\sum_{i}^{k}(\alpha(i)-1)\psi(\alpha(i))\end{split}$$

where $\alpha(0)=\sum_{i}^{k}\alpha(i)$ denotes the sum over all $K$ dimensions. Thus, the uncertainty score $u_{t}$ for the $t^{th}$ word in utterance $z$ can be computed as:

$$u_{t}=\begin{cases}H(\boldsymbol{\alpha}_{t})&\mbox{Dirichlet Prior RNN}\\ H(\tilde{\boldsymbol{\alpha}}_{t})&\mbox{Calibrated Dirichlet Prior RNN}\\ \end{cases}$$

where Dirichlet Prior RNN has $\boldsymbol{\alpha}_{t}=Me^{\boldsymbol{m}_{t}}=M\exp(RNN(x_{t};\theta))$ and Calibrated Dirichlet Prior RNN has calibrated $\tilde{\boldsymbol{\alpha}}_{t}=(\boldsymbol{\mathrm{I}}-\boldsymbol{\mathrm{W}}_{c})\boldsymbol{\alpha}_{t}$. When $u_{t}$ is larger than a threshold $\theta$, we set its uncertainty label as high(H) and mark it as “unknown” by ignoring its predicted slot label.

We extract unknown concept by expanding the set of high uncertainty words with dependency tree of the input utterance. For each word labeled as “unknown”, we traverse the dependency tree upwards along edges (including compound nouns, adjectival modifiers, or posessive modifiers) and locate the root of the corresponding noun phrase containing this word. Then, we mark this root node and all its descendants as “unknown”. This procedure helps to improve the recall by considering syntactically coherent phrases.

At last, we convert the labels to IOB format at phrase level by extracting all sets of consecutive words that are labeled as “unknown”. In each set, we label the first word as “B-unknown” and all following words as “I-unknown”.

We evaluate our method on three SLU benchmark datasets: Concept Learning Jia et al. (2017), Snips Snips (2017) and ATIS Hemphill et al. (1990). For Snips and ATIS datasets, we collect new concepts in a subset of slot types which have many potential uncovered concepts. The dataset details are in Table 2.

For Snips and ATIS datasets, we collect new concepts in a subset of slot types which have many potential uncovered concepts. We separate the slot types to be personalized and populational. We collect the new concepts from a group of crowd sourced workers. For personalized concepts (e.g., playlist), we allow them to freely create any new concept. For population concepts (e.g., movie name), we further evaluate the collected concepts by soliciting judgments from another group of crowd sourced markers. Each turker marker is presented 50 concepts of one slot type and required to select the correct ones. To judge the trustworthiness of each turker marker, we include 10 gold-standard concepts internally marked by experts. Finally, we keep all population concepts that are marked correct by trustworthy markers and all collected personalized concepts. At last, we substitute these new concepts in original basic test set (IND) and construct the new test set (OOD). In OOD test data, we label the unseen concept as unknown in IOB format to preserve phrase level information for evaluation.  Table 2 shows the data details and statistics.

We evaluate on the state-of-the-art RNN based slot filling model Goo et al. (2018). We use the state-of-the-art RNN based slot filling model Goo et al. (2018) in all of our experiments. We use all the hyperparameters in original paper Goo et al. (2018) for ATIS and Snips datasets. For Concept Learning dataset, we tested different hyperparameters including hidden state size 32, 64, 128 and 256 through validation. The same hidden state size 64 in original paper also gives the best results in Concept Learning dataset and we used it in our experiment.

For the only hyperparameter $\delta$ in our method, we experimented with different values of $\delta$ from 0.05 to 0.5 with uniform interval 0.05 (we try 10 values of $\delta$) and found that setting $\delta=0.1$ yield the best results. We implement our method in Tensorflow using Adam optimizer with 16 mini-batch size. On each dataset, we train both our RNN and Calibrated Prior RNN models with 20 epochs using single GPU. All experiments are run on NVIDIA Tesla V100 GPU 16GB.

Table 3 reports the slot filling results of IND test set. In all datasets, we observe that our Calibrated Prior RNN model only slightly affects performance (up to 0.2 F1 drop) against RNN baseline due to the calibration loss.

We refer to our method as Dirichlet Entropy + Syntax on (Dirichlet Prior) RNN and Calibrated Dirichlet Prior RNN.