Ranking Clarification Questions via Natural Language Inference

The advent of conversational systems like “Alexa”, “Siri” etc. has led to the identification of various interesting problems at the intersection of Natural Language Processing (NLP) systems and human interaction. One such problem is to ask clarifying questions when a system is presented with a natural language query. According to Rao and Daumé III (2018), the primary goal of a clarification question is to fill in the information gaps. This can be explained via the following example: suppose there is an underspecified (or ambiguous) post $p$ on a QA forum. Then, a good clarification question $q$, for $p$, would be the one whose answer leads to a resolution of the underspecification associated with $p$. In more simpler terms, a good question will be the one whose likely answer will be useful.

Rao and Daumé III (2018) pose the problem of coming up with clarification questions in an Information Retrieval setting, where the task is to retrieve clarifying questions from a pool of candidate questions with respect to a particular post. They propose a model based on the decision theoretic principle of Expected Value of Perfect Information (EVPI). Further, in order to facilitate training, Rao and Daumé III (2018) have provided a dataset based on three domains of StackExchange, which provides triplets of $(p,q,a)$, where $p$ is the original post, $q$ is the clarification question and $a$ is the most probable answer to the clarification question.

Based on the definition of a clarification question (as provided earlier), we can ascertain that the principal modeling objective of the problem is that a clarification question $q$ should help in increasing the probability of the correct answer to the given post $p$. An underlying assumption behind this statement is that the context of the answer will be useful in determining the most relevant question. In this work, we study this modeling problem as well as test its assumptions from a representation learning point of view. More specifically, we hypothesize and validate that ranking clarification questions could be treated as a special case of a Natural Language Inference task.

There are only a couple of datasets that explicitly contain clarification questions: Rao and Daumé III (2018) and ClarQ (Kumar and Black, 2020). We review two lines of related work: (1) Ranking of Clarification Questions and (2) Transfer Learning.

Ranking of Clarification Questions: There hasn’t been a lot of work on this problem. Recently, Aliannejadi et al. (2019) formulated the task of asking clarifying questions in open-domain information-seeking conversational systems, and provided a dataset, Qulac. They posit that a system can improve user satisfaction by proactively asking questions that can help satisfy users’ information needs. Additionally, they proposed a retrieval framework consisting of three components: question retrieval, question selection, and document retrieval. However, the dataset consists of keyword based queries and does not resemble the problems posed by natural language questions.

The work which is most relevant to ours is that of Rao and Daumé III (2018), which proposes a neural ranking model inspired by the theory of Expected Value of Perfect Information (EVPI) (Avriel and Williams, 1970) to rank clarification questions. The EVPI model is decomposed into two components (which is modelled jointly): (1) Answer Modelling and (2) Utility Calculator. The answer modelling component tries to assess the value of an answer associated with a post and a candidate clarification question. On the other hand, the utility calculator computes the utility of concatenating the post with the answer to the candidate clarification question.

Transfer Learning: The use of large-scale pre-trained models such as BERT (Devlin et al., 2018) for transfer learning has become prevalent in several Information Retrieval tasks. A number of recent works have further tried to improve upon the generic transfer learning methods of using representations and fine-tuning, and have proposed to further adapt BERT to their tasks, using additional training tricks such as multi-task learning and contrastive learning. In particular, Rosset et al. (2020) utilize a multi-task fine-tuning approach for the task of query suggestion (in context of ad-hoc search) and demonstrate that multi-tasking helps in improving the overall $CTR$ of search results.

We formally describe the problem as follows: Given an initial post $p_{i}\in P$ and a set of candidate clarification questions $q_{i}\in Q$, the task is to provide a ranked list of $N$ clarification questions $[q_{i1},q_{i2}\dots q_{iN}]\forall q_{ij}\in Q$, such that the more relevant questions are higher up in the ranked list. Each $q_{i}j$ is also paired with an answer $a_{i}j$ which can be used as an additional signal for model training. Note that $a_{i}j$ is an answer to $q_{i}j$ and not the post $p_{i}$ ¹¹1In all our experiments, $N$ is set to 10..

Before describing our proposed methods, we draw an analogy between the task of natural language inference and the pair regression task of ranking clarification questions.

Natural Language Inference (NLI) : The task of natural language inference is to determine whether, given a premise, a hypothesis can be inferred. The task has been widely studied in the Natural Language Processing community, where two large scale datasets, namely the Stanford Natural Language Inference (SNLI) corpus (Bowman et al., 2015) and the Multi-Genre NLI (MultiNLI) corpus (Williams et al., 2017) have been made available to evaluate different representations and models (Wang et al., 2018).

Task Analogy with NLI : In the task of evaluating a candidate clarification question for a given post, the classifier has to predict whether the question follows from the context of the post. This formulation can be juxtaposed to the task of natural language inference (NLI), where, given a premise, the classifier has to predict whether a hypothesis is true (entailment), false (contradiction) or neutral. However, the task of evaluating a candidate clarification question is narrower in scope. More precisely, the task of of evaluating a candidate clarification question for a given post is a binary inference problem, where the inference is made on the basis of latent information that is common to the context of both the question and the post. Consequently, we hypothesize that our problem is a special case of the natural language inference (NLI) task, since, given a post and a question, the problem is to determine whether the two share any common latent information, based on the context.

Leveraging NLI : The previous analogy suggests that pre-training the classifier on NLI datasets might be useful to bootstrap the model for ranking clarification questions. To this end, we use the Sentence-BERT models (SBERT) (Reimers and Gurevych, 2019) pretrained on SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2017) datasets. Using the SBERT model to extract representations for the posts, questions and answers, we build the following models:

1. (1)

   SBERT-PQ: We extract post (P) and question (Q) representations (768 dimensional) from SBERT and feed it to a three layer fully connected network, before outputting softmax probabilities, of the question being true (relevant) or false (non-relevant). The loss function is the standard cross entropy loss.

2. (2)

   SBERT-PQA: Besides the post (P) and question (Q), we also extract representations for the corresponding answer (A) from SBERT (768-dimensional). Rest of the model is the same as SBERT-PQ.

3. (3)

   SBERT-Large-PQ: We use SBERT-Large to extract the representations (1024-dimensional) for the posts (P) and questions (Q). Rest of the model is the same as SBERT-PQ.

4. (4)

   SBERT-Large-PQA: Besides the post (P) and question (Q), we also extract representations for the corresponding answer (A) from SBERT-Large (1024-dimensional). Rest of the model is the same as SBERT-Large-PQ.

In all the SBERT models, we extract the mean pooled final-layer representations of the tokens.

The key metric to compare the performance of the various models is P@k, which refers to Precision of the top-k results generated by the model. Tables 1 and 2 describe the performance of the various models, along with the results reported from Rao and Daumé III (2018). Rao and Daumé III (2018) only report Precision at 1, 3 and 5, while we also report the Precisions at 2 and 4 in both the tables.

We make the following observations based on the obtained results:

1. (1)

   The Bidi-PQA model significantly outperforms the EVPI model, especially on P@1 and P@5 metrics.

2. (2)

   The performance of all DistilBERT based models is lower than that of PQA, EVPI as well as Bidi-PQA. This clearly suggests that using the representations of DistilBERT was detrimental for the task at hand. One of the possible reasons could be that StackExchange posts consist of code snippets and character patterns that need to be recognized in an effective manner. Perhaps, off-the-shelf BERT (when it receives out-of-domain data) was ineffective in identifying such patterns as most of it gets treated as unseen tokens. On the other hand, since Bidi-PQA and EVPI utilize Glove embeddings pre-trained on the StackOverflow data, they are able to perform much better. Our finding here is consistent with a very recent work by Tabassum et al. (2020). Tabassum et al. (2020) show that Glove embeddings perform far better than a fine-tuned off-the-shelf BERT on the task of named entity recognition on the StackOverflow data. They also show that training BERT on StackOverflow data (with the use of a special tokenizer) achieves much better performance than a fine-tuned off-the-shelf BERT. We believe that adopting a similar strategy would be beneficial for our task as well, given the domain mismatch.

3. (3)

   All SBERT based models are in par or better than EVPI in terms of P@1 scores. In fact, SBERT-PQ model (which is the best performing SBERT model) provides a 60% improvement over EVPI when tested using Best Annotations and a 40% improvement over EVPI when tested using Valid Annotations (in terms of P@1 scores). The P@1 performance of SBERT-PQ model is also better than that of Bidi-PQA. The task that we are entailed with is a high precision task, and thus a model which has a higher P@1 can be considered to be a better model overall. The better performance of SBERT model also validates our hypothesis (Section 3) that the problem of ranking clarification questions is a special case of a natural language inference task. In order to predict a good clarification question, the model should be able to infer the latent information that is common to the context of both the question and the post. We hypothesize that this is the core source of the gains observed in this case.

4. (4)

   When comparing the PQ and PQA counterparts of the DistilBERT and SBERT models, we observe that PQ models perform better than PQA models. This means that a majority of the times, incorporating answers within the model was detrimental to the performance. We hypothesize that this could be an artifact of the data itself. Rao and Daumé III (2018) identify answers to each questions based on certain heuristics like time stamps, edit history etc. It could be very well possible that the answers might not be actual answers at all. For certain models, like SBERT, such answers might be causing a drift in the models understanding of a good clarification question i.e owing to lower semantic similarity, answers could be acting as noise.

Finally, regarding the overall performance of the various methods, we can observe that the SBERT based models generally obtain the highest P@1 scores thereby producing a new state-of-the-art for the given task.

Figure 1 shows the plot of the number of posts (bucketed by post length), for which each of the models got the correct clarification question at rank 1, when evaluated on the ‘Valid’ evaluation set. The models compared are: EVPI (the published state-of-the-art), Bidirectional-PQA and SBERT-PQ (our new state-of-the-art). From the figure, it is evident that for posts in buckets of length 81 or larger, the SBERT-PQ model significantly outperforms the other two. This is despite the fact that it uses no answer information (unlike the other two). Although, it is hard to disentangle whether the gains arrive due to better representations or due to better inference of the shared latent information, given that, at shorter lengths (upto 40), EVPI does almost as well as SBERT-PQ. This also shows that without any NLI-based representations, the inference task becomes significantly harder for longer length posts, where SBERT-PQ gains a clear advantage. Further, the trends are almost similar on the ‘Best’ evaluation set as can be seen from Figure 2.

We approach the problem of ranking clarification questions from a natural language inference perspective, which is in contrast to the previous approaches in existing literature (decision-theoretic, generative, etc.) and demonstrate that BERT representations pre-trained on SNLI and MultiNLI can achieve very high performance on the task. We also showed that the context provided by the answer isn’t helpful in learning representational similarities for the task.