Unsupervised Alignment-based Iterative Evidence Retrieval for Multi-hop Question Answering

AIR iteratively builds justification chains given a query. AIR starts by initializing the query with the concatenated question and candidate answer text³³3Note that this work can be trivially adapted to reading comprehension tasks. In such tasks (e.g., SQuAD Rajpurkar et al. (2018)), the initial query would contain just the question text.. Then, AIR iteratively repeats the following two steps: (a) It retrieves the most salient justification sentence given the current query using an alignment-IR approachYadav et al. (2019a). The candidate justification sentences come from dataset-specific KBs. For example, in MultiRC, we use as candidates all the sentences from the paragraph associated with the given question. In QASC, which has a large KB⁴⁴4In large KB-based QA, AIR first uses an off-the-shelf Lucene BM25Robertson et al. (2009) to retrieve a pool of candidate justification sentences from which the evidence chains are constructed. of 17.4 million sentences), similar to Khot et al. (2019a) candidates are retrieved using the Heuristic+IR method which returns 80 candidate sentences for each candidate answer from the provided QASC KB. (b) it adjusts the query to focus on the missing information, i.e., the keywords that are not covered by the current evidence chain. AIR also dynamically adds new terms to the query from the previously retrieved justifications to nudge multi-hop retrieval. These two iterative steps repeat until a parameter-free termination condition is reached.

We first detail the important components of AIR.

Alignment:

To compute the similarity score between a given query and a sentence from KB, AIR uses a vanilla unsupervised alignment method of Yadav et al. (2019a) which uses only GloVe embeddings Pennington et al. (2014).⁵⁵5Alignment based on BERT embeddings marginally outperformed the one based on GloVe embeddings, but BERT embeddings were much more expensive to generate. The alignment method computes the cosine similarity between the word embeddings of each token in the query and each token in the given KB sentence, resulting in a matrix of cosine similarity scores. For each query token, the algorithm select the most similar token in the evidence text using max-pooling. At the end, the element-wise dot product between this max-pooled vector of cosine-similarity scores and the vector containing the IDF values of the query tokens is calculated to produce the overall alignment score $s$ for the given query $Q$ and the supporting paragraph $P_{j}$:

$\displaystyle s(Q,P_{j})$

$\displaystyle=\sum_{i=1}^{|Q|}\mathit{idf}(q_{i})\cdot\mathit{align}(q_{i},P_{j})$

(1)

$\displaystyle\mathit{align}(q_{i},P_{j})$

$\displaystyle=\max_{k=1}^{|P_{j}|}\mathit{cosSim}(q_{i},p_{k})$

(2)

where $q_{i}$ and $p_{k}$ are the $i^{th}$ and $k^{th}$ terms of the query ($Q$) and evidence sentence ($P_{j}$) respectively.

Remainder terms ($Q_{r}$):

Query reformulation in AIR is driven by the remainder terms, which are the set of query terms not yet covered in the justification set of $i$ sentences (retrieved from the first $i$ iterations of the retrieval process):

$\displaystyle Q_{r}(i)$

$\displaystyle=\displaystyle t(Q)-\bigcup_{s_{k}\in S_{i}}t(s_{k})$

(3)

where $t(Q)$ represents the unique set of query terms, $t(s_{k})$ represents the unique terms of the $k^{th}$ justification, and $S_{i}$ represents the set of $i$ justification sentences. Note that we use soft matching of alignment for the inclusion operation: we consider a query term to be included in the set of terms in the justifications if its cosine similarity with a justification term is larger than a similarity threshold $M$ (we use $M$=0.95 for all our experiments - see section 5.2), thus ensuring that the two terms are similar in the embedding space.

Coverage ($Q_{c}$):

measures the coverage of the query keywords by the retrieved chain of justifications $S$:

$\displaystyle Q_{c}(i)$

$\displaystyle=\displaystyle\frac{|\bigcup_{s_{k}\in S_{i}}t(Q)\cap t(s_{k})|}{|t(Q)|}$

(4)

where $|t(Q)|$ denotes the size of unique query terms.

AIR’s justification chains can be fed into any supervised answer classification method. For all experiments in this paper, we used RoBERTa Liu et al. (2019), a state-of-the-art transformer-based method. In particular, for MultiRC, we concatenate the query (composed from question and candidate answer text) with the evidence text, with the [SEP] token between the two texts. A sigmoid is used over the [CLS] representation to train a binary classification task⁶⁶6We used RoBERTa base with maximum sequence length of 512, batch size = 8, learning rate of 1e-5, and 5 number of epochs. RoBERTa-base always returned consistent performance on MultiRC experiments; many runs from RoBERTa-large failed to train (as explained by Wolf et al. (2019)), and generated near random performance. (correct answer or not).

For QASC, we fine-tune RoBERTa as a multiple-choice QA ⁷⁷7We used similar hyperparameters as in the MultiRC experiments, but instead used RoBERTa-large, with maximum sequence length of 128. (MCQA) Wolf et al. (2019) classifier with 8 choices using a softmax layer(similar to Khot et al. (2019a)) instead of the sigmoid. The input text consists of eight queries (from eight candidate answers) and their corresponding eight evidence texts. Unlike the case of MultiRC, it is possible to train a MCQA classifier for QASC because every question has only 1 correct answer. We had also tried the binary classification approach for QASC but it resulted in nearly 5% lower performance for majority of the experiments in table 2.

In QA tasks that rely on large KBs there may exist multiple chains of evidence that support a correct answer. This is particularly relevant in QASC, whose KB contains 17.2M facts.⁸⁸8The dataset creators make a similar observation Khot et al. (2019a). Figure 1 shows an example of this situation. To utilize this type of redundancy in answer classification, we extend AIR to extract parallel evidence chains. That is, to extract $N$ parallel chains, we run AIR $N$ times, ensuring that the first justification sentences in each chain are different (in practice, we start a new chain for each justification in the top $N$ retrieved sentences in the first hop). After retrieving $N$ parallel evidence chains, we take the union of all the individual justification sentences to create the supporting evidence text for that candidate answer.

In addition to previously-reported results, we include in the tables several in-house baselines. For MultiRC, we considered three baselines. The first baseline is where we feed all passage sentences to the RoBERTa classifier (row 11 in table 1). The second baseline uses the alignment method of Kim et al. (2017) to retrieve the top $k$ sentences ($k={2,5}$). Since AIR uses the same alignment approach for retrieving justifications in each iteration, the comparison to this second baseline highlights the gains from our iterative process with query reformulation. The third baseline uses a supervised RoBERTa classifier trained to select the gold justifications for every query (rows 16–21 in table 1). Lastly, we also developed a RoBERTa-based iterative retriever by concatenating the query with the retrieved justification in the previous step. We retrain the RoBERTa iterative retriever in every step, using the new query in each step.

We considered two baselines for QASC. The first baseline does not include any justifications (row 7 in table 2). The second baseline uses the top $k$ sentences retrieved by the alignment method (row (8–12 in table 2).

For evidence selection, we report precision, recall, and F1 scores on MultiRC (similar to Wang et al. (2019b); Yadav et al. (2019b)). For QASC, we report Recall@10, similar to the dataset authors Khot et al. (2019a). We draw several observation from the evidence selection results:

1. (1)

   AIR vs. unsupervised methods - AIR outperforms all the unsupervised baselines and previous works in both MultiRC (row 9-15 vs. row 23 in table 1) and QASC(rows 0-6 vs. row 18). Thus, highlighting strengths of AIR over the standard IR baselines. AIR achieves 5.4% better F1 score compared to the best parametric alignment baseline (row 12 in table 1), which highlights the importance of the iterative approach over the vanilla alignment in AIR. Similarly, rows (4 and 5) of table 2 also highlight this importance in QASC.

2. (2)

   AIR vs. supervised methods - Surprisingly, AIR also outperforms the supervised RoBERTa-retriver in every setting(rows 16–21 in table 1). Note that the performance of this supervised retrieval method drops considerably when trained on passages from a specific domain (row 19 in table 1), which highlights the domain sensitivity of supervised retrieval methods. In contrast, AIR is unsupervised and generalize better as it is not tuned to any specific domain. AIR also achieves better performance than supervised RoBERTa-iterative-retriever (row 21 in table 1) which simply concatenates the retrieved justification to the query after every iteration and further trains to retrieve the next justification. The RoBERTa-iterative-retriever achieves similar performance as that of the simple RoBERTa-retriever (row 16 vs. 21) which suggests that supervised iterative retrievers marginally exploit the information from query expansion. On the other hand, controlled query reformulation of AIR leads to 5.4% improvement as explained in the previous point. All in all, AIR achieves state-of-the-art results for evidence retrieval on both MultiRC (row 23 in table 1) and QASC (row 18 of table 2).

3. (3)

   Soft-matching of AIR - the alignment-based AIR is 10.7% F1 better than AIR that relies on lexical matching (rather than the soft matching) on MultiRC (row 22 vs. 23), which emphasizes the advantage of alignment methods over conventional lexical match approaches.

For overall QA performance, we report the standard performance measures ($F1_{a}$, $F1_{m}$, and $EM0$) in MultiRC Khashabi et al. (2018a), and accuracy for QASC Khot et al. (2019a).

The results in tables 1 and 2 highlight:

1. (1)

   State-of-the-art performance:
   Development set - On both MultiRC and QASC, RoBERTa fine-tuned using the AIR retrieved evidence chains (row 23 in table 1 and row 14 in table 2) outperforms all the previous approaches and the baseline methods. This indicates that the evidence texts retrieved by AIR  not only provide better explanations, but also contribute considerably in achieving the best QA performance.

   Test set - On the official hidden test set, RoBERTa fine-tuned on 5 parallel evidences from AIR achieves new state-of-the-art QA results, outperforming previous state-of-the-art methods by 7.8% accuracy on QASC (row 21 vs. 20), and 10.2% EM0 on MultiRC (row 35 vs. 34).

2. (2)

   Knowledge aggregation - The knowledge aggregation from multiple justification sentences leads to substantial improvements, particularly in QASC (single justification (row 4 and 8) vs. evidence chains (row 5 and row 9) in table table 2). Overall, the chain of evidence text retrieved by AIR enables knowledge aggregation resulting in the improvement of QA performances.

3. (3)

   Gains from parallel evidences - Further, knowledge aggregation from parallel evidence chains lead to another 3.7% EM0 improvement on MultiRC (row 27), and 5.6% on QASC over the single AIR evidence chain (row 18). To our knowledge, these are new state-of-the-art results in both the datasets.

To understand the importance of modeling missing information in query reformulation, we analyzed a simple variant of AIR in which, rather the focusing on missing information, we simply concatenate the complete justification sentence to the query after each hop. To expose semantic drift, we retrieve a specified number of justification sentences. As seen in table 3, now the AIR(lexical)-uncontrolled and AIR-uncontrolled perform worse than both BM25 and the alignment method. This highlights that the focus on missing information during query reformulation is an important deterrent of semantic drift. We repeated the same experiment with the supervised RoBERTa retriever (trained iteratively for 2 steps) and the original parameter-free AIR, which decides its number of hops using the stopping conditions. Again, we observe similar performance drops in both: the RoBERTa retriever drops from 62.3% to 57.6% and AIR drops to 55.4%.

We evaluate the sensitivity of AIR to the 2 hyper parameters: the threshold ($Q_{r}$) for query expansion, and the cosine similarity threshold $M$ in computation of alignment. As shown in table 5, evidence selection performance of AIR drops with the lower values of $M$ but the drops are small, suggesting that AIR is robust to different $M$ values.

Similarly, there is a drop in performance for MultiRC with the increase in the $Q_{r}$ threshold used for query expansion, hinting to the occurrence of semantic drift for higher values of $Q_{r}$ (table 4). This is because the candidate justifications are coming from a relatively small numbers of paragraphs in MultiRC; thus even shorter queries ($=2$ words) can retrieve relevant justifications. On the other hand, the number of candidate justifications in QASC is much higher, which requires longer queries for disambiguation ($>=4$ words).

To verify if the MultiRC training data is sufficient to train a supervised justification retrieval method, we trained justification selection classifiers based on BERT, XLNet, and RoBERTa on increasing proportions of the MultiRC training data (table 6). This analysis indicates that all three classifiers approach their best performance at around 5% of the training data. This indicates that, while these supervised methods converge quickly, they are unlikely to outperform AIR, an unsupervised method, even if more training data were available.