NORMY: Non-Uniform History Modeling for Open Retrieval Conversational Question Answering

The Retriever module retrieves the k most relevant passages from a document collection $D$, given a query $q_{n}$ and context $C$. We considered two types of search algorithms: classic Information Retrieval BM25-style ranking methods, and dense retriever methods. Although dense retriever approaches like ORQA [25] and DPR [26] which use encodings of documents using ALBERT [27], have shown to provide better results, they require training data for fine-tuning. Their vanilla pre-trained models without training do not perform as well as BM25 (shown in Section IV). Further, BM25 is more scalable for large collections. Hence, given that the main focus of this paper is history modeling, we picked BM25 for our Retriever. Specifically, we index the documents using Lucene³³3https://lucene.apache.org/pylucene/. Then we retrieve top k documents using BM25, which is a term frequency based document ranking algorithm.

History Modeling. NORMY’s Retriever has two key novelties. First, we use a keyphrase extraction-based candidate selection algorithm to identify the key context from the whole conversational history. Second, we consider all passages returned by previous turns alongside passages returned by final turn as candidate passages, and rank them using history aware decay scoring method to return top $k$.

Our retrieval algorithm is shown in Algorithm 1. We extend the keyphrase extraction algorithm YAKE [28] to select $y$ best keywords per conversation turn using the YAKE formula shown in Equation (1). YAKE considers features like the casing of the word, word positions, word frequencies, word relatedness to context, etc. to assign a score $S(b)$ to each word $b$.

where $W_{Rel}$ is the relatedness to context score, $W_{Pos}$ is the word position score, $W_{Case}$ is the word casing score, $W_{freq}$ is the word frequency divided by the sum of mean term frequency and standard deviation $\sigma$, and $W_{DifS}$ is calculated based on how many times a particular word appears in other sentences. The detailed equations of all the terms can be found in [28]. We compute the union $R(C)$ of the reformulated questions $R(q_{0})\cdots R(q_{n})$ to retrieve the top $k$ passages (line 4-5). Each passage returned has a BM25 score assigned to it.

History-Aware Decay Scoring. After retrieving $k$ passages for the current turn $n$, we refine their scores by considering their similarity to the retrieved passages. Further, we assign less weight to passages returned from previous turns. Specifically, we use a decay weight $\lambda$ to update the scores of older passages. The passages returned from previous turns may be relevant for subsequent modules but they do not share equal weight to passages returned from current turn $n$. Next, to assess the relevance of each passage $P_{nj}$, $\{j=1\cdots k\}$ from turn $n$, we compute the average pairwise similarity with passages returned in the previous turn $P_{(n-1)i}$, $\{i=1\cdots k\}$. This ensures that the passages returned has relevance to the whole conversation. Passages returned from irrelevant conversation turns will be scored less. To compute the similarity, we use SBERT [29] to produce embeddings of passages and perform cosine similarity. We update the score of $P_{nj}$ using this similarity. Finally, we rank all the passages using updated scores $S_{rt}$ and select top $k$. The retriever score of a passage $p$ is shown in Equation (2).

where BM(.) is the BM25 score of a passage and sim(.) returns semantic similarity between two passages.

The Reranker module reranks the retrieved top $k$ passages using transformer-based encoders and a neural network to compute passage relevance score. The transformer based Reranker augments BM25 ensuring an extra layer of passage relevance. As $k<<$total size of collection, using a transformer encoder is inexpensive. A Reranker has been shown to improve the overall performance of the end-to-end system with little additional cost [10, 30]. However, we show that using the same history modeling as the previous module or using the context from all history turns do not give the best results as now we have grounded documents as evidence. Our experimental results show that using a context with a history window size $w$ works best. The input to the module is the final query $q_{n}$, the context $C$, and $k$ passages retrieved by the Retriever. A reranking score $S_{rr}$ is assigned to every passage.

Encoder. Our Reranker module uses BERT to encode the input representation. We use the last $w$ history turns before $q_{n}$ and concatenate them together to model the history. We then concatenate the retrieved passage $p_{j}$, $j=\{1\cdots k\}$ to the appended history turns to create the final input sequence $(q_{n},C,p_{j})=$ [CLS] $q_{n-w}$ [SEP] $\cdots$ [SEP]$q_{n-1}$ [SEP]$q_{n}$ [SEP]$p_{j}$. We use the contextualized vector representation of the input sequence ${\nu}_{[CLS]}$, and use it as input to a fully connected feed-forward layer that classifies the given passage as either relevant or non-relevant and outputs a classification score $S_{rr}$:

where ${\nu}_{[CLS]}\in\mathbb{R}^{T}$, T is the model embedding dimension, which is 768, and $W_{[CLS]}$ is a projection of the $[CLS]$ representation to obtain the sequence representation ${\nu}_{[CLS]}$. We compute the score for each passage in top $k$ independently and rerank them based on $S_{rr}$.

The Reader module inputs the final query $q_{n}$, the context $C$ and the reranked passages $\{p_{1},p_{2}....p_{k}\}$ and outputs a span from one of the passages as the answer.

History Modeling. As the documents have already been narrowed down using the conversational context in previous modules, we show that using a history modeling with full contextual information produces worse results than a history model that uses less context. This happens due to: 1) The passages already hold the necessary contextual information from the history, 2) Previous history questions in the context misdirects the BERT Reader model into predicting incorrect answer spans. The naive idea would be to use just the final query as input. However, the final query is prone to co-references and ellipses as users will not use self-contained utterances in a natural conversation. Thus, we use a co-reference resolution model to generate a resolved final query ${q_{n}}^{\prime}$ using the previous context. We adapt the huggingface neural co-reference model ⁴⁴4https://huggingface.co/coref/ which uses two neural networks to assign a score to each pair of mentions (or co-references) in the input and their antecedents [31]. The history turns $q_{0}$ to $q_{n-1}$ are concatenated and used to rewrite $q_{n}$ into ${q_{n}}^{\prime}$.

Encoder. Our Reader module uses similar BERT architecture as the previous module to encode the input. The input sequence ”$[CLS]{q_{n}}^{\prime}[SEP]p_{j}$” is used to generate a representation of all tokens in the input. Two sets of parameters, a start vector $W_{s}$ and an end vector $W_{e}$ are used to compute the score for the $m$-th token.

where $S_{s}$ is the start score of a token and $S_{e}$ is the end score. The span Reader score $S_{rd}$ is computed as the maximum score of each token being either the start or end token. The start token must appear before the end token in the input.

where $s$ is the answer span with the start token $[m_{s}]$ and end token $[m_{e}]$. The answer spans are re-ranked using the combined score of all three modules and the top answer is given as a prediction.

We use three datasets with different conversation structures. The first dataset: ORQUAC [10] is an aggregation of three existing datasets: QuAC, CANARD [32], and Wikipedia corpus that serves as a knowledge source for open retrieval. The Wikipedia corpus is a collection of 11 million passages that are created from splitting Wikipedia articles into 384 tokens.

The second dataset is doc2dial-OR, which was created by us, as an extension of doc2dial [22] dataset, which consists of natural information-seeking goal-oriented dialogues that are grounded in documents. doc2dial has more complex questions than ORQUAC, associated with multiple sections of a document. However, this dataset is only grounded to 480 long documents collected from different government websites, which is not ideal for an open retrieval task. doc2dial-OR extends doc2dial by having a much larger set of passages consisting of (a) 11 million Wikipedia passages⁵⁵5https://dumps.wikimedia.org/enwiki/20191020, and (b) the 480 documents of doc2dial split into 384-token chunks. The second difference between doc2dial-OR from doc2dial is that we convert free-text ground truth answers to exact text spans from the gold passage in the dataset, to make it suitable for a span prediction task, as is the case for ORQUAC.

The third dataset is ConvMix [23], which contains documents from heterogeneous sources: Wikipedia info boxes, tables, and text snippets (passages). They use the Wikipedia dump from 2022-01-31. To adapt this dataset for our task, we selected the 5.94 million textual snippets as our collection and the question turns in a conversation where the answer can be extracted from these text snippets. The dataset statistics are shown in Table II.

Competing History Models. To the best of our knowledge, there are no fully unsupervised non-uniform history modeling approaches. There are supervised systems (OrConvQA [10], WS-OrConvQA [11]) that uses history window of 6 for all the modules. ConvADR-QA [12] uses all history turns along with their predicted answers as context. However, they require annotated data to train the modules. We adapt their approaches to an unsupervised setting. There are also conversational closed retrieval systems (RL [14], RW [15]). We adapt such history modeling methods to an open-retrieval setting. Further, we also propose some standard history modeling techniques. Thus, we have identified the following baselines: 1) No History [2]: Where we do not perform any history modeling. The last question turn is used as input to each individual module. 2) First-Last: We propose an intuitive history modeling baseline, where we define the history as the combination of the immediately previous user utterance and the first utterance of the conversation. 3) Full History: With all previous turns concatenated. For the modules where we use transformer models with token limitation, we prune the earlier tokens if the total token size exceeds 384. The input sequence is $C=[CLS]q_{0}[SEP]q_{1}$ $[SEP]\cdots[SEP]q_{n}$ 4) YAKE [28]: Keyphrase extraction-based history modeling has not been used previously in any research work. We extract $y$ keyphrases per history turn and concatenate them with the last turn to create the input sequence. 5) Backtracking [RL]: We adapt the immediate reward based history selection proposed by Qiu et al. [14] for closed retrieval systems. We select a history turn if the similarity with previously selected history turns is greater than 0.5. For each history turn($i=1...n$) we calculate the immediate reward with SBERT sentence encoder. 6) Question Rewriting [RW]: We adapt the algorithm of Vakulenko et al. [15] for closed retrieval systems, which uses a question rewriting model to resolve ambiguous questions (co-references) into self-contained questions. Their model requires training data thus we use neuralcoref to resolve the co-references of the final query using previous history turns. There are other query rewriting models like QReCC [33] which also require annotated data. 7) Fixed Window [OrConvQA, WS-OrConvQA] [10, 11]: WS-OrConvQA is an improvement over OrConvQA but uses training data to learn weak supervision signals. Both models use a history window size $w$. Window size 6 is shown to produce the best results for both. To select the baseline, we also compared different window sizes (2,4,6,8) in a small validation set and $w=6$ has given the best results. 8) Fixed Window with Ans. [ConvADR-QA] [12]: This model predicts the answers for each historical question with a teacher model using annotated query rewrites and appends the predicted answer to the context along with historical questions. For a fully unsupervised pipeline, we adapt this approach to predict an answer for each question using OrConvQA and append the answer to the context. The input sequence is $C=[CLS]q_{0}a_{0}[SEP]q_{1}a_{1}$ $[SEP]\cdots[SEP]q_{n}a_{n}$, where $a_{n}$ is the predicted answer for turn $n$.

Implementation Details. NORMY is fully unsupervised without requiring any training data. The pre-trained models are implemented with the open-source library Huggingface. We index our document collection using PyLucene with StandardAnalyzer as tokenizer Indexing is done with term frequencies, document frequencies, and positions. Keyphrase extraction is done with open source library pke [34]. For computing similarity, we use SBERT’s pre-trained roberta-large model. For Reranker module, we use pre-trained BERT model finetuned on MSMARCO dataset [35]. For our Reader module we use a pre-trained bert-large model finetuned on SQUAD dataset. We make our full code available to the research community.

Evaluation Methodology. We use the commonly used Mean Reciprocal Rank (MRR) and Recall (R@k) methods to measure our retrieval performance. MRR calculates how far down the ranking the first relevant document is on average, where higher is better. R@k measures the fraction of times the correct document is found in the top $k$ predictions, where higher is better.
Results. We first experiment with different values of the number of keywords $y$ in the keyphrase extraction shown in Figure 3(a) and find that $y=5$ performs best. We use $y=5$, $k=10$ and $\lambda=0.1$ in Table III. We found that using a larger $k$ increases the execution time of the system with a very small accuracy benefit. Figure 3 shows our experimental results for different parameters used in NORMY.

We compare different history modeling techniques using BM25 in Table III. For all three datasets, we see that our NORMY Retriever performs significantly better than other baselines. This is due to a couple of factors: 1) We remove irrelevant information from each history turn rather than eliminating the history turn altogether, 2) We consider previously retrieved passages from previous history turns as candidate passages. We can also see that history models that use fewer contexts like Question Rewriting, First-Last, and No History perform significantly worse indicating we need more context while retrieving from millions of passages.

Evaluation Methodology. We use the same metrics as the Retriever as both of these modules produce top $k$ passages.
Results. From Table IV we see that the Reranker module significantly improves the ranks of relevant passages. Using a fixed window, which sits between a broader context like Full History and a narrower context like Rewriting produces the overall best results. We conduct experiments with different history window sizes $w$ and show the results in Figure 3(b). Fixed window of 6 works best for two of our datasets, which is supported by the literature [10]. For ConvMix, we see that YAKE performs slightly better than all other methods and the results vary very little. This is because the passages in the collection are single sentences only, leading to multiple passages being relevant to the question. The Reranker ranks such passages similarly whereas only one contains the gold answer. In the real world, it is unusual for the documents to be single sentences. Note that our hypothesis that the Reranker needs less context than the Retriever still holds, as YAKE has less context than Full History.

Evaluation Methodology. We treat the evaluation of Reader module as a span selection task and adopt token level F1 as the evaluation metric. F1 calculates the similarity between the ground answer and the predicted span, where higher means better.
Results. From Table V we can see significant performance drops when more contexts are added for all three datasets. As the candidate passages have been reduced to 1, transformer-based reader model performs significantly better when only one query is used. Narrower contexts like adding no history and question rewriting perform much better than broader context models further proving our hypothesis. Among them, question rewriting produces a better result. For ConvMix we can see very high F1 scores for appropriate history models as the answers are on average two tokens only, which makes the Reader model easily predict answer spans. However, history models with broader context have poor F1 scores for this same dataset indicating the model needs proper history models even in simpler scenarios.

In this section, we compare our pipeline with the state-of-the-art models [10, 11, 12], which either use a fixed window of 6 or full history with predicted answers uniformly in all modules. We see in Table VI that SOTA models perform poorly in a fully unsupervised setting. We also see that, ConvADR-QA performs worse than OrConvQA, as in an unsupervised setting, a wrongly predicted answer could misdirect the context to retrieve irrelevant passages. Our pipeline with non-uniform history modeling performs significantly better. SOTA models use fine-tuned dense retriever model (DPR) for their retriever module instead of BM25 which is used by NORMY. We use the vanilla pre-trained version of DPR here as we don’t have access to training data. We also compare to a variant of OrConvQA that uses BM25 instead of DPR for completeness. We see that BM25 performs better than dense retriever models. Note that, uniformly using other baseline history modeling techniques evaluated in previous subsections does not produce better results than NORMY in the end-to-end pipeline. We do not show these in Table VI for brevity.

The effectiveness of our model relies on some of the design choices we made. We investigate such choices our novel retriever NORMYRetr has from equation (2). We present the ablation results in Table VI. Specifically, we show two ablation settings as follows:

NORMY w/o decay. We showed that if we disregard previously returned passages we may miss out on some relevant information required for subsequent modules. However, if we gave the same weight to previous passages as passages returned from current turn n, we see a degradation in performance. This is due to the current turn holding the most amount of information. Thus passages returned from the current turn should be given the most weight.

NORMY w/o sim. If a history turn is related to its previous turn, the passages returned will also have some similarities. Here, we disregarded the average pairwise similarity with the previous turn’s returned passages from equation (2). We again see a decrease in model performance. By refining retriever scores with similarity score, we compute the relevance of passages with relevant conversational history. The ablation studies further verify that both decay and similarity scores are crucial for NORMY to perform best.