Explaining Documents’ Relevance to Search Queries

Document summarization is related to the defined task of search result explanation. As we aim to generate coherent and grammatically readable explanations, abstractive models for text summarization (Rush et al., 2015; Liu et al., 2015; Nallapati et al., 2016; Chopra et al., 2016) better suit as finishing components of explanation generation models, compared to extractive summarization models (Rossiello et al., 2017). Neural abstractive summarization started by generating headlines from the first sentence of news articles (Rush et al., 2015) and was then applied to different settings, such as longer text inputs (Liu et al., 2015; Nallapati et al., 2016; Chopra et al., 2016). Although search result explanation is beyond summarizing document contents, even exploiting abstractive summarization techniques to different types of documents in the open-domain Web is not straightforward, as they need domain-specific fine-tuning at the very least to produce reasonable outputs (Chu and Liu, 2019).

Dealing with no readily available labeled data, unsupervised training of sequence transduction tasks has been studied in some recent work. Since language understanding is required to generate fluent sequences (texts), using pre-trained language representation models, such as Word2vec and GloVe embeddings (Mikolov et al., 2013; Pennington et al., 2014), ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), and UniLM (Dong et al., 2019) can reduce the amount of training data required. The pre-trained language representation model Bert is used for ensuring fluency of generated text in the decoding step or representing input documents in the encoding step of text summarization models (Zhang et al., 2019; Liu and Lapata, 2019). Beyond using pre-trained components in conjunction with or as part of a sequence-to-sequence model, there are some pre-trained models for the sequence generation task (Song et al., 2019; Lewis et al., 2019). Unsupervised training of machine translation models as another example of sequence transduction has also been investigated in some studies (Artetxe et al., 2018; Lample and Conneau, 2019; Ren et al., 2019; Artetxe et al., 2019).

Snippet generation has been known a special type of document summarization, in which sentences, or sentence fragments, are selected to be presented in a search engine result page (SERP) (Turpin et al., 2007). It was also called query-biased summarization by Tombros and Sanderson (1998). Snippet generation is an active area of research and has been studied in the context of Web search (Verstak and Acharya, 2012), XML retrieval (Huang et al., 2008), semantic search (Wang et al., 2019a), and more recently dataset search (Wang et al., 2019b). Early Web search engines presented query-independent snippets consisting of the first tokens of the result document. Google was the first Web search engine to provide query-biased summaries (Verstak and Acharya, 2012; Turpin et al., 2007). Bast and Celikik (2014) proposed an efficient solution for extractive snippets by taking advantage of inverted index, a popular data structure used in most information retrieval systems. Recently, Chen et al. (2020) proposed abstractive snippet generation as a potential solution to circumvent copyright issues. The authors demonstrated that despite the popularity of extractive snippets in the current search engines, abstractive summarization is equally powerful in terms of user acceptance and expressiveness.

Although snippet generation, and in general query-focused document summarization, is closely related to GenEx explanation, they are fundamentally different. GenEx explanations are terse, consisting of a few words, while snippet generation models try to select or generate a few sentences or even a paragraph. In addition to the length, the goal of these tasks are different. Query-focused summarization tries to find a set of sentences or passages containing frequent and close occurrences of query tokens, while GenEx aims to describe what such sentences convey about the query, explicitly avoiding the query words themselves. While previous work on snippet generation tries to select or generate a few sentences or even a paragraph, explaining the documents’ relevance to a query in a few words is a challenging task.

How to define and evaluate interpretability and explainability of machine learning models are discussed in several studies (Doshi-Velez and Kim, 2017; Guidotti et al., 2018; Lipton, 2018; Miller, 2019; Ribeiro et al., 2018). But, application of machine learning techniques to different tasks can impose task-specific requirements on definition and evaluation of suitable explanations. Explaining search results has been briefly studied in recent years. Describing relationships between entities in queries is considered an explanation of search results and is generated based on incomplete descriptions of relationships in knowledge graph (Voskarides et al., 2015; Voskarides et al., 2017). As users’ information needs are very diverse, search result explanations requires description of much more relevance factors than relationships between entities, which we aim to extract and describe. Some models explain search results by providing a set of keywords (with their estimated weights) for each document by training another ranker to simulate the scores of a black-box ranker (Singh and Anand, 2018a; Verma and Ganguly, 2019a; Singh and Anand, 2018b, 2019; Roy et al., 2019). These models mainly explain search results following the posthoc explanation method for classifiers - LIME (Ribeiro et al., 2016). Specifically, Singh and Anand (Singh and Anand, 2018b) interpret a base ranker by training a second tree-based learning-to-rank model with an interpretable subset of content-based ranking features, such as frequency and TF-IDF. Training data for the second ranker is generated from the outputs of the base ranker. Sen et al. (2020) also use basic retrieval heuristics (frequency of a term in a document, frequency of a term in a collection, and length of a document) as explanation features. While these interpretable features seem to be useful for system engineers, how to use them to provide explanations for users is unexplored.

Singh and Anand (2019) uses LIME to explain the output of a ranker, which is based on perturbing the instance to be explained. The authors cast the ranking task as a classification task and obtain binary labels of relevant or non-relevant for perturbed documents based on three ways: top-k binary, score-based, and rank-based. These perturbed instances with their binary labels are then fed to the LIME explainability model and visualized as in the original, using bar-charts to show word contributions to the model’s decision.

Fernando et al. (2019) explore a model-introspective explainability method for neural ranking models. They use the DeepSHAP (Lundberg and Lee, 2017) model to generate explanations and defined five different reference to generate explanations: 1) document only containing OOV words, 2) document built by sampling words with low IDF values, 3) document consisting of words with low query-likelihood scores, 4) document sampled from the collection that is not available in the top-1000 ranked list, and 5) document that is sampled from the bottom of the top-1000 documents retrieved. They found that DeepSHAP’s explanations highly depends on a reference input that needs to be further investigated. The authors also compared DeepSHAP’s explanations with those generated by EXS (Singh and Anand, 2019) based on LIME and found that they are significantly different. They note that this difference by the two explanation models is concerning, especially in the absence of gold explanations. Verma and Ganguly (2019b) propose a model-agnostic approach based on a weighted squared loss to explain rankers as well as three sampling approaches to perturb the document in the instance to be explained: uniform, biased, and masked sampling. Explanation features in their work consist of words.

Recently, Singh et al. (2021) propose a local model-agnostic method for explaining learning-to-rank models. They define interpretability features based on IR heuristics and propose two metrics, validity and completeness, to generate explanations. They propose a greedy approach to find a subset of the features such that there is a high correlation between the rankings produced by the selected features and the original black-box model, i.e., high validity. They try to jointly maximize completeness, which is defined as the negative of the correlation between non-explanation features and the original ranking. Correlations between ranked lists are measured using the Kendall’s Tau.

Helping users interpret search results can be different than, and thus not possible through, presenting (partial) information about how search engines work, such as providing (interpretable) ranking features to users. In a recent study, Thomas et al. (Thomas et al., 2019) investigated how users perceive rankings provides by a search engine, with the goal of finding out what forms of explanations may help users. The authors identified six core concepts that used in ranking at Web scale such as relevance and diversity. Participants are asked about why each result is chosen. Although diversity had been identified as important ranking factor before collecting user responses, less than 1% of users found that search results are presented because of diversification to cover different intents and facets of queries. In more details, diversity has the lowest mentions in the collected responses. They mainly performed a set of user studies and online surveys to better understand the mental model of users while using the web search engines. In this work, we propose a model for explaining to what aspect (or facet) of the query, the document is relevant. We believe adding our explanations to SERP could address the issue found by Thomas et al. (2019) about query intents and facets.

Explainable recommendation has recently attracted considerable attention (Zhang and Chen, 2018). For instance, Ai et al. (2019) recently proposed a model based on dynamic relation embedding to produce explanation for recommendation in the context of e-commerce. Content-based models for explanation of recommender systems are closer to explanation of search results than those of collaborative filtering, yet structures in items and user-item interactions are not available in Web search.

Mining query aspects to diversify search results (Santos et al., 2015) is also related to our work. Most approaches in this category are based on query reformulations found in a query log (Radlinski and Dumais, 2006; Capannini et al., 2011) or existing taxonomy or knowledge bases such as Open Directory Project (Agrawal et al., 2009). Other than these sources, there are some work that extract query facets from the search results. For example, Wang et al. (2013) cluster the phrases from top retrieved documents to extract query aspects. Kong and Allan (2013, 2014) use pattern-based semantic class extraction, such as “NP such as NP, NP, …, and NP, to obtain a list of candidate query facets. Their proposed models based on the directed graphical model or clustering then filters out noisy candidates. Later, Kong et al. (2015) extend their model to utilize pre-search context in prediction of query intents. QDMiner (Dou et al., 2016) also extract query facets from top search results by using predefined patterns to obtain candidates from free texts. In another line, Ruotsalo et al. (2018) propose to model query intent by incorporating feedback from users. Although expected explanations to be generated can reveal query aspects, the defined task is different than existing models for mining query aspects as we are interested in explaining the relevant part of a single document’s content to a given query without using other sources of information. In addition, existing models for extraction of query aspects from top search results are based on predefined patterns, thus the aspects are generated in the extractive setting, while our model generates abstractive explanation for a given query-document pair.

Iwata et al. (2012) propose AspecTiles to present the degree of relevance of a document to each query aspect where the query aspects are given. Our focus in this work is not about what is a good way to present the generated explanations to users, but about how to generate high-quality explanations. Having explanations, AspecTiles can be one way to present them to users.

Given a query $\mathbf{q}$ and a document $\mathbf{d}$, the goal is to learn an abstractive explanation model $\mathbf{e}=\mathcal{F}(\mathbf{q},\mathbf{d})$ to generate a text sequence $\mathbf{e}$ that explains how the given document $\mathbf{d}$ is relevant to the query $\mathbf{q}$. The explanation generated by the model may include tokens that do not actually occur in the document. A training instance for this task is thus denoted as the triplet $(\mathbf{q},\mathbf{d},\mathbf{e})$.

We first tokenize queries, documents, and target explanations using subword tokenization following Wu et al. (Wu et al., 2016) and encode tokens with a pre-trained vocabulary embeddings (Devlin et al., 2019).

We then add positional encodings to the token embedding using the sine function following the Transformer model (Vaswani et al., 2017) to capture the relative positions of tokens in input and target sequences given to the encoder and decoder, respectively.

Finally, as the input to the explanation task consists of query and document parts, we add a special token at the end of each part of the input. We also construct a segment embedding in two ways as the input to our explanation models: (1) segment embeddings indicating whether a token belongs to the query or to the document, and (2) embeddings indicating whether a token occurs in the query or not. The choice of segment embedding is based on the model architecture and will be discussed later.

A straightforward solution to handle the two-input characteristic of the explanation task is to concatenate query and document tokens separated using a special token, and then feed the obtained vector to the encoder of a sequence-to-sequence model, such as the Transformer encoder (Vaswani et al., 2017). The input sequence to the model is thus as follows:

where $\sigma$ is a special separation token, similar to the input of several Bert based models for different tasks such passage/document ranking (Nogueira and Cho, 2019) or question answering. We observed, however, that concatenation of short keyword queries with long documents does not lead to sufficient attention weights from document tokens to the query tokens. This is while learning a query-focused representation of documents is crucial for generating explanations.

To learn proper attention from document tokens to query tokens, we first build two input sequences for a given query and document. These inputs are then separately fed to the Transformer-based encoder model with shared weights. The self-attention mechanism in the Transformer leads to contextual representations of query and document tokens. The query and document representations obtained by this encoding component are denoted as $\boldsymbol{z}_{q}$ and $\boldsymbol{z}_{d}$, respectively. This architecture also allows pre-computation of document representations similar to the recent Bert-based rankers (Khattab and Zaharia, 2020; MacAvaney et al., 2020).

To obtain query-focused representations of documents, we propose another encoder model on top of the learned contextual representations of document tokens. The architecture is shown in Figure 2. This second encoder model is based on the Transformer encoder where self-attention layers are replaced with masked-document query attention layers. This layer consists of three parameter matrices $W_{Q}$, $W_{K}$, and $W_{V}$. The query attention layer computes the following attention matrices based on inputs different from the self-attention mechanism:

where $\|$ donates concatenation of representations build from previous encoder layers. The representation of each document token is then updated using the scaled dot-product attention function of the Transformer as below:

where $k$ is the dimension of the keys $K$. However, we mask the scaled dot product in a way to prevent a document token from attending to other document tokens. Therefore, the output representation of a document token is computed as a weighted sum of the attention values corresponding to itself and query tokens. Lastly, representations are obtained by concatenation of outputs from multiple attention functions, each projecting its inputs to a different subspace. The input and output dimensionality of query attention layers are the same and a stack of $N$ query attention layers is used in the second encoder component. The representations of document tokens from the second encoder component constitute the input of the decoder.

Documents on top of a search engine result page often contain query tokens with high frequencies (Manning et al., 2008). In addition, our encoder consists of masked-document query attention layers, which highlight the query related part of the document. Therefore, it is likely that the decoder generates query tokens as the explanation of a document’s relevance. However, this is not desired; the model should generate explanations that provide more information other than the query itself, otherwise the generated explanation is useless. Therefore, we generate explanations by extending the Transformers Decoder architecture (Vaswani et al., 2017) using a query masking mechanism to reduce the probability of generating query tokens by the decoder. To achieve this, we use a masked multi-headed attention for the encoder-decoder attention layer in which the query tokens in both query and document are masked. This makes every decoder position to attend over only non-query tokens in the input sequence.

Note that query tokens are not replaced with a special mask token, because these tokens are central in building the representation of documents. Instead, the representations of query tokens are masked during the decoding process to allow the model to describe the query-related information that the document provides.

We constructed Wiki training data using Wikipedia articles. In this dataset, each section of a Wikipedia article is treated as a document. The section documents extracted from a Wikipedia article are all relevant to the query built from the article’s title, with their section headers as their explanation labels. For the work discussed here, we use a March 2019 dump of English Wikipedia. Only Wikipedia articles are used for building the dataset, and Wikipedia pages such as disambiguation and redirect pages as well as administration pages such as talk, user, and maintenance pages are removed. Articles with URL links as their titles are also removed. We extract all text from each Wikipedia article, filtering out unwanted data such as HTML markup, using WikiExtractor³³3https://github.com/attardi/wikiextractor and then discard any article that has fewer than 500 characters of text. A section is not included unless it contains at least 20 tokens. Sections with highly-frequent headers such as “references” and “see also” are removed. The obtained samples are randomly divided into train, validation, and test samples referred to as the Wiki test set. Statistics of Wiki dataset are reported in Table 2. The reported length values are based on tokens obtained by using sub-word tokenization. Explanations having a token not occurred in the query or document are counted as abstractive explanations. Almost 98% of explanations have a token not occurred in the input sequence.

We also built Anchor training data to improve the generalizability of learned explanation models. Ideally, we could have used Web search logs for mining different facets of a head query, but they are not publicly available due to concerns about user privacy. However, it has been shown that anchor text can effectively simulate real user logs for query reformulation techniques (Dang and Croft, 2010). Based on that work, we approximate query facets using widely available anchor text data, where different anchor texts linking to a particular page approximate query facets relevant to that page. For this purpose, we follow the subtopic clarification by keyword observation through user log analysis (Hu et al., 2012). We used the external version of the anchor text for ClueWeb09 dataset⁴⁴4http://lemurproject.org/clueweb09/anchortext-querylog/ to build training data for the explanation task. All anchor texts to one page that start with the same prefix words are grouped together. The common prefix is then considered as a query, and each suffix is considered as a facet of the query. The linked page is considered to be a relevant document to the query. We performed a number of post-processing steps on the extracted query facets. For examples, facets containing words such as “homepage” or “website” are disregarded. Stopwords such as “of” or “and” are removed from the beginning of query facets.

Clue-Res test data is generated from the ClueWeb09 category B dataset⁵⁵5http://lemurproject.org/clueweb09/ which is a standard TREC collection and has been used in the TREC Web track for several years (Collins-Thompson et al., 2014, 2015). Topics from TREC Web track 2009 to 2012 are used to build test samples. We use the title of a TREC topic as the query, and subtopics of relevant documents in judgments as the set of explanations. Some topics have multiple subtopics. Navigational subtopics as well as the first subtopic of queries are discarded since first subtopics mainly provide general description of queries and are not focused around a query facet. Subtopic descriptions are manually rephrased by removing phrases such as “I’d like to find information about”. This test set contains 543 samples.

We built Passex test data using the passage ranking dataset of TREC 2019’s Deep Learning Track⁶⁶6https://microsoft.github.io/TREC-2019-Deep-Learning/. The TREC passage ranking dataset is built based on the MS MARCO dataset where passages retrieved with respect to the test questions are judged for relevance in much more details. Questions having more than 30 relevant passages in the dataset are chosen for building our evaluation samples. Questions are then manually rephrased as short keyword queries and explanations. Some questions whose conversion to query and explanation were not straightforward are ignored in this step. Also relevant passages with less than 100 tokens are removed. In the end, there are 27 unique queries and 188 evaluation samples in the built Passex dataset.

Note that target outputs in the built test samples are not precise explanations. These test sets are constructed to have approximate out-of-domain samples to guide the development of a generalized explanation model. In the current study, we focus on explaining why a document is relevant to a query by providing the query aspect that the document covers as supportive evidence. Explanation of non-relevant documents to a query is left for future work.

Pre-processing steps. Document texts, queries, and ground truth explanations are lowercased, and encoded using sub-word tokenization following Wu et al. (2016) with Bert vocabulary⁷⁷7https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip of size 30,522. We consider only documents that have more than 20 tokens, and truncate long documents. Documents of the Wikipedia dataset are truncated if required by keeping the first $L$ tokens. We do not truncate the output sequence, but we ignore training samples whose explanations (sub-headings) are longer than 15 tokens. Documents chosen from the ClueWeb dataset for building anchor dataset are extractively summarized as below.

Extractive summarization of long documents. Documents are segmented into sentences using the NLTK toolkit⁸⁸8https://www.nltk.org/. Sentences with exact occurrences of query terms where document and query terms are stemmed using the Porter stemmer, are chosen to be in the query-biased extractive summary of the document. To capture all query-related information of the document, we also compare the Word2vec pre-trained embeddings of all pairs of query-document terms. A sentence whose most similar term to a query term has a score higher than a defined threshold of 0.8 is retained for the summarized version of the document. If the length of selected sentences is less than $L$, then additional sentences with the highest contextual similarity to the query are added in the order of their occurrence until no other sentences can be added to the summary while keeping the length lower than or equal to the length cap $L$.

Following the text summarization and machine translation communities, we use BLEU (Papineni et al., 2002) and different variations of the ROUGE metric (Lin, 2004) such as ROUGE-1 (unigrams), ROUGE-2 (bigrams), and ROUGE-L (longest-common substring)⁹⁹9https://github.com/google-research/google-research/tree/master/rouge for evaluation of generated explanations: the more words and n-grams that are in common between the predicted output and the target subtopics, the more likely the explanation is to be good. We also use BERTScore (Zhang* et al., 2020) to semantically compare generated explanations with reference explanations. BERTScore uses pre-trained contextual embeddings from BERT or its variants to compute cosine similarity between tokens in candidate and reference text segments. Zhang et al. (Zhang* et al., 2020) showed that BERTScore better correlates with human judgements and thus is a stronger metric for comparing text generation models. Two-tailed paired $t$-test is used to test whether the differences between performance of models are statistically significant. Content-based explanation of a single document with respect to a query does not produce any rankings, therefore metrics for evaluation of rankings such as Mean Average Precision are not suitable to measure the quality of explanations.

To the best of our knowledge, this is the first study to explain search results from a black-box ranker by generating abstractive and concise explanations beyond term matching. The unique characteristics of the defined problem make models for related task not suitable where the differences are discussed in Section 2. The first baseline is to use TextRank algorithm (Mihalcea and Tarau, 2004) to extract document keywords. The TextRank algorithm represents a document as a graph of terms linked by co-occurrence relation. Two terms are connected if they co-occur within a window of fixed size, set to 10 terms in our experiments. Then, the PageRank algorithm is applied to the graph to rank terms. The top ranked terms of a document are identified as its keywords. For the defined explanation problem, we are interested in describing a document with respect to a query. To accommodate this setting, we also tried the TextRank algorithm by using topic-sensitive PageRank (Haveliwala, 2002) to rank graph vertices. We refer to this modified version as topic-sensitive TextRank. We also compare with KeyBERT (Grootendorst, 2020) which uses BERT to extract keyphrases from a document. This baseline demonstrates the necessity of attending to queries for explanation of document relevance, therefore models for keyword or keyphrase extraction from documents cannot fulfill the task of relevance explanation with noun phrases. Due to the lack of training data for query-focused keyword extraction, we chose the unsupervised and widely used TextRank model as well as recent BERT-base model KeyBERT.

The next group of baselines is based on using local model-agnostic explanation methods to describe the relevance of a document with respect to a query. For this group, we use LIME (Ribeiro et al., 2016) and Sensitivity (Zeiler and Fergus, 2014) that explain the prediction of a black-box classifier for a given input sample. For the purpose of our task, explanation features are defined as document tokens (Ribeiro et al., 2016; Singh and Anand, 2019). Specifically, the LIME method generates a number of perturbed samples of the input and learns a linear model, based on the classifier’s predictions for perturbed samples. We trained a linear SVM model as the explanation model in our experiments. Features with large coefficients in the learned linear model are considered as explanation. The Sensitivity method estimates the score of a token by measuring the change in the predicted relevance probabilities of perturbed samples that do not include that token (Zeiler and Fergus, 2014). The higher the contribution of a token in the relevance probability is, the more important the token is in understanding the relevance of the document with respect to the query.

We used two different types of rankers as the black-box model whose predictions are to be explained by LIME and sensitivity. These rankers are used to get the relevance probabilities of perturbed documents with respect to the query. The first ranker for this purpose is the classic TF.IDF ranker. For this ranker, documents are represented as bag of words and perturbed samples are obtained by randomly changing one dimension of document vectors to zero. We also used the BERT-based ranker (Nogueira and Cho, 2019) since it has been shown to achieve the current state-of-the-art performance (Craswell et al., 2020; Gao et al., 2020). In this ranker, document tokens that are semantically similar to query tokens can impact the relevance probability of a document, in contrast to the TF.IDF ranker which is only based on exact term matching between a query and document.

The input of the BERT-based ranker is built by concatenating the query and document with a separate token. The pre-trained BERT-based model is fine-tuned for the ranking task using the training data of the MS MARCO passage-ranking dataset (Nogueira and Cho, 2019). As the BERT-based ranker uses sub-word tokens, the term-level score is obtained by summing the score of each token’s subword. For BERT-based ranker, a given input sample which is a query-document pair in the setting of our task, is perturbed by masking a token from a random position in the document. The final score of each document token is calculated by summing the scores obtained for all occurrences of the token in the document. Document tokens are sorted by the obtained scores and the top-ranked tokens are considered as the explanation of the document. The top-ranked tokens are considered to be tokens whose scores are not less than 10% of the token with the highest score. We consider the top three tokens if more than 3 are selected by the thresholding function. The cut-off value is chosen based on the average length of gold explanations in the test sets. Note that the average length of gold explanations is not used by the GenEx model.

The last baseline is training the original Transformer model where input sequences are constructed by concatenating query and document tokens in the training dataset as Eq. (1). The model input is constructed by adding segment embeddings differentiating query and document segments in addition to the separator token $\sigma$ between them. These segment embeddings have the value of 0 for query tokens and 1 for all document tokens. We refer to this model as Orig-Trans.

Ablation study. There are two main architectural choices in the proposed explanation model; query attention layer, and query-masking during the decoding. To show the effectiveness of each choice, we compare our final model with the following variants, gradually constructed on top of the Orig-Trans model. The first variant uses separate encoders for queries and documents with an additional encoder for documents which consists of query attention layers. However, query tokens are not masked in the decoder. We refer to this model as Sep-q-doc. We also test another variant by segment embeddings that differentiate query tokens from other tokens in the input sequence, having 0 for query tokens in the input sequence and a value of 1 for other tokens. Therefore, query tokens in a document are also masked during the decoding process, and only the final hidden vectors corresponding to non-query tokens in a document are fed into the decoder. This is similar to the decoder input of the GenEx model. This variant is referred to as Seg-q-toks model.

All baseline based on the Transformer architecture and GenEx are all built and trained using the same settings of hyperparameters and on the same training data to ensure a fair comparison between them. Hyperparameters are mainly set following the base Transformer model. Each encoder/decoder module consists of a stack of 6 identical layers. We used 8 attention heads in each layer, each with depth of 96. The input dimension is 768. Deep contextualized representations of queries and documents can be obtained by fine-tuning BERT. However, our pilot experiments showed that we can obtain reasonable contextual representation using 6-layer Transformer encoder, mainly because these representations will be updated by the next encoder component. Therefore, considering computational constraint and having large amount of weakly labeled training data, we decided to use Transformer encoder with pre-trained input embeddings from BERT, instead of using the entire pre-trained BERT model. Training samples are batched by their total length. Each batch consists of samples with approximately 2048 tokens in total. The input sequence to encoder modules are truncated if required by keeping the first 256 tokens. The models are trained for 10 epochs. We used the Adam optimizer (Kingma and Ba, 2014) with $\beta_{1}=0.9$, $\beta_{2}=0.98$, $\epsilon=10^{-9}$, and a fixed learning rate of 0.00001. Dropout rate of 0.1 is used during the training of models. In addition, target outputs are smoothed with a value of 0.1. Explanations for test samples are generated by greedy decoding.

In this section, we provide the evaluation results of the proposed model. Performance of GenEx and baseline models on Wikipedia test fold (Wiki), Clue-Res, and Passex datasets in terms of ROUGE and BLEU metrics is shown in Table 3. As baseline models do not consider the order of words that are selected as explanation, we do not compare them with GenEx based on the performance metrics that depend on higher order n-grams. The difficulty of the task at hand, the limitations of the noisy automatically-generated training data, and imprecise labels of test data are the main reasons for low performance values. GenEx outperforms all baselines over all three test sets where the improvements are mostly substantial and statistically significant. The BLEU improvements of GenEx over the best performing baseline are 168%, 18%, and 296% over Wiki, Clue-Res, and Passex datasets respectively. In terms of the BLEU metric, the best performing baseline over Wiki and Passex is TextRank, and KeyBERT shows the highest performance over the Clue-Res test set. In terms of the ROUGE metric, the best performing baselines over all datasets is TextRank. The ROUGE improvements of GenEx over TextRank are 151%, 41%, and 34%.

The first group of baselines is keyword extraction by TextRank and its topic-sensitive variant. TextRank provides a query-independent explanation of a document, while topic-sensitive TextRank is developed to explain documents with respect to a given query. As a small part of a document may be relevant to the query (TREC, 2000) and in such cases we are not interested in the document keywords related to its general topic, the topic-sensitive variant of the TextRank algorithm should intuitively select document words that are more suitable for explanation of document relevance. The reported results in Table 3 show that these models are among the best performing baselines. However, the TextRank algorithms shows higher performance than its topic-sensitive variant based on BLEU and ROUGE metrics. Further investigation of TextRank and its topic-sensitive variant, we observed that topic-sensitive TextRank provides superior keywords when all query tokens occur in the document to be explained. When the document contains only one or a subset of query tokens, topic-sensitive PageRank leads to lower weights for document tokens related to non-present query tokens in the document compared to those by the original PageRank algorithm, and this hurts the quality of explanations for such cases by topic-sensitive TextRank.

LIME and Sensitivity methods underperform the GenEx model where the performance difference is considerable and statistically significant. These models extract document tokens that highly impact the relevance probability of a document with respect to a query. Consequently, tokens that are exact match or semantically similar to query tokens, get the highest importance scores. While this type of explanation show document relevance in terms of keyword matching, they mostly fail to provide words related to query aspects in their top keywords. These models thus provide different but complementary explanation than the GenEx model. LIME explanation of the TF.IDF ranker achieves better performance than that of the BERT-based ranker over the Wiki and Clue-Res test sets. The TF.IDF and BERT-based rankers are used to get the relevance probability of perturbed documents. The lower performance of LIME for the BERT-based ranker could be related to the complex structure of BERT, where the widely used explanation based on linear approximation is not accurate enough to predicate its function.

GenEx significantly outperforms KeyBERT that shows explanations of interest are beyond finding important noun phrases of a text segment.

Tables 4 and 5 report the performance of the ablations of the GenEx model. As the results show, all model variants almost always outperform the original transformer model. The obtained improvements are also mostly statistically significant, with exceptional cases of the Bleu-2/R-2 performance. This observation demonstrates the effectiveness of adding query-attention layers and masked-query decoding.

Seg-q-toks and Sep-q-doc models show similar performance on test sets, however our analysis of their generated explanations reveals that the two model perform well on almost disjoint sets of query-document pairs. Comparing the generated explanations by these two models, the main difference seems to arise from query ambiguity. The Seg-q-toks model generates better explanations for ambiguous queries than the Sep-q-doc model. One example of ambiguous queries in the Clue-Res dataset is query 73 of TREC Web track 2010, “the sun”, with documents about the star in solar system, the U.K. newspaper, and the Baltimore Sun newspaper. In case of ambiguous queries, we believe that it would be helpful to encode queries using document tokens as their contexts. That is likely the main reason that the Seg-q-toks model outperforms the Sep-q-doc model for ambiguous queries.

To semantically compare the generated explanations with respect to gold explanations, we use the BERTScore metric. This evaluation is necessary for abstractive text generation as BLEU and ROUGE metrics only consider the exact matching between generated and gold text segments, while a concept can be expressed in different ways. The default setting of BERTScore, “roberta-large_L17_no-idf_version = 0.3.2 (hug_trans = 2.8.0)-rescaled”, is used in our evaluations. The results are shown in Table 6. GenEx outperforms the original Transformer by 9.9%, 10.7%, and 23.2% over Wiki, Clue-Res, and Passex datasets, respectively, and all improvements are statistically significant. These results demonstrate the higher quality of generated explanations by GenEx compared to those by the original Transformer. As both models are trained on the same data, these improvements also demonstrate the suitability of GenEx architecture for the task in hand.

The semantic evaluation results over Clue-Res and Passex datasets are almost consistent with those of Rouge and BLEU metrics; GenEx greatly outperforms the original transformer and its variants, and is not the best performing model variant on the Wiki test set. Although GenEx is not the best performing variant on the Wiki test set, it still outperforms the original Transformer and the improvements are statistically significant. The difference between model variants could be related to the special structure of Wikipedia articles that is not the case for all pages in the Web. As the goal of our study is to explain a relevant document to a query, a model that is not dependent on the structure of document’s content is more desirable. Thus, GenEx is designed and trained to perform well on general documents in Clue-Res and Passex without being trained on them.

Explanations of keyword-based baselines are not evaluated using this metric as the BERT representations of input sequences in BERTScore are sensitive to input grammaticality and the order of words.

Table 7 shows two examples of explanations generated by the GenEx model. Document contents are shown in Table 8. In addition to the generated explanations, the table shows the document titles and the top 3 keywords of documents obtained by the TextRank and LIME+TF.IDF models. Document titles show the general topic of the documents, which are not necessarily good explanations for queries that seek to find these documents. The GenEx model generates a reasonable explanation for the first example in the table. As this example shows query-focused explanation of the retrieved document is different than the document title which can be considered as general explanation of the document. The generated explanation for example 1 is more useful than the document title to reveal the relevancy of the document to the query. Note that document titles are not used as input of explanation generation models. Although it may be helpful to have document titles for contextual representation of documents, we do not want the model to rely on a feature that may not be available for some webpages and reduces the applicability of the model in practice.

Second example in Table 7 shows a failure of the GenEx model, where the generated explanation is part of the expansion of the given abbreviated query. A reasonable explanation for the second document with respect to the given query can be “symptoms”. Desired explanations to be generated by a model should not repeat query terms as long as explanation readability is not sacrificed. This is the reason that masked query decoding is proposed in our GenEx model. However, expansion tokens of abbreviated queries in documents have high similarity values with query tokens, and they are not masked during the decoding since query masking is done based on exact matching of tokens. This input characteristic makes it highly probable that the decoder generates expansion tokens of abbreviations as explanations. Abbreviated queries thus constitute one category of failure cases of our GenEx model.

For each query-document pair, workers are asked to answer whether: (1) GenEx explanation is better, (2) Explanation by a baseline (LIME-TF.IDF or TextRank) is better, (3) both are equally good, or (4) both are equally bad.

TextRank is mostly the best performing baseline in terms of BLEU and ROUGE metrics according to the results in Table 3. LIME is a successful and widely-used model for explanation of black-box models, which is also the state-of-the-art model in the explanation of black-box rankers (Singh and Anand, 2019). LIME has shown a better performance in explaining the TF.IDF ranker in our experiments with results reported in Table 3. LIME does not use any information about the structure of the TF.IDF ranker, it only uses its outputs for the perturbed instances of the document to be explained. This baseline is chosen as GenEx explains why a document is relevant to a query without having any knowledge of the underlying ranker model.

Table 9 summarizes the obtained results which demonstrate that the majority of workers preferred explanations generated by GenEx over those by LIME-TF.IDF and TextRank in 18%-53% more samples, respectively. In addition, the ratios of individual workers who preferred GenEx explanations over LIME-TF.IDF and TextRank are 2.3 and 3.3, respectively. Results of human evaluation show the higher quality of explanations generated by GenEx compared to baseline models.

We also conducted a user study to capture human evaluation of quality of the GenEx explanations. Three different dimensions are considered for this evaluation: linguistic quality, relevance of explanation to the query topic, and relevance of explanation to the document content. The last two questions provide a proxy for the utility of generated explanations’ ability to connect the query and document. We use the 5-point Likert Scale to evaluate the subjective quality of generated explanations, as Very poor, Poor, Acceptable, Good, and Very good, with assigned scores from 1 to 5, respectively. Because differences between the levels can be subtle, we calculated Fleiss’ Kappa agreement by collapsing negative scores (1 and 2) to no and the others to yes. Table 10 summarizes the obtained results.

Linguistic quality. We evaluated the grammaticality and coherence of the explanations generated by the GenEx model. As shown in Table 10, we observed that the average of grammaticality scores is 4.06. Only 9% of all answers to the grammaticality questions were Poor or Very poor. Part of this strong result can be due to the short length of explanations, as all generated explanations are terse (up to four terms). Yet as explanations are generated in an abstractive way, this result strongly indicates the potential of our GenEx model in generation relevance explanations.

Relevance to query and document pairs. We asked workers if the generated explanations were relevant to the query topic and document content. Note that, following annotation guidelines, explanations that do not provide additional information compared to the query, such as explanations that are a subsequence of queries, should be rated as very poor for both questions. An explanation that is relevant to the topic of the query without repeating the query, provides additional useful information with respect to the query that can help users in understanding the information space. When an explanation is relevant to the document, it means that the explanation is describing the content of the document, and the model is not generating a general or a high-frequent phrase in the training data, which is a common issue of text generation models.

The obtained results in Table 10 show that on average, workers rated GenEx explanations have acceptable degree of relevance to both query topic without repeating the query and document content. These results demonstrate that generated explanations can reasonably describe document content with respect to the query topic and provide more information than the given query.

We chose not to run this part of the study for LIME explanations, since by their nature, they are unlikely to be grammatical. On the other hand, extractive explanations by LIME are likely to be relevant to the document, and do not require the same evaluations as abstractive explanations by GenEx. Note that abstractive generation of texts at the level of noun-phrase has a lower risk of topic drift compared to abstractive snippet generation which has been motivated recently (Chen et al., 2020), and is confirmed by the results of human evaluation on relevance to document content.