PReGAN: Answer Oriented Passage Ranking with Weakly Supervised GAN

The general architecture of the PReGAN model is illustrated in Figure 2. It is a weakly supervised generative adversarial neural network containing three parts: a rank discriminator (for relevance), an answer discriminator (for answerability), and an answer-oriented rank generator.

The rank discriminator distinguishes the generated rank distributions from the ground-truth rank distribution. The answer discriminator tells if a generated sample contains the answer to the given question. The rank generator reads the passages under the guidance of the two discriminators and learns to generate the answer-oriented distribution on thoses passages for the given question. The discriminators therefore guide the reading-based generator with a REINFORCE process from two different perspectives.

At the end of the training, it is expected that passages that are relevant to the question and contain the answer text can be ranked highly. In other words, optimizing the three objectives modeled in the GAN framework will help discard noise and promote good passages that can truly answer the question. The utilization of the GAN framework as an effective denoising means (Tran et al., 2020) is not new, but we show for the first time that it is also particularly adapted to the noisy training environment in OpenQA.

Taking the answer-oriented ranking as a distribution over passages for a given question, the objective of the generator is to learn this distribution through the minimax game. The rank generator would try to select passages that the answer can be reasoned (by the reader) for the given question. The rank discriminator would try to draw a clear distinction between the ground-truth answer passages and the selected ones made by the generator. Since the positive training passages are noisy, the rank discriminator alone may be misleading for the machine reading-based generator. However, if the answer can be extracted from the passage and the answer discriminator confirms the correctness of the answer, the passage selected by the generator should be a good one. This principle can be formulated as the following overall objective function of the minimax game:

where the generative ranking model with parameter $\theta$ is written as $p_{\theta}(d|q_{n},a)$, and the rank discriminator $D_{\phi}^{r}$ parameterized by $\phi$ estimates the probability that a passage $d$ can answer the question $q$. The answer discriminator $D_{\xi}^{a}$ with parameter $\xi$ works as a regularizer of the generator emphasizing passages in which the answer text appears, no matter it can answer the question or not. The last item in the objective is another commonly used regularizer (Lin et al., 2018; Wang et al., 2018) pushing the overall ranking score distribution produced by the generator towards the ground-truth distribution by minimizing KL-divergence between them.

As shown in Figure 2, the rank discriminator takes as input the passages generated (selected) by the current optimal generator $p_{\theta^{*}}$ together with the observed passages with answers, and predicts the source of the passages. Denoting $D_{\phi}^{r}(d|q_{n})=\sigma(f_{\phi}(d,q_{n}))$, which is the sigmoid function of the rank discriminator score $f_{\phi}$, the loss function of rank discriminator is then represented as:

where the score function $f_{\phi}$ is implemented as a bi-encoder attention network between passages and questions, and is solved by stochastic gradient descent algorithm.

The answer discriminator is designed to cooperate with the generator to mitigate the effects of noise in the training data. The answer discriminator only tells if the answer text appears in a passage for the given question instead of trying to reason out a result. The generator, on the other hand, makes decisions based on a machine reading comprehension process and cares less about the existence of the answer texts in the passages. The idea is that the answer discriminator and generator make judgements from different perspectives, then cross check the judgements and guide the training of the generator towards passages on which they have mutual agreement. The training of the answer discriminator, however, is independent from the generator, and can be trained directly using passage and answer pairs for each question with a binary cross entropy loss as follows:

where, similar to the rank discriminator, the score function $f_{\xi}$ is also implemented as a bi-encoder attention network. Passages for question $q_{n}$ are divided into $A^{+}$ and $A^{-}$ depending on whether a passage contains the answer text or not.

The generator $p_{\theta}$ aims to minimize the objective, through which it tries to fool the rank discriminator by generating samples faking the ground-truth answer distribution. By keeping the two discriminators $f_{\phi^{*}}$ and $f_{\xi^{*}}$ fixed after their maximization, the generator’s loss function is as follows:

where minimizing the first item means to guide the generator $p_{\theta}$ toward generating passages which the rank discriminator considers as likely ground-truth passages. Minimizing the second item will drive the generator away from generating passages that the answer discriminator dislikes. The third item imposes a regularization on the overall answer distribution produced by the generator, forcing it to stay close to the ground-truth answer distribution. $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters controlling the impacts of the answer discriminator and the distribution regularizer on the generator.

The generator is trained under weak supervision using the noisy training data. It predicts the ranking score of each passage using a simple machine reading comprehension (MRC) component. Specifically, the MRC component takes a question and the passage candidates as input, and produces a joint distribution of the start position and end position of the answer in a passage. Then it takes the maximum of the joint probability as the predicted ranking score of the passage with respect to the question. The ranking scores are then normalized into a probabilistic distribution indicating how likely each passage can answer the given question. Similar to $f_{\phi}$ and $f_{\xi}$, the ranking scores are also generated by scoring networks. The details of the scoring networks are given in the next section. Our scoring networks are inspired by (Lin et al., 2018). We opt for it because of its high efficiency, which is required for our ranking. Other scoring schemes, such as transformer-based approaches, could also be adopted in our framework in the future.

From the loss function of the generator, its gradient can be derived as follows (We include the details about the derivation of Equation (4) in the Appendix A.1 for interested readers):

where $p_{true}=1/|d^{+}|$ ($|d^{+}|$ is the total number of passages containing correct answers) if the passage contains a correct answer, otherwise 0. This completes the approximation approach for generator optimization with policy gradient based reinforcement learning. The term $\log(1+\exp(f_{\phi}(q_{n},a)))+\lambda_{1}\log(1+\exp(f_{\xi}(q_{n},a)))$ acts as the balanced rewards from rank discriminator and answer discriminator for the generator $p_{\theta}(d|q_{n},a)$ to select passage $d$ for a given question $q_{n}$.

In this section, we will introduce the scoring functions we used in details. As can be seen from the Figure 2 and the Equation (1), the three scoring functions lie in the kernel of our proposed GAN framework.

Formally, given a question $q=(q^{1},q^{2},\cdots,q^{|q|})$ and $m$ passages returned by some retrieval component, which are defined as $D=\{d_{1},d_{2},\cdots,d_{m}\}$, where $d_{i}=(d_{i}^{1},d_{i}^{2},\cdots,d_{i}^{|d_{i}|})$ is the $i$-th retrieved passage composed of a sequence of words, the answer-oriented passage ranking aims to assign a score $P(d_{i}|q)$ to passage $d_{i}$ measuring the probability that it can answer the question $q$.

For $\mathbf{f_{\phi}}$ in rank discriminator $D_{r}$, we use a bi-encoder framework to build the scoring function $f_{\phi}(d,q_{n})$. Each passage $d_{i}$ and the question $q$ are encoded with an RNN network:

where $\hat{d}_{i}^{j}$ and $\hat{q}^{j}$ are the hidden representations of words in passage $d_{i}$ and question $q$ encoding its context information through RNN. We use a single-layer bidirectional LSTM (biLSTM) as our RNN unit. Following (Lin et al., 2018), the final representation of the question is obtained through a self-attention operation $\hat{q}=\sum_{j=1}^{|q|}\alpha^{j}\hat{q}^{j}$, where $\alpha^{j}=\text{softmax}(wq^{j})$ is the importance of each word in the question and $w$ is a learned weight vector. Then the score of each passage is obtained via a max-pooling layer and a softmax layer:

where $W$ is a weight matrix to be learned.

For $\mathbf{f_{\xi}}$ in answer discriminator $D_{a}$, even though it serves a different purpose with a different objective function from the rank discriminator $D_{r}$, the network framework is similar, except that the last layer is a sigmoid function instead of softmax function.

Finally, for $\mathbf{f_{\theta}}$ in rank generator $G$, similar to $\mathbf{f_{\phi}}$, we encode the passage $d_{i}$ in a sequence of hidden vectors, and apply a self-attention layer to attend to the question vectors to get a question embedding. Different from the discriminators, we ask the generator to score a passage through predicting the probability of the start and end positions of an answer span in the passage. The probability of a position $j$ as the start position of the answer span in the passage $d_{i}$ is calculated by $p_{s}^{j}=\text{softmax}(\hat{d}_{i}^{j}W_{s}\hat{q})$ with parameter $W_{s}$, and $p_{e}^{j}=\text{softmax}(\hat{d}_{i}^{j}W_{e}\hat{q})$ for the end position $p_{e}^{j}$ using $W_{e}$. The score of passage $d_{i}$ is then obtained as the maximum of the joint probability of the start and end positions:

Our scoring functions are inspired by (Lin et al., 2018). We opt for it because of its high efficiency, which is required for our ranking. Other scoring schemes, such as transformer-based approaches, could also be adopted in our framework in the future.

With all the components in place, the overall ranking algorithm is summarized in Algorithm 1. Before the adversarial training, we initialize the three component with random parameter weights. The answer discriminator, rank discriminator, and generator are pre-trained using the noisy training data.

From the ranked passages, the top-K (K=50 in our experiments) are passed to the subsequent reader to determine the answer span.

We use a BERT-based reader architecture in this paper, which follows (Karpukhin et al., 2020).¹¹1https://github.com/facebookresearch/DPR Let $d_{i}\in\mathbb{R}^{L\times h}$ ($i\leq i\leq k$) be a BERT representation of the $i$-th passage, where $L$ is the maximum length of the passage and $h$ is the hidden dimension.²²2We assume that the reader is familiar with the BERT training process, or could find more details in (Karpukhin et al., 2020). The probability of an answer span ($s,e$) in passage $d_{i}$ is defined as:

where $\hat{D}=[d_{1}^{[CLS]},\cdots,d_{k}^{[CLS]}]\in\mathbb{R}^{h\times k}$ where $d_{i}^{[CLS]}$ is the BERT-based sentence representation (with 12 transformer layers and hidden size of 768), and $w_{start}$, $w_{end}$, $w_{doc}\in\mathbb{R}^{h}$ are learnable parameters. Notice that the final answer’s probability $P(s,e,i)$ also uses the passage ranking score $P(d_{i})$, so that the reader will trust more the answers from highly ranked passages.

We evaluate our ranking model on five commonly-used datasets for OpenQA tasks:

Quasar-T (Dhingra et al., 2017) contains 43,012 trivia questions, each with 100 passages retrieved from ClueWeb09 data source using LUCENE.

SearchQA (Dunn et al., 2017) is a large-scale OpenQA dataset with question-answer pairs crawled from J! Archive and passages retrieved by Google. It contains 140k question-answer pairs. On average, each question has 49.6 passages.

TriviaQA (Joshi et al., 2017) includes 95K question answer pairs authored by trivia enthusiasts and independently gathered evidence passages. Each question has 100 webpages retrieved by Bing Search API. We only use the first 50 for training the ranker and the reader.

Curated TREC (Baudiš and Šedivỳ, 2015) is based on the benchmark from the TREC QA tasks, which contains 2,180 questions extracted from datasets of TREC 1999, 2000, 2001, and 2002.

Natural Questions (Kwiatkowski et al., 2019) consists of real anonymized, aggregated questions issued to the Google search engine. The answers are spans in Wikipedia articles identified by annotators. Each question is paired with up to five reference answers.

For Quasar-T, SearchQA, and TriviaQA, we use the same processed paragraphs provided by (Lin et al., 2018). For Curated TREC and Natural Questions, following existing work, we determine a subset of 50 passages for each question and apply our model to them (Lin et al., 2018; Min et al., 2020). ³³3This is done due to our limited computation resources – training our model on a larger set of long passages (webpages) would require more memory than we have. We call the generated datasets “Natural Questions Subset” (NQ-Sub). The statistics are shown in Table 1.

The ranking results are evaluated by two common metrics:

HITS@$N$ evaluates the ranking results by indicating the proportion of top-$N$ passages that contains answer texts. Notice that this measure only provides an approximation of the true ranking quality, as a passage containing the answer without support would also be considered relevant.

EM measures the percentage of answers found by the whole system (the reader) that match the ground-truth answers.

For comparison, we select four representative ranking-based models as baselines. These models do not exploit external resources.

(1) BM25+BERT: The passages ranked by BM25 are directly submitted to the BERT-based reader. This is intended to test how a basic retriever+reader approach could perform on the datasets.

(2) DPR (Karpukhin et al., 2020) is one of the latest methods proposed in the literature, and it produces state-of-the-art performance. It adopts a bi-step retriever-reader framework where the retriever utilizes a neural matching model. The reader is the same BERT-based network structure as ours. We will note that DPR requires large computation resources to run.

(3) DSQA (Lin et al., 2018) adopts a retriever-selector-reader framework similar to ours. However, its denoising is limited to exploiting the score of the reader as a confidence score. No explicit denoising is done within the selector. The comparison with DSQA will reveal the impact of the GAN framework for denoising in ranking.

(4) R³ (Wang et al., 2018) is a reinforced model using a ranker to select the most confident passage to pass to the reader. Interactions are allowed between the ranker and the reader – the reader provides feedback (rewards) to the ranker. This is similar to the principle of our ranker, but we use GAN to discriminate between different passages and force the ranker to meet multiple objectives. In addition, the same neural architecture is used in R³’s reader and ranker, while we employ a simplified reader in our ranker for higher efficiency.

For fairness, we do not include methods such as REALM (Guu et al., 2020) and ORQA (Lee et al., 2019a) which either utilize external resources or expensive additional pre-training tasks, while our method does not. These approaches are orthogonal to ours, and can be added on top of ours. We leave it to future work.

We tune our model on the development set. Restricted by the computation resources available – one Nvidia GTX Titan X GPU (12G VRAM), only limited parameter sets could be explored. The hidden size of the RNN used in the ranker is 128, the number of RNN layers for the passage and question are both set to 1. The maximum passage length is set according to the passage length distributions – 150 for quasar-T, SearchQA, TriviaQA, and Natural Questions, and 350 for Curated TREC. The batch size varies among {2, 4, 8, 16} across datasets depending on the maximum passage length in order to fit the model into the GPU RAM. We only use top-50 passages retrieved by BM25 to train the ranker and the reader. The hyper-parameters $\lambda_{1}$ and $\lambda_{2}$ are set to 0.25 and 1. We follow  (Lin et al., 2018) for the setting of other parameters. The complete optimal parameter settings and code will be made public in our Github repository later.

We first test the ability of our PReGAN ranker to better rank passages containing the answers. The baseline models have been used on different datasets. To compare with them directly, we run two sets of experiments: one on Quasar-T and SearchQA on which DSQA has been tested, and another on Curated TREC and TriviaQA on which DPR has been tested. In addition, we also compare DPR with our method on the Natural Question subset.

At the top of Table 2, we report the ranking results by BM25, DSQA, and PReGAN. It shows that PReGAN outperforms both baselines by a large margin. Recall that the main difference between DSQA’s selector and PReGAN lies in the denoising based on GAN. The improvements hence directly reflect the usefulness of the GAN mechanism.

On Curated TREC and TriviaQA, as the candidate passages are not provided in the datasets, but retrieved from Wikipedia, we consider two cases – retrieving 50 and 100 candidates with BM25 and submitting them to the ranker. Notice that DPR directly retrieves passages from the whole Wikipedia dump, and thus is not limited by the 50 or 100 candidate passages. Nevertheless, we can see that our ranker can produce similar ranking results (Hits@20) to DPR despite the more limited candidates. On the Natural Question subset, we run DPR retriever and our ranker on the top-50 results from BM25. We see that PReGAN can better rank passages in top positions than DPR. However, at lower rank positions (from HITS@20), DPR performs better. This can be attributed to the more sophisticated retrieval model in DPR. In the future, the score networks used in our current model could be replaced by more sophisticated ones as in DPR. Another factor that explains the better ranking for DPR when $N\geq 20$ is that HITS@$N$ is only an approximation of the ranking quality. This measure should be considered together with the EM measure of answers.

The most important observation to make is the improvements of PReGAN over BM25. They reflect the capability of our ranker to favor the passages that may contain an answer. We expect that this will benefit the reader, as we will show.

We adopt a BERT-based reader structure as in (Karpukhin et al., 2020) to extract answers from the selected passages. We test the QA answer for each question produced by the reader with the top-50 passages (together with their ranking scores). The results are shown in Table 3.

We can observe that on all the datasets, PReGAN (combined with DPR reader) leads to consistently better answer results than the baselines. As BM25 ranking results are submitted to the same reader as PReGAN, the differences between them are directly attributed to passage ranking. This comparison clearly shows the usefulness of adding a ranker to rerank the BM25 retrieval results.

The comparison with R³ shows the utility of adding discriminators. Recall that R³ also uses the rewards from the reader to rerank the retrieval results, but without contrasting between different passages. We can see that by adding discriminators, PReGAN is forced to better distinguish between truly good passages from noisy ones.

In this experiment, DSQA uses its own selector to rerank the BM25 candidates and its reader to find the answer, and DPR also uses both its retriever to retrieve candidates and its reader. This setting may play in favor of these models because the retriever, selector, and reader have been optimized together. More particularly, DPR is allowed to search for top-ranked candidate passages directly from the document collections in Curated TREC (top-100) and Natural Questions (top-50), instead of using the top-50 retrieved with BM25. The quality of these passages is better than that of BM25. Despite this, compared to DSQA and DPR, PReGAN can still produce better results. This result demonstrates that our QA approach can be competitive against the state-of-the-art approaches that have much more parameters. Taking the experiments on passage ranking and answer identification together, we can see that a better passage ranking generally leads to better answers.

In this section, we further analyze the impact of the components of PReGAN and the time efficiency.