AVA: an Automatic eValuation Approach to Question Answering Systems

The task of reranking answer sentence candidates provided by a retrieval engine can be modeled with a classifier scoring the candidates. Let $q$ be a question, $T_{q}=\{t_{1},\dots,t_{n}\}$ be a set of answer sentence candidates for $q$, we define $\mathcal{R}$ as a ranking function, which orders the candidates in $T_{q}$ according to a score, $p\left(q,t_{i}\right)$, indicating the probability of $t_{i}$ to be a correct answer for $q$. Popular methods modeling $\mathcal{R}$ include Compare-Aggregate Yoon et al. (2019), inter-weighted alignment networks Shen et al. (2017), and BERT Garg et al. (2020).

The evaluation of system Accuracy can be approached in two ways: (i) evaluation of the single answer provided by the target system, which we call point-wise evaluation; and (ii) the aggregated evaluation of a set of questions, which we call system-wise evaluation.

We define the former as a function: $\mathcal{A}\left(q,r,t_{i}\right)\rightarrow\{0,1\}$, where $r$ is a reference answer (GS answer) and the output is simply a correct/incorrect label. Table 2 shows an example question associated with a reference, a system answer, and a short answer¹¹1The latter can be very effective but it adds an additional annotation cost, thus we limit its use just for the baseline model. That is, we aim to have a lower cost AVA model.

A configuration of $\mathcal{A}$ is applied to compute the final Accuracy of a system using an aggregator function. In other words, to estimate the overall system Accuracy, we simply assume the point-wise AVA predictions as they were the GS. For example, in case of the Accuracy measure, we simply average the AVA predictions, i.e., $\frac{1}{|Q|}\sum_{q\in Q}\mathcal{A}(q,r,t_{i}[,s])$, where $s$ is a short answer (e.g., used in machine reading). It is an optional input, which we only use for a baseline, described in Section 4.1.

Given an input example, $\left(q,r,s,t\right)$, our classifier uses the following similarity features: $x_{1}$=sim-token$\left(s,r\right)$, $x_{2}$=sim-text$\left(r,t\right)$, $x_{3}$=sim-text$\left(r,q\right)$; and $x_{4}$=sim-text$\left(q,t\right)$, where sim-token between $s$ and $r$ is a binary feature testing if $r$ is included in $s$, sim-text is a sort of Jaccard similarity:

and $\emph{tok}\left(s\right)$ is a function that splits $s$ into tokens.

Let ${\bf x}=f\left(q,r,s,t\right)=\left(x_{1},x_{2},x_{3},x_{4}\right)$ be a similarity feature vector describing our evaluation tuple. We train ${\bf w}$ on a dataset $D=\{d_{i}:\left({\bf x}_{i},l_{i}\right)\}$ using SVM, where $l_{i}$ is a binary label indicating whether $t$ answers $q$ or not. We compute the point-wise evaluation of $t$ as the test ${\bf x}\!\cdot\!{\bf w}>\alpha$, where $\alpha$ is a threshold trading off Precision for Recall in standard classification approaches.

Transformer-based architectures have proved to be powerful language models, which can capture complex similarity patterns. Thus, they are suitable methods to improve our basic approach described in the previous section. Following the linear classifier modeling, we propose three different ways to exploit the relations among the members of the tuple $\left(q,r,s,t\right)$.

Let $\mathcal{B}$ be a pre-trained language model, e.g., the recently proposed BERT Devlin et al. (2018), RoBERTa Liu et al. (2019), XLNet Yang et al. (2019), AlBERT Lan et al. (2020). We use a language model to compute the embedding representation of the tuple members: $\mathcal{B}\left(a,a^{\prime}\right)\rightarrow{\bf x}\in\mathbb{R}^{d}$, where $\left(a,a^{\prime}\right)$ is a sentence pair, ${\bf x}$ is the output representation of the pair, and $d$ is the dimension of the output representations. The classification layer is a standard feedforward network as $\mathcal{A}\left({\bf x}\right)={\bf W}^{\intercal}{\bf x}+b$, where W and $b$ are parameters we learn by fine-tuning the model on a dataset $D$.

We describe different designs for $\mathcal{A}$ as follows.

We build a language model representation for pairs of members of the tuple, $x=\left(q,r,t\right)$ by simply inputing them to Transformer models $\mathcal{B}$ in the standard sentence pair fashion. We consider four different configurations of $\mathcal{A}_{0}$, one for each following pair $\left(q,r\right)$, $\left(q,t\right)$, $\left(r,t\right)$, and one for the triplet, $\left(q,r,t\right)$, modeled as the concatenation of the previous three. The representation for each pair is produced by a different and independent BERT instance, i.e., $\mathcal{B}_{p}$. More formally, we have the following three models $\mathcal{A}_{0}\left(\mathcal{B}_{p}(p)\right)$, $\forall p\in\mathcal{D}_{0}$, where $\mathcal{D}_{0}=\{(q,r),(q,t),(r,t)\}$. Additionally, we design a model over $(q,r,t)$ with $\mathcal{A}_{0}\left(\cup_{p\in\mathcal{D}_{0}}\hskip 3.00003pt\mathcal{B}_{p}(p)\right)$, where $\cup$ means concatenation of the representations. We do not use the short answer, $s$, as its contribution is minimal when using powerful Transformer-based models.

The models of the previous section are limited to pair representations. We improve this by designing $\mathcal{B}$ models that can capture pattern dependencies across $q$, $r$ and $t$. To achieve this, we concatenate pairs of the three pieces of text above. We indicate this string concatenation with the $\circ$ operator. Specifically, we consider $\mathcal{D}_{1}=\{(q,r\circ t),(r,q\circ t),(t,q\circ r)\}$ and propose the following $\mathcal{A}_{1}$. As before, we have the individual models, $\mathcal{A}_{1}\left(\mathcal{B}_{p}(p)\right)$, $\forall p\in\mathcal{D}_{1}$ as well as the combined model, $\mathcal{A}_{1}\left(\cup_{p\in\mathcal{D}_{1}}\hskip 3.00003pt\mathcal{B}_{p}(p)\right)$, where again, we use different instances of $\mathcal{B}$ and fine-tune them together accordingly.

We propose peer-attention to encourage the feature transferring between different $\mathcal{B}$ instances. The idea, similar to encoder-decoder setting in Transformer-based models Vaswani et al. (2017), is to introduce an additional decoding step for each pair. Figure 1 depicts our proposed setting for learning representation of two different pairs: $a_{0}=\left(a,a^{\prime}\right)$ and $g_{0}=\left(g,g^{\prime}\right)$. The standard approach learns representations for these two in one pass, via $\mathcal{B}_{a_{0}}$ and $\mathcal{B}_{g_{0}}$. In peer-attention setting, the representation output after processing one pair, captured in ${H}_{[CLS]}$, is input to the second pass of fine-tuning for the other pair. Thus, the representation in one pair can attend over the representation in the other pair during the decoding stage. This allows the feature representations from each $\mathcal{B}$ instance to be shared both during training and prediction stages.

The AS2 datasets from the previous section typically consist of a set of questions $Q$. Each $q\in Q$ has $T_{q}=\{t_{1},\dots,t_{n}\}$ candidates, comprised of both correct answers $C_{q}$ and incorrect answers $\overline{C_{q}}$, $T_{q}=C_{q}\cup\overline{C_{q}}$. We construct the dataset for point-wise automatic evaluation (described in Section 4) in the following steps: (i) to have positive and negative examples for AVA, we first filter the QA dataset to only keep questions that have at least two correct answers. This is critical to build positive and negative examples.

To test AVA at level of overall system Accuracy, we need to have a sample of systems and their output on different test sets. We create a dataset that has candidate answers collected from eight systems from a set of 1,340 questions. The questions were sampled from an anonymized set of user utterances. We only considered information inquiry questions. The systems differ from each other in multiple ways, including: (i) modeling: Compare-Aggregate (CNN-based) and different Transformers-based architectures with different hyper-parameter settings; (ii) training: the systems trained on different resources; and (iii) candidates: the pool of candidates for the selected answers are different.

We trained and test models using AVA-NQ and AVA-GPD datasets, described in Section 5.2. We also evaluate the point-wise performance on the WikiQA and TREC-QA datasets.

Table 5 summarizes the configurations we consider for training and testing. For the linear classifier baseline, we built a vanilla SVM classifier using scikit-learn. We set the probability parameter to enable Platt scaling calibration on the score of SVM.

We developed our Transformer-based evaluators on top of the HuggingFace’s Transformer library Wolf et al. (2019). We use RoBERTa-Base as the initial pre-trained model for each $\mathcal{B}$ instance Liu et al. (2019). We use the default hyperparameter setting for typical GLUE trainings. This includes (i) the use of the AdamW variant Loshchilov and Hutter (2017) as optimizer, (ii) the learning rate of  $1e\text{-}06$ set for all fine-tuning exercises, and (iii) the maximum sequence length set to 128. The number of iterations is set to 2. We also use a development set to enable early stopping based on F1 measure after the first iteration. We fix the same batch size setting in the experiments to avoid possible performance discrepancies caused by different batch size settings.

We study the performance of AVA in evaluating passage reranker systems, which differ not only in methods but also in domains and application settings. We employ the following evaluation strategies to benchmark AVA.

We evaluate the performance of AVA in predicting if an answer $t$ is correct for a question $q$, given a reference $r$. Table 6 shows the result: Column 1 reports the names of the systems described in Section 4, while columns 2 and 3 show the F1 measured on AVA-NQ and AVA-GPD, respectively.

We note that: (i) the F1 on AVA-GPD is much higher than the one on AVA-NQ, this is because the former dataset is much larger than latter;
(ii) $\mathcal{A}_{0}\left(\{\left(q,r\right)\}\right)$ cannot predict if an answer is correct as it does not use it in the representation, thus its Accuracy is lower than 7%;
(iii) $\mathcal{A}_{0}\left(\{\left(r,t\right)\}\right)$ is already a reasonable model mainly based on paraphrasing between $r$ and $t$;
(iv) $\mathcal{A}_{0}\left(\{\left(q,t\right)\}\right)$ is also a good model as it is as much powerful as a QA system;
(v) the $\mathcal{A}_{1}$ models that takes the entire triplet $q$, $r$ and $t$ are the most accurare achieving an F1 of almost 74%;
(vi) the use of combinations of triplets, e.g., $\mathcal{A}_{1}\left(\{\left(r,q\circ t\right),\left(t,q\circ r\right)\}\right)$, provides an even more accurate model; and finally,
(vii) the peer-attention model, i.e., $\mathcal{A}_{2}\left(\left(r,q\circ t\right),\left(t,q\circ r\right)\right)$ reaches almost 75%.

We evaluate the ability of AVA in predicting the Accuracy of QA systems as well as the performance of answer sentence reranking tasks. We conduct two evaluation studies with two public datasets, TREC-QA and WikiQA, and an internal ADS dataset.

Table 10 reports some example questions from TREC-QA test set, the top candidate selected by the TANDA system Garg et al. (2020), the classification score of the latter, and the AVA score. AVA judges an answer correct if the score is larger than 0.5. We note that even if the score of TANDA system is low, AVA assigns to the answer a very high score, indicating that it is correct (see the first three examples). Conversely, a wrong answer could be classified as such by AVA, even if TANDA assigned it a very large score (see the last two examples).