Leveraging Advantages of Interactive and Non-Interactive Models for Vector-Based Cross-Lingual Information Retrieval

Deep relevance matching model (DRMM, Guo et al. 2016) and kernel-based neural ranking model  (Xiong et al. 2017, KNRM,) are two representatives under interactive framework. These approaches concatenate and feed query-document pairs into a unified deep neural encoder to learn their representations for semantic matching, as shown in Figure 1(a). Accordingly, features in query and document are interacted and fused, resulting in promising quality on a variety of IR tasks (McDonald, Brokos, and Androutsopoulos 2018). Recent studies further initialize the encoder with large-scale pre-trained language model, such as BERT (Devlin et al. 2019), which brings better model abilities on representation learning and relevance scoring (Jiang et al. 2020). Specifically, given the input token embeddings of a query Q and a document D, the interactive model first encodes them into hidden states:

$$\textbf{H}=\mathrm{Encoder}([\textbf{Q},\textbf{D}])$$

(1)

where $\mathrm{Encoder}(\cdot)$ indicates the BERT model, $[\cdot]$ represents the concatenation of inputs. Then, a non-linear active function is adopted to calculate the relevance score s over the mean of the output hidden states $\overline{\textbf{H}}$:

$$\textbf{s}=\mathrm{sigmoid}(\overline{\textbf{H}})$$

(2)

Contrary to the synchronous encoding procedure in interactive architecture, the representations of queries and documents in non-interactive architecture (Figure 1(b)) are encoded asynchronously (Huang et al. 2013; Shen et al. 2014; Reimers and Gurevych 2019; Lu, Jiao, and Zhang 2020). It employs two encoders for query and document representation learning, respectively. Formally, the representations of query and document are calculated in a separate manner:

$$\textbf{H}_{Q}=\mathrm{Encoder}_{Q}(\textbf{Q}),~{}~{}~{}~{}\textbf{H}_{D}=\mathrm{Encoder}_{D}(\textbf{D})$$

(3)

A cosine function is assigned for the similarity calculation:

$$\textbf{s}=\cos(\overline{\textbf{H}}_{Q},\overline{\textbf{H}}_{D})=\frac{\overline{\textbf{H}}_{Q}\cdot\overline{\textbf{H}}_{D}}{||\overline{\textbf{H}}_{Q}||\cdot||\overline{\textbf{H}}_{D}||}$$

(4)

The parameters of both interactive and non-interactive models are optimized by minimizing the cross-entropy loss over the training set:

$$\textbf{L}_{cross}=-\sum_{\textbf{Q},\textbf{D}}\textbf{y}\mathrm{log}(\textbf{s})+(1-\textbf{y})\mathrm{log}(1-\textbf{s})$$

(5)

where $\textbf{y}\in\{0,1\}$ denotes the groundtruth relevance label of Q and D.

Several potential manners can collect relevant multilingual queries:

• •

  Ready-Made Queries. Several documents have already provided multilingual summarization or keywords. For example, the dataset WikiCLIR (Sasaki et al. 2018) collects large amount of pages of Wikipedia, each of which links with multilingual relevant keywords. For simplification, we can directly collect $N$ ready-made relevant queries labeled as “relevant” for each document.

• •

  Mining-Based Queries. For an existing cross-lingual search engine (no matter translation-based or vector-based), the relevant queries for each document can be extracted according to its clickthrough data. The click behavior reflects users’ interests and the latent semantic relationships between the input queries and the clicked documents (Radlinski, Kurup, and Joachims 2008; Ma et al. 2008). We can collect Top $N$ queries according to the click-through rate (CTR).

A special circumstances is that there is neither off-the-rack multilingual relevant queries, nor search log. In this paper, we do not take attention into this case, because a vector-based CLIR system even can not be well-trained without large-scale cross-lingual annotated data, letting alone considering the trade-off between efficiency and quality. Despite of that, several few-shot learning approaches in IR contexts can be carried out to handle this problem, such as generating pseudo data using MT (Chidambaram et al. 2019; Yang et al. 2020) and searching associated queries using a preliminary retrieval system (Litschko et al. 2018).

The distribution of word embeddings immediately affects the final representations, therefore plays a critical role in neural-based natural language understanding tasks (Levy and Goldberg 2014; Kusner et al. 2015). As cross-lingual queries and documents are jointly encoded by the interactive model, word embeddings have relatively less discrepancy caused by distinct languages. As a result, we reuse the well-trained word embedding layer of the teacher model to initialize that of non-interactive models. We expect this can carry a better initialization for model parameters, thus contribute to the subsequent training procedure.

Knowledge distillation approach (Ba and Caruana 2014; Hinton, Vinyals, and Dean 2015) enables the transfer of knowledge from a teacher model to a student model, which is improved in the process. Here, we exploit knowledge distillation to guide both non-interactive and semi-interactive model to learn the probability distribution output by a well-trained interactive model. Specifically, given the relevance probability $\textbf{s}{{}^{\prime}}$ predicted by the teacher model, the goal of distillation is to force our model to fit the probability distribution of the teacher:

$$\textbf{L}_{distill}=-\sum_{\textbf{Q},\textbf{D}}\textbf{s}{{}^{\prime}}\mathrm{log}(\textbf{s})$$

(7)

The final loss can be formally expressed as:

$$\textbf{L}=\alpha\textbf{L}_{cross}+(1-\alpha)\textbf{L}_{distill}$$

(8)

where $\textbf{L}_{cross}$ is the conventional cross-entropy loss as defined in Equation 5. $\alpha$ denotes a factor to balance the two loss, which is set to 0.7 as default according to ablation study as Section 5.2 described.

Recent studies (Tang et al. 2019; Lu, Jiao, and Zhang 2020; Izacard and Grave 2020) has successfully adopted such kind of methods to the area of monolingual IR. For example, Tang et al. (2019) and Lu, Jiao, and Zhang (2020) distill knowledge from the pre-trained language model BERT to a BiLSTM-based model and a non-interactive model, respectively. Contrast with these researches, we are the first to distill knowledge from a interactive model to the semi-interactive one, and examine the effectiveness of knowledge distillation under the cross-lingual scenario. Besides, the input of teacher model and student model are same in prior studies, while ours are different.

Semantic similarity tasks aim to predict whether query and document are semantically relevant or not. We evaluate our methods on WikiCLIR in five languages: Russian (Ru), Spanish (Es), French (Fr), Portuguese (Pt) as well as Arabic (Ar). For each language pair, we randomly extract 30K samples as the test set. To evaluate the performance, we use the area under the curve (AUC, Lydick, Epstein, and White 1995). Moreover, in order to verify the effectiveness of mining queries from search log, we also conduct a group of experiments on a dataset collected from a real-world search engine, as described in Section 5.4.

This task is to rank candidate documents by relevance with respect to a multilingual query. Here, we use the test data of MULTI-8 (Sun and Duh 2020) to further evaluate our proposed methods. MULTI-8 is also constructed from Wikipedia but allows for more finergrained levels of relevance. We evaluate our methods on seven multilingual queries (Arabic (Ar), German (De), Spanish (Es), French (Fr), Japanese (Ja), Russian (Ru), and Chinese (Zh)) with each language pair contains 100K samples. Following the default setting in Sun and Duh (2020), we calculate normalized discounted cumulative gain (NDCG, Järvelin and Kekäläinen 2002) to evaluate the search performance.

As shown in Table 2, our model significantly outperforms the non-interactive baseline across different language pairs, demonstrating the universal-effectiveness of the proposed method. In the meanwhile, our model yields over 4 times faster than its interactive counterpart at the inference time.

Concretely, although interactive model outperforms non-interactive model, it significantly increases the inference time and fails to be employed in a real-world IR system. We regard this model as an upper limit of retrieval quality. The proposed semi-interactive mechanism narrows the accuracy gap, which confirms our hypothesis that supplementing the document representation with its relevant multilingual contexts is conductive to V-CLIR. Moreover, the interactions among multilingual queries and documents of our approach are able to be conducted offline, maintaining the superiority of non-interactive model on processing speed.

We assess knowledge transferring on non- and semi-interactive models. Both the models boost the retrieval performance, indicating that transferring knowledge from a well-trained interactive model to non-interactive models is able to narrow the quality gaps among them. Besides, our results demonstrate that the Semi-Interact and KT are complementary to each other, progressively benefiting to the cross-lingual semantic matching.

Table 3 shows the NDCG@10 of different models on MULTI-8 document search tasks. Similar to semantic similarity tasks, our method yields consistently better results across language pairs. As a superiority of pre-trained multilingual language model lies in its ability on knowledge transferring, we can transfer knowledge from high-resource languages to low-resource ones in the downstream tasks. Accordingly, we conduct three retrieval tasks that are not considered at the training time. As seen, the proposed methods yield considerable performance on both the three tasks. It is encouraging to see that, although we do not enhance documents with relevant queries in these three languages, the improvements are still preserved on these tasks.

We plot Figure 3 (a) to show the effects of the number of relevant queries added to document representation. Specifically, via incorporating queries (N $>$ 0), the relevance accuracy increases, confirming that the associated queries are helpful for the document representation learning. We observe that the number with 3 is superior to other settings. When the number of queries goes up (N $>$ 3), the classification performance inversely drops. One possible reason is that the conventional semantic meaning of the document may be potentially overlooked when complementing superfluous multilingual queries. This results in biases in document representations and raises the difficulty on relevance prediction.

We verify the effect of semi-interactive on different stages, i.e. training and inference. As shown in Table 4, we found that exploiting relevant queries at the training time can consistently improve the performance, no matter relevant queries are utilized or not during inference. When adapting semi-interactive mechanism only in inference, the performance marginally changes. We attribute these to the fact that complementing multilingual contexts during training benefits the representation learning. Our results suggest that carrying out our method on both training and inference stages performs best.

Figure 3 (b) visuals the effects of the factor $\alpha$ used in loss function at knowledge distilling time. Our results reveal that the weight significantly affect the performance. A small $\alpha$ makes the teacher model plays a dominate role during training time, while too large values result in less affects from the distillation procedure. Weight with 0.7 is the best configuration to balance the learning from the groundtruth and the teacher model.

We further conduct experiments to examine knowledge transferring strategies. As shown in Table 5, reusing teacher model word embeddings (Initialize) and knowledge distillation (Distillation) improve the relevance quality separately and are complementary to each other. These results verify again that transferring knowledge from interactive model is facilitate to the representation learning and relevance scoring of non-interactive model.