Meta Learning to Rank for Sparsely Supervised Queries

Learning to Rank has gained much attention from the information retrieval research and industry community, which aims to optimize the perfect search results. A series of LTR models have been developed in the research community, mainly in two divisions. One division is based on Gradient Boosted Decision Trees (GBDT), which have been used by many production search systems (Yin et al., 2016); the other division is based on neural networks. With the intriguing interest in neural LTR models, a lot of papers have been published, and researchers keep proposing state-of-the-art methods (Huang et al., 2013; Dehghani et al., 2017; Xiao et al., 2019; Qin et al., 2021; Huang et al., 2013; Devlin et al., 2019; Dai and Callan, 2019). The DSSM model, cited as (Huang et al., 2013), belongs to the category of representation-based models. It operates by calculating the embeddings of a query and a document, which involves averaging the word embeddings from their respective text fields. On the other hand, the interaction BERT-based model, referenced as (Devlin et al., 2019), employs a different approach. It concatenates the query and document text fields into a single sentence, which is then fed through multiple transformer layers. This approach has proven to yield state-of-the-art results, as cited in (Dai and Callan, 2019). In recent few years, neural LTR models gradually start to be launched on various commercial search engines successfully, such as LinkedIn (Guo et al., 2020), Taobao (Yao et al., 2021). Han et al. (2020) introduced TFR-BERT, a generic document ranking framework that combines the power of LTR models and BERT. Wu et al. (2022a) proposed a multi-task learning approach for the ranking problem to optimize the different engagement signals simultaneously. Furthermore, some LTR models (Lucchese et al., 2017; Aslam et al., 2009; Kanoulas et al., 2011; Yang et al., 2020) used effective negative sampling technique to improve the efficiency of model training process and effectiveness of the resulting model by filtering out noise and reducing the redundancy of the query-document pairs. Lucchese et al. (2017) demonstrated the proportion of relevant documents to non-relevant documents highly affect the quality of the learning-to-rank collections. Kanoulas et al. (2011) illustrated the relevance grade distribution in the training set is an important factor for the effectiveness of learning to rank algorithms. While Neu-IR and traditional LTR models have shown promising results on large, well-labeled datasets, their performance decreases when faced with sparsely labeled data. Conversely, the global ranking function in LTR may not be optimal for document retrieval since it ignores differences between feature distributions for each query (Ai et al., 2018). Although Can et al. (2014); Dehghani et al. (2017) constructed a ranking function specific to each query, this approach is limited by its high cost and low generalization ability due to the almost infinite number of queries and the unpredictable feature distribution of unseen queries. Our proposed MLTR model, however, can effectively improve the generalization of the LTR model on sparse datasets.

While traditional Learning to Rank (LTR) models rely on manually annotated labels for supervision, click-based LTR models harness user interaction logs as a guiding resource (Joachims, 2002). Initially, these models were grounded in an online dueling-bandit framework (Schuth et al., 2016; Yue and Joachims, 2009), evolving later to an online pairwise method (Oosterhuis and de Rijke, 2018) for unbiased pairwise optimization in LTR. However, that click data are not entirely reliable indicators of user preference due to various influencing factors beyond just preference (Joachims et al., 2005; Radlinski et al., 2008). Addressing this, Yuan et al. (2023) and later Joachims et al. (2017) developed the first theoretically unbiased LTR method, based on the premise that a user’s likelihood of examining an item is tied to its rank position, with clicks primarily on items that are examined (Craswell et al., 2008; Wang et al., 2018a). They employed counterfactual IPS estimation (ROSENBAUM and Rubin, 1983) to adjust for the biases inherent in these examination probabilities. IPS-based approaches, combating biases like position bias (Joachims et al., 2005), trust bias (Agarwal et al., 2019), and selection bias (Oosterhuis and de Rijke, 2020), are prevalent in Information Retrieval (IR). Oosterhuis and de Rijke (2021) enhanced these methods, accounting for potential changes in the logging policy during data collection. Despite their widespread use, IPS methods do grapple with high variance issues (Gupta et al., 2023). This opens avenues for future research into alternative bias mitigation techniques, such as the doubly-robust method (Oosterhuis, 2023). Sample-based approximation (Oosterhuis, 2021; Ustimenko and Prokhorenkova, 2020) are typically employed in LTR scenarios where relevance is clearly established. Notably, Oosterhuis (2021) recently introduced the PL-Rank method, an efficient approach for unbiased gradient estimation based on sampled rankings. Click-based LTR models aim to derive unbiased ranking models from inherently biased user behavior data, demonstrating efficiency in unbiased learning to rank area. However, our research doesn’t primarily focus on the bias in data collection or usage. Instead, we concentrate on the application of meta-learning techniques to train robust LTR-based models effectively, particularly in scenarios characterized by sparse data sets.

On the other hand, some works implement online LTR methods. For instance, Yue and Joachims (2009) introduced the first online learning to rank method, which utilizes online evaluation by sampling model variants and comparing them with interleaving to identify better rankers, thereby improving the entire system. Hofmann et al. (2013) extended this by guiding exploration through reusing previous interactions. However, these approaches are not always efficient and sometimes their performance at convergence is much worse than offline approaches, especially in online settings where performance cannot be measured, making early-stopping unfeasible (Schuth et al., 2016). Addressing these issues, our MLTR model can ensure more efficient optimization of traditional ranking model and keep stable model generalization with low computational cost in sparse data experimental settings.

Supervised learning algorithms have faced a challenge in handling sparse and imbalanced labeled data. Previous techniques have typically addressed the negative impact of sparse data and imbalanced data distribution by optimizing the model or through data augmentation methods. From a model perspective, Zhou et al. (2007) proposed that existing methods for dealing with the challenge of few labeled examples often rely on semi-supervised learning techniques that exploit both labeled and unlabeled data. Moreover, Sun and Hardoon (2010) introduced an active learning strategy for identifying informative examples that require manual labeling, which is particularly beneficial when manual labeling resources are limited. Nonetheless, it is crucial to note that the inductive bias of a model can be significantly impacted by having a limited number of examples, commonly referred to as sparse data, as noted in (Baxter, 2000). From data augmentation perspective, resampling is an typical technique for handling data imbalance in machine learning (Good, 2005). Data oversampling was introduced by Chawla (2003), who sampled the minor classes from the available data and included them in the training process to mitigate the imbalance between major and minor classes. One of the popular oversampling techniques is SMOTE (Chawla et al., 2002), which has various adaptations such as those proposed by (Han et al., 2005; He et al., 2008), and others. However, the learned supervised model has limit improvement with duplicated data without new information and high risk of overfitting. In the same way, Liu et al. (2009b) employed data undersampling as a technique to achieve a comparable amount of training data in various classes by reducing the number of data in the major classes. The article referenced as (Yang et al., 2020) reports on the use of a two-tower neural model that was trained utilizing a mixed negative sampling technique alongside batch random negatives. However, this method may lead to a loss of information during the reduction of training data through sampling. Data generation models for informative data augmentation in LTR are proposed by Yu and Lam (2019) and Qiu et al. (2021), as they believe that generating informative data is more beneficial than using resampling techniques. Those models generated informative synthetic data based on Adversarial Autoencoder (AAE) (Makhzani et al., 2015) and Gaussian Mixture Variational Autoencoder (GMVAE) (Dilokthanakul et al., 2016), respectively. Given the strong text generation capabilities of large language models (LLMs), many researchers (Bonifacio et al., 2022; Dai et al., 2023; Peng et al., 2023) propose using LLM-driven methods to generate pseudo queries or relevance labels from existing collections. Both of them could generate new data given different query types and different relevance levels. Resampling methods and data augmentation techniques have the potential to mitigate the effects of imbalanced data in the training set, however, they have limited improvement on overall model generalization.

Meta-learning is also known as learning to learn, which aims to learn better algorithms, including better parameter initialization, optimization strategy (Andrychowicz et al., 2016), network architecture (Zoph and Le, 2017) and distance metrics (Gao et al., 2019). Finn et al. (2017) proposed a Model-Agnostic Meta-Learning (MAML) algorithm, which trains a model on a variety of tasks, such that the model can be easily generalized to a new task with a small number of gradient steps from a small number of data from that task. Also, a lot of existing works have implemented the meta-learning approach in other research areas. Lee et al. (2019) proposed Meta-Learned User Preference Estimator (MeLU), which utilizes meta-learning approach to deal with the cold start problem in the recommendation system. Cui et al. (2022) proposed a novel approach to address the challenge of data sparsity in next POI recommendation, called Meta-SKR, which leverages a meta-learning approach to generate user-conditioned parameters for a sequential-knowledge-aware embedding module. Bansal et al. (2020) proposed a MAML-based meta-learning method LEOPARD for domain adaptation tasks in NLP. In addition, there are some works on information retrieval. Carvalho et al. (2008) proposed a meta-learning algorithm to suppress the undesirable outlier effects of the pairs of documents using the pairwise ranking function. Zabashta et al. (2015) presented a meta-learning model for selecting rank aggregation algorithms based on a specific optimality criterion. Wu et al. (2022b) introduced a novel Bayesian Online Meta-Learning Model (BOML) tailored for personalized product search. BOML harnesses meta-knowledge acquired from inferences made about other users’ preferences, enabling accurate predictions even in situations where historical data is limited. By addressing the challenge of data sparsity, BOML can significantly enhance the accuracy of recommendations in personalized product search. Wang et al. (2022) proposed Meta-learning based Fair Ranking (MFR), which alleviates the data bias and achieves better fairness metrics in the ranking model through an automatically weighted loss. Sun et al. (2021) proposed the MetaAdaptRank, which is a domain adaptive learning method for few-shot Neu-IR based on meta-reweighted weak supervision data selection during the different periods of the training process. However, to the best of our knowledge, there have been no work using meta-learning on ranking models with the sparsely labeled queries.

Let $Q=\{q_{1},q_{2},...,q_{N}\}$ denote the collection of $N$ queries, $D=\{d_{1},d_{2},...,d_{M}\}$ denote the collection of $M$ documents, and $L=\{1,2,...,l\}$ denote the collection of $l$ labels. There has an order of labels $l>l-1>...>1$, where $>$ denote the sequence of the label order.

For every query $q_{i}$, there is a corresponding related document collection $D_{i}=\{d_{i,1},d_{i,2},...,d_{i,J}\}$ and the corresponding label collection $y_{i}=\{y_{i,1},y_{i,2},...,y_{i,J}\}$. Above all, the original data $\mathcal{S}$ can be denoted as $\mathcal{S}=\{(q_{i},D_{i}),y_{i}\}_{i=1}^{N}$. The object is to train the ranking model of a given query $q_{i}$ and corresponding related document collection $D_{i}$ with the ranking label $y_{i}$, mathematically as $\hat{y}_{i}=f(\mathbf{x}_{i};\theta)$ where $f(\cdot)$ is a ranking function, $\theta$ represent all the learnable parameters in $f(\cdot)$ and $\mathbf{x}_{i}$ denote the concatenated feature vector generated from the query and documents $(q_{i},D_{i})$. $\mathbf{x}_{i}=concat(\phi(q_{i}),\psi(D_{i}),\mathbf{r}_{i})$, where $\phi(\cdot)$ and $\psi(\cdot)$ denote the query encoder and the document encoder respectively; $\mathbf{r}_{i}$ denotes the numeric ranking features for each query and corresponding related document collection $(q_{i},D_{i})$. Generally, we learn the optimized $\theta^{\ast}$ by $min_{\theta}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(\hat{y}_{i},y_{i})$ and $\mathcal{L}$ could be used as any ranking loss functions.

Our work is inspired by optimization-based meta-learning, specifically Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017), which optimizes globally shared parameters over several tasks, so as to rapidly adapt to a new task with just one or a few gradient steps based on a small number of examples.

In the search and ranking setting, we define each task as ranking items for a given query. Our MLTR framework trains a model with a good generalization which can quickly adapt to a new query based on the query’s sparse engagement information. We divide the raw data into $\mathcal{S}$ and $\mathcal{T}$. We limit each query task (including the query and all its corresponding items) within only one set, such that there is no query overlap between $\mathcal{S}$ and $\mathcal{T}$.

For each task query $q_{i}$ ($\in Q$) in $\mathcal{S}$, its corresponding items are randomly divided into a training set $\mathcal{S}_{train,i}$ and a test set $\mathcal{S}_{test,i}$ to optimize the model during various stages. For each task query $q_{i}$ ($\in Q$) in $\mathcal{T}$, its corresponding items are randomly split into a fine-tuning set $\mathcal{S}_{tuning,i}$ and an evaluation set $\mathcal{S}_{eval,i}$ to fine-tune the model and assess its performance, respectively. For further information regarding the notation employed in this paper, please refer to Table 1.

The MLTR framework’s key concept is to create robust model parameters through many query-based ranking tasks in meta-training, then quickly adapt these parameters for new tasks in meta-testing with a few gradient steps. In meta-training, it performs local and global updates. The local update adjusts the parameter of the query-specific ranker on each training set (learning process). The global update trains the parameter of the global ranker to minimize the meta losses with the adapted parameters on the test sets (learning-to-learn process). The proposed meta-learning approach to ranking considers that individual queries may have distinct optimal parameters for their rankers, which is unlike traditional Learning to Rank (LTR) models that learn a global ranking model applicable to all queries. Fig. 1 illustrates the architecture of the proposed meta-training process. To estimate document relevance for a new query in meta-testing, the ranker can then be fine-tuned based on the limited amount of labeled documents. The following subsections provide the details of the proposed approach.

We evaluate the performance of our MLTR framework in the setting of sparsely supervised queries using four different datasets. The datasets include MQ2007, MQ2008, MSLR-10K ¹¹1https://www.microsoft.com/en-us/research/project/letor-learning-rank-information-retrieval and Istella-S LETOR ²²2http://quickrank.isti.cnr.it/istella-dataset, which are public datasets widely used as benchmarks for LTR models (Qin and Liu, 2013). These datasets consist of queries, retrieved documents, and labels provided by human experts. Furthermore, we used a real-world e-commerce dataset collected from a one-month user log on Walmart.com. The focus of the dataset is on non-frequent tail queries, meaning the label distribution is extremely sparse. This dataset includes user search queries and the corresponding products in the search results, with labels (rating scores) ranging from 1 to 15 based on the level of user engagement. Query-product pairs that have been purchased receive the highest scores, whereas products that have received only clicks are assigned scores ranging in the middle. Products that have only received impressions are assigned scores lower than that of the click-only products. Negative items are assigned a score of 1. Scores for ordered products are calculated using a smoothed estimation of their order rate ($rate=\frac{order+\alpha}{impressions+\alpha}$), where $\alpha$ is the smoothing factor.

For all the above datasets, we first divide the raw data into meta-train, meta-validation, and meta-test sets, with a ratio of 80%, 10%, and 10%, respectively. Each query-document (item) pair is associated with a relevant rating label, which has different ranges for each dataset. Table 2 provides more details about the data statistics.

To simulate the sparse labeled queries, we further process the train, validation, and test datasets. In our experiments, we primarily control the number of positively labeled documents since it is usually limited in the real world and the negative documents can often be sampled from the dataset in a relatively large quantity. We perform a quantitative comparison on the simulated imbalanced datasets during training and testing.

We use $\mathcal{S}$ with superscript $p\cdot n\cdot$ to denote the number of sampled positive/negative labeled items per query in the training data (e.g., $\mathcal{S}_{train:p1n9}$ means the sampled training data with 1 positive-labeled items and 9 negative-labeled items per query). Thus, $\mathcal{S}_{train}$ is chosen from

and $\mathcal{S}_{test}$ is chosen from

Similarly, we use $\mathcal{T}$ with superscript $p\cdot n\cdot$ to denote the number of sampled positive/negative labeled items per query for model fine-tuning, $\mathcal{T}_{tuning}$ is chosen from

and the rest of the items of each query to evaluation our model with $\mathcal{T}_{eval:rest}$.

The validation set will be used to find the best model and hyper-parameters during the meta-training process, and will be split in the same manner as the meta-test dataset.

For the evaluation of the ranking results in MLTR, we apply Normalized Discounted Cumulative Gain (NDCG) which is suitable for ranking where users are usually sensitive to the ranked position of the relevant items (Järvelin and Kekäläinen, 2002).

To verify the efficiency and compatibility of our proposed model, we refrained from directly using overly complex baseline models. Instead, we conducted experiments on simple models and observed the resulting improvements to verify the effectiveness of our meta-learning based method for overall performance improvement in LTR models. On the other hand, due to the lack of semantic features for queries and corresponding documents in the public datasets MQ2007, MQ2008, MLSR-10K, and Istella-S, we did not utilize the corresponding text embedding representation features in our tests. Nevertheless, we supplemented the corresponding text embedding representation features using the BERT model with pre-trained weights from distillbert-base-uncased³³3https://huggingface.co/distilbert-base-uncased based on the text information of the query and document in the subsequent Walmart.com dataset. We then conducted experiments to verify the effectiveness of these features. We compare MLTR with the following competitive baselines:

• •

  LTR: The LTR baseline is a 3-layer Multi-Layer Perceptron (MLP) with a ReLu activation function. The ranking loss functions are introduced later in this section. To ensure fair comparisons, we perform fine-tuning on the test stage.

• •

  LTR+SMOTE (Chawla et al., 2002): This method is resampling-based and generates a resampled list using SMOTE, a popular oversampling strategy. We added the resampled data to the original training data and followed the same training and testing protocol as the LTR baseline model.

• •

  LTR+GMVAE (Qiu et al., 2021): This method is based on data augmentation and utilizes GMVAE to generate additional synthetic items. The GMVAE model is pre-trained with the entire training dataset, and then the augmented lists are produced. We added the synthetic data to the original training data and followed the same training and testing approach used with the LTR baseline model.

• •

  LTR+Policy-Gradient (Oosterhuis, 2022): Plackett-Luce (PL) ranking models, a decision theory-based approach to ranking. This model employs Gumbel sampling techniques for efficient sampling of multiple rankings from a PL model. Following this, algorithms PL-Rank-1, PL-Rank-2, and PL-Rank-3 are applied to these samples. This process enables an unbiased approximation of the gradient of a ranking metric in relation to the model. For our experiments, we have adhered to the model parameters and implementation as detailed in the official repository ⁴⁴4https://github.com/HarrieO/2022-SIGIR-plackett-luce and adapted the data-loader to fit our experimental setup.

• •

  LTR+Unbiased (Oosterhuis, 2023): Unbiased click-based LTR models, tailored specifically to adjust for position bias in click feedback. In our experiments, we deploy three distinct estimators. First, the inverse-propensity-scoring (IPS) approach employs counterfactual IPS estimation to mitigate the selection bias linked to examination probabilities. Next, we utilize DM and DR approaches that account for position bias, trust bias, and item-selection bias, offering a more flexible criterion for unbiasedness compared to the widely used IPS method. Our implementation follows the model parameters outlined in the official repository ⁵⁵5https://github.com/HarrieO/2022-doubly-robust-LTR with $N=10^{-4}$ for comparison, and we have adapted the data-loader to suit our experimental framework.

To ensure a fair comparison, most components in our proposed MLTR model employ the same model structure as the LTR baseline model. Regarding MLTR, we made corresponding adjustments to the model training process and data usage.

In traditional LTR models, three different types of loss functions, namely Pointwise, Pairwise, and Listwise (Liu et al., 2009a), are usually used depending on the task and data. We use the following representative losses for the LTR baseline and MLTR: RankMSE, RankNet, LambdaRank, and ListNet losses.

Pointwise loss: It only takes into account a single document $d_{i,j}$ at a time for a query $q_{i}$. RankMSE algorithm (Cossock and Zhang, 2008) is as follows,

Pairwise loss: It considers a pair of documents $<d_{i,j},d_{i,s}>$ at a time for a query $q_{i}$ if $y_{i,j}>y_{i,s}$ ($d_{i,j}$ should be ranked before $d_{i,s}$) (Burges, 2010). RankNet algorithm (Burges et al., 2005) and LambdaRank algorithm (Wang et al., 2018b), with their loss functions shown in Equation (6) as follows:

where $\varphi$ denotes the Sigmoid function for RankNet loss, $\varphi(u)=\Delta NDCG(j,s)\log_{2}(1+e^{-\sigma u})$ for LambdaRank loss, where $\sigma$ is a hyper-parameter and $\Delta NDCG(j,s)$ is the absolute difference between the NDCG values when two documents $d_{i,j}$ and $d_{i,s}$ are swapped.

Listwise loss: It directly looks at the entire list of documents $D_{i}$ and tries to come up with the optimal ordering for each query $q_{i}$ (Cao et al., 2007). For example, the loss function for the ListNet algorithm is as follows,

where $L(\cdot)$ denote the cross-entropy loss. $f(\mathbf{x}_{i})$ is the predict label for query $q_{i}$. $y_{i}$ denote the true label of each document in query $q_{i}$.

An extensive set of experiments were designed to address the following questions of the proposed research:

RQ1: Can the proposed MLTR framework achieve improved performance on sparsely labeled queries over the baseline methods? (Section 3)

RQ2: How does the training and test mechanism designed for MLTR effectively improve model performance compared to traditional model training processes? (Section 5.2.1)

RQ3: Without fine-tuning towards a specific query in test data, can MLTR still improve model generalization? (Section 5.2.2 and 5.2.3)

RQ4: Can MLTR alleviate the data sparsity issue and domain shift problem? How much NDCG lift can the MLTR models gain over the traditional LTR models? Is the amount of NDCG relative gain correlated with training/test data’s sparseness? (Section 5.3)

RQ5: Can MLTR be effective in real-world applications with limited labeled data and result in improved performance? (Section 5.4)

We compare the performance of MLTR to traditional LTR models when handling sparsely labeled queries. We tested our models on the four public datasets by simulating sparse data scenarios. The process involved training on $\mathcal{S}_{train:p1n9}$ and $\mathcal{S}_{test:p1n9}$, followed by fine-tuning on $\mathcal{T}_{tuning:p1n9}$ and evaluating the results on $\mathcal{T}_{eval:rest}$. The results shown in Table 3 indicate that our MLTR models, regardless of being in a without fine-tuning or with fine-tuning setting, outperform the baseline models in all metrics (NDCG@1, NDCG@5, and NDCG@10) across all four datasets, with the exception of the unbiased click-based LTR baseline models. This improved performance is maintained across various loss functions. Notably, on the MQ2007 and MQ2008 datasets, where the positive sample distribution is relatively sparse, the MLTR model shows significant improvement across all loss functions, providing further evidence that the meta-learning approach can enhance the model’s predictive ability in sparse data. While the MSLR dataset has a relatively even distribution of positive and negative samples overall, the MLTR model still manages to improve the model’s predictive results in most of the loss functions. The results also suggest that the SMOTE resampling technique can alleviate the issue of sparse data and improve performance compared to the traditional LTR model. The GMVAE-based data augmentation method outperforms the SMOTE resampling method in most cases as it can incorporate more informative data. However, data augmentation-based methods do not significantly enhance model generalization compared to our proposed MLTR approach. In comparing the PL-rank methods with MLTR, we found that the PL ranking methods lacks consistent stability, particularly underlined by our experiments that highlight the sparse nature of positive samples during training. In contrast, the test results indicate that MLTR yields more stable outcomes in scenarios characterized by a limited number of training samples or a sparser distribution of positive samples. On the other hand, when examining the results of Unbiased click-based LTR methods, these methods show a notable advantage when the training samples contain rich features in the query and document pairs. For instance, on the Istella-S dataset, DM and DR achieved the best performance in NDCG@1 and NDCG@5. However, in the other three datasets, the MLTR model displayed a more consistent performance advantage. In our proposed MLTR model, we did not strictly address the bias present in the data. The experimental outcomes of unbiased click-based LTR methods indicate that incorporating unbiased methods like IPS into the training process of meta-learning might further enhance the performance of meta-learning-based LTR models. We plan to implement and evaluate this approach in our future work. On the other hand, regardless of whether the MLTR model utilizes the fine-tuning process during meta-testing, it consistently demonstrates competitiveness compared to traditional methods. The results of MLTR without fine tuning still lead in most experiments, surpassing traditional LTR models as well as baseline models with other optimization approaches. Furthermore, if the fine-tuning process in meta-testing is employed, we find that MLTR can adapt more rapidly to changes in queries, thereby further enhancing the model’s performance on test evaluation data. When dealing with sparsely labeled queries, our MLTR model can achieve better adaptability with a small proportion of labeled data, leading to improved overall model performance.

This section demonstrates the superiority of our meta-learning model over the baseline when dealing with extremely sparse data and a low positive-to-negative label ratio. The evaluation was conducted on the MSLR-10K dataset and various experimental scenarios were simulated by sampling subsets of the data with varying ratios of positive and negative labels per query.

The study is performed on the real-world Walmart.com dataset, which has sparse positive-labeled queries as shown in Table 2. It is worth conducting the robustness experiments to evaluate the model generalization, as the data in the real world are often more dynamic with drifted distributions.