Enhanced Facet Generation with LLM Editing

Search clarification has been an area of interest in information retrieval for a long time (Radlinski and Craswell, 2017; Vtyurina et al., 2017). Users send queries with various sub-intents to the search system and expect various search results. These sub-intents are called facets. For instance, facets could include "warcraft game", "warcraft movie", "warcraft book", "warcraft history" and more when a user searches for "warcraft". Previous studies (Hashemi et al., 2021; Samarinas et al., 2022) introduce facet generation task as generating facets from the query. If a search system can predict the query facets in advance, it can provide more diverse and higher-quality search results.

The previous studies (Hashemi et al., 2021; Samarinas et al., 2022; Hashemi et al., 2022; Liu, 2023; Zhao et al., 2023) demonstrated that models can improve their performance in generating various query facets by leveraging Search Engine Result Page (SERP). The most commonly used information in SERP is the snippet of the retrieved document. Constructing input with both query and document snippets provides the model with richer information, leading to improved performance in facet prediction. However, there are several challenges to commercialize these methods. First, public search engines like Bing or Google are continuously updated. Search algorithms change over time, and user documents are continually updated. The external researchers cannot grasp the principles and changes of the private search algorithm. Therefore, if the search engine is used as part of the model, SERP changes between training and testing, leading to a drop in performance. The second challenge is that public search engines only search for public documents. If the systems need to create a query facet for an in-house service, the facet distribution to target will be different. However, there is a significant cost involved in constructing a separate search engine to leverage in-house documents. Finally, external communication is essential for SERP. Therefore, the previous methods are difficult for customers who want on-premise services to consider.

We focus on a framework that operates independently of the search engine by using only a query as input during testing. We propose two strategies to predict query facets without SERP. The first is multi-task learning, which uses SERP only in the training and not in the test. The approach of simply concatenating documents as input for training is not efficient during the test (refer to Section 4). Therefore, we consider SERP as the target to improve the performance of our model. The second is editing facets using LLM. Recently, LLM has made remarkable progress since InstructGPT (Ouyang et al., 2022), achieving high performance in a variety of tasks. However, simply instructing an LLM to generate query facets can result in inaccurate facet generation. Because LLM does not know the distribution of the dataset, it is difficult to predict the facet that fits the target. We improve performance by editing the facets predicted by the fine-tuned small model with LLM. This is the effect of allowing LLM to generate accurate facets by informing LLM of the distribution of the dataset through a small model learned with the training dataset. In other words, LLM editing is more effective than end-to-end generation because a fine-tuned small model generates intermediate results to the target facet. We also demonstrate that LLM editing works effectively on the previous models as well.

Previous studies leverage SERP to enhance facet generation performance.  Hashemi et al., 2021 proposes the NMIR framework, which learns multiple intent representations by taking query and retrieved documents as input.  Samarinas et al., 2022 introduces five methods based on query and retrieved documents. FG (Facet Generation) generates a facet by inputting the retrieved document. SL (Sequence Labeling) determines whether a token is a facet through sequence labeling in the retrieved document. EFC (Extreme Facet Classification) considers terms that frequently appear in facets as classes and trains a classifier to find facet terms in documents.  Hashemi et al., 2022 proposes a permutation-invariant approach based on NMIR.  Liu, 2023 proposes a method that leverages related queries in addition to documents to enhance the performance.  Zhao et al., 2023 finds better facets for queries by combining external structured information in addition to documents. SR (Structured Relation) uses hypernyms through external knowledge (Concept Graph Wang et al. (2015) and WebIsA Seitner et al. (2016)) and list structure through HTML as input.

Search Clarification is closely related to interactive search systems (Sekulić et al., 2021; Aliannejadi et al., 2021). This is because the search system can clarify the user’s intent and provide more accurate services. Therefore, for ambiguous queries, the search system asks a clarifying question (Aliannejadi et al., 2019).  Zamani et al., 2020a identifies the taxonomy of clarification and generates clarifying questions.  Rao and Daumé III, 2018 builds a neural network model for the task of ranking clarification questions.

Facet identification of a query is also related to learning the query representation or expansion. Traditionally, query representations were constructed from term frequencies in search logs Salton et al. (1975).  Rocchio Jr, 1971; Lavrenko and Croft, 2017 introduce query representation using query expansion and relevance feedback.  Mikolov et al., 2013 learns the relationships between adjacent words in the corpus and represents words as embeddings. Words and queries can be expressed through pre-trained language models learned with large corpora such as BERT Devlin et al. (2018) and GPT2 Radford et al. (2019). Recently, Wang et al., 2023; Jagerman et al., 2023 introduce query expansion through LLM.

This paper focuses on generating facets based on only queries. In the training, $T_{train}=\{(q_{1},D_{1},R_{1},F_{1}),...,(q_{N},D_{N},R_{N},F_{N})\}$ where $q_{i}$ represents a query, $D_{i}=\{d_{i1},...,d_{im}\}$ consists of snippets from $m$ retrieved documents, $R_{i}=\{r_{i1},...,r_{it}\}$ contains $t$ related queries, obtained from query logs, and $F_{i}=\{f_{i1},...,f_{ik}\}$ represents $k$ target facets. In the test, $T_{test}=\{(q_{1},F_{1}),...(q_{M},F_{M})\}$ where we generate $F_{i}$ from $q_{i}$. In contrast to the training, $D$ and $R$ cannot be used during the test.

The previous methods are frameworks that learn a model by using the SERP for the query as input. Similar to the previous method, the model in Table 1 is fine-tuned to generate facets based on BART-base (Lewis et al., 2019) by receiving queries and documents. The performance of the fine-tuned model changes depending on the input configuration of training and testing. The performance will significantly decrease if SERP used for training is not used in testing. As a result, the most ideal scenario is a situation where the input configuration in training and test is the same.

Our method assumes a scenario where SERP is not available for test. Therefore, we utilize multi-task learning by placing information in the target rather than the input of the model. Additionally, we leverage vast knowledge by combining LLM and small models. The term "small model" denotes a model that can be trained on a single GPU. In our experiments, this corresponds to BART-base. The term "LLM" denotes a model of size 7B or larger and can be used in various tasks through pretraining and instruction-tuning.

We followed previous studies (Hashemi et al., 2021; Samarinas et al., 2022; Liu, 2023; Hashemi et al., 2022; Zhao et al., 2023) and used BART-base as a small model. ChatGPT (OpenAI, 2022) or GPT4 are private LLMs and have cost issues. Additionally, OpenAI’s models are constantly updated, making it difficult to reproduce our results, so we use open-source LLMs with public parameters. At the time of our experiments, we used UP 30B (Upstage, 2023), which ranks high on the LLM leaderboard Beeching et al. (2023).

The proposed method generates facets using only queries, which eliminates the dependency on search engines. To address the limitation of not being able to utilize SERP as input, we propose two strategies: multi-task learning and LLM editing. Multi-task learning helps the small model better understand the query. LLM receives prior information from a small model and generates improved facets. Even without SERP, FD+M+E shows similar performance to FG in automatic evaluation and achieves the best performance in LLM-based evaluation. LLM editing is a way to effectively combine small models and LLMs in various NLP tasks, rather than using them separately. Therefore, our method can be extended to various NLP tasks.