Incorporating External Knowledge and Goal Guidance for LLM-based Conversational Recommender Systems

Building on the advancements of LLMs over general LMs in language generation and reasoning, we explore their inherent response generation and recommendation capabilities, with and without external knowledge or goal guidance. Our analysis comprises three settings, as shown in Figure 2a:

• •

  Direct Generation (DG). LLMs directly generate system responses and recommendations without any external inputs (Figure LABEL:ICLa).

• •

  Chain-of-thought Generation (COT). LLMs internally reason their built-in knowledge and goal-planning scheme for both CRS tasks (Figure LABEL:ICLb).

• •

  Oracular Generation (Oracle). LLMs leverage gold-standard external knowledge and dialogue goals to enhance performance in both CRS tasks, providing an upper bound (Figure LABEL:ICLc).

Additionally, we conduct an ablation study of different knowledge types on both CRS tasks by analyzing 1) factual knowledge, referring to general facts about entities and expressed as single triple (e.g., [Jiong–Star sign–Taurus]), and 2) item-based knowledge, related to recommended items and expressed as multiple triples (e.g., [Cecilia–Star in–<movie 1, movie 2, …, movie n>]). Our primary experimental approach utilizes in-context learning (ICL) on the DuRecDial dataset Liu et al. (2021). Figure LABEL:ICL provides an overview of the ICL prompts, with examples detailed in Appendix LABEL:Prompt and experiments detailed in § 5. For response generation, we evaluate content preservation ($bleu$-$n$, $F1$) and diversity ($dist$-$n$) with knowledge and goal prediction accuracy. For recommendation, we evaluate top-K ranking accuracy ($NDCG@k,MRR@k$).

We summarize our three main findings given the results of the response generation and recommendation tasks shown in Tables 1 and 2.

Our analysis reveals that integrating both factual and item-based knowledge can significantly boost the performance of LLM-based CRS. However, knowledge-enhanced approaches for LLM-based CRS present unique challenges that have been relatively unexplored compared to prior training-based methods in CRS or retrieval-augmented (RA) methods in NLP Zhang (2023); Di Palma (2023).

Training-based methods, which train LMs to memorize or interpret knowledge representations through techniques like graph propagation, have been widely adopted in prior CRS research Wei et al. (2021); Zhang et al. (2023). However, such approaches are computationally infeasible for LLMs due to their input length constraints and training costs. RA methods, which first collect evidence and then generate responses, face two key limitations in CRS Manzoor and Jannach (2021); Gao et al. (2023). First, without a clear query formulation in CRS, RA methods can only approximate results rather than retrieve the exact relevant knowledge Zhao et al. (2024); Barnett et al. (2024). Especially when multiple similar entries exist in the knowledge base (KB), precisely locating the accurate knowledge for CRS becomes challenging. Second, RA methods retrieve knowledge relevant only to the current dialogue turn, whereas CRS requires planning for potential knowledge needs in future turns, differing from knowledge-based QA systems Mao et al. (2020); Jiang et al. (2023). For instance, when discussing a celebrity without a clear query (e.g., “I love Cecilia…”), the system should anticipate retrieving relevant factual knowledge (e.g., “birth date” or “star sign”) or item-based knowledge (e.g., “acting movies”) for subsequent response generation or recommendations, based on the user’s likely interests.

To address this challenge, we employ a relation-based method which allows LLMs to flexibly plan and quickly retrieve relevant “entity–relation–entity” knowledge triples $K$ by traversing along the relations $R$ of mentioned entities $E$ Moon et al. (2019); Jiang et al. (2023). Firstly, entities for each utterance is directly provided by extracting entities in the knowledge bases from the dialogue utterance Zou et al. (2022). Relations that are adjacent to entity $E$ from the KB are then extracted as candidate relations (denoted as $F1$) and LLMs are instructed to plan the knowledge retrieval by selecting the most pertinent relation $R^{*}$ given the dialogue history $C_{j}$. Knowledge triples $K^{*}$ can finally be acquired using entity $E$ and predicted relation $R^{*}$ (denoted as $F2$). The process is formulated in Figure 3 and demonstrated with an example in Figure LABEL:KG-b. Given the dialogue utterance “I love Cecilia…” and the extracted entity [Cecilia], the system first extracts all potential relations for [Cecilia], from which the LLM selects the most relevant relation, [Star in]. The knowledge retrieval agent then fetches the complete knowledge triple [Cecilia–Star in–<movie 1, movie 2, …, movie n>].

When there are multiple entities in one utterance, we perform the knowledge retrieval one by one and in the scenario where there are multiple item-based knowledge triples, we randomly selected a maximum of 50 item-based knowledge due to the limitations of input token length. We implement N-shot ICL to guide LLMs in choosing knowledge relations and we show the detailed ICL prompt and instruction with examples in Table LABEL:table:_KR (§ LABEL:AK).

Accurately predicting the dialogue goals is crucial for 1) proactive response generation and 2) balancing recommendations versus conversations in CRS. Utilizing goal annotations for each dialogue utterance from CRS datasets, we leverage an existing language model, adjusting it for goal generation by incorporating a Low-Rank Adapter (LoRA) approach Hu et al. (2021); Dettmers et al. (2023). This method enables parameter-efficient fine-tuning by adjusting only the rank-decomposition matrices. For each dialogue history $C^{k}_{j}$ ($j$-$th$ turn in dialogue $k$; $j\in T$, $k\in N$), the LoRA model is trained to generate the dialogue goal $G^{*}$ for the next utterance using the prompt of dialogue history, optimizing the loss function in Eq 2 with $\theta$ representing the trainable parameters of LoRA. The detailed prompt and instructions are shown in Table LABEL:table:_GP (§ LABEL:AG).

In ChatCRS, the knowledge retrieval and goal-planning agents serve as essential tools for CRS tasks, while LLMs function as tool-augmented conversational agents that utilize these tools to accomplish primary CRS objectives. Upon receiving a new dialogue history $C_{j}$, the LLM-based conversational agent employs these tools to determine the dialogue goal $G^{*}$ and relevant knowledge $K^{*}$, which then instruct the generation of either a system response $s_{j+1}^{system}$ or an item recommendation $i$ through prompting scheme, as formulated in Eq 3. The detailed ICL prompt can be found in § LABEL:Prompt.

Datasets. We conduct the experiments on two multi-goal Chinese CRS benchmark datasets a) DuRecDial Liu et al. (2021) in English and Chinese, and b) TG-ReDial Zhou et al. (2020) in Chinese (statistics in Table LABEL:data). Both datasets are annotated for goal guidance, while only DuRecDial contains knowledge annotation and an external KB–CNpedia Zhou et al. (2022) is used for TG-Redial.
Baselines. We compare our model with ChatGPT⁴⁴4OpenAI API: gpt-3.5-turbo-1106 and LLaMA-7b/13b Touvron et al. (2023) in few-shot settings. We also compare fully-trained UniMIND Deng et al. (2023), MGCG-GLiu et al. (2023b), TPNetWang et al. (2023a), MGCG Liu et al. (2020) and SASRec Kang and McAuley (2018), which are previous SOTA CRS and RS models and we summarise each baseline in § LABEL:A1.
Automatic Evaluation. For response generation evaluation, we adopt $BLEU$, $F1$ for content preservation and $Dist$ for language diversity. For recommendation evaluation, we adopt $NDCG@k$ and $MRR@K$ to evaluate top K ranking accuracy. For the knowledge retrieval agent, we adopt Accuracy ($Acc$), Precision ($P$), Recall ($R$) and $F1$ to evaluate the accuracy of relation selection (§ LABEL:AK).
Human Evaluation. For human evaluation, we randomly sample 100 dialogues from DuRecDial, comparing the responses produced by UniMIND, ChatGPT, LLaMA-13b and ChatCRS. Three annotators are asked to score each generated response with {0: poor, 1: ok, 2: good} in terms of a) general language quality in (Flu)ency and (Coh)erence, and b) CRS-specific language qualities of (Info)rmativeness and (Pro)activity. Details of the process and criterion are discussed in § LABEL:A2.
Implementation Details. For both the CRS tasks in Empirical Analysis, we adopt N-shot ICL prompt settings on ChatGPT and LLaMA* Dong et al. (2022), where $N$ examples from the training data are added to the ICL prompt. In modelling framework, for the goal planning agent, we adopt QLora as a parameter-efficient way to fine-tune LLaMA-7b Dettmers et al. (2023). For the knowledge retrieval agent and LLM-based conversational agent, we adopt the same N-shot ICL approach on ChatGPT and LLaMA* Jiang et al. (2023). Detailed experimental setups are discussed in § LABEL:A1.

ChatCRS significantly improves LLM-based conversational systems for CRS tasks, outperforming SOTA baselines in response generation in both datasets, enhancing content preservation and language diversity (Table 4). ChatCRS sets new SOTA benchmarks on both datasets using 3-shot ICL prompts incorporating external inputs. In recommendation tasks (Table 5), LLM-based approaches lag behind full-data trained baselines due to insufficient in-domain knowledge. Remarkably, ChatCRS, by harnessing external knowledge, achieves a tenfold increase in recommendation accuracy over existing LLM baselines on both datasets with ICL, without full-data fine-tuning.

Human evaluation highlights ChatCRS’s enhancement in CRS-specific language quality. Table 6 shows the human evaluation and ablation results. ChatCRS outperforms baseline models in both general and CRS-specific language qualities. While all LLM-based approaches uniformly exceed the general LM baseline (UniMIND) in general language quality, ChatCRS notably enhances coherence through its goal guidance feature, enabling response generation more aligned with the dialogue goal. Significant enhancements in CRS-specific language quality, particularly in informativeness and proactivity, underscore the value of integrating external knowledge and goals. Ablation studies, removing either knowledge retrieval or goal planning agent, demonstrate a decline in scores for informativeness and proactivity respectively, confirming the efficacy of both external inputs for CRS-specific language quality.

CRS datasets typically contain a huge volume of knowledge. By analyzing dialogues from the DuRecDial datasets, categorized by goal types, we calculated a “Knowledge Ratio” dividing the number of utterances with annotated knowledge $N_{K,G}$ by total number of utterances $N_{G}$ in each goal type (Eq 4) to measure the necessity of relevant knowledge in CRS task completion. Our analysis, depicted in Figure 4, shows that recommendation tasks rank highly in terms of knowledge necessity, with “POI recommendation” dialogues requiring pertinent knowledge in 75% of cases.

Internal knowledge of LLMs is insufficient for handling domain-specific CRS tasks, which is reaffirmed by the analysis of knowledge retrieval accuracy in Table 7. Contrasting with traditional RS which relies on user data for collaborative recommendations, CRS mainly depends on context/content-based recommendation He et al. (2023). This shift highlights the limitations of LLMs in harnessing internal knowledge. ChatCRS overcomes these limitations by interfacing LLMs to plan and reason over external KBs through entities and relations. Therefore, it largely improves the recommendation accuracy, outperforming the training-based approach using full data. Given the limitations in LLM-based CRS tasks, Zhang (2023); Di Palma (2023), we anticipate future studies to further explore such approaches in CRS.

Factual knowledge guides the response generation process, mitigating the risks of generating implausible or inconsistent responses. The “Asking questions” goal type which has the highest knowledge ratio, demonstrates the advantage of leveraging external knowledge in answering factual questions like “the zodiac sign of an Asian celebrity” (Table 8). Standard LLMs produce responses with fabricated content, but ChatCRS accurately retrieves and integrates external knowledge, ensuring factual and informative responses.

Goal guidance contributes more to the linguistic quality of CRS by managing the dialogue flow. We examine the goal planning proficiency of ChatCRS by showcasing the results of goal predictions of the top 5 goal types in each dataset (Figure LABEL:GOALS). DuRecDial dataset shows better balances among recommendation and non-recommendation goals, which exactly aligns with the real-world scenarios Hayati et al. (2020). However, the TG-Redial dataset contains more recommendation-related goals and multi-goal utterances, making the goal predictions more challenging. The detailed goal planning accuracy is discussed in § LABEL:A-goal.

Dialogue goals guide LLMs towards a proactive conversational recommender. For a clearer understanding, we present a scenario in Table 8 where a CRS seamlessly transitions between “asking questions” and “movie recommendation”, illustrating how accurate goal direction boosts interaction relevance and efficacy. Specifically, if a recommendation does not succeed, ChatCRS will adeptly pose further questions to refine subsequent recommendation responses while LLMs may keep outputting wrong recommendations, creating unproductive dialogue turns. This further emphasizes the challenges of conversational approaches in CRS, where the system needs to proactively lead the dialogue from non-recommendation goals to approach the users’ interests for certain items or responses Liu et al. (2023b), and underscores the goal guidance in fostering proactive engagement in CRS.