Empowering Few-Shot Recommender Systems with Large Language Models-Enhanced Representations

In contrast to implicit feedback derived primarily from user behavior observations, explicit feedback is openly and actively provided by users themselves to reflect their preferences and attitudes. The concept of explicit feedback mentioned in the book Recommender Systems: An Introduction encompasses ratings and annotations[18], while Konstan and Riedl[19] broaden its definition to include diverse forms of user-contributed content such as reviews, tags, blog posts, tweets, Facebook updates, among others.

In previous studies, ratings have been regarded as a crucial form of explicit feedback that enhances the performance of recommender systems[20][21] and can be combined with implicit feedback to cater to diverse recommendation tasks[22][23]. Text, serving as another manifestation of explicit feedback, can also be leveraged by recommender systems. Textual explicit feedback is commonly manifested as user reviews and comments[24] that are generated in various languages[25]. Other forms of textual explicit feedback include but are not limited to Tweets[26], web chats[27], messages accompanied by geographic information[28], and Tags[29]. Therefore, natural language processing (NLP) plays a crucial role in constructing recommender systems that rely on textual explicit feedback. Text mining has long been considered as an essential prerequisite in various recommendation models[24], encompassing techniques such as Latent Dirichlet Allocation (LDA)[30], TF-IDF[29], word segmentation[25], rule-based classifiers[31], and more. The processed text can be leveraged to support recommender systems built through approaches such as collaborative filtering[25] [30], content-based filtering[27] or knowledge-based[28].

In recent years, the embedding process has emerged as a prominent focus in recommendation studies due to advancements in related research. The utilization of LLMs in recommendation has been increasingly prevalent owing to their proficiency in comprehending and processing human natural language[32]. Transformer architecture models (e.g., BERT, GPT, and T5[33]) have been extensively employed in aspects including Pre-training, Fine-tuning, and Prompting [32]. Attention mechanism has also been integrated in the development of recommender system models. For instance, NARRE[34], a neural attention recommendation framework utilizing user reviews, is introduced to simultaneously predict users’ ratings towards items and generate review-level explanations for the prediction. Other attention models such as TARMF[35] and MPCN[36] that leverage textual explicit feedback also exhibit superior performance across diverse recommendation tasks compared to existing deep learning-based recommendation models (e.g., ConvMF[37], DeepCoNN[38]).

Released by OpenAI in 2022, ChatGPT[39] is an advanced LLM and dialogue system that has demonstrated exceptional performance across various vertical domains. It showcases remarkable capabilities in context-based comprehension, summarization, and text generation[12]. The investigation into the methodology of transferring and employing ChatGPT’s extensive knowledge and paradigm acquired from large-scale corpora to recommendation scenarios has emerged as a cutting-edge pursuit in the academic domain.

ChatGPT can independently serve as a versatile recommendation model capable of handling various recommendation tasks. Liu et al.[14] consider ChatGPT as a self-contained recommender system and construct a benchmark to track its performance in specific recommendation tasks, such as rating prediction and direct recommendation. ChatGPT can also serve as a component of existing recommender systems. Gao et al. [16] introduce Chat-REC, which employs ChatGPT as an interface for conversational recommendations, thereby enhancing the performance of existing recommendation models and rendering the recommendation process more interactive and explainable. Dai et al.[13] propose ChatAug that utilizes ChatGPT to rephrase sentences for textual data augmentation, simultaneously demonstrating the effectiveness of ChatGPT as a text summarization tool when accompanied by pretrained language models (BERT).

In terms of natural language generation tasks, ChatGPT demonstrates remarkable proficiency in generating persuasive recommendation interpretations and advertisements under specific conditions[40] [14]. Related research also suggests that the engagement of ChatGPT could be a innovative solution to address few-shot learning challenges[41]. However, recent research[14] reveals that when employed independently in few-shot scenarios as a recommender system, ChatGPT’s performance falls short compared to a series of classical recommendation models across diverse recommendation tasks, such as top-N direct recommendations. The aforementioned studies inspire us to explore the utilization of ChatGPT as an explicit feedback processing method indirectly participating in few-shot recommendation scenarios.

ChatGPT is designed to excel in user-oriented tasks, enabling us to adopt prompting paradigms[42] to target specific tasks without the need for fine-tuning. Drawing partly from relevant studies[14], our experiment initially utilizes the well-established ChatGPT model, gpt-3.5-turbo, to generate textual user and item representations by providing ChatGPT with tailored prompts. Each prompt consists of three components: review injection, task description, and format indicator. The review injection is designed to provide ChatGPT with a sequence of reviews from the same subject (a specific user or item). The task description aims to elucidate the input materials and establish clear task requirements. The format indicator serves to standardize response formats and constrain content scope. Additionally, we set a limiter when generating prompts to prevent them from exceeding the maximum token limit in ChatGPT.

Given that the API interface of ChatGPT necessitates its invocation in a conversational format, we assume the template ${\tau}$, which denotes the procedure for employing ChatGPT to generate a textual representation of a specific subject by utilizing its review collections. Formally, this can be expressed as

where Y represents a slot subsequently filled by ChatGPT’s response, suffix (i.e., task description and format indicator) identifies certain text specifically designed to guide ChatGPT in accomplishing representation generation, and input X is a sequence of reviews r pertaining to the specific subject, formally:

The example in Fig.1 illustrates the generation of a user representation through template ${\tau}$. It is noteworthy that the generation of the item representations also adheres to template ${\tau}$, albeit with a slightly different suffix; we modify the task description context for item representations generation to “(…Based on your understanding of these movie reviews, summarize the movie’s tag and scenes, associate and infer what type of audience and fans may be attracted by this movie.” In response to this description, ChatGPT would provide associations and inferences such as “Audiences who prefer heartwarming scenes and happy endings” for item representations.

ChatGPT incorporates a certain degree of randomness to ensure the diversity of generated response, which may pose challenges in terms of reproduction and evaluation. The implementation of the format indicator component has been observed to effectively standardize the responses and mitigate irrelevant variations. During preliminary training with small sample sizes, ChatGPT exhibits exceptional association and inference capabilities that surpass our initial expectations. In certain instances, ChatGPT accurately “guesses” a specific movie and subsequently retrieves comprehensive information from its own database, even when the movie title is not explicitly mentioned in the original reviews. To ensure controlled variables, we explicitly instruct ChatGPT to exclusively focus on materials provided by us when generating representations.

After generating textual user and item representations, we employ MacBERT[43], a pre-trained LLM for Chinese, to embed them to become our experimental dataset. Simultaneously, we construct a control dataset by concatenating reviews that belong to the same subject (item or user), embedding them with MacBERT, and merging the outputs. Additionally, we use a pre-trained Chinese Word2vec model[44] that does not employ attention mechanism to generate embeddings as an extra reference in some cases.

The length of each embedding generated using MacBERT is 1,024, while the length of those generated using Word2vec is 200. Considering the superior efficiency of MacBERT in natural language embedding tasks, we primarily utilize MacBERT-processed embeddings as our main control datasets and only refer to experimental results obtained from using Word2vec-processed embeddings under specific conditions. The model selection as well as the embedding process partially drew upon a relevant study[13].

The dataset employed in our experiment is the publicly available Douban Chinese Moviedata-10M[51], which shares similarities with the benchmark MovieLens dataset[52] in terms of content and format. The Douban dataset encompasses a substantial amount of explicit feedback provided by platform users, each sample presented as a user-item interaction comprising a user ID, an item ID, a piece of movie review, and other pertinent information such as a rating and a timestamp. In contrast to the MovieLens dataset, the Douban dataset primarily comprises Chinese text, and encompasses a substantial number of colloquial expressions, internet memes, emojis, and other intricate linguistic corpora. We intentionally perform minimal data cleansing to thoroughly evaluate ChatGPT’s proficiency in handling real-world explicit feedback.

To construct our experimental dataset, a cohort of 1,000 users is randomly selected. We extract the historical user-item interaction samples of these users and sort them in chronological order. The item IDs corresponding to the two most recent interactions are extracted as test and validation samples, respectively, and are concatenated into their respective sets. The remaining interaction samples of these users constitute the training dataset for inputting into ChatGPT to generate textual user representations. To simulate a few-shot scenario, we artificially control the number of interaction samples per user by randomly discarding excess samples while ensuring at least one sample per user remains. Detailed statistical information about the user training dataset is provided in Tab.I.

After extracting all the samples corresponding to the aforementioned 1,000 users, we construct the remaining samples as the item training dataset. Each item in the dataset is designed to have at least one corresponding sample. We apply filtering rules to ensure that all items present in the user training dataset, validation dataset, and test dataset exist simultaneously in the item training dataset. Moreover, in order to maintain control variables when constructing the control datasets, we restrict the number of samples per item to a maximum of 10. Following filtration, approximately 98.05% of the reviews in these samples are less than 150 words long (with 79.84% being under 100 words).

Eventually, during the stage of constructing the training dataset, we obtain a user training dataset consisting of 1,000 users, encompassing a total of 7,270 interaction samples; additionally, we create validation and test datasets with each comprising 1,000 samples (one per user); finally, an item training dataset is compiled containing over 300,000 interaction samples from 38,750 items. By providing ChatGPT with tailored prompts derived from the two training datasets (elaborated in Section 3), we generate a corresponding number of textual user and item representations, which are then combined to form textual user and item representation datasets and subsequently embedded by language models to form experimental datasets. The workflow for constructing the datasets is illustrated in Fig.3.

As outlined in Section 3, we employ MacBERT and Word2vec to embed the textual item and user representations for generating embedding datasets. Additionally, we build control datasets in accordance with the methodology detailed in the section.

In total, we acquire the following datasets:

• •

  A pair of textual representation datasets (user & item).

• •

  A pair of experimental datasets (user & item): ChatGPT-processed + MacBERT-embedded

• •

  Three pairs of control datasets (user & item): Only MacBERT-embedded; Only Word2vec-embedded; ChatGPT-processed + Word2vec-embedded.

In this section, we quantify the semantic relationships between embeddings of each subject (user/item) across the experimental dataset (ChatGPT-processed + MacBERT-embedded) and the control dataset (MacBERT-embedded). The evaluation method proposed in Section 4.A is employed to obtain the statistical measurement results presented in Tab.II.

The cosine similarity metric primarily focuses on the angular relationship between two vectors in a multi-dimensional space. When comparing two semantically similar sentences, regardless of their length, the angle between their vectors becomes smaller, resulting in a higher value for cosine similarity. Euclidean distance and Manhattan distance calculations encompass both direction and magnitude, which can serve as a complementary measure to cosine similarity.

When comparing the experimental and control datasets, both in terms of items and users, we observe that the result of Mean Cosine distance approaches 1, indicating a significant semantic similarity between the representations generated by ChatGPT and the original reviews. We also note that the Mean Euclidean and Manhattan distances deviate significantly from zero. Based on these results, we suggest that while the ChatGPT-generated representations demonstrate comparable semantics to the original reviews, they do exhibit significant disparities in terms of information content and quantity. This discrepancy may be attributed to their truncated length and refined content. In general, the aforementioned findings partially substantiate the effectiveness of ChatGPT in extracting a substantial portion of salient features and crucial information from the original reviews, albeit with a reconfigured textual composition and altered content. The reconfiguration and alteration will be examined in Section 4.E through a detailed case study.

Fig.4 depicts the workflow for conducting ablation experiments on two recommendation tasks using user-item interactions and the user and item embeddings from both experimental and control datasets. Notably, we conduct 10 independent repetitions to train each model in the two recommendation tasks and report the average results, aiming to comprehensively investigate their overall performance.

For the interaction prediction task, we concatenate the user embedding and item embedding from the same interaction samples (including randomly generated negative samples) and input them into binary classification models along with their labels for training. Subsequently, we assess the model’s Accuracy, Precision, and F1 Score on the ground truth test dataset.

For the direct recommendation task, we initialize the recommendation model with user-item interaction dataset. Upon model initialization, the BPR and NCF models automatically generate random embeddings for users and items, which are subsequently updated during training based on the models’ learning from the user-item interaction dataset (with ratings). After model training, these fine-tuned embeddings serve as a foundation for recommending items to selected users. In our study, we eliminate ratings by substituting them with a uniform constant to prevent the recommendation model from relying on ratings. As compensation, we replace the model-automatically-generated embeddings with the user and item embeddings in our experimental and control datasets. We assess the performance of the models using HR (Hit Rate) and MRR (Mean Reciprocal Rank), while additionally considering scenarios where these embeddings continue to undergo fine-tuning or remain fixed during model training.

In addition to conducting ablation experiments, we perform a comprehensive case study on the textual user and item representations to complement our findings and uncover potentially overlooked information within the embedding process. Our manual observations suggest that ChatGPT demonstrates exceptional proficiency in processing explicit textual feedback.

Specifically, it consistently demonstrates precise recognition and comprehension of contextual information with varying sentiment tendencies, even in the absence of quantitative metrics such as ratings. Notably, ChatGPT effectively handles reviews that contain positive, neutral, and negative snippets simultaneously by either disregarding the negative portion or considering an opposing viewpoint for recommendations. Additionally, ChatGPT adeptly identifies quotations within the reviews (e.g., movie lines, plots, extra materials) and utilizes them appropriately. The aforementioned observations collectively suggest that ChatGPT holds the potential to enhance the generalization capability of recommendation models by providing adaptability for diverse recommendation scenarios, such as social media platforms that exclusively comprising textual content.

Meanwhile, in contrast to conventional language models, ChatGPT demonstrates a unique ability to generate expansion context even when provided with limited information. While traditional NLP approaches primarily focus on keyword identification and extraction, ChatGPT goes beyond by introducing new content that may deviate from the original corpus. For instance, as depicted in Fig.1, ChatGPT suggests the keyword ”furry lovely animals,” possibly due to the user’s preference for documentaries featuring bears and animations. Essentially, ChatGPT “refines and reinforces” initial representations by augmenting them with supplementary information through association and inference. This could partially account for the observed semantic similarity yet content disparity between the experimental and control datasets, as evidenced by the findings in Section 4.C. Furthermore, the effectiveness of the refined and reinforced representations is demonstrated with support from the experimental results presented in Section 4.D. This partially indicates that the additional information contained within these representations, generated through ChatGPT’s association and inference, carries significant implications. In other words, these supplementary pieces of information reasonably reflect users’ underlying thoughts to a certain extent. To summarize, ChatGPT demonstrates its effectiveness in handling few-shot recommendation scenarios compared to conventional language models, owing to its distinctive capabilities in associative thinking and logical reasoning.

It is noteworthy that in this experiment, ChatGPT functions as a symbolic representation of emerging LLMs endowed with generative and logical reasoning capabilities. Considering the continuous advancements in technology, forthcoming LLMs equipped with enhanced proficiencies in association and inference may ultimately supplant ChatGPT within our experimental framework. Nevertheless, the insights derived from this investigation retain significant reference value for future studies.