Beyond Utility: Evaluating LLM as Recommender

The remarkable capabilities demonstrated by LLMs have sparked the interest of researchers in utilizing them for recommendation (Kang et al., 2023). Early work established the now widely adopted LLM-as-recommender paradigm (Liu et al., 2023a, b) by converting recommendation data into natural language inputs and obtaining recommendation lists in natural language form from LLMs. Under this paradigm, prompt design is one of the crucial factors affecting performance. Some studies manually design various efficient prompt strategies (Wang and Lim, 2023; Dai et al., 2023; Sun et al., 2023b) and in-context learning methods (Hou et al., 2024; Liu et al., 2023a; Wang and Lim, 2024) through experimentation. Others propose automated prompt engineering processes (Wang et al., 2024; Sun et al., 2023a; Du et al., 2024) leveraging the reflective capabilities of LLMs, reinforcement learning methods, and so on. This work has already demonstrated promising performance in some scenarios, but due to the lack of training on recommendation data, the results theoretically still have the potential for additional enhancement.

To further improve the recommendation capabilities of LLMs, researchers try to tune LLMs on recommendation data. The main objective is to align the natural language space with the recommendation space and the item space (Bao et al., 2023b, a; Harte et al., 2023; Zhang et al., 2023d). Work such as OpenP5 (Geng et al., 2022; Xu et al., 2024a) and POD (Li et al., 2023a) propose strategies for unifying various recommendation tasks using templates and training them jointly with the language model’s loss. Another set of studies (Hua et al., 2023; Zhang et al., 2024, 2023b; Lin et al., 2024) explores different item indexing methods, aiming to incorporate more collaborative information through indexing.

The above mentioned work primarily focuses on improvements in accuracy. Nevertheless, there are many other aspects that are equally noteworthy in recommendation systems. Therefore, a more comprehensive evaluation of LLMs as recommenders is required.

Accuracy is one important dimension in evaluating RSs, but it is not the only important dimension. In traditional recommendation scenarios, substantial work has been done to assess factors beyond accuracy that are also essential to user satisfaction and platform sustainability, which can be categorized into following dimensions: novelty (Ge et al., 2010; Zhou et al., 2010; Kaminskas and Bridge, 2016), popularity bias (Abdollahpouri et al., 2019; Zhu et al., 2020; Abdollahpouri et al., 2021; Zhu et al., 2021), diversity (Kaminskas and Bridge, 2016; Lin, 1991; Hurley and Zhang, 2011), and fairness (Wang et al., 2023c; Ge et al., 2021; Li et al., 2021).

Existing evaluation work on LLMs as recommenders often considers only these conventional evaluation dimensions. Palma et al. (Palma et al., 2023) investigate the performance of ChatGPT with various prompts in terms of novelty, popularity bias, and diversity in top-K recommendation, cold-start scenarios, and re-ranking tasks. Other work explores the fairness of LLMs as recommenders (Li et al., 2023b; Shen et al., 2023). FaiRLLM (Zhang et al., 2023a) and CFaiRLLM (Deldjoo and di Noia, 2024) evaluate fairness among user groups with different sensitive attributes, such as gender and age. FairEvalLLM (Deldjoo, 2024a) further extends the research by addressing intrinsic fairness. IFairLRS (Jiang et al., 2024) focuses on item-side fairness, observing the fairness of items belonging to different categories, while Deldjoo (Deldjoo, 2024b) considers both individual and group-level item-side fairness.

However, evaluating only these conventional dimensions is insufficient to fully uncover the characteristics of LLMs as recommenders, as their introduction brings new opportunities and challenges. Researchers have assessed LLMs from various angles since their emergence (Li et al., 2023c; Hendrycks et al., 2021; Zheng et al., 2023; Wang et al., 2023a; Qi et al., 2023), revealing multiple distinctive advantages and drawbacks of LLMs. These unique characteristics will cause LLM recommendations to exhibit distinct features compared to traditional ones. More specifically, one systematic bias that LLMs may exhibit is a particular preference for responses at certain positions (Wang et al., 2023b), which might lead to an input position bias in candidate lists in recommendations (Hou et al., 2024). The potential for hallucination (Xu et al., 2024b; Zhang et al., 2023c) may result in LLMs composing and recommending nonexistent items. Moreover, the text and generation capabilities of LLMs make it possible for them to create textual user profiles, which can also be integrated into the recommendation process and have an impact on the results. None of these aspects are a concern for traditional recommendation models; however, when using LLMs as recommenders, they need to be examined more thoroughly.

We propose a multidimensional evaluation framework of LLM-as-recommender that encompasses two traditional dimensions and four newly introduced dimensions related to LLMs’ characteristics. The overview of our framework is shown in Fig.1. Our framework adopts the commonly-used LLM-as-recommender paradigm and considers two specific tasks, ranking and re-ranking, the primary difference of which lies in the formation methods of the candidate sets. The task settings are described in detail in Section 3.2. Considering the cost, our framework supports evaluation on small sample datasets, with the sampling method of the datasets detailed in Section 3.3. After obtaining the recommendations from the LLM, our evaluation includes six dimensions: two conventional dimensions, utility and novelty, and four newly proposed dimensions. These four new dimensions focus on the potential new impacts that LLMs might bring to recommendations. Detailed descriptions of the evaluation dimensions are provided in Section 3.4.

Using LLMs as recommenders typically involves three steps: converting recommendation task and data into prompts, obtaining LLM inferred outputs, and extracting recommendation lists from those outputs. For input instructions, a common paradigm is to provide either off-the shelf or fine-tuned LLMs with the task instruction, user history $h_{u}$ and candidate item set $C_{u,y}$. The prompting strategies $\mathcal{P}$ usually index items by their titles $\mathcal{T}$. With the instructions, LLMs select top-K items $R_{u}$ from the candidate set for recommendations.

More specifically, we support evaluations on two tasks: ranking and re-ranking, which are the two main settings in recommendation scenarios. We construct these tasks through different formation methods of the candidate sets.

Ranking Task. The ranking task aims to identify the top-K recommendations from the entire item set $\mathcal{I}$. However, limited by the input length of LLMs, we can only simulate this task using the candidate set. In this case, we sample m negative items randomly and combine them with the positive item to form $C_{u,y}^{rank}$.

Re-Ranking Task. The re-ranking task is the process of further refining and personalizing recommendations after obtaining preliminary results from some recommendation models. These models select the top-K items from $\mathcal{I}$, while re-ranking models reorder the items they have selected. In this case, $C_{u,y}^{rerank}$ should be the aggregation of the top-K recommendation of multiple traditional models. These recall models $f_{i}$ are trained on the training set $\mathcal{D}_{train}$, and required to make recommendations on $\mathcal{I}$ for each sample in $\mathcal{D}_{test}$.

Any recommendation datasets that include the titles of items $\mathcal{T}$ and user-item interactions $\mathcal{D}$ can be used to conduct the evaluation for LLMs as recommenders in our framework.

Also, our framework supports evaluations on a small sample test set, considering that running inference on the full dataset can be time-consuming and costly in the context of LLMs. We utilize the leave-one-out strategy to divide $\mathcal{D}$ into $\mathcal{D}_{train},\mathcal{D}_{valid}$ and $\mathcal{D}_{test}$, and randomly sample $n$ users from the entire user set $\mathcal{U}$ to form the sample test set $\mathcal{SD}_{test}$. When using the small sample setting, we replace all $\mathcal{D}_{test}$ mentioned above with $\mathcal{SD}_{test}$, while the training set keeps unchanged as the entire $\mathcal{D}_{train}$.

To minimize the distributional difference between the sample and the full dataset, we perform a Kolmogorov-Smirnov (K-S) test on the hypothesis in (Eq. 4). Only samples with no significant differences will be passed on to further evaluation; otherwise, a new random sample is taken.

where $f_{\Theta}$ represents a chosen traditional recommendation model, and $Metric$ denotes any utility or beyond-utility metrics.

The evaluation covers six dimensions. Utility and novelty aspects are important areas in traditional RSs. We also conduct a more comprehensive evaluation of other beyond-utility aspects, including fairness and diversity. Due to coverage in previous work and limited space in this paper, we include them in the appendix. More notably, we take four additional dimensions into consideration, focusing on how the strengths and weaknesses of LLMs impact recommendations.

In the following part, we provide the detailed introduction of dimensions measured and metrics used. Considering that our evaluation should adapt to the scene of small samples, we carefully select and design metrics that can work on both large and small samples.

The utility of the recommendations from different LLMs and traditional models is reported in Tab.3. In Beauty, Sports, and ML-1M, LLMs demonstrate some recommendation capabilities; however, the overall recommendation accuracy of these models is inferior to that of traditional models. The two open-source models with 7b parameters cannot outperform the MostPop method, but the larger Llama3-70b and closed-source models achieve similar or better accuracy compared to MostPop. The best-performing LLM, GPT-4o, exhibits performance that even surpasses GRU4Rec in Sports, yet it still falls short of LightGCN.

More impressively, in LastFM, LLMs exhibit stronger recommendation capabilities. Several LLMs achieve significantly better recommendation accuracy than the best traditional model, LightGCN. Since LLMs might have a better understanding of singers in the LastFM dataset due to the reason that these singers are more likely to have appeared in the training corpus, this result indicates that LLMs can effectively leverage their knowledge in areas where they excel and possess rich knowledge for recommendations. In this case, employing LLMs in recommendations can be beneficial to utilize their world knowledge.

In summary, for ranking tasks, LLMs have the potential to make better recommendations because of their world knowledge, while it is also crucial for LLMs to learn collaborative filtering information. The above results can be summarized as the first key observation:

Observation 1. Overall, current LLMs are less accurate than traditional models, but they can exhibit greater accuracy in domains where they possess more extensive knowledge.

In this section, we evaluate whether niche items that users are interested in can be discovered. It might be assumed that when LLMs make recommendations, they tend to suggest popular items based on inherent stereotypes. However, our results show that off-the-shelf LLMs are far less affected by popularity bias than traditional models. In Fig.2, regarding Popularity performance, besides BM25, the other top three methods are all LLMs, while the strong performance of BM25 on this metric may be a tradeoff for its relatively lower accuracy. LLMs’ understanding of popular trends does not cause them to focus on the biggest hit items in the dataset. In LastFM, GPT4o even demonstrates not only better accuracy but also less popularity bias at the same time. Moreover, this provision of exposure for niche items is not merely a fairness concern; LLMs can indeed accurately identify and recommend the correct niche items. Considering Serendipity in Fig.2, the metric that favors more accurate and unexpected results, although the accuracy of LLMs’ recommendations is inferior in Beauty, the Serendipity of the best-performing LLMs can still outperform traditional models. This result suggests a major advantage of LLMs, which is their ability to recommend less popular items, enhancing both fairness and accuracy.

Observation 2. LLMs are adept at recommending more niche items correctly.

The variation of recommendation accuracy with input history length is shown in Fig.3. Focusing on the performance of LLMs, we can conclude that they can effectively utilize user history data, as they perform much better with than without history (0 history length) as input, except for Mistral and Qwen2 in ML-1M. In Beauty and Sports, when user history is not provided, the HR@5 of LLMs is essentially equal to random recommendations (0.25); however, in ML-1M and LastFM, LLMs achieve a significantly higher HR@5 than 0.25 when the history length is 0, indicating that they already possess knowledge of movies and singers.

Nevertheless, the performance of the LLM does not consistently improve with the increasing length of input user history. The latest and second latest history inputs generally bring substantial improvements to the performance of LLMs, but the growth rate quickly slows down thereafter, and they often reach their peak performance with fewer than 10 history inputs. In comparison, traditional models require a longer history to achieve their best performance, keeping improving smoothly as the length of history increases. What is noteworthy is that, when there are only 1-2 historical records, best LLMs can demonstrate greater accuracy than traditional models. This suggests LLMs may offer insights into solving the cold start problem. However, it is also necessary to consider how to address the issue of LLMs not being able to utilize longer histories.

Another notable issue is that Qwen2 and Mistral exhibited abnormal performance in ML-1M. With user history input, their performance deteriorates. This may indicate that these 7b models experience a conflict between injected knowledge and their existing knowledge, leading to poorer results.

Observation 3. LLMs require only brief histories to perform well, while longer histories do not always benefit LLMs. LLMs can beat the traditional models in the cold-start scenario.

In this section, we observe the differences in the accuracy of users with positive items placed at different positions within the input candidate list. In general, candidate position bias is prevalent in LLM-based recommendations. According to Fig. 4, although the performance of traditional models and LLMs both fluctuates across different user groups when the positive item appears at different positions, traditional models remain relatively steady overall. In contrast, the performance of LLMs deteriorates as the positive item is placed further down the input list. This trend is particularly pronounced in Beauty, Sports, and ML-1M, and also observable in the LastFM dataset among Qwen2, Llama3, Claude3 and GPT4o-mini. The only exception is Mistral, which maintains relatively stable performance across the first three datasets; however, in LastFM, it exhibits a bias favoring items placed in later positions.

This candidate position bias can adversely affect the accuracy of recommendations, since it suggests that LLMs tend to make indiscriminate recommendations for the top-positioned item. When the top item is not positive, this indiscriminacy will affect the priority of the real positive items in the recommended lists. Therefore, mitigating this candidate position bias and fostering LLMs to rank items based on their actual relevance and quality are necessary to enhance accuracy. It is worth adding that previous work (Hou et al., 2024) has discovered that placing the positive items in the middle results in the highest accuracy. However, after we expand the user sample size and the number of datasets, our results show that most LLMs tend to prefer items positioned earlier in the list.

Observation 4. LLMs suffer from a severe candidate position bias, favoring items at the beginning most of the time.

In this part, we examine the multi-faceted performance when incorporating the LLM user profile generation step. For more details on the step, please refer to Section 3.4.5. When given the task of inferring a user’s personality and preferences, LLMs can generate personalized user profiles based on the user’s history. Fig.5 shows a summary of the user profile provided by GPT4o-mini of a user in Beauty. The LLM can identify the subcategories of products that the user prefers and the key factors they value, which can help enhance the understandability of the recommendation process.

In Beauty and Sports, the generated user profile contains most of the patterns useful for LLMs to make recommendations. According to Fig.6, we can see that removing the history and using only the generated profile, LLMs can achieve almost similar performance in terms of accuracy and better performance regarding popularity. Utilizing both the profile and history simultaneously leads to improvements across all four dimensions, regardless of history length. The trend in Sports is consistent with Beauty. In ML-1M and LastFM, LLMs can also generate personalized profiles that capture key patterns, but likely due to LLMs having deeper knowledge of specific items in ML-1M and LastFM, using the profile results in some loss of accuracy compared to using only the history.

Observation 5. LLMs can generate user profiles that capture a majority of the key patterns useful for recommendations, which can help enhance the recommendation interpretability.

This section examines the issue of hallucinations in LLM-based recommendations. When dealing with ranking tasks, LLMs still exhibit an overall deficiency in following instructions. In Tab.4, we can see that the proportion of non-existent items generated is mostly around 1% and less than 5% in models other than Mistral, while Mistral outputs quite a lot of non-existent items, with a proportion around 23% in Beauty. Common errors made by LLMs may include omitting part of the original titles, combining multiple titles, or misstating elements such as capacity. When the first two types of errors occur, it is difficult to map the generated item back to the corresponding item in the original dataset. A well-designed mapping strategy is highly needed; otherwise, it will result in many invalid recommendations. Hallucination will be one of the critical factors affecting the recommendation quality of Mistral. Although the hallucination issue is relatively less severe in other models, around 1% occurrence can still impact the user experience.

Observation 6. In general, most LLMs tend to generate below 5% of non-existent items, while some LLMs significantly hallucinate more items.

Next, we assess LLMs’ performance on the re-ranking task from the above multiple dimensions. According to Tab.5, GPT-3.5 already shows higher re-ranking performance than four traditional models, both enhancing accuracy and promoting the exposure of niche items. One reason might be that LLMs both explicitly and implicitly utilize new features beyond those used by the four traditional models to rank the candidates, thereby better distinguishing between items in the candidate set, while the four traditional models find it more difficult to re-rank their own top recalled items. In practice, it might be a viable option to apply LLMs in the re-ranking stage.

The candidate position bias issue is similarly alleviated. In this scenario, considering that some candidate sets do not contain positive items, we calculate $CandDif$ based only on samples that include positive examples. Compared to $CandDif_{HR}$ and $CandDif_{NDCG}$ of 0.5172 and 0.4713 in ranking setting in Beauty, GPT-3.5 also shows relative insensitivity to position when re-ranking (Tab.5). Moreover, we can actually design the order of candidates according to the original recall model’s ranking to leverage this position bias for better recommendations in the re-ranking task.

Observation 7. Compared to ranking tasks, LLMs are better at re-ranking tasks regarding utility and beyond.