Less Can Be More: Exploring Population Rating Dispositions with Partitioned Models in Recommender Systems

Collaborative filtering recommenders are based on the idea that each user benefits from recommendations based on the ratings of other users. Implicitly, this means that the other users’ data are in some way appropriate to use – that users base their ratings on some common value system (even if they have different preferences) or that past agreement is predictive of the future agreement. We already know that this assumption doesn’t extend arbitrarily. Prior work has shown that in those cases it may result in better recommendations if the rating dataset is partitioned based on topic modeling and users’ persona (Wilson et al., 2014).

Currently, we know 3 different biases that control what a rating looks like - 1) Selection bias arises from user choices of items to consume. Typically, users prefer items that they anticipate enjoying (Dalvi et al., 2013). 2) Under-reporting bias results from users’ decisions of which items to rate. Empirical studies suggest that most users’ average ratings are close to or even higher than the scale’s average, as consumers are more likely to provide feedback on items they feel positively about. Additionally, users are more likely to rate items if they are either satisfied or dissatisfied rather than if they feel neutral about the product (Hu et al., 2017). 3) Rating scale bias reflects personal differences in the use of rating scales across users (Gena et al., 2011; Cena et al., 2017).

Looking at data from a movie recommender, we wonder whether it might also be useful to partition users who come to the recommender with very different goals and ways of using the system. We call this way of using the recommender system – the choice of items to rate and how they’re rated – the user’s rating disposition, which encapsulates the three biases mentioned above. We want to explore the usage patterns by implicitly clustering users based on empirical distributions of their ratings. We frame our study into three research questions:

To answer RQ1, we first split users into two populations based on their negative rating percentages. We also further categorize more granular subgroups based on positive rating peak distribution for two populations. Built upon the findings from RQ1, we run a preliminary partitioning experiment on two different populations for RQ2 by training the recommender model separately for optimistic and pessimistic users. Finally, we move to RQ3 with more empirical trials and show that we can potentially improve the recommendation quality and better serve each population with partitioned model training.

In the following sections, we first discuss related work, then we introduce our data collection and rating disposition identification process. A preliminary binary population partitioning trial and more thorough partitioned modeling are then demonstrated, and we conclude with discussion and potential future implications.

We use data from the movie recommender system MovieLens (https://movielens.org). Movielens is a non-commercial, personalized recommender that gathers users’ ratings on movies they have watched and provides predictions of new movies with different recommendation algorithms. Our data contains 3,908,657 ratings from 6724 users on 68,044 movies. Those are active users who have logged in to the website over 12 times from 2019 to 2020 and rated over 20 movies since their registration. Administrative users are excluded. The dataset can be downloaded at https://grouplens.org/datasets/rating-disposition-2023.

Our first approach for quantifying rating disposition is to categorize users based on their negative rating percentage. Previous studies usually treated 1-star or 2-star as negative ratings on a 5-star scale (Zeng et al., 2011). In our dataset, the median of rating is 3.5, so we adjust the definition based on our average and define the negative rating as less than or equal to 3 stars, and positive ratings as ¿=3.5 stars, since Movielens only has half-star intervals. After that, we introduce the concept of optimistic population and pessimistic population.

To find out the threshold for splitting two different populations, we compute the percentage of negative ratings among all ratings for each user, and then aggregate individual data into six different groups: [0-10%, 10-30%, 30-50%, 50-70%, 70-90%, 90-100%], each indicating the percentage bucket of negative ratings. The justification for the non-equal bin width is to segregate out the most optimistic (0-10%) and pessimistic (90-100%) user groups and potentially test their impact on other groups, even if their absolute number of users is small. Based on the distribution observation, 57.5% users have less than or equal to 30% ratings among all of their ratings less than or equal to 3 stars, while 42.5% users have more than 30% negative ratings. For better representation and split equality, we choose 30% as the optimistic and pessimistic user threshold and hypothesize that users with different percentage of negative ratings reflect their potentially dissimilar dispositions. Based on the hypothesis, we also plot out the ratio of extremely positive ratings (4.5-star and 5-star) over all positive ratings (¿= 3.5 stars) for both populations in the last graph of Fig. 1, which shows that pessimistic users give fewer high ratings and potentially have different positive rating scales from optimistic users.

In the next section, we show how to use the positive rating peak along with the optimistic and pessimistic population definitions to categorize more detailed disposition types.

In this section, we seek to mimic a more efficient recommendation model that only trains and tests each different user population within their own group of neighbors. We hypothesize that the data from each disposition group will be most relevant to recommendations to the save group and possibly harmful to other population group recommendations.

We conduct three different experiments to evaluate recommendation model performance on the original data as well as on the partitioned data based on our definition of populations. We choose three classic recommender algorithms to evaluate model performance: User-User CF, Item-Item CF, and the latent factored CF using matrix factorization approximation Funk SVD. All algorithms are impelmented with the Surprise API(Hug, 2020). For User-user and Item-item CF, they both make use of KNNWithMeans model, with parameters min_K = 1, max_K = 20, and similarity measure = Mean Squared Difference (MSD). For the SVD algorithm, we set n_factors = 10, and n_epochs = 20. We note that different algorithms may be more or less robust to having users from different rating dispositions together. We hypothesize that User-User CF will be highly sensitive to impacts from their nearest neighbors and more likely to benefit from partitioning, since users with different dispositions are rarely neighbors. But it also could be that they often become ”false neighbors” that correlate well on the data they have rated in common but then turn out to disagree badly on other items. By contrast, latent factor methods such as SVD create a single dimensionalization (the latent factors) from all users with consideration of model user-specific bias – having different subpopulations might have little effect because they can all be served by the same factors even if they weigh different factors differently, or having them together might result in a dimensionalization that’s worse for each group. We carry out experiments to explore these questions.

With the definition of two groups of population and other disposition indices, we identified rating disposition as a useful concept, and demonstrate its value in an online movie recommender community. We ran multiple sets of partitioning experiments limiting a model to users with the same rating disposition and explored how can we improve the recommendation models’ performance, at least for certain dispositions. The preliminary binary population model showed the benefit of building a recommender for each user group with different dispositions. We then tested with a more granular negative percentage group partitioning that had demonstrated its efficiency and strong performance boost, mainly for the User-User CF algorithms. Finally, the positive rating peak partition brought quite positive changes for the user-user CF model and also worked for certain groups in the item-item CF model, with corrected statistical significance. Compared to the results in some other previous works on de-biasing user ratings, such as the post-hoc rating adjustment technique by Adomavicius et al., in which the predictive accuracy (measure with RMSE) did not see significant improvement with de-biased adjustment in both user-user and item-item CF (Adomavicius et al., 2014), our results showed the effectiveness of population partitioning in group recommendation not only for more efficient model training with its less amount of data, but also in improving the recommendation performance with collaborative filtering algorithms.

In this study, we explored the patterns behind raw ratings from different user populations with a movie recommendation dataset. Our 3-step method involved identifying different rating dispositions, testing partitioned recommenders, and defining training subgroups. We demonstrate that the partitioned models can work more efficiently while also achieve positive impacts on User-User CF algorithms, and latent factor methods seem to be robust with rating dispositions.

This work is not meant to be an exhaustive exploration of group partitioning recommendation, but a case example to demonstrate the potential benefits and feasibility of the approach. We admit several limitations: 1) We focused on a single domain and limited to 5-point Likert scale; 2) We did not explore other similarity measures or more advanced algorithms; 3) We have not conducted interviews to understand users’ subjective rating preferences; 4) Our taxonomy of two user groups is simple and needs further refinement. Future work should confirm and expand our approach for a deeper understanding of when ratings disposition is useful and how to effectively segregate or exclude ratings.