Can One Embedding Fit All? A Multi-Interest Learning Paradigm Towards Improving User Interest Diversity Fairness

User interest diversity aims to measure the dissimilarity of the items interacted with each user in the training data (i.e., users’ historical interactions). Based on whether category information is available, we define interest diversity based on item category or item embedding.

Essentially, $\text{D}_{\text{cat}}(u)$ measures the probability that two randomly sampled items are from different categories, and $\text{D}_{\text{emb}}(u)$ measures the dissimilarity between the interacted items in their embedding space. For both scenarios, a larger value indicates a higher level of interest diversity. Unless specified, we use $\text{D}_{\text{cat}}$ for default.

Given users’ interest diversity, we group users with k-means clustering (MacQueen et al., 1967). The number of clusters is determined using the commonly-used elbow method (Thorndike, 1953). The assignment of clusters subsequently defines the group partition, with a higher group ID indicating a higher diversity of interests. It’s worth noting that there are alternative methods to group users, e.g., dividing users into equal sized groups based on number of users, or range of user interest diversity. Unless specified otherwise, we primarily rely on k-means clustering in the experiments.

Given the exceptional performance of utilizing graphs in RSs, we select two graph-based models for evaluation: LightGCN (He et al., 2020) and CAGCN^∗ (Wang et al., 2023b). The former is a widely recognized and frequently used model. The latter is a newer development and improves the overall utility by reducing the emphasis of neighbors not adhering to the main interest which is closely related to our topic.

We evaluate them on four datasets including ml-1m, epinion, embmetics, and anime, the details of which will be described in Sec. 5.1.1. The preliminary results across different scenarios are illustrated in Fig. 2. Specifically, Fig. 2(A) is the group utility performance (Recall) where groups are divided based on k-means clustering with $\text{D}_{\text{cat}}$ as the diversity metric. The curves suggest a trend that as interest diversity increases, the group utility performance generally decreases. This pattern is observable across multiple datasets and models. We also explore another diversity definition $\text{D}_{\text{emb}}$ in Fig. 2(B) which shows a similar trend. Additionally, we obtain results based on different group partitions including the equal user number and equal user interest diversity range in Fig. 2(C). The results show a consistent trend across various datasets, models, diversity metrics, and group partitions that users with diverse interests generally receive a lower recommendation quality. This indicates the existence of user interest diversity unfairness, which jeopardizes the user experience for user with diverse interests.

Fig. 5 shows the multi-interest framework where each user/item has different types of embeddings, including (1) center embeddings $\mathbf{E}_{C}^{l}\in\mathbb{R}^{N\times d}$ representing users/items main characteristic/features where $N$ is the number of node (including users and items) and $d$ is the dimension; (2) interest (virtual) embeddings $\mathbf{E}_{V}^{l}\in\mathbb{R}^{N\times K\times d}$ which relate to specific interests where $K$ is the number of interests (for simplicity, we denote $\mathbf{E}_{V}$ as virtual embeddings hereafter). Among these embeddings, center embeddings are learnable parameters while the virtual embeddings are calculated based on center embeddings via attentions. This mechanism avoids introducing a large number of learnable parameters by sharing the global interest $\mathbf{w}_{k}^{l}$ in the attention mechanism. We represent the $k$-th virtual embedding of node $v_{u}$ as $\mathbf{E}_{V}^{L}[v_{u},k]$ and the user center embedding as $\mathbf{E}_{C}^{L}[v_{u}]$. Similar notations apply to the item side. We illustrate the motivation of using multiple embeddings for items in Appendix C.

With these notations, the framework is as follows: (1) Given the user-item bipartite graph, user and item embeddings are obtained through the multi-interest representation layers (details in Sec. 4.2);(2) After obtaining the embeddings, the relevance score $\hat{y}_{ui}$ for user, item pair $(v_{u},v_{i})$ is calculated based on the last layer representations where L is the number of hops:

(3) These predicted relevance scores are optimized via Bayesian Personalized Ranking Loss (BPR) loss (Rendle et al., 2012) $\mathcal{L}_{\text{BPR}}$.

Note that the relevance score in Eq.(4) is different from the calculation in previous recommendation models (Wang et al., 2023b; He et al., 2020) or multi-interest-based session recommendation (Zhang et al., 2022; Cen et al., 2020). In previous works, because user and item only have single embeddings, the dot product between the user and the item embedding (i.e., ${\mathbf{e}_{u}}^{\top}\mathbf{e}_{i}$) denotes their relevance score. In multi-interest based session recommendation, only items have learnable parameters and users/sessions are calculated based on items ($K$ embeddings with $\mathbf{e}_{u}^{k}$ denoting the $k$-th interest) and therefore $\max_{k=1}^{K}{\mathbf{e}_{u}^{k}}^{\top}\mathbf{e}_{i}$ is sufficient to update the item embeddings. However, similar to LightGCN, we have both user and item embeddings to learn. Simply optimizing the traditional multi-interest relevance score that is commonly used in session-based representation cannot utilize user embeddings, indicating it is not suitable in our case. Therefore, we use the symmetric scores shown in Eq.(4) where both user and item embeddings are optimized.

Next, we introduce the details of multi-interest representation layer, which is at the core of the architecture and designed to learn, calculate and aggregate multiple embeddings. The model is composed of stacked layers to deliver the final user and item embeddings.

We utilize the BPR loss (Rendle et al., 2012) ($\mathcal{L}_{\text{BPR}}$) to train our multi-interest RS.

where $\mathcal{D}=\{(u,i,j)|u\in\mathcal{U}\land i\in\mathcal{I}_{u}^{+}\land j\in\mathcal{I}_{u}^{-}\}$ is the training dataset and $\mathcal{U}$ is the total user set, $\mathcal{I}_{u}^{+}$/$\mathcal{I}_{u}^{-}$ are the item sets that user $u$ has/hasn’t interacted with. $\sigma(\cdot)$ is Sigmoid function. $\Theta$ denotes the model parameter with $\lambda_{\Theta}$ controlling the $L_{2}$ norm regulation to prevent over-fitting. $\hat{y}_{ui}$ is the predicted preference/relevance score computed based on Eq.(4).

We present the utility and fairness scores in Table 3 (The results based on another utility metric NDCG is included in Appendix G). Since the standard deviations for all methods across various seeds are negligible compared with the main performance, we leave them out. From the result, we draw several observations:

• •

  The multi-interest framework has the best fairness-utility trade-off in general. Our proposed method achieves the best and runner up performance in most of the times when compared with other methods. Upon calculating the average rank for each method, ours emerges as the leader in both backbones. While the current rank of $1.92$ indicates some room for enhancement towards the optimal rank of $1$, it underscores the efficacy and potential of the multi-interest framework in balancing fairness and utility.

• •

  The multi-interest framework works better with large dataset. In cosmetics dataset, which has the highest count of items and categories, our method consistently delivers enhanced performance in both fairness and utility. Given the diversity of items and categories, learning varied interests becomes more essential, amplifying the advantages. We dive into more details about the impact of items and categories in Appendix H.

• •

  The multi-interest framework is more stable across backbones compared with other fairness baselines (i.e., DRO and ARL). DRO and ARL rank higher than the base model CAGCN^∗, however, their rank drops when integrated into LightGCN. This underscores the complexity of maintaining an optimal balance across different models. Furthermore, such distinct performance variations of DRO and ARL across different backbones can be attributed to their inherent design. These methods were specifically designed to enhance the performance for instances with suboptimal recommendations. While Fig. 2 demonstrates that the user group with diverse interests has the poorest average performance and is expected to gain the most, other factors, such as the percentage of under-performing users in each group, play a role. If other groups have a higher proportion of users with poor recommendations, they might obtain greater benefits, thereby increasing the unfairness. Therefore, we can observe in some cases (e.g., DRO in ml-1m with LightGCN backbone) that the utility improves and the fairness drops. Such percentage in each group can vary across models, resulting in high instability of DRO and ARL due to their heavy reliance on the performance distribution. This suggests that DRO and ARL are not universally effective in the current context. In contrast, the multi-interest framework relies on the underlying interests rather than performance, which is more closely related to the current setting and more stable.

Multiple embeddings are expected to learn a better embedding distribution compared with single embeddings (Fig. 4), especially for the embedding alignment between user and interacted items. To evaluate this, we calculate the average alignment based on the backbones and their multi-version. Table 4 shows that multi-interest improves the alignment consistently. This suggests that the framework effectively brings users and their interacted items closer in the embedding space. However, an intriguing observation arises when examining performance metrics. While the improved alignment in CAGCN^∗ leads to superior utility performance compared to LightGCN in Table 3, the enhanced alignment in the multi-version does not always result in better utility performance relative to the backbones. This inconsistency may arise from the trade-off between alignment and uniformity (Wang et al., 2022a). Specifically, while alignment improves, it could lead to reduced uniformity in the multi-version due to more user embeddings, which offsets the anticipated enhancements. The nuanced interplay between alignment and uniformity, and strategies to effectively balance them, present intriguing avenues for future exploration in multi-interest scenario.

Beyond evaluating overall alignment, we delve into embedding alignment at the group level. In Fig. 6, there’s a discernible trend when comparing the backbone to its multi-version: alignment appears more evenly distributed across different groups. Since alignment is closely related to the utility performance, it contributes to a fair recommendation across groups, which follows our expectation.

For each user, our multi-interest framework initially assigns the same number of interests (i.e., $K$). Given the underlying assumption that users exhibit varied levels of interest diversity, can the model autonomously adjust the number of interests even if it begins with an equal allocation? To answer this question, we obtain the set of interests that matches the recommended items (i.e., for each item, the matched interest is the specific interest that has the maximum relevance score) and calculate the average matched interest number for each group. Results in Fig. 7 show that for the first three groups, users with more diverse interests have been assigned a larger interest number, indicating that our model has the ability to distinguish different interest diversity and can automatically cater to user preferences to some extend. However, the trend for the last two groups is not consistent, which leave us a future direction to explicitly assign interest number based on user interest diversity in addition to the current implicit way.

We measure the diversity of the recommended item sets. The results are presented in Table 5 based on two diversity metrics: $\text{D}_{\text{cat}}$ in Eq.(1) and $\text{D}_{\text{emb}}$ in Eq.(2). First, the cosmetics dataset, which has the highest number of categories among the datasets, consistently exhibits the greatest diversity in comparison to the other datasets. Second, CAGCN^∗ has a slightly reduced $\text{D}_{\text{cat}}$ than LightGCN. This is attributed to CAGCN^∗’s mechanism: it assigns higher pre-computed topological-based weights to neighbors that are more densely connected to the center node (i.e., nodes that are topologically more similar). While certain nodes gain emphasis, others get overshadowed. This reduces the likelihood of recommendations based on less-similar users, resulting in the drop in diversity. Third, multi-CAGCN^∗ has a consistent diversity enhancement (in both $\text{D}_{\text{cat}}$ and $\text{D}_{\text{emb}}$) compared with the backbone (with enhancements in $7/8$ cases). We hypothesize that CAGCN^∗ learns more accurate user interests and incorporating higher-quality embeddings amplifies the advantages of our multi-interest framework.

There are two hyperparameters in the model: the number of interests and the number of hops. From Fig. 8, we draw the following observations. A larger interest number could contribute to the utility performance but not necessary maintain a higher performance. This could be due to the increasing learning difficulty and over-fitting risk. Our multi-model prefers a smaller hop since (1) the multi-interest representation layer in Sec. 4.2 aggregates neighborhood information, serving as an implicit hop; (2) more layers would result in a higher level of smoothness which hides the diversity.

The majority RS development is concentrated predominantly on utility performance enhancement. However, emergent concerns regarding the equitable treatment of diverse user groups have motivated the advent of fairness-aware recommender systems (Li et al., 2022; Zhao et al., 2023; Wang et al., 2023a). Researchers have divided users into groups and investigated the group-level unfairness based on various criteria which can be summarized into two primary categories (Zhao et al., 2023): (1) explicit features, which involve sensitive features such as gender (Deldjoo et al., 2021; Wu et al., 2021), race (Gorantla et al., 2021; Zhu et al., 2018) and age (Farnadi et al., 2018); (2) implicit features, which are extracted from interactions such as the number of interactions (i.e., degree) and the amount of purchases (Li et al., 2021; Wu et al., 2022; Fu et al., 2020; Rahmani et al., 2022). While the explicit features are vital to fairness discourse, they are often inaccessible due to privacy policies or users’ reluctance to share such information. Consequently, our research focuses on implicit features given the profusion of user interactions in recommendation scenarios. Despite the significance of all previously mentioned features, our study explores a novel perspective within the realm of implicit features called user interest diversity considering its close relationship with the RS goal and its high relevance to real-world applications. Additionally, while most works adopt group information during training (Wang et al., 2022a; Li et al., 2022), recent works have also explored group-agnostic directions with the assumption that group partitions are not available during the optimization (Lahoti et al., 2020; Hashimoto et al., 2018). In this work, we follow this setting considering there are various ways to divide users into groups. Our goal is to develop a model that upholds fairness across diverse group divisions rather than catering to specific partition.

The main idea of multi-interest solutions is that single embedding is insufficient to represent node’s features, hence necessitating the deployment of multiple embeddings. This idea has been extensively employed in session-based recommendations - depending on how the interests are obtained, the solutions fall into attention-based and category-based methods. The attention-based methods extract interests from the interactions into interest embeddings based on the attention mechanism. MIND (Li et al., 2019) initializes the effort to extract interests based on dynamic capsule routing. After that, ComiRec (Cen et al., 2020) leverages self-attention to learn multiple interests. While ComiRec (Cen et al., 2020) considers the item-to-interest relationship, Re4 (Zhang et al., 2022) models interest-to-item relationship by adding regularizations. The cluster-based methods perform clustering on the interacted items and obtain representative embedding per cluster to depict interests. PinnerSage (Pal et al., 2020) clusters interacted items with Ward hierarchical clustering method (Ward Jr, 1963), and utilizes the embedding of the center item, which minimizes distance sum to other items within the cluster, to depict user’s interests. MIP (Shi et al., 2022) assigns each interest as representation of the latest interacted item in each cluster. Additionally, MIP learns weight to represent preference over each interest and integrates it into the relevance score.

Beyond their application within RSs, multi-interest idea has also been applied in other representation learning tasks. For instance, the multi-interest-based random walk (Park et al., 2020; Yan et al., 2021) assigns each node a target embedding along with multiple context embeddings. Similarly, in the multi-interest-based Graph Neural Network (GNN)(Choudhary et al., 2022; Yan et al., 2023a), each node is characterized by several embeddings and an additional membership embedding that signifies the association with each interest. The principle of node partitioning(Epasto and Perozzi, 2019) has also been adapted to accommodate multi-interest strategies, where a node is divided into several virtual nodes based on neighborhood structure, whose embeddings represent the original node (Yan et al., 2023b).

In contrast, we delve into multi-interest in direct recommendation, emphasizing the importance of learning both user and item embeddings. Notably, in contrast to numerous studies (Epasto and Perozzi, 2019; Choudhary et al., 2022; Park et al., 2020; Yan et al., 2022) that increase parameter size for user representation, we employ shared global interest parameters for all users. This approach allows us to compute virtual interests in a parameter-efficient manner.