Exploring Popularity Bias in Session-based Recommendation

In this section, we will explain in detail our methodology of calculating the propensity scores, based on which we stratified our data in the initial experiments. As stated in Jadidinejad et al. (2021), a propensity score $p_{i,u}$ is defined as the probability that the deployed model exposes item i to user u under a closed loop feedback scenario. Since the true probability for a model exposing some items is always unknown to us, we will estimate the propensity score using the raw observations from the dataset with a method proposed by Yang et al. (2018). Based on their framework, we introduce the following assumptions and observations.

Assumption 1: the propensity score $p_{i,u}$ is user-independent. This assumption was first made in Yang et al. (2018) to address the lack of auxiliary user information in many user-item interaction records. We will keep this assumption in our method because user information is usually not utilized anyways under a session-based context. With this assumption, we are able to write the propensity score of a certain item as

$$p_{*,i}=p_{*,i}^{select}\cdot p_{*,i}^{interact|select}$$

(1)

where $p_{*,i}^{select}$ is the probably that item $i$ is recommended and $p_{*,i}^{interact|select}$ is the conditional probability of the user interacting with item $i$ given that it is recommended.

Assumption 2: $p_{*,i}^{interact|select}=p_{*,i}^{interact}.$ That is, a user’s preference over items is not affected by what is recommended to her. That makes the probability that a user will interact with an item conditioned on it being recommended equal to the true probability that a user will interact with the item. Since this probability is user independent, it is proportional to the item’s true popularity $n_{i}$:

$$\hat{p}_{*,i}^{interact}\propto n_{i}$$

(2)

Assumption 3: $p_{*,i}^{select}$ is proportional to $(n_{i}^{*})^{\gamma}$, where $n_{i}^{*}$ is the number of observed interactions. This assumption comes from a common template of popularity bias introduced in Steck (2011) based on empirical observations.

With one more observation that $n_{i}^{*}$ is sampled from a binomial distribution parameterized by $n_{i}$, we are able to write the estimated propensity score of an item as

$$\hat{p}_{*,i}\propto(n_{i}^{*})^{\frac{\gamma+1}{2}}$$

(3)

In the following section, we used this method to calculate the propensity score for each item in the dataset, with one caveat: instead of stratifying our dataset with one item as a basic unit, we need to calculate context-aware propensity for each action that aggregates the item propensity of all previously seen items in the current session.

We follow the evaluation metrics in Ludewig et al. (2020) to evaluate each model’s performance on stratified test data. We will focus on the following main metrics, since non-neural network models consistently outperform neural network models in terms of them.

Hit Rate (HR) is the percentage of actions where the model’s predicted item is the target item.

$$HR=\frac{|A_{hit}^{L}|}{|A_{all}|}$$

(4)

Mean Reciprocal Rank (MRR) measures the relevance of recommendations by taking reciprocal rank of each recommended item and averaging over all actions.

$$MRR=\frac{1}{|A_{all}|}\Sigma_{u=1}^{|A_{all}|}RR(a)$$

$$RR=\Sigma_{i\leq L}\frac{relevance_{i}}{rank_{i}}$$

There are mainly two models involved in our work. The SKNN represents nearest neighbor based models and GRU4REC Hidasi et al. (2016) represents the deep-learning based models.

For our experiments, we adopt two e-commerce recommendation datasets, Diginetica and Retailrocket. These datasets are of the same domain and moderate size. Additionally, we adopt 30MU dataset that contains the music listening log obtained from Last.fm to diversify the domain of the our experiments. The specifications are given in Table 1. Compared to the other two datasets, 30MU generally has longer sessions and more involved items. Following previous work, we remove sessions with length one, as well as items that are interacted with less than 5 times.

To answer the first question of whether such strata exist in session-based recommendation system, we leverage the evaluation method proposed in Jadidinejad et al. (2021). The evaluation method calculates propensity score for each item and stratifies each action by context-aware propensity.

To run the approach to session-based recommendation system datasets, we design two methods to calculate score for each action in a session. And we use these two methods to split the evaluation data into two parts, Q1 (actions with low score) and Q2 (actions with high score).

Stratification based on the propensity of the target item. In this method, we directly use the propensity of the target (ground truth) item. But in reality, our algorithm is not supposed to have access to an action’s target item until the evaluation of this action is done. So, this is an ideal method but somehow violates the evaluation protocol by peeking into the future.

Stratification based on the propensities of historical items in the session. In this method, instead of using the information from the target item, we use the information of all previous seen items in the session. A straightforward way to aggregate these items is to take the average of the propensity scores of the historical items in the session. Here, we make the assumption that this metric is correlated to the propensity of the target item. (In Figure 5, we provide a visualization to demonstrate the empirical correlation between the metrics for each dataset).

For our experiments, we used SKNN as our KNN-based models and GRU4REC as the deep-learning based model. We decided to include them in our experiments because VKNN model itself has long been favored by practitioners We used GRU4REC as the neural-net model for evaluation because as it is one of the pioneers and often served as a strong baseline for new neural network models in the field.

Our training and inference framework were primarily inherited from the Python framework session-rec, we implemented additional propensity-score based ensemble models, inference scripts and rewritten parts of evaluation classes to suit our evaluation goals better. The hyper-parameters we have chosen for each model were mainly coming from Ludewig et al. (2020) and are listed here:

1. 1.

   For SKNN, we used k=100, sample size of 500 and cosine similarity.

2. 2.

   For GRU4REC, we used the bpr-max loss function (Hidasi et al., 2016), dropout rate of 0.3, learning rate of 0.03 and momentum of 0.1. The learning algorithm was Adagrad.

We first visualize the item propensity for each dataset to grasp their characteristics. The calculation of item propensity is based on Eq. 3 and the power law algorithm is fitted independently for each dataset.

We show the distribution of log item propensity for the three datasets in Fig. 2. Since we use the power law, the range of the resultant item propensity is similar in different datasets. However, the distribution of Diginetica and RetailRocket is more spread out and that of 30MU is more centered toward the lower end.

We stratify evaluation data based on action-wise propensity into varying proportions and report the results of different models on different strata.

Specifically, we calculate the action-wise propensity score using the two methods elaborated in Sec. 4.2. And we experiment with different $x$ values such that the $x\%$ percentile of propensity score value split the whole evaluation dataset into two portions, Q1 and Q2. We then evaluate two different models, SKNN and GRU4REC on both portions to analyze the results of different models on actions with different propensities.

The results for Dignetica are shown in Fig. 3 and Fig. 4. Due to space limit, we move the results for RetailRocket and 30MU to the Appendix (Sec .1). For e-commerce datasets (Dignetica and RetailRocket), as the percentile decreases, which means the average popularity of items decreases, GRU4REC clearly gains advantage over SKNN. However, in 30MU, there is no such trend and the gap between the two models is consistent across different $x$.

Additionally, we visualize the correlation between the two stratification methods, shown in Fig. 5. There is strong correlation in 30MU dataset and low correlation in the two e-commerce datasets. However, the correlation is not necessarily related to the experimental results we achieve- even for 30MU, the trends from the line plot in Fig. 8 & 9 are not alike for these two stratification methods.

To examine the robustness of deep learning models against unpopular items, we take the S1, the set of actions with bottom 10% propensity score, and S2, the set of actions with top 10% propensity score. We evaluate the performance of the two models on each set, and compute the ratio between score of S1 and score of S2. The result is shown in Table 2. GRU4REC has higher ratio compared to SKNN for all datasets, suggesting that it is more robust against unpopular items.

Specifically, for each action, we set a threshold based on our analysis of the action-wise propensity score distribution, which is dataset-specific. We assign different weights to the prediction scores of GRU4REC and SKNN model and use the weighted average as the final prediction score of the action. The weights and the threshold are hyper-parameters. We use a symmetrical weight assignment approach where the weights for GRU4REC and SKNN are reversed in two strata.

The ensemble method clearly benefits e-commerce datasets, bringing $~{}10\%$ relative improvement over HitRate@20 on Dignetica and $~{}6\%$ over HitRate@20 on RetailRocket. And the gain is not sensitive to the ensemble weights. On the 30MU dataset, although the GRU4REC model outperforms SKNN on every strata, the ensemble methods still benefit from incorporating SKNN’s predictions.

In the previous method, we fix the weight of different models on different propensity strata, where all the actions in the same strata, regardless of its real propensity score, will be assigned the same set of weights for ensembles. Therefore, we propose the dynamic-weight ensemble method where the weights of each action are dynamically computed using their propensity score. This ensures each action gets a set of weights that better reflects its propensity. The weights $w_{1},w_{2}$ are computed with the following formula, denote the propensity score of action $i$ as $p_{i}$:

$$p_{normalized,i}=\dfrac{log(p_{i}+10^{-9})-\hat{p}_{mean}}{\hat{p}_{std}}$$

(5)

$$w_{1}=\dfrac{1}{1+e^{p_{normalized,i}+\alpha}},w_{2}=1-w_{1}$$

(6)

The formula is essentially an x-shifted Sigmoid transformation on the negation of the log-normalized propensity score. The hyper-parameters are $\alpha,\hat{p}_{mean}$ and $\hat{p}_{std}$. $\hat{p}_{mean}$ and $\hat{p}_{std}$ are the approximation of the mean and standard deviation of the actions in the entire test dataset. It can be approximated using the actions in the training set, since the items in two sets are completely overlapped. The $\alpha$ governs how much does the Sigmoid function shifts in the x direction, it is optimized using grid search.
The results of our dynamic-weight ensemble methods is shown in table 4. It is compared with three other fixed-weight ensemble methods. We could see that the dynamic-weight method gives the highest HitRate among all the methods but the MRR metric did not improve as much.