Quantifying the Effects of Recommendation Systems

The actual data was organized in a matrix where the rows represented the users and the columns were the jokes. There was an additional column designated to record the number of jokes a user rated. Users had rated at least 36 jokes. For the jokes that were unrated, their values were 99 rather than leaving them blank. The graph in Figure 2 shows the popularity based on the real dataset. The shape of the graph in Figure 2 strengthens the concept of popularity bias and “long-tail” items mentioned in the beginning. It shows that bias is already present before conducting our experiment, which means there could have been recommendation inequality using Eigentaste.

Dataset 1 has no correlation between the number of ratings for a joke and its mean rating, as shown in Figure 3. More ratings for a joke does not necessarily mean it has a higher mean rating. This gives an idea of how Eigentaste made recommendations. Since it is a CF algorithm, it was focused more on recommending based on a higher mean rating rather than a high number of ratings (which could contain high and low value ratings) for a certain joke.

These are the 2 cases we used for experimental analysis.

Case 1: single training. For every test user, if the recommended joke does not have a rating, it is labeled as “no rating,” but the joke ID is still recorded. Unlike Case 2, any ratings and data on each test user are not incorporated into the training set.

Case 2: repeated training. For every test user, if the recommended joke does not have a rating, we move to the next joke with the second best recommendation score. If we do not find a joke with a rating within the top 3 recommended jokes, then we simply do not have a rating for the test user. The new inputs that are included in the retraining of the training set are the ratings of the random jokes during the profiling phase and the rating of the recommended joke. For example, if user Y was given 10 random jokes to rate and was given 1 recommendation afterwards, then the ratings of those 11 jokes are incorporated into the training set for retraining. We set $k=100$, where new inputs are incorporated only after every $k$th iteration of an algorithm.

In this section, we will lay out the general procedure for conducting our experiments.

Reorganize data and filter users who have rated at least 50% of the jokes. Little data processing and cleaning was needed other than adding labels to categorize the data for easy manipulation. We used about 20,000 users who had rated at least 50% of the jokes to decrease the chance of not having a rating for a joke that could be recommended through our system. Since we are only using the data from Dataset 1, any rating predictions we make are used as a guide and are not treated as actual data as they will not be true to the user. The analysis we are performing is reliant on true data from the user so we can preserve their true preferences that are conveyed through the previously rated jokes. Because of this aspect, we attempt to lower the chances of making a recommendation that has no rating data by taking users who have rated at least 50% of the jokes.

Divide users into 2 groups: training set and test set. For the training set, we randomly picked 500 users for a smaller training set to serve as the starting point since for the filtering methods, we need data in the system first before we can make recommendations. For the test set, we randomly chose about 4000 users from the data to represent the “new users” being added through our system. As we run this experiment multiple times, we randomly change the training and test set with different users each time.

Create 2 different cases for filtering methods. Case 1 is single training (static) and Case 2 is repeated training (dynamic). To quantify the effect of the feedback-loop in recommendation systems, we compare dynamic training vs a one time static training of the recommendation methods. Both cases will start with 500 users from the training set. In single training, the number of users in the training set stays constant during the whole experiment. Static training will only utilize ratings of the static set to recommend jokes to the test users (“new users”). In repeated training, the starting set of users will be updated, thus the number of users in the system will not stay constant during the experiment. After every 100 recommendations, rating data from test users are incorporated into the starting set and their data will be considered in the calculation when making recommendations for the rest of the test users. Dynamic training mimics how real online platforms update data. Because of how often new users enter the system, the data that the model is making predictions on must have a similar distribution as the data on which the model was trained in order to make accurate predictions. Data distributions can be expected to drift over time, so retraining a model on newer data is common in online platforms since the data distribution could have deviated significantly from the original training data distribution [15]. Online platforms retrain their models periodically to ensure the accuracy of their recommendations. By using multiple methods, we are able to see the inequalities that are present and make comparisons about the impact of retraining on the system.

Run Case 1 and Case 2 each multiple times on 10 jokes for the filtering methods. The gauge set in the data consisted of 10 jokes that every user in Dataset 1 has rated. This serves as the common set of jokes used to profile the users before making recommendations. We run the algorithm to simulate the action of giving the test users 10 jokes from the gauge set to rate. We follow the same procedure for both cases for the above methods. Based on those ratings, we compare the test user to the current database of users and its predicted ratings to generate a recommendation.

Record which jokes were recommended, the order of recommended jokes, and their ratings. For each test user, only 1 recommendation is made. We record the joke ID and the rating of the recommended joke for each test user for both static training and dynamic training. We also record the order of the recommended jokes as it is needed to calculate the inequalities overtime. For the jokes that have no rating, we categorize them as “no rating,” but they are still part of the recommendation order and inequality calculation. For dynamic training, since we are retraining, it is ideal to obtain a recommended rating rather than no rating. More details will be discussed in the next section on how the algorithm works for the 2 cases.

Use the Gini coefficient to measure the inequalities overtime. To measure the inequalities, we calculate the frequencies of joke i (x_i) and joke j (x_j) being recommended out of n number of recommended jokes. These frequencies are then inputted into the equation defined below to calculate the Gini coefficient G:

The value output from the formula is between the range of 0 and 1— 0 being perfect equality and 1 being maximal inequality. During the experiment, we calculate the Gini every 100 recommendations to produce a visualization of the inequality overtime.

For this method, we expect the Gini coefficient to be a low value because the randomness of the recommendations should give each joke an equal chance at being recommended. This means that all 100 jokes should have been recommended during the experiment. The jokes are treated the same in quality, which means their ratings have no advantage when using the random method. Thus, there is very little recommendation inequality as the Gini coefficient is low. Although, there is little inequality, it is imbalanced in terms of joke quality. Chances of getting poor recommendations is high, so it is important to find a balance between recommendation inequality and joke quality. A recommendation system would not be effective if users get recommended items that do not match their preferences. Using random filtering methods increases the chances of getting poor recommendations since they are not personalized.

Our popularity method takes the most popular jokes, the ones with the highest average rating, and recommends one of them to the test users. In other words, a joke’s probability of being recommended is proportional to its rating. This means that if a joke has a high probability of being recommended, then it has a high mean rating. We expect this type of popularity method to have the highest inequality because it only recommends a select amount of jokes that have high probabilities and high mean ratings. The Gini coefficient for this method is the highest out of all the filtering methods. However, if there are many popular jokes, such that among the 100 jokes, the majority had similar ratings that exhibit their popularity, then the inequality would not be as obvious. In the real world, recommendation systems do not always recommend only a few popular items. Depending on the platform and the amount of user data it has, popularity could span to a majority of items, which could lower the overall recommendation inequality.

We will not be focusing on the popularity method with maximal inequality mentioned previously as most online platforms would not use this type of algorithm. Instead, we will be using these two popularity methods: 1) one that uses the joke’s mean rating scaled to an exponent of 1 and the other that uses the joke’s mean rating scaled to an exponent of 2. The methods are defined by the equation $P_{j}$ below:

$P_{j}$ is the probability of joke $j$ being recommended, $r_{j}$ is the mean rating of joke $j$, $k$ is the exponent, i.e 1 or 2, and $\sum_{i=1}^{n}(r_{i})^{k}$ is the sum of the scaled mean ratings of $n$ jokes. Scaling it to exponent 2 increases and exaggerates the probability of a joke being picked based on its popularity, making it even more likely to pick a popular joke but not restricting it to only a select amount of popular jokes.

We modified a simple CF Pearson correlation algorithm written in 2012 by Wai Yip Tung [17]. The main idea is that we compute the similarity score between a user X from the training set for all users and a user Y from the test set. Using that similarity score as the weight, we calculate a value, in which we label as the similar rating score, by combining the weight with each joke rating. We repeat this with all users in the training set. In other words, each test user will have at least 500 similarity scores, one for each user in the training set, and 100 similar rating scores, one for each joke. Although 100 similar rating scores are given, we only use at most 90 ratings since we exclude the gauge set. In the original dataset from Jester 5.0, the gauge set was not part of the recommendations so we excluded it so our recommendation process would be similar to Jester. The similarity score is calculated using Pearson correlation $\rho$X,Y, which is defined as:

Covariance cov for n total ratings is

where X_i are the ratings of user X, $\bar{X}$ is the mean of X_i, Y_j are the ratings of user Y and $\bar{Y}$ is the mean of Y_j. The covariance is the joint variability of the 2 users. Pearson correlation gives a more accurate range (from 0 to 1) on user similarity. Close to 0 means that users have different preferences and close to 1 means users have similar preferences.

To calculate and normalize the recommendation score for each joke, we divide the sum of the similar rating scores by the sum of the similarity scores. We sort the jokes with the highest recommendation score and exclude any recommended jokes that are part of the gauge set. The joke with the highest recommendation score is recommended. There are some cases where the top recommended joke does not have a rating recorded in Dataset 1. This is because that our CF algorithm behaves differently and would not necessarily give the exact same recommendations as Eigentaste. Nevertheless, our algorithm handles that aspect differently in both cases.

For MF, we modified another simple algorithm by Nick Becker in 2016 [2]. The process of using the training set and test set is similar to the CF Pearson algorithm, but the way it creates recommendations is different. This algorithm utilizes singular value decomposition (SVD), which decomposes a matrix $R$ to a smaller approximation of the original matrix $R.$ Mathematically, it decomposes $R$ into two unitary matrices and a diagonal matrix:

where $R$ is a user ratings matrix, $U$ is the user “features” matrix, $\Sigma$ is the diagonal matrix of singular values (weights), and $V^{T}$ is the joke “features” matrix. $U$ represents how much users “like” each feature and $V^{T}$ represents how relevant each feature is to each joke. To get the lower rank approximation, we take these matrices and keep only the top $k$ features, which we think of as the $k$ most important underlying taste and preference vectors [2]. In other words, these features are not explicitly passed when computing the recommendations. It is through matrix multiplication that it had generated some hidden “features” and picked up underlying preferences of users. After multiplying the 3 matrices, we get prediction ratings for each user. If prediction ratings for a user are high for certain jokes, then we recommend those jokes to the user.

We use this model to compare against the real preferences of people. It serves as the baseline for what people actually like, which encourages recommendation systems to improve their frameworks to meet people’s expectations. In the optimal method, the Gini coefficient stabilizes at a value very quickly. Optimal does not necessarily mean perfect equality, but in a sense that all users like the items being recommended to them. Note that this is very difficult in the real world. Recommendation algorithms will recommend items close to a user’s preferences, but they are not always accurate. There are times where users will reject the recommendations because they did not match their interests.

Since we defined the optimal case as having all users like the items being recommended, we assume those items have the highest ratings. Thus, the method is defined as:

where $S_{u}$ is a set of ratings for each user $u$. There are a total of $n$ joke ratings for each user. To find the optimal joke $o_{u}$ for user $u$, we find the highest or maximum rating in $S_{u}$, which is defined below:

.

Unlike the cases for single training and repeated training, all recommended jokes will have ratings recorded previously because we are taking the highest rating that exists in Dataset 1 for each test user. For each user in this experiment, we find the joke that has the highest rating and record the joke ID and its rating. From here, the procedure for finding the Gini is the same as the above cases.