Keen2Act: Activity Recommendation in Online Social Collaborative Platforms

We define the activity recommendation problem as follows: For a given user, which items should (s)he choose and what activities should (s)he perform on those items? The proposed Keen2Act approach addresses this joint item-activity recommendation problem by breaking it down into two sub-problems: the Keen and Act tasks. The former aims to identify the set of items that a user is potentially interested in, while the latter aims to subsequently determine the set of activities to perform on the items of interest. An outline of our proposed Keen2Act model is given in Figure 1.

We first denote a particular user, item, and activity using the notation $u$, $v$ and $z$, respectively. We also let $U$, $V$ and $Z$ denote the set of all users, all items, and all activities, respectively. For a given user $u$, the Keen component determines whether an item $v$ should be included in the set $V^{+}_{u}$ of items selected by that user, as follows:

where $\mathcal{K}(u,v)$ is the Keen ranking score for user-item pair $(u,v)$, and $\delta_{\mathcal{K}}(v)$ is the Keen decision threshold for $v$. Subsequently, the Act component determines whether an activity $z$ should be part of the set $Z^{+}_{u,v}$ of activities performed by user $u$ on a selected item $v$:

where $\mathcal{A}(u,v,z)$ is the Act ranking score for the tuple $(u,v,z)$, and $\delta_{\mathcal{A}}(z)$ is the Act decision threshold for $z$.

The main intuition behind the Keen followed by Act steps is that, prior to determining an activity $z$ on item $v$, user $u$ must be sufficiently keen in item $v$ in the first place. When there is a lack of keenness (i.e., $\mathcal{K}(u,v)<\delta_{\mathcal{K}}(v)$), the user should not perform any activity at all on item $v$. Conversely, only when the user shows a sufficient level of keenness (i.e., $\mathcal{K}(u,v)\geq\delta_{\mathcal{K}}(v)$), he/she can proceed with selecting which activities to be performed on item $v$. This two-stage decision process helps not only reduce the search space (by filtering out less relevant items) but also improve the quality of the final item-activity recommendation.

The ranking scores $\mathcal{K}(u,v)$ and $\mathcal{A}(u,v,z)$ can each be realized using any machine learning model. In this work, we choose to use a multilinear model called Factorization Machine (FM) (Rendle, 2012), which has shown good performance in a variety of recommendation tasks based on sparse data. It is also worth noting that the decision thresholds $\delta_{\mathcal{K}}(v)$ and $\delta_{\mathcal{A}}(z)$ are parameters that are learnable from data and are specific to each item $v$ and activity $z$, respectively.

Finally, a Recommendation Aggregation process takes place to combine $V^{+}_{u}$ and $Z^{+}_{u,v}$ in order to arrive at the final list $R^{+}_{u}$ of recommended item-activity pairs. More specifically, for a given user-item-activity triplet $(u,v,z)$, the Recommendation Aggregation process corresponds to a decision function $\mathcal{D}(u,v,z)$:

where $\mathbb{I}[.]$ is an indicator function and $\land$ is an AND logical operator.

We further elaborate the formulation of learning mechanisms behind the Keen and Act components in Sections 3.2 and 3.3, respectively. We then describe the Recommendation Aggregation process in greater detail in Section 3.4. Finally, we recap the overall Keen2Act learning procedure in Section 3.5.

Keen ranking. We consider the problem of identifying the parameters $\theta_{\mathcal{K}}$ of the Keen model $\mathcal{K}(u,v)$ as a learning-to-rank task, and it is thus sensible to use a loss function optimized for ranking. Among the myriad of ranking loss functions, the Weighted Approximately Ranked Pairwise (WARP) loss (Weston et al., 2011) in particular has been shown as a good criterion for recommendation tasks. An appealing trait is that WARP works for data that have only implicit feedback, making it well-suited to our problem. The key idea of WARP is that, for a given positive (observed) example, the remaining negative (unobserved) examples are randomly sampled until we find a pair of positive and negative examples for which the current model incorrectly ranks the negative example higher than the positive one. We can then perform a model update only based on these violating examples.

Adopting WARP to the context of our Keen model, we can write the loss function with respect to the Keen model parameters $\theta_{\mathcal{K}}$ as:

where $\Phi(.)$ transforms the rank of a positive item $v\in V^{+}_{u}$ into a weight. Here the $rank_{v}$ function can be defined as a margin-penalized rank of the following form:

Choosing a transformation function such as $\Phi(n)=\sum_{i=1}^{n}\frac{1}{i}$ would allow the model to optimize for a better Precision@$k$. However, directly minimizing such a loss function via gradient-based algorithms would be computationally intractable, as equation (5) sums over all items, resulting in a slow per-gradient update. We can circumvent this issue by replacing $rank_{v}$ with the following sampled approximation (Weston et al., 2011): we sample $N$ items $v^{\prime}$ until we find a violation, i.e, $\mathcal{K}(u,v)<1+\mathcal{K}(u,v^{\prime})$, and subsequently estimate the rank, i.e., $rank_{v}(\mathcal{K}(u,v))$, by $\frac{|V\setminus V^{+}_{u}|-1}{N}$. In addition to the WARP loss, we need an appropriate regularization term to control our model complexity. In this work, we employ L2 regularization, which is differentiable and suitable for gradient-based methods as well. This leads to an overall, regularized ranking loss function $\mathcal{L}_{\text{rank}}$:

where $\lambda_{\mathcal{K}}>0$ is a user-specified $l_{2}$-regularization parameter. The term $\|\theta_{\mathcal{K}}\|^{2}$ is used to mitigate data overfitting by penalizing large parameter values, thus reducing the model complexity.

Keen thresholding. Once the Keen ranking step is done, we need to determine the appropriate decision thresholds $\delta_{\mathcal{K}}$ in order to decide whether to include item $v$ into $V^{+}_{u}$. Ideally, we wish to identify a threshold such that the set of selected items matches the set of ground-truth, observed items as close as possible. However, it is difficult to learn the thresholds using such a set matching objective. We therefore relax this via the cross-entropy loss function, and for the Keen model this would be:

where $CE(x,y)=-\left[y\ln(\sigma(x))+(1-y)\ln(1-\sigma(x))\right]$ is the cross entropy function, $\sigma(x)=\frac{1}{1+\exp(-x)}$ is the sigmoid function, and $I(v\in V^{+}_{u})$ is an indicator function reflecting the ground truth for whether item $v$ belongs to the set of items selected by user $u$.

Act ranking. The ranking loss formulation for the Act model is similar to that of the Keen model except that the former deals with ranking of activities at the level of user-item pair. In particular, the WARP loss function associated with the Act model is given by:

where the rank function is likewise estimated by a sampled approximation $rank_{z}(\mathcal{A}(u,v,z))$ $\approx\frac{|Z\setminus Z^{+}_{u,v}|-1}{N}$. Accordingly, adding the L2 penalty to control the model complexity, the overall regularized WARP loss for the Act model $\mathcal{A}$ is given by:

where $\lambda_{\mathcal{A}}>0$ is the L2-regularization parameter.

Act thresholding. Similar to the thresholding in the Keen model, we can estimate the decision threshold of the Act model using the following cross-entropy loss:

where $I(z\in Z^{+}_{u,v})$ is an indicator function reflecting the ground truth for whether activity $z$ belongs to the set of activities carried out by user $u$ on a selected item $v$ (i.e., $v\in V^{+}_{u}$).

The final step in the Keen2Act model is to generate a sorted list $R^{+}_{u}$ of recommended item-activity pairs for a given user $u$, by aggregating the selected item set $V^{+}_{u}$ and chosen activity set $Z^{+}_{u,v}$ as computed by the Keen and Act models respectively. To achieve this, we generate $R^{+}_{u}$ by enumerating the selected items $v\in V^{+}_{u}$ starting from the one with the highest $\mathcal{K}(u,v)$, and then for each selected item, we enumerate the selected activities $z\in Z^{+}_{u,v}$ starting from the highest $\mathcal{A}(u,v,z)$. This results in a recommendation list whereby the item ranking takes precedence over the activity ranking.

To minimize the ranking losses $\mathcal{L}_{\text{rank}}(\theta_{\mathcal{K}})$ and $\mathcal{L}_{\text{rank}}(\theta_{\mathcal{A}})$ as well as threshold losses $\mathcal{L}_{\text{thres}}(\delta_{\mathcal{K}})$ and $\mathcal{L}_{\text{thres}}(\delta_{\mathcal{A}})$, we devise an incremental learning procedure based on a variant of stochastic gradient descent called Adaptive Moment Estimation (Adam) (Kingma and Ba, 2015). Algorithm 1 summarizes the procedure, which takes in the Keen interactions $\mathcal{I}_{\mathcal{K}}=\{(u,v):u\in U\land v\in V^{+}_{u}\}$ and Act interactions $\mathcal{I}_{\mathcal{A}}=\{(u,v,z):u\in U\land v\in V^{+}_{u}\land z\in Z^{+}_{u,v}\}$ as inputs, and outputs the model parameters $\theta_{\mathcal{K}}$ and $\theta_{\mathcal{A}}$ as well as thresholds $\delta_{\mathcal{K}}$ and $\delta_{\mathcal{A}}$.

The overall algorithm consists of two phases: rank learning and threshold learning. In the first phase, we carry out Adam to update the model parameters $\theta_{\mathcal{K}}$ and $\theta_{\mathcal{A}}$ by finding a pair of positive and negative examples that violate the desired (Keen or Act) ranking. In the second phase, we also perform Adam update on $\delta_{\mathcal{K}}$ and $\delta_{\mathcal{A}}$ by enumerating at the item and activity levels respectively.

Table 2 shows the MAP@$k$ results averaged over the 5 runs of our activity recommendation experiments. We observe that Keen2Act consistently outperforms all the other methods. Specifically, Keen2Act outperforms FM_BPR and FM_WARP by 175% and 29% respectively in Stack Overflow, and 94% and 28% respectively in GitHub. As we increase $k$, we also notice deterioration in MAP@$k$ results. This can be attributed to the Precision@$k$ metric favoring a model that outputs a ranked list with the relevant (i.e., observed) item-activity pairs leading the list. That is, as $k$ increases, it is likely that more and more irrelevant item-activity pairs would appear in between the relevant item-activity pairs, pushing the Precision@$k$ lower.

Additionally, we can see that the improvement of Keen2Act over the baselines is greater in Stack Overflow than in GitHub. A possible reason is due to the sparsity of user activities. More specifically, GitHub users are observed to perform more activities concentrated on a small set of items (i.e., denser user by item-activity interaction matrix), whereas Stack Overflow users tend to perform fewer activities spread across a large set of items. The denser interaction matrix for GitHub allows the baseline methods to have sufficient observations to achieve competitive results.

Comparing the different variants of our proposed model, we can see that Keen2Act outperforms the Keen model, suggesting that it is inadequate to learn only the item-level interests of the users when recommending activities. Keen2Act also outperforms the Act model, which demonstrates the importance of the Keen step in learning the item-level interests before activity-level interests. It is also interesting to see that the Keen model performs fairly well in comparison to the other methods. We can attribute this to reduced sparsity in the problem space it is operating at, i.e., the Keen model only makes item-level recommendation while the other methods recommend at the activity level. Finally, it is worth noting that the MAP@$k$ scores of various models are generally low, showcasing the complexity of the activity recommendation problem.