Disentangling Long and Short-Term Interests
for Recommendation

Motivated by recent works (Zhao et al., 2018; Yu et al., 2019; An et al., 2019; Lv et al., 2019) that learn LS-term interests separately with two different models, we design two separate attentive encoders, $\phi$ and $\psi$, to capture the two aspects, respectively. First, we generate query vectors for LS-term interests as follows,

where we use a look-up embedding table and a Gated Recurrent Unit (GRU) (Chung et al., 2014) to capture different dynamics over time. In order to impose extra self-supervision on embedding similarity, all the embeddings need to be in the same semantic space. Thus, we use the historical sequence of items as keys of the attentive encoders, thus the obtained LS-term interests representations are in the same item embedding space as follows,

where $\bm{u_{l}^{t}}$ and $\bm{u_{s}^{t}}$ are the learned representations of LS-term interests. We now introduce the proposed encoders for LS-term interests.

Figure 2 (B) illustrates the proposed long-term interests encoder $\phi$. We use attention pooling to extract long-term interests representations, and the attention score of each item $x^{u}_{j}$ can be computed as follows,

where $\bm{W_{l}}$ is a transformation matrix, $\tau_{l}$ is a multi-layer perceptrons (MLP) network, and $\|$ denotes the concatenation of embeddings. The final learned long-term interests representation is a weighted aggregation of the entire interaction history, with weights computed from the above attentive network, formulated as follows,

Sequential patterns of user interaction play a crucial role in short-term interests modeling, thus we utilize another attentive network on top of a recurrent neural network (RNN). Specifically, we feed the historical item embeddings to a RNN model and use the output of RNN as the keys, which can be formulated as follows,

where $\bm{W_{s}}$ is a transformation matrix and $\rho$ represents a RNN model. In Section 4, we conduct experiments to evaluate different implementations of the RNN model, including LSTM (Hochreiter and Schmidhuber, 1997), GRU (Chung et al., 2014) and Time4LSTM (Yu et al., 2019). Similar as Eqn (18) and (19), we use $\bm{q_{s}^{u,t}}$ as the query vector, and obtain attention scores $b_{k}$. Then the learned representation for short-term interests can be computed as follows,

Although separate encoders are adopted, disentanglement of LS-term interests is not guaranteed since $\bm{u_{l}^{t}}$ and $\bm{u_{s}^{t}}$ are extracted in an unsupervised manner (Locatello et al., 2019). Particularly, there is no labeled data to supervise the learned interests representations. Therefore, we propose to design contrastive tasks which can achieve disentanglement with self-supervision and overcome the challenge of lacking labeled data.

As introduced previously, long-term interests provide a holistic view of user preferences which summarize the entire historical interactions, while short-term interests evolve dynamically over time which reflect recent interactions. Therefore, we can obtain proxies for LS-term interests from the interaction sequences themselves to supervise the two interests encoders. Specifically, we calculate the mean representation of the entire interaction history as the proxy for long-term interests, and use the average representation of recent $k$ interactions as the proxy for short-term interests. Formally, the proxies of LS-term interests for a given user $u$ at timestamp $t$ can be calculated as follows,

where $\bm{E}(x)$ means the embedding of item $x$. Note that we only calculate proxies when the sequence length is longer than a threshold $l_{t}$, since there is no need to distinguish long and short-term if the whole sequence only contains a few items (Li et al., 2011). The threshold $l_{t}$, the length of the recent-behavior sequence $k$ are hyper-parameters in our method. Furthermore, we use mean pooling here for its simplicity and the performance turns out to be good enough. In fact, our self-supervised paradigm is capable of exploiting more complex design for proxies which we leave for future work.

With proxies serving as labels, we can utilize them to supervise the disentanglement of LS-term interests. Specifically, we perform contrastive learning between the encoder outputs and proxies, which requires the learned representations of LS-term interests to be more similar to their corresponding proxies than the opposite proxies. We illustrate the contrastive tasks in Figure 2 (A). Formally, there are four contrastive tasks as follows,

where Eqn (19)-(20) supervise long-term interests, and Eqn (21)-(22) supervise short-term interests, and $sim(\cdot,\cdot)$ measures embedding similarity. Take long-term interests modeling as an example, Eqn (19) encourages the learned long-term interests representation, $\bm{u_{l}^{t}}$, to be more similar to the long-term proxy, $\bm{p_{l}^{u,t}}$, than to the short-term proxy, $\bm{p_{s}^{u,t}}$. Meanwhile, Eqn (20) requires that $\bm{u_{l}^{t}}$ is closer to $\bm{p_{l}^{u,t}}$ compared with the short-term interests representation, $\bm{u_{s}^{t}}$. With four symmetric contrastive tasks on the similarity between encoder outputs and proxies, we add self-supervision on LS-term interests modeling which can achieve stronger disentanglement compared with existing unsupervised approaches.

We implement two pairwise loss functions based on Bayesian Personalized Ranking (BPR) (Rendle et al., 2012) and triplet loss to accomplish contrastive learning in Eqn (19)-(22). Formally, the two loss functions, which use inner product and Euclidean distance to capture embedding similarity, are computed as follows,

where $\sigma$ is the softplus activation function, $\langle\cdot,\cdot\rangle$ denotes inner product of two embeddings, $d$ denotes the Euclidean distance, and $m$ denotes a positive margin value. Both $\mathcal{L}_{bpr}$ and $\mathcal{L}_{tri}$ are designed for making the anchor $a$ more similar to the positive sample $p$ than the negative sample $q$. Thus the contrastive loss for self-supervised disentanglement of LS-term interests can computed as follows,

where we omit the superscript of interest representations and proxies, and $f$ can be either $\mathcal{L}_{bpr}$ or $\mathcal{L}_{tri}$.

In summary, we implement two separate encoders $\phi$ and $\psi$ to learn representations for LS-term interests, respectively. In order to achieve disentanglement of LS-term interests, we compute proxies from the historical interaction sequences. We further propose contrastive-learning loss functions that guide the two encoders only to capture the desired aspect in a self-supervised manner.

With the learned disentangled representations by self-supervised learning, how to aggregate the two aspects to predict interactions remains a challenge. Simple aggregators, such as sum and concatenation, assume that contributions of LS-term interests are fixed, which is invalid in many cases. In fact, whether long or short-term one is more important depends on the historical sequence. For example, users are mainly driven by short-term interests when they are continuously browsing items from the same category. Meanwhile, the importance of LS-term interests also depends on the target item. For instance, a sports lover may still click on a recommended bicycle due to long-term interests, even after he/she browses several books. Therefore, we include both the historical sequence and the target item as input of the aggregator, where historical sequence is compressed with a GRU. The proposed attention-based adaptive fusion model is illustrated in Figure 2 (D), which dynamically determines the importance of LS-term interests to aggregate $\bm{u_{l}^{t}}$ and $\bm{u_{s}^{t}}$. Formally, the final fused interests are obtained as follows,

where $\sigma$ is the sigmoid activation function, and $\tau_{f}$ is a MLP for fusion. Here $\alpha$ denotes the estimated fusion weight based on historical interactions, target item, and user’s LS-term interests.

To predict the interaction, we use the widely adopted two-layer MLP (Yu et al., 2019) shown in Figure 2 (E). Then the estimated score given a user $u$ and an item $v$ at timestamp $t+1$ can be predicted as follows,

Following the existing works’ settings(Yu et al., 2019), we use the negative log-likelihood loss function as follows,

where $\mathcal{O}$ is the set composed of training pairs of one positive item $x^{u}_{t+1}$ and $N-1$ sampled negative items. We train the model in an end-to-end manner with multi-task learning on two objectives. Specifically, the joint loss function with a hyper-parameter $\beta$ to balance objectives, can be formulated as follows,

where $\lambda\|\Theta\|_{2}$ denotes the $L2$ regularization for addressing over-fitting. The computation complexity of our implementation is $\mathcal{O}((M+N)d+Q)$ where $Q$ denotes the complexity of MLP and GRU, which is on par with the state-of-the-art SLi-Rec method (Yu et al., 2019).

In CLSR, we utilize two separate representations for LS-term interests. Therefore, it is crucial that each side only captures the desired single aspect. In order to evaluate the effectiveness of each side, we reserve one-side interests and discard the other side of CLSR and SLi-Rec. Results on two datasets are illustrated in Figure 3, from which we can observe that CLSR outperforms SLi-Rec in all cases. Specifically, on Taobao dataset, CLSR improves AUC against SLi-Rec by about 0.03 with short-term interests and full interests. On Kuaishou dataset, the improvements of AUC are about 0.1, 0.2, and 0.4 for long-term interests, short-term interests, and full interests, respectively. It indicates that CLSR attains more meaningful representations for both LS-term interests. Moreover, for both methods on both datasets, combining LS-term interests achieves better performance than using one-side interests. This further supports our motivation to model both long and short-term interests for accurate recommendation.

Learning disentangled representations of underlying factors is very helpful especially when the importance of different factors changes (Zheng et al., 2021; Schölkopf, 2019; Schölkopf et al., 2021). For example, behaviors of higher costs, such as purchase (cost of money) in Taobao dataset and like (cost of time) in Kuaishou dataset, tend to be more driven by users’ long-term interests, and behaviors of lower costs such as click in both datasets indicate more about short-term interests, which has been acknowledged by existing works (Grbovic and Cheng, 2018). Therefore, to investigate whether CLSR achieves disentanglement of LS-term interests, we design counterfactual evaluations where the importance of different interests changes. Specifically, we use models well-trained on click data to predict both clicked items and purchased/liked items, where the importance of LS-term interests is different. Since purchase/like behavior reflects more long-term interests, the importance of long-term interests is supposed to be higher when predicting purchase/like than click. In other words, when the model predicts purchase/like behavior, it is expected that the attention weight for long-term interests when fusing the two aspects, i.e. $\alpha$, to be also larger than predicting click behavior.

Table 3 illustrates the AUC and the average of $\alpha$ for clicked items and purchased/liked items. We have the following findings:

• •

  CLSR outperforms SLi-Rec for all behaviors. Although predicting purchase/like with models trained on click data is challenging, AUC of CLSR is significantly larger than SLi-Rec by over 0.03. Meanwhile, the average $\alpha$ of CLSR is much lower in all cases, unlike SLi-Rec whose average $\alpha$ is even over 0.7 on Kuaishou dataset. In fact, low $\alpha$ in CLSR is consistent with previous findings in Table 2 that long-term interests are less important than short-term interests, which means that LS-term interests are successfully disentangled in CLSR. On the contrary, high $\alpha$ in SLi-Rec indicates that the learned long-term interests representations contain much information of the undesired short-term interests, i.e. the two aspects entangles with each other.

• •

  Since purchase/like reflects more long-term interests than click, $\alpha$ is supposed to be also larger when predicting purchase/click. On Taobao dataset, $\alpha$ of CLSR for purchase behavior is larger than click by about 4%. However, for SLi-Rec, $\alpha$ for purchase is even less than click by over 6%. On Kuaishou dataset, though $\alpha$ for like is larger than click in both SLi-Rec and CLSR, the relative increment of $\alpha$ for CLSR is over two times larger than SLi-Rec (+9.06% v.s. +3.91%). This further validates that CLSR achieves much stronger disentanglement of LS-term interests.

Meanwhile, we also evaluate under special cases where long or short-term interests are blocked by re-arranging interaction sequences with two protocols, namely shuffle and truncate, as illustrated in Figure 4. The details are as follows.

• •

  Shuffle: The historical sequence is randomly shuffled, and thus short-term interests are removed under this protocol.

• •

  Truncate: Early history is discarded and only recent history is available. Thus long-term interests are weakened.

Table 4 shows the results under shuffle protocol on two datasets. Since shuffling operation blocks short-term interests, predicting click behavior is much more difficult than the original case, while predicting purchase behavior is relatively easier. We can observe that the results in Table 4 compared with Table 3 is consistent with the expectation. Specifically, for both SLi-Rec and CLSR, AUC decreases by over 0.04 and increases by about 0.02 on click-prediction task and purchase/like-prediction task, respectively. Meanwhile, CLSR improves the AUC of click-prediction by over 0.04, and improves the MRR of purchase-prediction by over 30%, against SLi-Rec. Although short-term interests are invalid under this protocol, CLSR can still achieves better performance since LS-term interests are disentangled and long-term interests can still take effect.

We further present the results of CLSR under truncate protocol with varying available length ($k$) of historical sequences in Figure 5 (a). We can observe that the performance of purchase prediction improves significantly as $k$ grows. Meanwhile, the performance of click prediction increases much slower when $k$ grows larger. This observation verifies our assumption that short-term interests can be effectively captured by mining the recent history, while for long-term interests, it is essential to take the entire history into consideration. In addition, we also show the performance of only using long-term interests representation under truncate protocol in Figure 5 (b). We can find that the accuracy of purchase-prediction increases drastically as $k$ getting larger, while the accuracy of click-prediction is barely changed. The different trends of click and purchase tasks confirm that the learned long-term interests representations only capture the desired interests and distill short-term interests.

In summary, by comparing CLSR and SLi-Rec, which both explicitly model LS-term interests, we empirically show that disentanglement of the two aspects is the reason of better recommendation performance. Moreover, it is insufficient to disentangle LS-term interests in an unsupervised way, and CLSR effectively overcomes the challenge of lacking labeled data with self-supervision.

Contrastive tasks on the similarity between learned representations and proxies for LS-term interests help achieve stronger disentanglement than existing unsupervised methods. We conduct ablation study to compare the performance of CLSR with and without the contrastive loss $\mathcal{L}_{\text{con}}$. In addition, we also evaluate the performance of replacing the short-term interests encoder $\psi$ with DIEN. Figure 6 (a) illustrates the results on Kuaishou dataset. We can find that GAUC of CLSR drops over 0.01 after removing the contrastive tasks which verifies the necessity of self-supervision. Meanwhile, adding self-supervision can also significantly improve the performance of DIEN, which means that CLSR can serve as a general framework to disentangle LS-term interests for existing recommendation models. We also investigate the performance under different loss weights of $\mathcal{L}_{\text{con}}$. Figure 6 (b) illustrates the results on Kuaishou dataset. We can observe that 0.1 is an optimal value, and too large $\beta$ may contradict with the main interaction prediction task which leads to low accuracy.

In CLSR, we propose to aggregate LS-term interests adaptively according to the target item and the historical sequence. Here we investigate whether this adaptive fusion is effective. To be specific, we compare with a static version, which means using a fixed $\alpha$ when combining the two aspects. Figure 7 shows the recommendation performance on two datasets, where the dashed line represents the performance of adaptive fusion. We can discover that adaptive fusion outperforms all different values of fixed $\alpha$. These results verify the necessity of adaptive fusion of LS-term interests, and our proposed attention-based network successfully accomplishes this goal.

To conclude, we conduct extensive experiments to show the superior performance of the proposed CLSR model. Counterfactual evaluations demonstrate that LS-term interests are successfully disentangled. More experimental results are left in Section A.4.