Prospective Preference Enhanced Mixed Attentive Model for Session-based Recommendation

In the last few years, numerous session-based recommendation methods have been developed, particularly using Markov Chains (MCs), attention mechanisms and neural networks such as RNNs and GNNs, etc. MCs-based methods [9] use MCs to capture the transitions among items for recommendation. For example, Rendle et al.  [9] employs a first-order MC to generate recommendations based on the transitions of the last item in each session. Attention-based methods [1, 10] model the importance of items for the recommendation. For example, Liu et al.  [10] developed a short-term attention priority model ($\mathop{\mathtt{STAMP}}\limits$), which adapts a gate mechanism to capture users’ short-term preferences. Recently, RNNs-based methods such as $\mathop{\mathtt{GRU4Rec+}}\limits$ [11] and $\mathop{\mathtt{NARM}}\limits$ [1] have been developed to model the temporal patterns among items primarily using GRUs.

GNNs-based methods are also extensively developed for the session-based recommendation. Wu et al.  [2] converted sessions to direct graphs, and developed a GNNs-based model ($\mathop{\mathtt{SR\text{-}GNN}}\limits$) to generate recommendations based on the graph structures. Qiu et al.  [12] re-examined the importance of item ordering in session-based recommendations and developed a GNN-based model ($\mathop{\mathtt{FGNN}}\limits$), which included self-loop for each node in graphs to better capture users’ short-term preferences. Chen et al.  [4] showed that the widely used directed graph representations can not fully preserve the sequential information in sessions. To mitigate this problem, they converted sessions to multigraphs, and developed a GNNs-based model ($\mathop{\mathtt{LESSR}}\limits$) to generate recommendations. Xia et al.  [3] developed hyper graph-based model ($\mathop{\mathtt{DHCN}}\limits$), which leverages hyper graphs and hyper graph convolutional networks to capture the high-order information among items.

Sequential recommendation aims to generate recommendations for the next items of users’ interest based on users’ historical interactions. It is closely related to session-based recommendation except that in sequential recommendation, we could access users’ historical interactions in a long-time period (e.g., months). In the last few years, neural networks (e.g., RNNs) and attention mechanisms have been extensively employed in sequential recommendation methods. For example, RNNs-based methods such as User-based RNNs [13] incorporate user characteristics into GRUs for personalized recommendation. Skip-gram-based methods [14] leverage the skip-gram model [15] to capture the co-occurrence among items in a time window. Recently, Convolutional Neural Networks (CNNs) are also adapted for sequential recommendation. Tang et al.  [16] developed a CNNs-based model, which uses multiple convolutional filters to model the synergies [17] among items. Yuan et al.  [18] developed another CNN-based generative model NextItRec to better capture the long-term dependencies in sequential recommendation. Besides CNNs, attention-based methods [19, 20, 21] are also developed for sequential recommendation. Kang et al.  [19] developed a self-attention based model, which adapts the self-attention to better model users’ long-term preferences. Sun et al.  [20] further developed a bidirectional self-attention based model to improve the representational power of item embeddings.

Previous studies [19, 17, 16] have shown that recently interacted items are more indicative than earlier ones of the next item. Following these, we focus on the most recent $n$ items (i.e., the last $n$ items) in a session to generate recommendations. Particularly, given a session $S=\{s_{1},s_{2},\dots\,s_{|S|}\}$, we transform it to a fixed-length sequence $A=\{a_{1},a_{2},\dots,a_{n}\}$, which contains the last $n$ items in $S$ (i.e., $a_{i}$ = $s_{|S|-n+i}$, $i=1,\cdots,n$). If $S$ is shorter than n, we will pad empty items at the beginning of $A$ until length n.

In $\mathop{\mathtt{P^{2}MAM}}\limits$, we represent the items in sessions using learnable embeddings. Specifically, we learn an item embedding matrix $\mbox{$\mathop{V}\limits$}\in\mathbb{R}^{m\times d}$, in which the $j$-th row $\mathbf{v}_{j}$ is the embedding of item $j$, and $m$ and $d$ is the number of all the items and the dimensionality of embeddings, respectively. Given $\mathop{V}\limits$, we represent the items in $A$ by a matrix $\mbox{$\mathop{E}\limits$}\in\mathbb{R}^{n\times d}$ as follows:

where $\mathbf{v}_{a_{i}}$ is the embedding of item $a_{i}$. Following the previous work [22, 19, 21], we use a constant zero vector as the embedding of padded empty items.

We also learn embeddings for the $n$ positions in the fixed-length sessions to capture the temporal patterns. Specifically, following Vaswani et al.  [22], we learn a position embedding matrix $\mbox{$\mathop{P}\limits$}\in\mathbb{R}^{n\times d}$:

in which the $t$-th row $\mathbf{p}_{t}$ is the embedding of the $t$-th position.

It has been shown [8, 10, 17] that the temporal patterns play an important role in predicting users’ preferences. Existing session-based recommendation methods model the temporal patterns primarily using GRUs-based methods [7, 23], which implicitly learn weights over items. However, as demonstrated in the literature [8], the complicated GRUs-based models may not be well learned for the notoriously sparse recommendation datasets, and it also suffers from poor interpretability [24] and limited parallelizability [8, 19].

In $\mathop{\mathtt{P^{2}MAM}}\limits$, different from other methods, we develop a novel position-sensitive attention mechanism to model the temporal patterns, and generate predictions of users’ preferences accordingly. Specifically, we use a dot-product attention mechanism as follows:

where $\mathop{C}\limits$ combines items embeddings and position embeddings in the session, and $\mathbf{c}_{i}$ is the $i$-th row in $\mathop{C}\limits$, $\bm{\alpha}$ is a vector of attention weights, $\mathbf{q}\in\mathbb{R}^{1\times d}$ is a learnable vector shared by all the sessions, and $\mathop{\mathbf{h}^{o}}\limits$ is the position-sensitive preference prediction. The intuition of our position-sensitive preference prediction is that items themselves (their contents) and their relative orders in the sessions represent meaningful information related to user preferences; we can use data-driven attention weights from item embeddings and position embeddings to capture such information. Different from existing attentive session-based methods [10, 1, 25], which learn attention weights without explicitly considering the position information, we explicitly incorporate position embeddings to learn position-sensitive attention weights. Compared to GRUs-based methods [7, 23], the simple and light-weight attention mechanism is easier to learn well on the sparse recommendation datasets, and it could also provide better interpretability and parallelizability.

As presented in Section 1, users’ prospective preferences reveal important items, and thus, could benefit the recommendation. Motivated by this, in $\mathop{\mathtt{P^{2}MAM}}\limits$, we develop a novel strategy that leverages the preference prediction (i.e., $\mathop{\mathbf{h}^{o}}\limits$) from $\mathop{\mathtt{PS}}\limits$ as an estimate of users’ prospective preferences to enable better attention weights over items. Specifically, we employ a multi-head attention mechanism [22] as follows:

where $\bm{\beta}_{i}$ is the vector of attention weights from the $i$-th head, $Q_{i}\in\mathbb{R}^{d\times\frac{d}{b}}$, $K_{i}\in\mathbb{R}^{d\times\frac{d}{b}}$, and $W_{i}\in\mathbb{R}^{d\times\frac{d}{b}}$ are learnable projection matrices in the $i$-th head, and $\mathbf{head}_{i}$ is the output of the $i$-th head, $b$ is the number of heads, $W\in\mathbb{R}^{d\times d}$ is a learnable projection matrix shared by all the heads, and $\mbox{$\mathop{\mathbf{h}^{p}}\limits$}\in\mathbb{R}^{1\times d}$ is the prospective preference enhanced preference prediction. The intuition of our strategy (i.e., $\mathop{\mathtt{P^{2}E}}\limits$) is that we learn predictive attention weights based on the estimate of users’ prospective preferences. As will be shown in Section 6.1, this strategy could significantly improve the recommendation performance. Note that $\mathop{\mathtt{P^{2}E}}\limits$ serves as a general strategy that is adaptable to other estimates of users’ prospective preferences. We also tried the dot-product attention as in $\mathop{\mathtt{PS}}\limits$ but empirically found that it produces inferior performance.

In $\mathop{\mathtt{P^{2}MAM}}\limits$, we calculate recommendation scores based on the predictions of users’ preferences (i.e., $\mathop{\mathbf{h}^{o}}\limits$ and $\mathop{\mathbf{h}^{p}}\limits$). Specifically, we develop three different methods to calculate the scores. Based on the scores, the items with the top-$k$ largest scores will be recommended.

Following the literature [10, 2, 4], we adapt the cross-entropy loss to minimize the negative log likelihood of correctly recommending the ground-truth next item as follows:

where $T$ is the set of all the training sessions, $\mathbf{r}_{i}$ is a one-hot vector in which the dimension $j$ is 1 if item $j$ is the ground-truth next item in the $i$-th training session or 0 otherwise, $\hat{\mathbf{r}}_{i}$ is the vector of recommendation scores for the $i$-th training session, and $\Theta$ is the set of learnable parameters (e.g., $\mathop{V}\limits$, $\mathop{P}\limits$ and $W$). All the learnable parameters are randomly initialized, and are optimized in an end-to-end manner. We use this objective to optimize all the $\mathop{\mathtt{P^{2}MAM}}\limits$ variants (i.e., $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$, $\mathop{\mathtt{P^{2}MAM\text{-}P}}\limits$ and $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$).

We compare $\mathop{\mathtt{P^{2}MAM}}\limits$ with the five state-of-the-art baseline methods:

• •

  $\mathop{\mathtt{POP}}\limits$ [2] recommends the most popular items of each session.

• •

  $\mathop{\mathtt{NARM}}\limits$ [1] employs GRUs and attention mechanisms to model the temporal patterns for the recommendation.

• •

  $\mathop{\mathtt{SR\text{-}GNN}}\limits$ [2] transforms sessions into directed graphs, and employs GNNs to model complex transitions in sessions.

• •

  $\mathop{\mathtt{LESSR}}\limits$ [4] transforms sessions into directed multigraphs and generates recommendations using GNNs.

• •

  $\mathop{\mathtt{DHCN}}\limits$ [3] transforms sessions into hypergraphs and generates recommendations using a hypergraph convolutional network.

Note that $\mathop{\mathtt{DHCN}}\limits$ and $\mathop{\mathtt{LESSR}}\limits$ have been compared against a comprehensive set of other methods including $\mathop{\mathtt{GRU4Rec+}}\limits$ [11], $\mathop{\mathtt{STAMP}}\limits$ [10] and $\mathop{\mathtt{FGNN}}\limits$ [12], and have outperformed those methods. Thus, we compare $\mathop{\mathtt{P^{2}MAM}}\limits$ with $\mathop{\mathtt{DHCN}}\limits$ and $\mathop{\mathtt{LESSR}}\limits$ instead of the methods that they have outperformed. For all the baseline methods, we use the implementations provided by their authors (Section A in Appendix).

We compare $\mathop{\mathtt{P^{2}MAM}}\limits$ with the baseline methods in the following benchmark datasets that are widely used in the literature [1, 2, 4]:

• •

  Diginetica ($\mathop{\texttt{DG}}\limits$) ²²2http://cikm2016.cs.iupui.edu/cikm-cup is from the CIKM Cup 2016, and contains anonymized browsing logs and transactions. Following the literature [1, 2, 4], we only use the transaction data in our experiments.

• •

  Yoochoose ($\mathop{\texttt{YC}}\limits$) ³³3http://2015.recsyschallenge.com/challenge.html is from the RecSys Challenge 2015 containing sessions of user clicks within 6 months.

• •

  Gowalla ($\mathop{\texttt{GA}}\limits$) ⁴⁴4https://snap.stanford.edu/data/loc-gowalla.html is a point-of-interests dataset and includes user-venue check-in records with timestamps.

• •

  Lastfm ($\mathop{\texttt{LF}}\limits$) ⁵⁵5http://ocelma.net/MusicRecommendationDataset/lastfm-1K.html is a dataset collecting streams of music listening events in the Last.fm. Following the literature [4], we focus on the music artist recommendation in our experiments.

• •

  Nowplaying ($\mathop{\texttt{NP}}\limits$) ⁶⁶6http://dbis-nowplaying.uibk.ac.at/#nowplaying is another dataset describing the music listening events of users.

• •

  Tmall ($\mathop{\texttt{TM}}\limits$) ⁷⁷7https://tianchi.aliyun.com/dataset/dataDetail?dataId=42 is from the IJCAI-15 competition, and it includes anonymized shopping logs on the online retail platform Tmall.

Following the literature [1, 2, 4, 3], for $\mathop{\texttt{GA}}\limits$, we only keep the top 30,000 most popular locations, and view the check-in records in one day as a session [4, 8]. For $\mathop{\texttt{LF}}\limits$, we only keep the top 40,000 most popular artists, and view the listening events in 8 hours as a session [4]. For all the datasets, we filter out sessions of length one and items appearing less than five times over all the sessions.

Table III presents the overall performance of different methods at recall@$k$, MRR@$k$ and NDCG@k in recommending the next item. Due to the space limit, we do not present the results on recall@5, MRR@5 and NDCG@5. However, we observed a similar trend on these metrics. Table III shows that overall, $\mathop{\mathtt{P^{2}MAM}}\limits$ (i.e., $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$, $\mathop{\mathtt{P^{2}MAM\text{-}P}}\limits$ and $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$) is the best performing method on the six benchmark datasets. In terms of recall@10 and recall@20, $\mathop{\mathtt{P^{2}MAM}}\limits$ achieves the best performance on all the six datasets, with significant average improvement of 4.2% and 5.5%, respectively, compared to the best baseline method at each dataset. Note that in session-based recommendation, improvement of above 1% is generally considered as significant [2, 4]. At MRR@$k$, $\mathop{\mathtt{P^{2}MAM}}\limits$ also achieves competitive performance over the baseline methods. For example, on $\mathop{\texttt{DG}}\limits$ and $\mathop{\texttt{GA}}\limits$, $\mathop{\mathtt{P^{2}MAM}}\limits$ achieves statistically significant improvement of 2.7% and 1.9% at MRR@10 and MRR@20, respectively. On $\mathop{\texttt{YC}}\limits$ and $\mathop{\texttt{TM}}\limits$, $\mathop{\mathtt{P^{2}MAM}}\limits$ achieves the second best performance at MRR@10 and MRR@20. We found a similar trend at NDCG@$k$. For example, in terms of NDCG@$10$, $\mathop{\mathtt{P^{2}MAM}}\limits$ substantially outperforms the baseline methods on $\mathop{\texttt{DG}}\limits$ and $\mathop{\texttt{GA}}\limits$; at NDCG@$20$, $\mathop{\mathtt{P^{2}MAM}}\limits$ is the best method on four out of six datasets except $\mathop{\texttt{YC}}\limits$ and $\mathop{\texttt{LF}}\limits$. These results demonstrate the strong recommendation performance of $\mathop{\mathtt{P^{2}MAM}}\limits$. We notice that on $\mathop{\texttt{YC}}\limits$, $\mathop{\texttt{LF}}\limits$ and $\mathop{\texttt{TM}}\limits$, at MRR@10 and MRR@20, $\mathop{\mathtt{P^{2}MAM}}\limits$ appears considerably worse than the baseline methods (e.g., $\mathop{\mathtt{LESSR}}\limits$). These results indicate that on certain datasets, even though $\mathop{\mathtt{P^{2}MAM}}\limits$ may be less effective than the baseline methods on ranking the ground-truth next items on the very top (e.g., top-1), $\mathop{\mathtt{P^{2}MAM}}\limits$ is still on average more effective on recommending the correct items among on top.

As shown in Table III, among the three variants of $\mathop{\mathtt{P^{2}MAM}}\limits$, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ has the best performance overall. In terms of recall@10, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ outperforms the other variants on $\mathop{\texttt{YC}}\limits$, $\mathop{\texttt{GA}}\limits$ and $\mathop{\texttt{LF}}\limits$, and achieves the second best performance on the other three datasets. In terms of recall@20, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ is the best method at four out of six datasets except $\mathop{\texttt{DG}}\limits$ and $\mathop{\texttt{NP}}\limits$. We found a similar trend at MRR@$k$ and NDCG@$k$. For example, in terms of NDCG@10 and NDCG@20, $\mathop{\mathtt{P^{2}MAM}}\limits$ achieves significant improvement over the other variants on three out of six datasets (i.e., $\mathop{\texttt{YC}}\limits$, $\mathop{\texttt{GA}}\limits$ and $\mathop{\texttt{LF}}\limits$). On the other three datasets, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ is still ranked as the second best method.

Compared to $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ learns attention weights conditioned on the estimate of users’ prospective preferences (i.e., $\mathop{\mathtt{P^{2}E}}\limits$), while $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ does not have this strategy. The superior performance of $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ over $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ on four out of six datasets indicates that on most of the datasets, incorporating the estimate of prospective preferences could enable better recommendations. Compared to $\mathop{\mathtt{P^{2}MAM\text{-}P}}\limits$, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ generates recommendations using the preference predictions from temporal patterns (i.e., $\mathop{\mathbf{h}^{o}}\limits$) and users’ prospective preferences (i.e., $\mathop{\mathbf{h}^{p}}\limits$), while $\mathop{\mathtt{P^{2}MAM\text{-}P}}\limits$ only use $\mathop{\mathbf{h}^{p}}\limits$ to generate recommendations. The strong improvement of $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ compared to $\mathop{\mathtt{P^{2}MAM\text{-}P}}\limits$ indicates that when used together, the two preference predictions could reinforce each other and improve the recommendation performance. We notice that on $\mathop{\texttt{DG}}\limits$ and $\mathop{\texttt{NP}}\limits$, the performance of $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ is slightly worse than that of $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ at recall@20. This might be due to that as will be shown in Section 6.5, in certain datasets (e.g. $\mathop{\texttt{DG}}\limits$ and $\mathop{\texttt{NP}}\limits$), the temporal patterns are highly strong, and could denominate the learning process. As a result, incorporating the $\mathop{\mathtt{P^{2}E}}\limits$ component may not improve the recommendation performance.

Existing methods [1, 2, 4] leverage GRUs to capture the temporal patterns, while in $\mathop{\mathtt{P^{2}MAM}}\limits$, we model the temporal patterns using a position-sensitive attention mechanism (i.e., $\mathop{\mathtt{PS}}\limits$). Here, we compare the performance of $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$, the $\mathop{\mathtt{P^{2}MAM}}\limits$ variant purely based on $\mathop{\mathtt{PS}}\limits$, and the GRUs-based baseline methods including $\mathop{\mathtt{NARM}}\limits$, $\mathop{\mathtt{SR\text{-}GNN}}\limits$ and $\mathop{\mathtt{LESSR}}\limits$. As shown in Table III, $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ achieves superior performance over GRUs-based baseline methods on five out of six datasets except $\mathop{\texttt{LF}}\limits$ at recall@$k$. On $\mathop{\texttt{LF}}\limits$, the performance of $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ is slightly worse than that of $\mathop{\mathtt{SR\text{-}GNN}}\limits$ and $\mathop{\mathtt{LESSR}}\limits$ but $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ is still able to outperform $\mathop{\mathtt{NARM}}\limits$ at all the metrics. GRUs model the temporal patterns in a recurrent fashion using complicated non-linear layers, while $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ explicitly models the temporal patterns using attention weights over items in the session. The significant improvement of $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ over GRUs-based baseline methods indicates that on sparse recommendation datasets, our simple attentive method could be easier to learn well than the complicated GRUs-based methods and thus, enable better performance.

Graph-based methods [2, 4, 3, 12] are extensively developed for session-based recommendation, and have been demonstrated the state-of-the-art performance. However, as shown in Table III, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ as a sequence-based method, significantly outperforms the state-of-the-art graph-based methods (i.e., $\mathop{\mathtt{SR\text{-}GNN}}\limits$, $\mathop{\mathtt{LESSR}}\limits$, $\mathop{\mathtt{DHCN}}\limits$) on the benchmark datasets. For example, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ achieves significant improvement compared to the state-of-the-art graph-based methods at both recall@10 and recall@20 on all the datasets. Graph-based methods convert sessions to directed graphs or hyper graphs, and learn the complex temporal patterns [2] leveraging the graph structure (i.e., topology). However, the graphs are constructed based on some assumptions by design such as an item should link to all the subsequent items [4]. Such assumptions may introduce noises or unnecessary/unrealistic relations in the graphs. Meanwhile, the sparse nature of recommendation datasets, and thus of the graphs, may not support the complicated learning of GNN-based models very well. The superior performance of $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ over graph-based methods signifies that the sequence representation could be more effective than graphs for the recommendation.

We conduct an analysis to verify the importance of position embeddings in $\mathop{\mathtt{P^{2}MAM}}\limits$. Specifically, in $\mathop{\mathtt{P^{2}MAM}}\limits$, we remove the position embeddings (i.e., $\mathop{P}\limits$) in Equation 3 and Equation 4, and calculate the attention weights using item embeddings (i.e., $\mathop{E}\limits$) only. We denote $\mathop{\mathtt{P^{2}MAM}}\limits$ without position embeddings as $\mathop{\mathtt{P^{2}MAM}}\limits$ ​\$\mathop{\mathtt{PE}}\limits$, and report the performance of $\mathop{\mathtt{P^{2}MAM}}\limits$ and $\mathop{\mathtt{P^{2}MAM}}\limits$ ​\$\mathop{\mathtt{PE}}\limits$ in Table IV. Due to the space limit, we do not present the performance of $\mathop{\mathtt{P^{2}MAM\text{-}P}}\limits$ but we observed a similar trend in $\mathop{\mathtt{P^{2}MAM\text{-}P}}\limits$.

As presented in Table IV, without position embeddings, the performance of $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ ​\$\mathop{\mathtt{PE}}\limits$ and $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ ​\$\mathop{\mathtt{PE}}\limits$ degrades significantly on four out of six datasets (i.e., $\mathop{\texttt{DG}}\limits$, $\mathop{\texttt{YC}}\limits$, $\mathop{\texttt{GA}}\limits$ and $\mathop{\texttt{NP}}\limits$). For example, on $\mathop{\texttt{DG}}\limits$ and $\mathop{\texttt{YC}}\limits$, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ ​\$\mathop{\mathtt{PE}}\limits$ underperforms $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ at 3.8% and 5.5%, respectively. Recall that in $\mathop{\mathtt{P^{2}MAM}}\limits$, we learn position embeddings to incorporate the position information into the model, and better model the temporal patterns. These results demonstrate the importance of the position information for session-based recommendation. These results also reveal that the temporal patterns on the four datasets (e.g., $\mathop{\texttt{DG}}\limits$ and $\mathop{\texttt{NP}}\limits$) are strong, and explain the similar performance of $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ and $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ on $\mathop{\texttt{DG}}\limits$ and $\mathop{\texttt{NP}}\limits$ as discussed in Section 6.2. We also notice that on $\mathop{\texttt{LF}}\limits$ and $\mathop{\texttt{TM}}\limits$, $\mathop{\mathtt{P^{2}MAM\text{-}O}}\limits$ ​\$\mathop{\mathtt{PE}}\limits$ and $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ ​\$\mathop{\mathtt{PE}}\limits$ still achieve performance similar to that with position embeddings. For example, on $\mathop{\texttt{LF}}\limits$, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ ​\$\mathop{\mathtt{PE}}\limits$ achieves 0.2449 at recall@20, and $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ achieves 0.2454 (difference: 0.2%). Similarly, on $\mathop{\texttt{TM}}\limits$, at recall@20, the performance of $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ ​\$\mathop{\mathtt{PE}}\limits$ and $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ is 0.3468 and 0.3438, respectively (difference: 0.9%). As will be shown in Section 6.7, on some datasets (e.g., $\mathop{\texttt{LF}}\limits$), the position information may not be crucial for the recommendation. Therefore, on these datasets, without position embeddings, the model could still achieve similar performance.

In $\mathop{\mathtt{P^{2}MAM}}\limits$, we leverage the position-sensitive preference prediction (i.e., $\mathop{\mathbf{h}^{o}}\limits$) from $\mathop{\mathtt{PS}}\limits$ (Section 4.2) as the estimate of users’ prospective preferences (Section 4.3). We notice that in the literature [10, 2, 19], another strategy is to estimate the users’ prospective preferences from the last item in each session. This strategy is based on the recency assumption [10, 2] that the most recently interacted item could be a highly strong indicator of the next item of users’ interest. We conduct an analysis to empirically compare the two strategies in $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$. Specifically, in Equation 4, instead of $\mathop{\mathbf{h}^{o}}\limits$, we use the embeddings of the last item and its position in each session (i.e., $\textbf{v}_{a_{n}}\!+\mathbf{p}_{n}$) to calculate the attention weights, and denote the resulted variant as $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$. Table V presents the performance of $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ and $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$. Similarly to that in $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$, we tune hyper parameters for $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$ using grid search, and report the results from the identified best performing hyper parameters.

As presented in Table V, overall $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ achieves considerable improvement compared to $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$ on four out of six datasets (i.e., $\mathop{\texttt{DG}}\limits$, $\mathop{\texttt{GA}}\limits$, $\mathop{\texttt{LF}}\limits$ and $\mathop{\texttt{TM}}\limits$). At recall@5, recall@10 and recall@20, on average, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ achieves significant improvement of 3.5%, 3.4% and 3.5%, respectively, over $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$. We find a similar trend at MRR@$k$ and NDCG@$k$. For example, in terms of MRR@5 and NDCG@5, over the four datasets, $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ still significantly outperforms $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$ with an average improvement of 4.9% and 4.4%, respectively. The primary difference between $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ and $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$ is that $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ uses a learning-based method to estimate users’ prospective preferences in a data-driven manner, while $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$ generates the estimate based on the recency assumption. The above results indicate that on most of the datasets, data driven-based estimate is more effective than recency-based estimate. We also notice that on $\mathop{\texttt{YC}}\limits$ and $\mathop{\texttt{NP}}\limits$, the performance of $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ is competitive with that of $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$. For example, on $\mathop{\texttt{YC}}\limits$, the performance difference between $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ and $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$ is 0.2% and 0.1% at recall@20 and MRR@20, respectively. On $\mathop{\texttt{NP}}\limits$, the difference at recall@20 is also as small as 0.5%. As shown in the literature [10], some datasets such as $\mathop{\texttt{YC}}\limits$ have strong recency patterns. The similar results between $\mathop{\mathtt{P^{2}MAM\text{-}O\text{-}P}}\limits$ and $\mathop{\mathtt{LAST\text{-}O\text{-}P}}\limits$ on $\mathop{\texttt{YC}}\limits$ and $\mathop{\texttt{NP}}\limits$, indicate that on datasets with strong recency patterns, our learning-based strategy may implicitly capture the patterns, and still produce competitive results.