A Pre-trained Zero-shot Sequential Recommendation Framework via Popularity Dynamics

We learn to represent items based on their popularity dynamics, i.e., changes in their popularity histories. Intuitively, popularity can be calculated by two horizons: long-term and short-term. Long-term horizons reflect the overall popularity of items, whereas short-term horizons should capture the recent trends in the domain. For example, the long-term popularity of a winter coat measures how popular is the coat in general, while its short-term popularity reflects more temporal changes, e.g., season, weather conditions, and fashion trends. Therefore, consider an item $v_{j}$ that has interaction at time $t$, denoted as $v_{j}^{t}$, we define two popularity representations for $v_{j}^{t}$: popularity $\mathbf{p}_{j}^{t}\in\mathbb{R}^{k}$ over a coarse period (e.g., month) and popularity $\mathbf{h}_{j}^{t}\in\mathbb{R}^{k}$ over a fine period (e.g., week).

To calculate $\mathbf{p}_{j}^{t}$ and $\mathbf{h}_{j}^{t}$, we first calculate the popularities of $v_{j}^{t}$ over the two horizons, denoted as $a_{j}^{t}\in\mathbb{R}^{+}$ (coarse period number of interactions) and $b_{j}^{t}\in\mathbb{R}^{+}$ (fine period number of interactions). Specifically, we calculate them as:

where $\gamma\in\mathbb{R}^{+}$ is a pre-defined discount factor and $c_{a}(v_{j}^{m})$ is the number of interactions of $v_{j}$ over a coarse time period $m$. Similarly, $c_{b}(v_{j}^{t})$ denotes the number of interactions of $v_{j}$ over a fine period $t$. We do not impose the discounting factor when computing $b_{j}^{t}$ since we want it to capture the current popularity information, whereas $a_{j}^{t}$ captures the cumulative popularity of an item over a longer horizon.

To make item popularity comparable across domains, we calculate the percentiles of $a_{j}^{t}$ and $b_{j}^{t}$ relative to their corresponding coarser and finer popularity distributions over all items at time $t$, denoted as $P(a_{j}^{t})\in\mathbb{R}^{+}$ and $P({b_{j}^{t}})\in\mathbb{R}^{+}$, respectively.

We now encode the popularity percentiles $P(a_{j}^{t})$ and $P({b_{j}^{t}})$ into $k$ dimensional vector representations $\mathbf{p}_{j}^{t}$ and $\mathbf{h}_{j}^{t}$ respectively. Denote the popularity encoder as $E_{p}:\mathbb{R}^{+}\rightarrow\mathbb{R}^{k}$, which takes in a percentile value. Suppose given the popularity percentile $P(a_{j}^{t})\in\mathbb{R}^{+}$ over a coarse time period $t$, the coarse level popularity vector representation $\mathbf{p}_{j}^{t}\in\mathbb{R}^{k}$ is computed as follows:

where $\lfloor\cdot\rfloor$ denotes the floor, $\{\cdot\}$ denotes the fractional part of a number, and $(\mathbf{p}_{j}^{t})_{i}$ denotes the $i$-th index of $\mathbf{p}_{j}^{t}$. For example, if $k=11$, $E_{p}(40.1)=[0,0,0,0,0.99,0.01,0,0,0,0,0]$. The interpretation of this would be considering the $10$ deciles for $i\in\{0,1,\ldots,9,10\}$ as basis vectors, and this popularity encoding as a linear combination of the nearest (in percentile space) two basis vectors. The fine level popularity vector is calculated identically, i.e., $\mathbf{h}_{j}^{t}=E_{p}(P(b_{j}^{t}))$. In this example, we’ve fixed the vector representation size to be 11, but this approach is fully generalizable to other sizes and would just require changing the multipliers in the encoding function. We also experimented with sinuoisal encodings of the same size, but found that the linear encoding empirically performed better.

We now define the popularity dynamics of $v_{j}$ at time $t$ over the coarse period (long-term horizon) to be $\mathcal{P}_{j}^{t}=\{\mathbf{p}_{j}^{1},\mathbf{p}_{j}^{2},...,\mathbf{p}_{j}^{t-1}\}$, and over the fine period (short-term horizon) as $\mathcal{H}_{j}^{t}=\{\mathbf{h}_{j}^{1},\mathbf{h}_{j}^{2},...,\mathbf{h}_{j}^{t-1}\}$. We use $t-1$ to constrain access to future interactions and prevent information leakage, i.e., we do not have access to the popularity statistics of $v_{j}$ at time $t$ if we are at time $t$. For example, say an interaction happens on the second Wednesday in February, we consider the coarser and finer time period up until the end of January and the end of the first week in February respectively. To limit computation, we constrain window sizes $m,n$ for $\mathcal{P}$ and $\mathcal{H}$ respectively. Formally, the coarse popularity dynamics of $v_{j}$ at time $t$ is $\mathcal{P}_{j}^{t}=\{\mathbf{p}_{j}^{t-m},\mathbf{p}_{j}^{t-m+1},...,\mathbf{p}_{j}^{t-1}\}$, and the fine popularity dynamics of $v_{j}$ at time $t$ is $\mathcal{H}_{j}^{t}=\{\mathbf{h}_{j}^{t-n},\mathbf{h}_{j}^{t-n+1},...,\mathbf{h}_{j}^{t-1}\}$.

Finally, we compute the embedding of item $v_{j}$ at time $t$ via the universal item representation encoder, defined as a function $\mathcal{E}(\mathcal{P}_{j}^{t},\mathcal{H}_{j}^{t})$ that learns to encode the popularity dynamics $\mathcal{P}_{j}^{t}$ and $\mathcal{H}_{j}^{t}$ into a $d$ dimension vector representation $\mathbf{e}_{j}^{t}$. Specifically, we have:

where $\|$ denotes the concatenation operation, and ${\bm{W}}_{p}\in\mathbb{R}^{d\times k(m+n)}$ is a learnable weight matrix.

In addition, we define $\|_{i=t-m}^{t-1}\mathbf{p}_{j}^{i}\coloneq\mathbf{p}_{j}^{t-m}\|\mathbf{p}_{j}^{t-m+1}\|...\|\mathbf{p}_{j}^{t-1}$ and $\|_{i=t-n}^{t-1}\mathbf{h}_{j}^{i}\coloneq\mathbf{h}_{j}^{t-n}\|\mathbf{h}_{j}^{t-n+1}\|...\|\mathbf{h}_{j}^{t-1}$. The item popularity dynamics encoder can effectively capture the popularity change of items over different time periods. Most importantly, it does not take explicit item IDs or auxiliary information as input to compute the item embeddings. Instead, it learns to represent items through their popularity dynamics, which is universal across domains and applications.

We also consider the time interval between two consecutive interactions when modeling sequences. Differences in time intervals might indicate differences in the users’ behaviors. While previous works explore absolute time intervals (Li et al., 2020), different domains exhibit diverse time scales, thus making modeling absolute time intervals ungeneralizable. Therefore, we propose to encode relative time intervals into modeling sequences. Given an interaction sequence $\mathcal{S}_{u}=\{v_{u,1},v_{u,2},...,v_{u,L}\}$ of user $u$, we define the time interval between $v_{u,j}$ and $v_{u,j+1}$ as $t_{u,j}=t(v_{u,j+1})-t(v_{u,j})$, where $t(v_{u,j})$ is the time that user $u$ interacts with item $v_{u,j}$. We then rank the time intervals of user $u$. Define the rank of relative time interval of $t_{u,j}$ as $r_{u,j}=\text{rank}(t_{u,j})$. The relative time interval encoding of interval $t_{u,j}$ is then defined as ${\bm{T}}_{r_{u,j}}\in\mathbb{R}^{d}$, where ${\bm{T}}\in\mathbb{R}^{L\times d}$, following the same setup in (Vaswani et al., 2017), is a fixed sinusoidal encoding matrix defined as:

We also tried a learnable time interval encoding, but it yielded worse performance. We hypothesize that the sinusoidal encoding is more generalizable across domains and the learnable encoding is more prone to overfitting.

As we will see in § 4.1.5, the self-attention mechanism does not take the positions of the items into account. Therefore, following (Vaswani et al., 2017), we also inject a fixed positional encoding for each position in a user’s sequence. Denote the positional embedding of a position $l$ as ${\bm{P}}_{l}\in\mathbb{R}^{d}$, where ${\bm{P}}\in\mathbb{R}^{L\times d}$. We compute ${\bm{P}}$ using the same formula in Equation 3. Again, we also tried a learnable positional encoding as presented in (Kang and McAuley, 2018; Sun et al., 2019), but it yielded worse results.

We follow previous works in sequential recommendation (Kang and McAuley, 2018; Sun et al., 2019; Li et al., 2020) and propose an extension to the self-attention mechanism by incorporating universal item representations (§ 4.1.2), relative time intervals(§ 4.1.3), and positional encoding (§ 4.1.4).

Firstly, we transform the user sequence $\{v_{u,1},v_{u,2},...,v_{u,|\mathcal{S}_{u}|}\}$ for each user $u$ into a fixed-length sequence $\mathcal{S}_{u}$ = $\{v_{u,1},v_{u,2},...,v_{u,L}\}$ via truncating the oldest interactions or padding, where $L$ is a pre-defined hyper-parameter controlling the maximum length of the sequence. Given a user sequence $\mathcal{S}_{u}=\{v_{u,1},v_{u,2},...,v_{u,L}\}$, we compute its input matrix ${\bm{E}}_{u}$ as:

$\mathbf{e}^{t}_{u,1},\mathbf{e}^{t^{\prime}}_{u,2},...,\mathbf{e}^{t*}_{u,L}$ is computed from Equation 2, ${\bm{T}}_{r_{u,1}},{\bm{T}}_{r_{u,2}},...,{\bm{T}}_{r_{u,L}}$ and ${\bm{P}}_{1},{\bm{P}}_{2},...,{\bm{P}}_{L}$ are computed following the procedure in § 4.1.3 and § 4.1.4 respectively.

Multi-Head Self-Attention. We adopt a widely used multi-head self-attention mechanism (Vaswani et al., 2017), i.e., Transformers. Specifically, it consists of multiple multi-head self-attention layers (denoted as MHAttn($\cdot$)), and point-wise feed-forward networks (FFN($\cdot$)). The multi-head self-attention mechanism is defined as:

where ${\bm{E}}_{u}$ is the input matrix computed from Equation 4, $h$ is a tunable hyper-parameter indicating the number of attention heads, ${\bm{W}}_{i}^{Q},{\bm{W}}_{i}^{K},{\bm{W}}_{i}^{V}\in\mathbb{R}^{d\times d/h}$ are the learnable weight matrices, and $\mathbf{W}^{O}\in\mathbb{R}^{d\times d}$ is also a learnable weight matrix. Attn is the attention function and is formally defined as:

The scale factor $\sqrt{d/h}$ is used to avoid large values of the inner product, which can lead to numerical instability.

Causality: In sequential recommendation, the prediction of the $L+1$ item should only depend on the first $L$ items that the user has interacted with in the past. However, the $L$-th output of the multi-head self-attention layer contains all the input information. Therefore, as in (Li et al., 2020; Kang and McAuley, 2018), we do not let the model attend to the future items by forbidding links between $Q_{i}$ and $K_{j}$ ($j>i$) in the attention function.

Point-Wise Feed-Forward Network: To add nonlinearity and interactions between different embedding dimensions, we follow previous works in sequential recommendation (Sun et al., 2019; Kang and McAuley, 2018; Li et al., 2020) and apply the same point-wise feed-forward network to the output of each multi-head self-attention layer. Formally, suppose the output of the multi-head self-attention layer is $\textbf{z}_{u}$, the point-wise feed-forward network is defined as:

where ${\bm{W}}_{1}\in\mathbb{R}^{d\times d}$ and ${\bm{W}}_{2}\in\mathbb{R}^{d\times d}$ are learnable weight matrices, and $\mathbf{b}_{1}\in\mathbb{R}^{d}$ and $\mathbf{b}_{2}\in\mathbb{R}^{d}$ are learnable bias vectors.

Stacking Layers: As shown in previous works (Kang and McAuley, 2018), stacking multiple multi-head self-attention layers and point-wise feed-forward networks can potentially lead to overfitting and instability during the training. Therefore, we follow previous works (Kang and McAuley, 2018; Sun et al., 2019; Li et al., 2020) and apply layer normalization (Ba et al., 2016) and residual connections to each multi-head self-attention layer and point-wise feed-forward network. Formally, we have:

$g(\mathbf{x})$ is either the multi-head self-attention layer or the point-wise feed-forward network. Therefore, for every multi-head self-attention layer and point-wise feed-forward network, we first apply layer normalization to the input, then apply the multi-head self-attention layer or point-wise feed-forward network, and finally apply dropout and add the input $\mathbf{x}$ to the layer output. The LayerNorm function is defined as:

where $\odot$ denotes the element-wise product, $\mu$ and $\sigma$ are the mean and standard deviation of $\mathbf{x}$, $\alpha$ and $\mathbf{\beta}$ are learnable parameters, and $\epsilon$ is a small constant to avoid numerical instability.

Given a sequence $\mathcal{S}_{u}$ of user $u$ as input, we denote $\mathbf{q}_{u}$ as the output of the popularity dynamics-aware transformer. Suppose at time $t^{+}$, we want to predict the likelihood of $v_{j}$ being the next item in the sequence, we first compute the item representation $\mathbf{e}_{j}^{t^{+}}$ from § 4.1.2. Then, we predict the score as the inner product of $\mathbf{q}_{u}$ and $\mathbf{e}_{j}^{t^{+}}$, formally:

Note that there is no information leakage in the prediction process, i.e., we do not assume access to the popularity statistics of $v_{j}$ at time $t^{+}$ if we are at time $t^{+}$ (§ 4.1.2).

We follow the zero-shot inference setting introduced in § 4.3 and report the results in Table 2. We also include the results of PrepRec and the best-performing sequential recommenders trained on the target dataset for reference. In the zero-shot setting, PrepRec shows minimal performance reduction in the target datasets (i.e., 6% maximum and 2% average reduction in R@10). The best zero-shot transfer results from PrepRec only fall short against the selected sequential recommendation baselines by up to 4% and even outperform (by up to 6.5%) them on the Epinions and Office. We found that PrepRec trained on douban-movie and amazon-tools show the highest generalizability, even outperforming the target-trained models on Music (0.811 vs. 0.782 on R@10). We conjecture that this is because Movie is the largest dataset in terms of the number of interactions. Overall, these results show PrepRec ’s effectiveness in zero-shot transfer without any training on interaction data or side information. In addition, this experiment also demonstrates that the popularity dynamics-based item and sequence representations are generalizable across domains.

We further investigate the robustness of PrepRec to possible noise in zero-shot transfer by adding Gaussian noise $\sim\mathcal{N}(0,\sigma)$ to the item popularity statistics and evaluate the zero-shot transfer performance on Douban-Music and Epinions from Douban-Movie. We randomly choose some percentage of items in the sequence to add noise, as indicated in  Figure 3. We find that PrepRec is relatively robust to noise, maintaining robust performance across different noise levels at 20% noised interaction. We attribute this to the model’s ability to learn from the overall popularity dynamics, which is less affected by noise in individual item popularity statistics. In addition, when the noise level is relatively low, e.g., $\sigma\leq 5$, even if 100% of the sequence is noised, PrepRec still holds the performance, indicating significant item popularity shifts exist in the sequence (Figure 1).

We show comparisons of PrepRec against state-of-the-art sequential recommenders in the regular sequential recommendation tasks (Table 3), i.e., all models are trained from scratch. PrepRec achieves competitive performance—within 2% in R@10 and 5% in N@10, with the state-of-the-art baselines. On Epinions, PrepRec even outperforms all baselines by 7.3%, particularly impressive since PrepRec has significantly fewer model parameters (Table 5).

PrepRec explicitly models popularity information and the MostPop demonstrates decent performance compared to the remaining baselines, thus we conduct an additional experiment (Table 4) where we sample the unobserved (negative) items based on their popularity (Sun et al., 2019). As shown in Table 4, MostPop’s performance dropped significantly, while PrepRec shows even more competitive performance on some datasets (e.g., Music and Epinions). This suggests PrepRec learns discriminative item and sequence representations.

PrepRec learns item representations through popularity dynamics, which is conceptually different from learning representations specific to each item ID. Therefore, we propose a simple post-hoc interpolation to investigate how much can popularity dynamics explain the performance of state-of-the-art sequential recommenders. We interpolate the scores from PrepRec with the scores from BERT4Rec as follows: $\hat{y}_{intp}(v^{t}_{j}|\mathcal{S}_{u})=\alpha*\hat{y}_{O}(v^{t}_{j}|\mathcal{S}_{u})+(1-\alpha)*\hat{y}_{S}(v_{j}|\mathcal{S}_{u})$, where $\hat{y}_{O}(v^{t}_{j}|\mathcal{S}_{u})$ and $\hat{y}_{S}(v_{j}|\mathcal{S}_{u})$ are the scores from PrepRec (Equation 10) and BERT4Rec, respectively. We set $\alpha=0.5$ for all datasets. After interpolation, the performance significantly boosts by up to 34.9% in N@10. Gains are largest in the medium and low-density datasets (Epinions, Amazon), indicating that our model complements existing methods in sparse datasets where item embeddings are less informative. Therefore, it is crucial to consider popularity dynamics to maximize performance.

We analyze the performance of PrepRec in detail. We separate test items into equally sized groups based on their popularity in the training set then compute the average R@10 and N@10 for each group (Figure 4). PrepRec achieves better performance on item group with the least interactions, i.e., long-tail items, while the SasRec and BERT4Rec show stronger performance on popular items. Long-tail item recommendation is a particularly challenging task explored by many previous works (Sankar et al., 2021) and requires recommenders able to learn high-quality representations with just a few interactions. This corresponds to our observation that PrepRec is more robust to data sparsity and can learn discriminative item and sequence representations (§ 5.4.1), showing that long-tail item recommendation can benefit from PrepRec ’s popularity dynamics-based item representations.

Here, we assess the importance of different components crucial to PrepRec , i.e., relative time encoding (§ 4.1.3), positional encoding (§ 4.1.4), popularity encoder $E_{p}$ (§ 4.1.1), and resolutions for popularity dynamics (§ 4.1.2). We find that removing relative time encoding ${\bm{T}}$ results in the largest performance drop on both the Music and Office datasets. This suggests that the relative time encoding is crucial for effectively capturing the popularity dynamics. Removing positional encoding ${\bm{P}}$ results in a maximum of 2.2% drop in R@10 on the Office dataset, indicating positional encoding is important for capturing sequential information. In addition, changing $E_{p}$ to the non-linear sunusoidal encoding shows worse performance on all datasets, meaning that the linear encoding is more suitable for capturing the popularity dynamics. On the music dataset, removing coarse popularity encoding $\mathcal{P}$ results improves the performance by 2% in R@10, while removing fine popularity encoding $\mathcal{H}$ results in a 7.5% drop in R@10. This suggests that the music domain is more sensitive to recent trends in popularity. Coarse and fine popularity encodings complement each other on other datasets.

We examine the effect of different preprocessing weights $\gamma$ used in popularity calculation (§ 4.1.1). In particular, $\gamma=1$ corresponds to the cumulative popularity, or in other words, at a given time period $t$, the overall number of interactions up to period $t$. On the other hand, $\gamma=0$ corresponds to the current popularity, or percentiles are calculated over interactions just in $t$, same as $b_{j}^{t}$ in Equation 1. When $\gamma=0.5$, it can be interpreted as interactions being exponentially weighted by time, with a half-life of 1 time period. We find that $\gamma=0.5$ outperforms the other two settings, with the largest gains of around 12% R@10 and 27% N@10 over cumul-pop in the dense Music dataset, and the largest gains over curr-pop in the sparser Office (5% N@10 and 4% N@10) and Epinions ($15\%$ R@10 and 17%N@10) datasets. We suspect this is due to cumulative measures in denser datasets failing to capture recent trends due to the large historical presence, while current-only measures in sparser datasets convey too little or noisy information and lose the information of long-term trends. curr-pop shows decent performance on the Music dataset, suggesting that Music trends might be more cyclical and thus the current popularity is more informative.

We study the effect of different time horizons to PrepRec . We found that in general, long-term horizons of 30 days and short-term horizons of 7 days perform worse than the other settings. This is likely because the long-term horizon might lead to the lack of resolutions in popularity statistics. We also find that depending on the dataset, the effect of different time horizons also varies. For example, both Music and Epinions show larger performance decrease from short to long-term horizons than Office. This could be because Music and Epinions are more sensitive to recent trends than Office, or their data are denser in terms of time granularity.