How to Index Item IDs for Recommendation Foundation Models

This paper studies the indexing problem based on P5 (Geng et al., 2022). P5 is a representative recommendation foundation model which enhances the generalization capabilities of existing recommender systems by integrating various tasks and personalized instruction prompts to pre-train a foundation model for recommendation. These tasks include sequential recommendation, rating prediction, explanation generation, review summarization, and direct recommendation. P5 is trained using input-target pairs of texts generated from a collection of prompt templates featuring personalized fields for distinct users and items: an example input prompt for sequential recommendation can be a description of user-item interactions such as “According to the places user_1 has visited: location_1123, location_4332, location_8463, location_12312, can you recommend another place for the user?” and the output text is the next generated item such as “Output: location_1934”. In this study, we focus on the sequential recommendation task since it explicitly relies on the item interactions presented in the input prompt, making it highly sensitive to different indexing methods.

In this paper, we need to introduce Out-of-vocabulary (OOV) tokens to construct item indices in some indexing methods, which are tokens that are not part of the normal vocabulary of the language model. In our case, they are tokens that do not exist in the default T5 vocabulary (Raffel et al., 2020). To distinguish the newly created OOV tokens from existent tokens, we use angle brackets “$\langle\rangle$” to represent the newly created OOV token, and use text without “$\langle\rangle$” to represent an existent token in the default tokenizer. All OOV tokens are randomly initialized in the model and thus the text enclosed in “$\langle\rangle$” does not influence embeddings of the OOV tokens. The text within angle brackets “$\langle\rangle$” could be words or numbers, but no matter which case, the text within angle brackets only functions to distinguish different OOV tokens and it is irrelevant to the existent tokens. For example, $\langle$restaurant$\rangle$ $\langle$Greek$\rangle$ $\langle$2$\rangle$ is the index for an item in Yelp consisting three OOV tokens, where $\langle$restaurant$\rangle$ is a different token from the plain English word “restaurant”, and $\langle$2$\rangle$ is a different token from the number “2”. When we need to use the existent plain word tokens, we will use them without the angle brackets, such as “restaurant” and “2”.

Experiments are conducted on Amazon Sports $\&$ Outdoors, Amazon Beauty, and the Yelp dataset. The Amazon datasets (Ni et al., 2019)¹¹1https://cseweb.ucsd.edu/ jmcauley/datasets/amazon_v2/ are sourced from Amazon.com for product recommendations, while the Yelp dataset²²2https://www.yelp.com/dataset provides a collection of user ratings and reviews for business recommendation. We use transaction records from Jan 1, 2019 to Dec 31, 2019, as in the original P5 paper (Geng et al., 2022). The detailed statistics for these datasets can be found in Table 1.

These datasets organize user-item interactions by individual users. We split the datasets into training, validation, and testing by the frequently used leave-one-out setting: for each user’s interaction sequence, we put the second-to-last item into the validation set, put the last item into the testing set, and put all other items of the sequence into the training set. For example, suppose the interaction sequence of user_i is {item_i,1, item_i,2, item_i,3, $\cdots$, item_i,k-1, item_i,k}. Then the prediction of item_i,k-1 based on the sequence {item_i,1, item_i,2, item_i,3, $\cdots$, item_i,k-2} is used for validation and the prediction of item_i,k based on the sequence {item_i,1, item_i,2, item_i,3, $\cdots$, item_i,k-1} is used for testing.

We motivate the exploration of indexing methods starting from three trivial indexing methods:

• •

  Random Indexing (RID): Assigning each item with a random number as the item ID. The number is further tokenized into a sequence of sub-tokens based on the SentencePiece tokenizer (Sennrich et al., 2016), as did in P5 (Geng et al., 2022). For example, a Yelp item is randomly assigned the number “4332”, and “4332” is represented as a sequence of tokens “43”“32”.

• •

  Title Indexing (TID): Using the item title to represent the item which is also tokenized by SentencePiece (Sennrich et al., 2016). For example, the Yelp item “Las Vegas Cigar Outlet” is represented as a sequence of tokens “Las”“Vegas”“Ci”“gar”“Outlet”.

• •

  Independent Indexing (IID): Creating an independent OOV extra token that needs to be learned for each item. For example, a Yelp item is represented as $\langle$IID5$\rangle$ which is an independent extra token specifically allocated for this item. In the rest of the paper, tokens created for IID will always start with the letters “IID”.

RID generates random numerical indices, leading to potential overlaps between unrelated items after tokenization. For example, two items “4332” and “4389” would be tokenized into “43”“32” and “43”“89”, respectively, which means that they always share the same sub-token “43” even though the two items may be totally unrelated with each other. This unintended overlap may establish arbitrary relationships among items, introducing unwanted bias to model training. As the overlaps stem from the index structure, they are impossible to eliminate no matter how the model learns from data. Consequently, RID is considered an unfavorable method.

TID makes the task more challenging since the model needs to memorize and generate lengthy item titles. Besides, certain words or expressions in the title could be unrelated to the real content of the item, also, very different items may share overlapping tokens in their title, and thus semantics derived from the titles may introduce strong linguistic biases (Zhang et al., 2021). For example, the movies “The Lord of the Rings” and “The Lord of War” share many tokens in their titles (“the”, “lord”, “of”), but they are two very different movies: the former is an epic fantasy, while the later is a crime drama. In general, two irrelevant items could have very similar titles, such as Apple the fruit and Apple the company, while two closely related items may have very different titles, such as the classic “beer and diaper” example in data mining (Li et al., 2023). As a result, using title as ID may encode misleading semantics into the generation process, similar as the problem of random indexing.

IID uses single-token indices for items without assuming any prior information about the items, making the item representations easier for language models to learn compared to RID and TID. Though better than RID and TID, it still has limited performance due to considering all items independent from each other when creating item IDs. It could also incur prohibitively long training time if a large number of new tokens are required to create.

The aforementioned analysis implies that none of the three methods is optimal. To validate this, we provide experimental results to show their suboptimal performance. We evaluate the three indexing methods against two strong and widely-used baselines: SASRec (Kang and McAuley, 2018) and S³-Rec (Zhou et al., 2020). Results are shown in Table 2, where the best result for each metric is highlighted in bold and the second-best result is underlined with wavy lines. Based on Table 2, RID and TID underperform relative to the baselines, while IID offers minor gains at the cost of introducing more learnable tokens because each item is considered as an independent new token. As a result, these indexing methods are considered suboptimal and we will further explore nontrivial indexing methods in the next section.

Sequential indexing is a straightforward method to leverage collaborative information for item indexing. Items interacted consecutively by a user are assigned consecutive numerical indices, reflecting their co-occurrence. Take Table 3 as an example, items are assigned with IDs consecutively starting from the first user and all the way to the last user. If an item has already been indexed in previous users’ interaction sequence, such as item 1001 in User 2’s sequence (and all other squared items in the table), then the item’s already assigned ID will be used, otherwise, an incremental new ID will be created and assigned to the item. Notice that the item indexing process only depends on the training sequences, while the validate and testing items do not participate in the indexing process. After the indexing process is finished, the validation and testing items are assigned the corresponding IDs that have already been established during the indexing process. Upon tokenization based on the SentencePiece tokenizer (Sennrich et al., 2016), an ID such as “1001” will be tokenized into “100”“1”, while “1002” will be tokenized into “100”“2”, resulting in the shared token “100” for these two consecutive items. This gives us encoding similarity between those items that co-appear in at least one user’s sequence. As a result, this simple sequential indexing method is able to capture collaborative information on some occasions.

One minor note is that we initiate item index enumeration at 1001. We initiate at 1001 instead of 1 for two reasons: 1) the SentencePiece tokenizer does not tokenize some numbers smaller than 1000 into multiple sub-tokens, such as the number 12, and thus items assigned with these small numbers will be completely independent of each other, 2) after tokenization, smaller numbers could become complete subsets of larger tokenized numbers, e.g., ID “12” can be a subset of ID “12”“34”, which may enforce false correlation between items.

Nevertheless, sequential indexing also has limitations: 1) Adjacently indexed items not interacted together by the same user may erroneously share tokens; for instance, the last item of User 2 is indexed as 1014 (tokenized as “10”“14”) and the first item of User 3 is indexed as 1015 (tokenized as “10”“15”), then the token “10” will be shared despite a lack of co-occurrence between the two items, 2) it cannot capture similarities based on the frequency of co-occurrence; for example, suppose items 1001 and 1002 co-occur once while items 1002 and 1003 co-occur ten times, both pairs will still share only one token, failing to convey frequency information, and 3) user ordering in the training data affects the results; for example, if we exchange the rows of User 1 and User 2 in Table 3, then the indexing result would be different. Although sequential indexing has its shortcomings, it can still yield relatively favorable results that are close to, or even surpass, the baselines.

Sequential Indexing is a preliminary method for integrating collaborative information into item indexing. To effectively capture the essence of collaborative filtering, we explore the Collaborative Indexing (CID) approach, which employs spectral clustering based on Spectral Matrix Factorization (SMF) (Von Luxburg, 2007; Ng et al., 2001) to generate item indices. This method is based on the premise that items with more frequent co-occurrence are more similar and should share more overlapping tokens in index construction. The core concept involves constructing a co-occurrence graph for all items based on the training dataset and using spectral clustering to group items into clusters, ensuring that items within the same cluster share tokens when constructing indices.

Semantic (content-based) Indexing (SemID) utilizes item metadata to construct IDs for items. As shown in Figure 3, items’ categories form a hierarchical structure (Zhu et al., 2018), with each non-leaf node (large yellow node) representing a category and each leaf node (small blue node) representing an item. Each non-leaf node is assigned an independent extra token, and each leaf node receives a unique extra token under its parent node. To create an item index, the tokens of non-leaf nodes and leaf nodes are concatenated along the path from root to leaf. Take the bolded path in Figure 3 as an example, the item’s categories range from coarse to fine-grained as $\langle$Makeup$\rangle$, $\langle$Lips$\rangle$, $\langle$Lip_Liners$\rangle$, and its leaf node token is $\langle$5$\rangle$, which differentiates this item from other items under the Lip Liners category, then the item would be indexed as $\langle$Makeup$\rangle$$\langle$Lips$\rangle$$\langle$Lip_Liners$\rangle$$\langle$5$\rangle$.

Hybrid Indexing (HID) is not a single specific indexing method but a category of methods. It concatenates multiple indices introduced above into one index, such as SID+IID, CID+IID, SemID+IID, SemID+CID, etc. This approach aims to leverage the advantages of different indexing techniques to produce better indices. In this paper we implement four combinations and here are the details:

For SID+IID: we append an independent extra token at the end of the sequential ID for each item. Suppose the SID of an item after tokenization is “10”“18”, and its IID index is $\langle$IID982$\rangle$, then the HID index will be “10”“18”$\langle$IID982$\rangle$. Thus it contains some item co-appearance information from SID and meanwhile ensure the item distinction through IID.

For CID and SemID, before we concatenate them with IID, we first remove the last token (the leaf node token) from them since the last token simply functions to differentiate an item from others under the same parent non-leaf node. For CID+IID: suppose an item’s CID is $\langle 1\rangle\langle 9\rangle\langle 5\rangle\langle 4\rangle$, and its IID is $\langle$IID28$\rangle$, then the item’s HID would be $\langle 1\rangle\langle 9\rangle\langle 5\rangle\langle$IID28$\rangle$. For SemID+IID: suppose an item’s SemID is $\langle$Makeup$\rangle$$\langle$Lips$\rangle$$\langle$Lip_Liners$\rangle$$\langle$5$\rangle$, and its IID is $\langle$IID1023$\rangle$, then the HID is $\langle$Makeup$\rangle$$\langle$Lips$\rangle$$\langle$Lip_Liners$\rangle$$\langle$IID1023$\rangle$. The final index incorporates both collaborative information from CID (or metadata content information in SID), and a special IID token that differentiates the item from all others, ensuring item distinction while preserving the advantages of the CID (or SID).

For SemID+CID: we concatenate the SemID and CID in either order, hoping to combine both metadata content information and collaborative information. Since both SemID and CID contain leaf node tokens to distinguish items under one parent node, we only need to retain one of them, e.g., we retain the CID leaf node token. Suppose the SemID is $\langle$Makeup$\rangle$$\langle$Lips$\rangle$$\langle$Lip_Liners$\rangle$$\langle$5$\rangle$ and the CID is $\langle 1\rangle\langle 9\rangle\langle 5\rangle\langle 4\rangle$. If we put SemID first, the final HID index is $\langle$Makeup$\rangle$$\langle$Lips$\rangle$$\langle$Lip_Liners$\rangle$$\langle 1\rangle\langle 9\rangle\langle 5\rangle\langle 4\rangle$; otherwise, the HID index is $\langle 1\rangle\langle 9\rangle\langle 5\rangle\langle 4\rangle\langle$Makeup$\rangle$$\langle$Lips$\rangle$$\langle$Lip_Liners$\rangle$.

In the following experiments, we will evaluate and compare the various different HIDs.

The datasets and their pre-processing methods have been introduced in Section 3.3. In this section, we introduce the baselines. We apply the various item indexing methods into the P5 framework (Geng et al., 2022) for sequential recommendation and compare with several representative sequential recommendation methods as baselines: Caser (Tang and Wang, 2018): This approach treats sequential recommendation as a Markov Chain and utilizes convolutional neural network to model user interests. HGN (Ma et al., 2019): This approach leverages hierarchical gating networks to learn user behaviors from both long-term and short-term perspectives. GRU4Rec (Jannach and Ludewig, 2017): Originally proposed for session-based recommendation, this approach employs GRU to model the user click history sequence. BERT4Rec (Sun et al., 2019): This approach mimics BERT-style masked language modeling, learning a bidirectional representation for sequential recommendation. FDSA (Zhang et al., 2019b): Focusing on feature transition patterns, this approach models the feature sequence with a self-attention module. SASRec (Kang and McAuley, 2018): Adopting a self-attention mechanism in a sequential recommendation model, this approach reconciles the properties of Markov Chains and RNN-based approaches. S³-Rec (Zhou et al., 2020): Leveraging self-supervised objectives on meta information of items, this approach helps the sequential recommendation model to better discover the correlations among different items and their attributes. For comparison, we utilize the implementation of S³-Rec and its baselines.

Following the P5 framework (Geng et al., 2022), our implementation utilizes T5 as the backbone (Raffel et al., 2020): there are 6 layers for both encoder and decoder, the model dimensionality is 512 with 8-headed attention. For tokenization, we use the default SentencePiece tokenizer (Sennrich et al., 2016) with a vocabulary size of 32,128 for parsing sub-word units. All independent extra tokens are not further tokenized. We use the same sequential recommendation prompts as P5 (Geng et al., 2022) to convert sequential information into texts. We pre-train P5 for 20 epochs using AdamW optimizer on two NVIDIA RTX A5000 GPUs with a batch size of 64, a peak learning rate of 1e-3. We apply warm-up for the first 5$\%$ of all training steps to adjust the learning rate.

RID, TID, and SID do not involve creating OOV tokens since their item indices comprise tokens from the default T5 tokenizer, while IID, CID, SemID, and HID involve creating extra OOV tokens, extending the original vocabulary. All tokens used in these indexing methods, excluding TID, are randomly initialized rather than using T5’s pre-trained embeddings for initialization. This is due to our observation that the pre-trained T5’s a priori semantics about numbers adversely impact the learning of item semantics and the recommendation performance during experimentation. We use T5’s pre-trained token embeddings for initializing TID tokens since TID only involves plain word tokens.

The overall experimental results are presented in Table 4 with all baselines. The best result for each metric is highlighted in bold, while the second-best result is underlined with wavy lines. For each indexing method, if the result surpasses the best baseline result, it is emphasized by underlining with straight lines. In general, RID, TID and IID cannot beat the baseline results in most cases, while most of the advanced indexing methods (SID, CID, SemID and the HIDs) surpass the baseline results. A more detailed breakdown analysis is as follows.

In Table 4, the first block contains all the baseline results. The second block contains the basic indexing methods, where RID and TID consistently perform worse than baselines, while IID in general performs better. The third block contains three advanced indexing methods. We can see that SID performs worse than CID and SemID on Amazon datasets but better on Yelp, while CID performs better than SemID across different datasets, indicating that constructing indices using collaborative information is more beneficial than using metadata, because CID can better capture item relationships from user behaviors by collaborative learning from the wisdom of the crowd, which could be more effective than only using items’ metadata. The fourth block in the table contains HID results with several different implementations: SID+IID, CID+IID, SemID+IID, and SemID+CID. CID+IID and SemID+IID perform much better than all other indexing methods while SID+IID and SemID+CID perform worse. In the following subsections, we will further analyze the results in the third and fourth blocks in detail based on more comprehensive experiments.

Table 4 shows that though simple in nature, SID can generate favorable results that are close to or surpass baselines. In Section 4.1, we explored the construction of SID and its limitations, specifically, the indexing result can be influenced by the user ordering, e.g., if we exchange the rows of User 1 and User 2 in Table 3, then the indexing result would be different. In this section, we present the results of SID using four different user orderings, which substantiate this claim and also suggest the most effective ordering to use:

1. (1)

   Time-Sensitive Ordering (TSO): Users are ordered chronologically in the original dataset based on their initial interaction with the system. Subsequent interactions are recorded and new records are created for previously unrecorded users upon their first interaction with the system. By sorting and processing interactions based on their timestamps, we ensure that users with earlier initial interactions are recorded first.

2. (2)

   Random Ordering (RO): Users are ordered randomly.

3. (3)

   Short-to-Long Ordering (S2LO): Users are organized according to their number of interactions, arranged in ascending order from the fewest to the most interactions.

4. (4)

   Long-to-Short Ordering (L2SO): Users are sorted in descending order from the most to the fewest interactions.

Table 5 presents the performance of the four settings. Our observations indicate that, in general, the relative performance is as follows: Time-Sensitive $>$ $\{$Long-to-Short, Short-to-Long$\}$ $>$ Random. The observations imply that time plays an important role in sequential indexing: items that are interacted at similar times, even by different users, may be more similar to each other compared to items being interacted at vastly different times. As a result, items that occurred at similar times are more likely to be co-interacted by certain users. Thus, using the time-related information when ordering users is likely to improve the performance.

Considering these observations, we recommend that future implementations of the simple SID method consider using the time-sensitive user ordering strategies to enhance performance. Note that the original Amazon and Yelp datasets already used a time-sensitive ordering to arrange the users. As a result, to generate indices using SID, we simply need to incrementally index the items from the first user all the way to the last user.

CID involves two hyper-parameters: $N$ and $k$, where $N$ is the number of clusters at each level of the clustering, and $k$ is the maximum number of items allowed in the final cluster. Varying these hyper-parameters results in different numbers of independent extra tokens and recommendation performances.

In Figure 4, we present hit@10 results for various $N$ and $k$ value combinations on the Beauty dataset. When $k$ = 50, the performance is below 4.5%, which is significantly lower than the baselines and some basic indexing methods. However, when $k$ is greater than 100, the performances improve considerably. Furthermore, Table 6 shows hit@10 results for multiple configurations with $k\in\{200,500,1000\}$, $N\in\{10,20\}$ and on all three datasets. In these different settings, nearly all the CID results outperform the baselines, indicating that CID is relatively easy to fine-tune with respect to its hyper-parameters.

Based on our observations, we can draw the following conclusions: (1) Extremely small $k$ values lead to suboptimal performance regardless of the chosen $N$. When $k$ = 50, the performance is below the baselines. This can be attributed to the limited expressiveness of a small number of new tokens, which cannot adequately capture the diversity of items. (2) Different $k$ and $N$ combinations yield varying ID lengths (i.e., the number of tokens in an ID). We compute the average ID length for each $k$ and $N$ hyper-parameter setting, and the results are shown in Figure 5 (for Beauty) and Table 7 (for all datasets). Combining Figure 4 and 5, as well as Table 6 and 7, we find that the optimal recommendation results are usually observed when the average ID length is between 3 and 4. For example, the squared points in Figure 5 shows all cases whose average ID length is between 3 and 4 for the Beauty dataset, and we can see that these points also correspond to the optimal performance on each line in Figure 4. Similarly, the best or second-best results in Table 6 also corresponds to 3$\sim$4 ID lengths in Table 7 in most cases.

Based on these observations, we recommend that future CID implementations use hyperparameters that generate an average ID length between 3 and 4. However, it is worth noting that different datasets may require slightly different lengths for optimal performance.

SemID uses metadata to construct item indices. In our experiments, we observe that if the categories follow a hierarchical tree structure, then the performance tends to improve. Category information in datasets is usually not a tree structure because in some cases, one category name can occur under different parent categories, which makes the categories into a graph but not a tree. Table 8 are two examples in Amazon Beauty, where the category “Eyes” occurs under both “Skin Care” and “Makeup Remover”, and the category “Creams” occurs under both “Skin Care” and “Moisturizers”.

To test whether the tree structure in categories is crucial, we compare two different settings in our experiments:

1. (1)

   Non-tree-structure setting: we directly use the category names to create the corresponding independent OOV extra tokens. For example, an item under “Beauty”, “Skin Care”, “Eyes”, and another item under “Beauty”, “Makeup”, “Makeup Remover”, “Eyes” will share the token $\langle$Eyes$\rangle$.

2. (2)

   Tree-structure setting: we enforce a tree structure on the categories by creating different OOV tokens when the same category name occurs at different places. For example, the category “Eyes” under “Beauty”, “Skin Care” will correspond to token $\langle$Eyes1$\rangle$ while that under “Beauty”, “Makeup”, “Makeup Remover” corresponds to $\langle$Eyes2$\rangle$.

Table 9 illustrates the importance of hierarchical information for SemID’s effectiveness. The more closely the categories adhere to a hierarchical structure, the better the performance of the model. This is likely because a hierarchically organized category list helps reduce the search space during the generation process. Consequently, this finding highlights the importance of properly organizing and structuring category information when implementing SemID in recommendation foundation models.

Based on the results presented in Table 4, CID+IID and SemID+IID show much better performance compared to their respective CID and SemID counterparts. But SID+IID does not improve on SID, and SemID+CID not only does not improve but decreases the performance a lot. Both CID+IID and SemID+IID are constructed by assigning each item an independent extra token and concatenating it after the sequence of cluster IDs or category IDs. These combinations maintain the original index lengths while preserving the hierarchical structure. The improved performance can be attributed to the increased expressiveness of the indices provided by the extra token, as well as the retention of either collaborative information or metadata information within the hybrid index. This combination of factors contributes to the performance enhancement observed in CID+IID and SemID+IID methods.

SID+IID is created by appending an independent extra token after the original sequential index, increasing the ID length by 1. SID+IID does not improve the performance possibly because the additional token interferes the time-sensitive information encoded as a numerical style in the original sequential indices. SemID+CID, which is created by concatenating category IDs with cluster IDs or vice versa, exhibits suboptimal performance, as shown in Table 4. This holds true for both concatenation orders: category IDs followed by CID indices and cluster IDs followed by SemID indices. The reason behind this suboptimal performance is that it generates excessively long indices and disrupts the hierarchical structure encoded in both SemID and CID. Considering our findings, we recommend employing CID+IID and SemID+IID as hybrid indices for recommendation foundation models, as they have demonstrated superior performance in such scenarios.