Unleash LLMs Potential for Recommendation by Coordinating Twin-Tower Dynamic Semantic Token Generator

To sufficiently integrate the semantic knowledge and the collaborative knowledge, we propose the dynamic knowledge fusion framework for TTDS recommender. Within this framework, the TTDS recommender system consists of a twin-tower semantic token generator and a LLM-based recommender backbone. The former aims to discretize the continuous representation of user/item into corresponding semantic index. The latter is not only in charge of converting the textual content of user/item into continuous representation, but also responsible for the next-item prediction based on the user/item semantic index and human instruction.

Consider a system of $K$ items and $J$ users, the interactive item sequence of user $U_{j}$ can be denoted as $S_{j}=[I_{k_{1}},I_{k_{2}},\cdots,I_{k_{n}}]$, where the subscript $k_{x}$ identifies the interactive item and $n$ is the sequence length. First, the LLM backbone $f$ embeds the related textual content of user $U_{j}$ and item $I_{k_{x}}$, denoted as $C_{j}^{U}$ and $C_{k_{x}}^{I}$, into continuous representation, formulated as follows,

where $C$ refers to the textual content $C_{j}^{U}$ and $C_{k_{x}}^{I}$, $X$ corresponds to the continuous representation $X_{j}^{U}$ and $X_{k_{x}}^{I}$. Afterwards, the twin-tower semantic token generator takes the continuous representation $X$ as input and produce semantic index for both user and item. To be more specific, the semantic token generator functions as vector quantization, which adopts a learnable codebook $\mathcal{C}=\{c_{i}\}_{i=1}^{|\mathcal{C}|}$ to approximate the input representation, formulated as follows,

where $\mathcal{D}$ represents distance metric and $i^{*}$ is the semantic index of $X$. Within the learnable codebook $\mathcal{C}$, vector $c^{*}$ (i.e., $c_{i^{*}}$) is the closest one to the input representation $X$.

The twin-tower semantic token generator quantizes the continuous embedding $X$ into the discrete index $i^{*}$ (i.e., semantic index) which serves as an identifier. Note that in Formula 7, the semantic index $i^{*}$ is selected on the basis of representation distance and thus the similar representations will be allocated with similar indices. Accordingly, the vocabulary of LLM backbone is extended with the newly generated semantic token, to predict the next item in an end-to-end manner. Based on the semantic index and the interaction history, we prompt the LLM backbone by natural language human instruction. Specifically, the original item index and user index within the interaction history are both replaced with the corresponding semantic index. The rewritten interaction history is organized as a semantic index sequence in chronological order and the sequential recommendation instruction is also merged into the prompt. An example instruction prompt is given as follows: You are an expert in sequential recommendation. Please predict the most suitable item for user $<i^{*}_{U_{0}}>$, based on the historical interactions: $<i^{*}_{I_{3}}>,<i^{*}_{I_{7}}>,\cdots,<i^{*}_{I_{15}}>$. Denoting the human instruction as $P$, the LLM backbone $f$ first transforms natural language $P$ into hidden representation $X_{P}$ based on Formula 6. Then a language model head is appended to project the hidden state $X_{P}$ into the semantic token vocabulary, formulated as follows,

where $\hat{i}$ is the semantic index of the recommendation result and an inverse look-up operation opposite to Formula 7 can identify the original item index. The sequential recommendation task based on human instruction prompt can be conveniently formulated as language generation task. The optimization objective is defined as the negative log-likelihood as follows,

where $F$ is the combination of the LLM backbone and the LM head, $B$ represents the batch-size, $i^{*}_{b}$ and $P_{b}$ is the ground-truth semantic index and the human instruction of the $b$-th batch, respectively.

Considering the computation increment caused by the extended vocabulary, TTDS recommender adopts hierarchical index mechanism whose expressive space increases exponentially with the index length. Within the hierarchical index mechanism, $M$ semantic tokens constitute one semantic index to represents one user/item. Taking item $I_{k}$ as an example, the quantization procedure in Formula 7 will be conducted $M$ times in a residual manner, formulated as follows,

The semantic index of item $I_{k}$ can be represented as

Therefore, a $M$ length hierarchical index mechanism with $N$ as base can theoretically represents $N^{M}$ distinct items and the newly introduced tokens are merely $N*M$.

Within the proposed dynamic knowledge fusion framework, the newly generated user/item semantic tokens are used to extend the LLM vocabulary, to predict the next-item in an end-to-end manner. However, the LLM backbone is unfamiliar with the under-trained semantic tokens, compared with the natural language tokens well pre-trained on large-scale corpus. To facilitate the LLM understanding on semantic tokens, we propose a dual-modality variational auto-encoder (DM-VAE) which constructs multi-grained alignment between the natural language token and the semantic token. Given the continuous embedding $X^{I}_{k}$ and the semantic index ${\rm Sid}_{I_{k}}$ of item $I_{k}$, we note that $X^{I}_{k}$ and ${\rm Sid}_{I_{k}}$ are two different modalities of the same object. Therefore, the LLM representation of ${\rm Sid}_{I_{k}}$ ought to be similar to $X^{I}_{k}$. We design the semantic index level reconstruction based on the optimization objective defined as follows,

where $X^{S}_{k}$ is the LLM representation of ${\rm Sid}_{I_{k}}$. Notably, based on the hierarchical index mechanism, a single semantic index consists of $M$ semantic tokens. We design the semantic token level reconstruction based on the optimization objective defined as follows,

The above two objective functions are also defined for the user branch. Overall, the optimization objective of DM-VAE is defined as follows,

where $\beta$ is the weighted parameter to control the loss magnitude.

Most of the existing LLM-based recommendation systems merely focus on the item-related information and design item-centred fine-tuning tasks, neglecting the high-order user-item interaction patterns implicated in the user historical behavior. On the basis of the interactive item sequence, the item co-occurrence pattern is easy to capture. However, the high-order user-item interaction patterns, such as the user co-purchase pattern and the user preference pattern, are implicit and difficult to modeling. To this end, we specifically customize a series of fine-tuning tasks oriented for the high-order user-item interaction patterns. First, inspired by the collaborative filtering technique, we design a user prediction task which is the counterpart of item sequential recommendation. We extract the users who purchase the same item and organize them as an interactive user sequence. The LLM backbone is prompted with the former several users and the human instruction is to predict the next user who is interested in the same item. An example instruction prompt is given as follows: You are an recommendation expert. The item ${\rm Sid}_{I_{1}}$ has been clicked by the following users: ${\rm Sid}_{U_{9}},{\rm Sid}_{U_{5}},\cdots,{\rm Sid}_{U_{23}}$. Can you predict another user who is interested in this item?

Furthermore, we also design fine-tuning tasks based on the user recent comments and search queries. The LLM backbone is prompted with the user historical interaction sequence and the human instruction is to deduce the user comment or the search query, which is able to reflect the user preference to a certain degree. Notably, the designed fine-tuning tasks oriented for the high-order user-item interaction patterns can also be formatted as language generation task in a sequence-to-sequence manner, with the optimization objective defined in Formula 9. The overall objective function of TTDS recommender system is defined as follows,

where $\alpha$ is the weighted hyper-parameter.

It is worth noting that the hierarchical index mechanism formulated by Formula 10 and 11 is unable to guarantee the semantic index uniqueness, i.e., different items/users are allocated with distinct semantic indices. Through the argmin operation over distance metric may result in semantic index collision, the corresponding recommender system is still practicable with a fairly low collision rate (Zheng et al., 2024). In our practice, the collision rate is no more than 1%. However, on one hand, the collision rate is affected by the dataset scale, the model architecture, the training parameter, and so on, which is extremely uncontrollable. On the other hand, within some recommender systems, the index uniqueness is significant and must be guaranteed (Rajput et al., 2023; Liu et al., 2024). Inspired by the rehashing method in hash collision, we append an additional token into the semantic index, representing the order inside the collision set (Rajput et al., 2023; Liu et al., 2024). For example, two different items share the same semantic index $<1,2,3>$. Then, the semantic indices will be remapped to $<1,2,3,p_{0}>$ and $<1,2,3,p_{1}>$, which scrupulously guarantee the index uniqueness.

For the LLM-based generative recommender systems, the index mechanism is the fundamental of item representation during both model training and service deployment. CLLM4Rec (Zhu et al., 2024) adopts vanilla item index such as ¡item_1¿ and considers the item index as a special token which can be appended into the language model vocabulary, resulting in a linearly expanded vocabulary. Furthermore, semantic index based methods, including TIGER (Rajput et al., 2023), LC-Rec (Zheng et al., 2024), and LETTER (Wang et al., 2024), adopt residual vector quantization technique to hierarchically allocate item index (e.g., $<10,1,6>$), which reduce the vocabulary scale to a great extent. However, all of the aforementioned methods separate the item indexing process from the downstream sequential recommendation task, hindering the integration of the semantic and collaborative knowledge, thus leading to a sub-optimal performance. To overcome this limitation, we propose a dynamic knowledge fusion framework, where the semantic index generator are jointly optimized with the sequential recommender. Accordingly, the semantic knowledge and the collaborative knowledge are able to be fused sufficiently. Furthermore, we construct multi-grained alignment between the semantic index representation and the textual feature representation of the same item, to facilitate the understanding of newly introduced semantic tokens.

Existing LLM-based generative recommender systems taking the user historical interaction sequence as input and output an item index as the prediction result. Semantic index based methods, including TIGER, LC-Rec, and LETTER, all neglect the high-order user-item interaction patterns, such as the user co-purchase pattern and the user preference pattern, provide next-item prediction solely based on the interactive item sequence which contains the local item co-occurrence pattern. CLLM4Rec takes user comments into consideration and exploits the user preference to some extent. However, the user co-purchase pattern is also overlooked in CLLM4Rec and the vanilla user index fails to capture the user similarity, leading the moderate recommendation performance. In this paper, we propose to tokenize user representation into semantic user index and customize several high-order user-item interaction patterns exploitation tasks, promoting the TTDS recommender to model the collaborative knowledge.

We evaluate the proposed TTDS recommender all the baseline models on public benchmarks from the Amazon Product Review dataset (He and McAuley, 2020), containing user review data from May 1996 to October 2018. Particularly, we extract three categories for the sequential recommendation task, i.e., ”Musical Instruments”,”Arts, Crafts and Sewing”, and ”Video Games”. Within the above datasets, each item is associated with a series of textual contents, including the item title, the detailed description, the item category, and so on. Similar to the user entity, the associated textual contents include the user comment, the search query, and so on. Following standard procedure, inactive users/items with less then 5 interactions are filtered out and the user interactive sequence is created according to the chronological order. The dataset statistics and detailed pre-processing are presented in Appendix A.

To comprehensively demonstrate the multifaceted superiority of TTDS recommender, the evaluation includes the following baselines based on various methodologies.

• •

  MLP-Based: FMLP-Rec (Zhou et al., 2022) proposes an all-MLP model with learnable filters for sequential recommendation, ensuring efficiency and reduces noise signals.

• •

  CNN-Based: Caser (Yuan et al., 2019) captures user behaviors by applying horizontal and vertical convolutional filters.

• •

  RNN-Based: HGN (Ma et al., 2019) utilizes hierarchical gating networks to capture both long-term and short-term user interests from historical behaviors. GRU4Rec (Jannach and Ludewig, 2017) is an sequential recommendation model that utilizes GRU (Chung et al., 2014) to encode the item sequence.

• •

  GNN-Based: SR-GNN (Wu et al., 2019) models session sequences as graph structured data, to capture complex transitions of items. GC-SAN (Xu et al., 2019) utilizes both graph neural network and self-attention mechanism, for session-based recommendation.

• •

  Transformer-Based: BERT4Rec (Sun et al., 2019) adopts a bidirectional Transformer model and combines it with a mask prediction task for the item sequences modeling. SASRec (Kang and McAuley, 2018) exploits a unidirectional transformer model to capture the item sequences and predict the next item. FDSA (Zhang et al., 2019b) focuses on the transformation patterns between item features, modeling both item-level and feature-level sequences separately through self-attention networks. S³-Rec (Zhou et al., 2020) utilizes mutual information maximization to pre-train a self-supervised sequential recommendation model, learning the correlation between items and attributes.

• •

  PLM/LLM-Based: P5-CID (Geng et al., 2022) organizes multiple recommendation tasks in a text-to-text format and models different tasks uniformly using the T5 (Gomez-Uribe and Hunt, 2016) model. TIGER (Rajput et al., 2023) adopts the generative retrieval paradigm for sequential recommendation and introduces a semantic ID to uniquely identify items. CLLM4Rec (Zhu et al., 2024) proposes mixed prompting strategy based on heterogeneous tokens to fulfill sequential recommedation task. LC-Rec (Zheng et al., 2024) extends the TIGER framework and further adopts LlaMA (Touvron et al., 2023) as the recommeder backbone. Moreover, LC-Rec also design several semantic index oriented fine-tuning tasks.

We adopt the Top-K Hit-Rate (HR@K) with K = 1, 5, 10 and the Normalized Discounted Cumulative Gain (NDCG@K) with K = 5, 10 to evaluate the sequential recommendation performance. Following standard setting, the leave-one-out strategy is adopted. Specifically, the most recent item serves as the evaluation data, the second most recent item servers as the validation data, and the remaining interactive items form the training data. As an end-to-end generative recommender system, TTDS performs full ranking evaluation over the whole item set, instead of the sample-based or candidate-based evaluation.

We employ the open source large language model LlaMA (Touvron et al., 2023) developed by Meta and introduce low-rank adaption technique (Hu et al., 2022) for LlaMA fine-tuning. The representation dimension of LlaMA model is 4096. A language model head is appended to translating the hidden representation into the extended vocabulary. For the twin-tower dynamic semantic token generator, the semantic index length $L$ is set as 4, and thus both the item and the user branch have 4 hierarchically connected codebooks respectively. Each codebook consists of 256 quantization embeddings whose dimension is set to 256. The encoder module within the semantic token generator contains a series of linear layers $[4096,2048,1024,512,256]$, gradually projecting the LlaMA representation into the codebook hidden space. We prepare a warm-up phase with a learning rate of 1e-3 for the twin-tower dynamic semantic token generator, by freezing the LlaMA-based recommender. For the TTDS recommender training phase, we adopt the AdamW (Kingma and Ba, 2015) optimizer with a learning rate of 5e-4.

To demonstrate the effectiveness of the proposed innovations, including the dynamic knowledge fusion framework, the dual-modality variational auto-encoder, and the high-order user-item interaction pattern specific tasks, we design ablation study to investigate the contribution of each module. In ablation study, TTDS refers to the twin-tower dynamic semantic recommender which is the unabridged model, STDS refers to single-tower dynamic semantic recommender that removes the TTDS user branch, TTSS refers to twin-tower static semantic recommender that removes the dynamic knowledge fusion framework, , TTDS-DM removes the DM-VAE module, STDS-DM removes both the DM-VAE module and the TTDS user branch, and TTDS-HO removes the specific tuning tasks for the high-order user-item interaction patterns.

The sequential recommendation performance on Amazon Instruments dataset is presented in Table 2. The data presented in Table 2 reveals that progressively integrating various semantic alignment tasks into the sequential recommendation process, which was initially based solely on collaborative semantics, leads to a notable enhancement in performance. The inclusion of these instruction tuning tasks has proven to be advantageous for boosting the effectiveness of sequential recommendations. Furthermore, there is an indication that performance could be further optimized by incorporating an even greater number of semantic alignment tasks into the model.

Furthermore, we investigate the impact of different index mechanism. The dynamic semantic index adopted by TTDS recommender is able to sufficiently integrate the semantic and collaborative knowledge together. To comprehensively justify its superiority, we introduce dynamic locality sensitive hashing (LSH) index, static LSH index, and static random index, denoted as TT-DLSH, TT-SLSH, TT-SR. Since the TTDS recommeder in main experiment adopts dynamic semantic index consisting of 4 tokens, we further train two comparison version of TTDS recommender with the dynamic semantic index length equals 3 and 5, denoted as TTDS-3 and TTDS-5, respectively. Note that the variant TTSS that removes the dynamic knowledge fusion framework adopts the static semantic index, also serving as a contrastive model.

The sequential recommendation performance on Amazon Instruments dataset is presented in Table 3. We can conclude the following observations from the presented result. First, representing each item/user with 4 semantic tokens (i.e., TTDS in main experiment) is appropriate such an dataset volume (ten thousand). TTDS-3 with less semantic index length may suffer from inadequate expressiveness and TTDS-5 with larger representation space will increase the difficulty of target item generation. Second, comparing TTDS recommender with the variants that adopts static semantic index TTSS, the superiority of dynamic semantic index optimized by the dynamic knowledge fusion framework is significant. The outperformance of TT-DLSH over the static variant TT-SLSH further verifies the effectiveness of the proposed dynaminc knowledge fusion framework for the locality sensitive hashing index as well. Third, comparing the semantic index based recommender and the LSH index based recommender, we can note that for the static index mechanism, the LSH index outperforms the semantic index, since the hyperplane projection in locality sensitive hashing is more stable than the vector quantization technique. Conversely, within the dynamic index situation, the capacity of semantic index is sufficiently exploited and thus TTDS recommender achieves significant improvement over the LSH index based variant.