LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application

The User Tower comprises a Content EXtraction (CEX) module and a Preference Alignment (PAL) module, as depicted in Fig. 3. The input of the user tower is a sequence of history items that interact with the user. Each item is described textually according to the prompt template shown in Fig. 3. The prompt is designed to be highly concise to effectively assess the informativeness of the textual description. The CEX module processes these item descriptions employing a pre-trained LLM and an average pooling layer to generate content embeddings $E^{c}$. During training, parameters of the pretrained LLM remain frozen, and the hidden states of the final decoder layer are used as output embeddings, which are subsequently sent to the pooling layer, as illustrated in Fig. 3(a). For the entire history interaction sequence $U^{hist}$, the CEX module converts the textual description of each item into a content embedding $E^{c}$, forming a content embedding sequence. Each item is processed independently by the CEX module.

The Preference ALignment (PAL) module captures user preference and outputs user embeddings based on the content embedding sequence. The PAL module starts with a content adaptor to perform dimension transformation. Subsequently, the transformer with 12 layers serves as the backbone network, following the configuration of BERT-base (Devlin et al. 2018) model. This transformer is specifically designed to learn implicit item relationships and model user preferences. Unlike bidirectional attention in BERT (Devlin et al. 2018), our module employs the causal attention mechanism to model sequential dependency by focusing exclusively on past items, in line with the chronological nature of user preferences. The output embeddings of our transformer are further processed through online projection layers to produce user embeddings $E^{user}\in\mathbf{R}^{64}$, which are directly utilized for online e-commerce recommender in Fig. 5.

The item tower processes textual descriptions of item content and outputs item embedding $E^{item}$ tailored for the recommendation domain. As depicted in Fig. 2, three variants of the item tower are proposed.

ItemTower(a) employs the same causal attention mechanism as the User Tower, whereas ItemTower(b) adopts the self-attention mechanism where each item attends exclusively to its own content. Despite this difference, both variants maintain an identical model architecture and share the same weights with the UserTower. To bridge the substantial gap between open-world and collaborative domains, both variants project content embeddings from the LLM into user/item embeddings used in recommendation system, following the “LLM-to-Rec” approach. In contrast, ItemTower(c) directly leverages the content embedding $E^{c}$ as the item embedding $E^{item}$ to guide user perference learning in the “Rec-to-LLM” manner. In the training stage, ItemTower(a) processes the entire user target sequence $U^{tar}$ as input, while ItemTower(b) and ItemTower(c) handle individual items independently. In the inference stage, all three variants take a single item as input and generate item embedding independently. Due to superior performance in Tab. 5, ItemTower(a) is adopted as the default setting.

To bridge the gap between content embeddings in the open-world domain of LLMs and user/item embeddings in the collaborative domain of RS, we align our training objectives with those of the online ranking model. In an online RS, the ranking model computes similarities between the user embedding and the embeddings of all items in the gallery. The top k items with the highest similarity scores are identified as those likely to interest the user. Thus, we employ a self-supervised contrastive learning mechanism to model user preferences, aligning with the goals of the online RS. This approach maximizes the similarity between the user embedding and the embeddings of relevant items while minimizing similarities with irrelevant items. We sample user embeddings from the user history sequence and item embeddings from the target sequence of the same user to construct positive sample pairs. Target item embeddings of other users in the same batch are sampled as negative. To fully exploit user interactions and capture user long-term interest, we adopt the dense all action loss (Pancha et al. 2022). $N_{h}$ user embeddings are sampled from the history sequence and the $N_{t}$ element embeddings are sampled from the target sequence. This allows us to construct $N_{h}\times N_{t}$ positive sample pairs from single user interaction data to apply dense all action loss. $N_{h}$ and $N_{t}$ are set to 10 by default.

Although items have been sampled based on the importance of behavior during the construction of the industry dataset, the length of user sequences remains excessively long due to training resource constraints. To address this, we design a two-stage sampling strategy during the training phase. In the first stage, we perform random sampling from the complete user history/target interactions to serve as input for the User Tower, ensuring that the data used for modeling user interests is unbiased. Visualization of user interaction data revealed that items – users recently interacted with – better reflect user current interests and are more relevant to the target items of user preferences. Thus, in the second stage, when constructing positive and negative sample pairs, we implement a sample weighting strategy that prioritizes recent items. The weighting $\tilde{w}_{i}$ of the $i$-th item in the history/target sequence is computed as:

$\displaystyle\tilde{w}_{i}=\frac{w_{i}}{\max(w)},\quad\text{where}\ w_{i}=\log(\alpha+i\cdot\frac{\beta-\alpha}{N-1}).$

(1)

The hyperparameters $\alpha$ and $\beta$ are set to 10 and 10000. $N$ is the length of user history/target interactions sampled from the first stage.

For industry application, we build a large-scale practical recommendation dataset from the e-commerce platform of a short video application. The industry dataset compose 12 million users interacted with 31 million items over a 10-month period from June 2022 to April 2023. The interactions from the first nine months are used as historical data, while the interactions from the last month are designated as target data. Six types of information (title, category, brand, price, keywords, attributes) are collected to form the item description. To make a fair comparison, we adopt the widely used Amazon Review dataset (Ni, Li, and McAuley 2019) and follow the same settings as RecFormer (Li et al. 2023). Seven categories are selected as pretraining data and the other six categories are selected as finetuning data to evaluate our method. Three types of information (title, category, brand) are used to form the item description. The statistics of public and industry datasets are shown in Tab. 1 and Tab. 2.

Baichuan2-7B (Baichuan 2023) is adopted as the LLM to extract content embedding based on item text description due to its robust capabilities in understanding Chinese and English text. The LLM parameters are frozen during the training stage. All experiments take the AdamW optimizer and the cosine scheduler as default settings. For industry datasets, the training batch size is set to 240 and the length of user history and target interaction are set to 80 and 40 due to memory limitation. Training epochs are set to 10. The hit rate (H@50, H@100) and recall (R@50, R@100) at Top50 and Top100 is used for performance evaluation. For the Amazon Review datasets, the batch size is set to 1024 during pretraining and 16 during fine-tuning. The learning rate is 5e-5 and 2e-5, respectively, for the two stages. Training epochs are set to 20 for pretraining and 200 for fine-tuning. We follow the evaluation settings proposed by RecFormer, applying the leave-one-out strategy (Kang and McAuley 2018) for the evaluation. Three metrics — NDCG@10 (N@10), Recall@10 (R@10), and MRR — are used to ensure a fair comparison. Given the limited interaction sequence length in the Amazon Review dataset, we opt not to apply any sampling strategy in the training stage of LEARN.

To verify the effectiveness of our method, performance on Amazon Review is reported in Tab. 3. We compare LEARN with three categories of methods: ID-Only methods (GRU4Rec (Hidasi et al. 2015), SASRec (Kang and McAuley 2018), BERT4Rec (Sun et al. 2019), RecGURU (Li et al. 2022)) , ID-Text methods (FDSA (Zhang et al. 2019), S³-Rec (Zhou et al. 2020)), and Text-only methods (ZESRec (Ding et al. 2021), UniSRec (Hou et al. 2022), RecFormer (Li et al. 2023)). Our method achieves significant improvements compared to the ID-only, ID-Text, and Text-only methods. Specifically, compared to the SOTA method RecFormer, our method LEARN brings improvements of 10.08%, 17.87%, 5.39%, 10.41%, 29.45%, and 10.50% in Recall@10 on the Scientific, Instruments, Arts, Office, Games, and Pet datasets, respectively. Instead of using masked language modeling (MLM) loss and a two-stage fine-tuning process like RecFormer, the proposed LEARN model is supervised solely by user-item contrastive loss in a single fine-tuning stage. Our method achieves significant performance improvements despite fewer loss constraints and a simpler training process, further demonstrating the effectiveness of our framework. We also conduct experiments in zero-shot settings (only with the pre-training stage) following the RecFormer setting. Results in Fig. 4 demonstrate that our LEARN framework can be taken as a pretrained recommendation model and performs well in downstream subscenarios. The performance of other methods in Fig. 4 is referenced in RecFormer.

The insight behind our performance improvement is that: a significant gap between the collaborative domain of the recommenders and the open-world domain of LLMs. Given the informative content embedding as inputs, one efficient way to bridge this gap is taking user-item interactions as alignment target to transform content embeddings into user/item embeddings. We demonstrate our insight in Tab. 4. First, we calculate the user embedding by averaging the content embeddings of all items the user has interacted with and use content embeddings of items as item embeddings directly. Since there is no alignment between the LLM and recommendation domains, we term this method “w/o Align”. We find that the performance of the “w/o Align” is very poor, which confirms our hypothesis that there is a significant gap between the open-world knowledge of the LLM domain and the collaborative knowledge of the recommendation domain. Therefore, content embeddings generated by LLMs are not suitable for direct use in recommendation tasks. Second, we use the content embedding generated by the LLM as the alignment target, with ItemTower(c) shown in Fig. 2. ItemTower(c) variant achieves domain alignment by transforming the recommendation domain into the LLM domain, following the same “Rec-to-LLM” adaptation of previous work. Experiments reveal that ItemTower(c) variant is inferior to LEARN. This is because the distinct characteristics of recommendation knowledge (target domain) are not well represented in the LLM’s open-world knowledge (source domain). In contrast, by projecting the source domain into the target domain space, the LEARN model is better attuned to the complex and intricacies of the recommendation (target) distribution, leading to improved performance.

It is worth noting that for the Amazon Review dataset, ItemTower(a) is equivalent to ItemTower(b), as the length of the user target sequence is 1 due to the leave-one-out setting (Kang and McAuley 2018; Li et al. 2023).

To further validate the rationality of our model design, we conduct ablation studies on the large-scale dataset collected from the real industry scenario. Dataset details are given in Tab. 2. As shown in Tab 5, “w/o Align” achieves the worst performance due to the significant gap between LLM and recommendation domains. Among the alignment strategies, LEARN with ItemTower(a) achieves the best performance, followed by ItemTower(b), with ItemTower(c) performing the worst. We believe that LEARN with ItemTower(a), which uses sequence-to-sequence alignment, allows the model to better capture long-term user interests compared to the sequence-to-item alignment used in ItemTower(b). LEARN with ItemTower(c), which uses “Rec-to-LLM” adaptation to align with the content embedding target, performs worse than ItemTower(a). This result is consistent with the findings on the Amazon Review datasets. Due to the lengthy user interactions spanning more than ten months, we apply the sample weighting strategy proposed in Eq.1 and compare it with a random sampling strategy. As shown in Tab.5, LEARN with sample weighting achieves 7.13% and 7.18% improvement in H@100 and R@100.

Given the limitations of ID embeddings in semantic representation and generalization, we explor the feasibility of alternatives to ID embeddings in large-scale real-world industrial scenarios. We used three approaches for item representation: learnable ID embeddings, frozen content embeddings extracted from the pretrained BERT (Yang et al. 2022) and pretrained LLM. The ID-emb dimension is set to 64 to align with the online system. As shown in Tab. 6, LLM-based content embeddings deliver significant performance improvement over ID embeddings, with H@100 increasing from 0.0504 to 0.0751, representing 49.01% enhancement. Compared to BERT-based content embeddings, LEARN with LLM embeddings achieves 30.38% improvement. This can be attributed to the richer information contained in the LLM embeddings, which are trained on extensive text corpora. Our experimental results give a promising direction to replace ID embedding with semantic content embedding.

Given the superior capabilities of LLMs in text comprehension and common sense reasoning, we replace the transformer layers trained from scratch with the pretrained LLM (Baichuan2-7B) in the PAL module of Fig. 2. To retain open-world knowledge, we fine-tune the LLM using LoRA (Hu et al. 2021) and adjust different settings to vary the number of trainable parameters.

As shown in Tab. 7, as the number of trainable parameters increased from 134M to 572M, the performance of variant “w/ LLM” improves from 0.0376 to 0.0513. However, this performance still falls short compared to LEARN, which uses 12 transformer layers trained from scratch as the backbone. Considering the working mechanism of LoRA (Hu et al. 2021), the output features of the “w/ LLM” variant are a blend of the original feature trained in the open-world domain and the LoRA feature trained in the recommendation domain. Due to the significantly larger number of frozen parameters in the LLM compared to the trainable LoRA parameters, the original features tend to dominate. These original features are learned from the open world domain and are supervised by next-token prediction loss. In contrast, LoRA features are trained in the recommendation domain and are supervised by contrastive loss. This disparity between the two types of features prevents the mixed features from achieving optimal performance.

To better align the user and item embeddings generated by LEARN with the online ranking model, we introduced the LEARN-Aaptor based on the baseline model. As illustrated in Fig.5, the baseline model consists of the original ranking model, which takes the existing online features as inputs (denoted as “Other” in Fig.5). The LEARN-Adaptor module includes a fusion module (two linear layers) and an MLP, which aggregate user and item embeddings into the fusion embedding through ConVersion Rate (CVR) loss. The fusion embedding, along with the user and item embedding of LEARN, and existing online features, are concatenated and fed into the ranking model.

AUC evaluation is conducted on a billion-scale dataset from the e-commerce platform of the short video application. Following common industry practices, we adopt UAUC and WUAUC metrics, as they more accurately assess the ranking performance for each user and better reflect user experience. Specifically, UAUC provides insight into the performance of long-tail users by applying uniform weights. As depicted in Tab. 8, our method outperforms the baseline model, achieving improvements of 0.84 percentage points (pp) in UAUC and 0.76 pp in WUAUC. These gains can be attributed to the superior generalization capabilities of the LEARN framework, which effectively captures the interests of long-tail users. To further validate this hypothesis, we conducted a more comprehensive analysis. We categorize users and items into three types based on historical interaction frequency. As shown in Tab. 9, LEARN delivers significant performance enhancements, particularly for cold-start and long-tail users and items, further confirming the generalization of our approach for users and items with sparse purchase histories.

We allocate 20% of the platform’s traffic to our proposed LEARN and baseline models. The experiments are conducted in a real-time system over 9 days. The results are shown in Fig. 6. The LEARN framework demonstrates a steady and significant increase in both revenue and CVR. Considering that the revenue for online recommendation models is measured in tens of millions, even a 2% improvement is highly significant.