Enhancing CTR Prediction through Sequential Recommendation Pre-training: Introducing the SRP4CTR Framework

For downstream CTR tasks, we introduce a uni cross-attention that enables the predicted item to transfer corresponding information from the pre-trained model. As illustrated in Figure  2, in uni cross-attention, the queries are composed of predicted items, while Key and Value are derived from the user behavior sequence representation at the same layer. Although the predicted item shares the same representational information as the input of the pre-trained model, the attention mechanism learned under self-supervised differs significantly from the attention mechanism required for CTR tasks. Therefore, we untie the parameter sharing between the uni cross-attention block and the pre-trained model, only sharing the input’s embedding parameters and the projection parameters for Key and Value at each layer. After multi-head self-attention, the encoded feature of the predicted item is passed through the feed-forward network and residual networks, as commonly processed in the transformer-based layer. Besides, a learnable variable is set as position embedding about the predicted item to circumvent issues with changes in positional semantics.

During the deployment phase, the uni cross-attention block only adds a minimal amount of extra computation. The primary computational load, carried by the pre-trained model, can still be accelerated using folded inference.

Though encoding from the pre-trained model, the features generated may not align perfectly with the information required for CTR tasks, because the goal of pre-training mismatches with CTR training objective. For better alignment, as shown in Figure  2, we introduce a novel Querying Transformer that employs $K$ learnable queries as input (where $K<L$) and encoded by cross-attention with the output of the pre-trained model as Key and Value. This module is expected to achieve discrimination of user interests. Additionally, in practical scenarios, we also use user or context features, such as gender, age, or hour, as the initialization for queries, which are then mapped into multiple different queries to accommodate the diverse interest distributions of different users. Finally, the Querying Transformer only processes user information, which enables it to be folded inference.

We conducted experiments using two offline public datasets: MovieLens-20M(Harper and Konstan, 2015) and Taobao(Zhou et al., 2018). In the case of MovieLens, similarly to (Zhou et al., 2018), the task is to predict whether a user is likely to assign a rating exceeding 3(denoted as a positive label). We leverage features derived from users’ historical interactions to evaluate the most recent 10 interactions. We employ movie_id and movie_cate_id as item-related features and the rating score from the log as the behavior-related feature. For the Taobao dataset, we use the item cate_id and brand_id as item-related features, and the behavioral tag serves as the behavior-related feature. Additionally, we have executed a data sampling in the Taobao dataset, selecting the top 50% of users with the most behaviors.

In this work, we utilize the Adam optimizer and apply polynomial decay to the learning rate. During the pre-training phase, we use the same architecture as in (Sun et al., 2019) consisting of a two-layer transformer with the maximum sequence length $L=200$. For the MovieLens dataset, we train with a batch size of 512 for 100,000 steps. For the Taobao dataset, due to GPU memory constraints, we train with a batch size of 256 for 200,000 steps. During the fine-tuning phase, with Taobao’s token count reaching 500,000 compared to MovieLens’s token count of only 20,000, there is a higher risk of overfitting for MovieLens. Consequently, all parameters in the pre-trained model are learnable in the Taobao dataset but frozen in the MovieLens. We train for 10 epochs and select the most effective checkpoint as the benchmark for performance evaluation. Additionally, we use Area Under the Curve(AUC) as a metric to evaluate the performance of the CTR task.

The overall performance of SRP4CTR is presented in Table  1. We primarily compared five traditional CTR-based methods: PNN(Qu et al., 2016), BST(Chen et al., 2019),DIN(Zhou et al., 2018), DIEN(Zhou et al., 2019), and CAN(Bian et al., 2022). Our approach demonstrated notable improvements over conventional end-to-end modeling methods among various datasets.

Furthermore, we explored the impact of different pre-training methods on SRP4CTR, including the original BERT(Sun et al., 2019), ELECTRA(Clark et al., 2020), S3(Zhou et al., 2020), and our proposed FG-BERT. Each experimental group included two models. We compared the performance of MP(Yuan et al., 2020) and SRP4CTR on different pre-trained models. MP is a simple method of transferring a pre-trained model to downstream tasks. In the fine-tuning phase, this method inserts a learnable model patch into the pre-trained model while the rest of the parameters are frozen. Subsequently, the representation produced by the pre-trained model is concatenated with the embedding of the estimated item for prediction. MP enables a more direct comparison of the effectiveness of different pre-training methods. From Table 1, our proposed FG-BERT outperforms other pre-training approaches. Furthermore, on the MovieLens dataset, the pre-trained model can surpass CTR-based methods solely through MP. However, for the sparser Taobao dataset, MP underperforms compared to CTR-based models.

In Table  2, we further explored the effectiveness of SRP4CTR for recommending long-tail items. We defined the bottom 20% of items in terms of their occurrence in the dataset as long-tail items and compared the performance of our method and traditional CTR prediction methods on this sub-dataset. Traditional CTR prediction methods like DIN and CAN tend to learn more effectively from hot items, resulting in a significant discrepancy between the AUC for long-tail items and the average AUC across all items. However, through pre-training, both the MP and SRP4CTR can greatly narrow the learning gap between long-tail items and other items.

We conducted detailed ablation studies on SRP4CTR, with the results presented in Table  3. The comparisons include our proposed querying transformer(qFormer) and uni cross-attention module, as well as the impact of other training paradigms like scratch. Scratch denotes training SRP4CTR directly in an end-to-end manner. Moreover, for the uni cross-attention, we also examined the impact of parameter tying. The results validate the effectiveness of our proposed methods.

In Table  4, we have conducted a comparison of the inference costs between SRP4CTR and other methodologies. We propose a new metric efficiency-FLOPs, which denotes the FLOPs when the inference batch size is set to one. This metric is indicative of the computational complexity that truly impacts the precision of the model within a folded inference framework. Furthermore, inference-FLOPs represents the computational complexity for a single batch when inference is carried out using folded inference with a batch size of 100 ¹¹1Our online deployment performs inference with a batch size of 100., thereby reflecting the cost of the model’s inference.

As the original implementations of DIN and CAN did not take into account inference performance, we have introduced two variants, DIN$\dagger$ and CAN$\dagger$. These variants can be accelerated by folded inference, designed to align as closely as possible with the original versions in terms of efficiency-FLOPs. From Table  4, SRP4CTR is capable of delivering an increase in efficiency-FLOPs by more than double in comparison to DIN$\dagger$ and CAN$\dagger$, with only a slight rise in inference-FLOPs.