AutoAttention: Automatic Field Pair Selection for Attention in User Behavior Modeling

Before introducing AutoAttention, we first present several straightforward approaches to interact user behaviors with all fields within attention: MLP with All fields (MAF in short) and DotProduct.

In above base models, all available fields are considered as the query in the attention function. However, there are still several concerns: 1) simply involving all fields as the query ignores the fact that some fields are more important than others when being interacted with each behavior field; 2) in real-world industry systems, the number of fields is large, including all of them as the query may increase the computation cost of the model; 3) some field pairs are irrelevant or noisy for capturing user interests, leading to performance deterioration when included in the attention.

To tackle the above mentioned challenges, we assign a weight $R_{B_{p},F_{j}}$ to model the interaction strength for each field pair between a behavior field $B_{p}$ and a query field $F_{j}$. These field pair wise weights are learnable and trained together with all the other parameters. We name our approach as AutoAttention. Mathematically,

Directly including all query fields in the attention function and interacting all of them with the behavior fields may introduce irrelevant or noisy field pairs. In order to identify and remove them, we conduct automatic field pair selection by pruning on the field pair wise weights $R$. There are lots of empirical studies in weight pruning area [13, 14, 15]. For simplicity, we adopt a standard iterative pruning algorithm used in [15]. An illustration of AutoAttention is presented in Fig. 2.

The algorithm of pruning is depicted in Alg. 1. We first train the model a few epochs to initialize the weights $R$ and then conduct pruning on $R$ to remove those with the bottom-$S\%$ lowest magnitude values. We gradually increase the sparsity rate $S\%$ such that it increases faster in the early phase when the network is stable and slower in the late phase when it becomes sensitive. Other approaches such as regularization [16] can also be used here.

Existing methods heavily rely on expert knowledge on selecting relevant fields to involve them in the attention. For example, in DIN [4], the authors manually select three fields from the target item side: item_id, shop_id and category_id. Such expert knowledge is not always feasible and accurate. AutoAttention avoids such reliance on expert knowledge by automatic field pair selection.

Follow [4], after we extract a user’s interest $\bm{v}_{u}$, we concatenate it with all the other features and feed them into an MLP:

We then minimize the following cross-entropy loss during model training:

where $N$ denotes the number of training samples, $\Theta$ denotes all trainable parameters, $y_{i}\in\{0,1\}$ denotes the label, and $\lambda\left\|\Theta\right\|_{2}$ denotes the $L_{2}$ regularization term.

We use the following two datasets for performance comparison. Their statistics are presented in Tab. II.

• •

  Alibaba Dataset²²2https://tianchi.aliyun.com/dataset/dataDetail?dataId=56 [6] is a public advertising dataset released by Alibaba. It randomly samples 1,140,000 users from the website of Taobao from 8 days of click logs (26 million records) to generate the original dataset. Following [6], we use the first 7 days’ samples as the training set (2017-05-06 to 2017-05-12), and the next day’s samples as the testing set (2017-05-13). We keep users’ most recent 50 behaviors. Please note that we only extract the user click behaviors whose click time is before the target item to prevent information leakage.

• •

  Tencent CTR Dataset is collected by sampling user click logs for one week from Tencent’s advertising CTR log. We use samples from 2021-09-05 to 2021-09-10 as the training set, and samples on 2021-09-11 as the testing set. The data preprocessing strategy is the same as that of the Alibaba dataset.

We compare AutoAttention with the following baseline approaches:

• •

  Sum Pooling conducts a sum pooling without weights on the user’s behavior embeddings to generate a fixed-length user interest representation.

• •

  DIN [4] conducts a weighted sum pooling over user behaviors. The attention weight is calculated over the user behavior and several manually selected fields from the item side. The attention is implemented as an out product between the user behavior embedding and those selected item side embedding, followed by an MLP.

• •

  DIEN [5] uses a GRU encoder to capture the behavior dependencies, followed by another GRU with an attentional update gate to depict interest evolution.

• •

  DSIN [6] captures users’ homogeneous interests in each session and heterogeneous interests in different sessions.

• •

  GRU4Rec [19] uses a GRU with ranking based loss to model user sequences for session based recommendation.

• •

  SASRec [20] uses a left-to-right Transformer to capture users’ behaviors for sequential recommendation.

• •

  BERT4Rec [21] uses a bi-directional self-attention to model user behaviors.

• •

  BST [22] uses self-attention and target-attention together to model user behaviors.

• •

  CFI [8] considers the corresponding field interactions for sequential recommendation.

All methods are implemented in Tensorflow 1.4 with Python 3.5, which are trained from scratch on a NVIDIA TESLA M40 GPU with 24G memory. For baseline methods, we follow the hyper-parameter settings in their original papers but also finetune them on our datasets. For both datasets, We set the maximum user behavior length $H$ to 50. The embedding dimension $K$ is 64 for all features. The dimension of each hidden layer of the three-layer MLP is 200, 80, and 1, with activation functions PReLU, PReLU, and Softmax, respectively. We use Adagrad [23] as the optimizer with a learning rate of 0.01. The batch size is 4,096 and 16,384 for the training and testing set, respectively. For DIN, DIEN, and DSIN, the dimension of the two-layer MLP in the local activation unit is 200 and 1, with the dice activation function [4]. For DSIN, we divide user behavior sequences into 5 sessions. The maximum user behavior length of each session is 10. For AutoAttention, the target sparsity rate $S$ is 0.6 and 0.8 for Alibaba and Tencent datasets respectively. The damping ratios $D$ and $U$ is set to 0.8 and 100, respectively.

We use user-weighted AUC as the evaluation metric [4], which measures the goodness of samples ranking for each user. A vanilla AUC is first calculated for all samples of each user, then we conduct a weighted average over these AUCs, using the number of samples of each user as the weights. We still refer it as AUC in this paper for simplicity.

where $n$ denotes the number of users, $\#impression_{i}$ and $\mbox{AUC}_{i}$ denote the number of impressions and AUC of the $i$-th user, respectively.

The experiment results of comparison between existing methods and our proposed AutoAttention on both datasets are shown in Tab. III. All experiments are repeated 5 times and the averaged results are reported.

The sum pooling method is treated as a baseline. DIN gets 0.52% and 0.67% relative AUC lift on two datasets compared with sum pooling, since it considers the different importance of each behavior. GRU4Rec, SAS4Rec, BERT4Rec, and BST achieve similar performance, which considers sequence dependencies. DIEN and DSIN take both into account and further improve AUC. CFI gets 1.51% and 3.14% AUC lift respectively, which shows the advantage of corresponding field interaction.

AutoAttention significantly lifts the AUC on two datasets by 2.19% and 3.58%, respectively. Please note that even a 0.1% AUC lift is huge and usually leads to a decent Gross Merchandise Volume (GMV) lift in online advertising systems [24].

So far, all baselines only consider several item side fields as specified in their papers, while the proposed AutoAttention uses all fields as the query and then conducts field pair selection. One may wonder whether the performance lift is mainly due to including more fields, or due to the selection.

AutoAttention can be treated as a general purpose attention unit. In addition to using it standalone to model a user’s interest, we can also replace the original attention unit within DIEN and DSIN by the proposed DotProduct and AutoAttention. Note that we can not obtain the original field embeddings of user behaviors since DIEN and DSIN use hidden states to represent behavior within attention. Therefore, we only assign field-wise weights for fields from the target item rather than the field pair wise weights. In DIEN, we replace its attention function by AutoAttention:

where $M^{\prime}$ denotes the number of fields from the target item side, $\bm{h}_{t}$ is the hidden state of user behavior. While in DSIN, we replace the attention function in the session interest activating layer. The formulation is similar to Eqn. (10), except that we use an embedding vector of session interest instead of a hidden state of user behavior.

As shown in Tab V, the performance of DIEN and DSIN can be further boosted with the two proposed attention units. Note that replacing the original attention unit by AutoAttention introduces marginal additional parameters and computational cost, which can be ignored compared with the cost of the two methods themselves.

We would verify whether the learnt field pairs weights really reflect the importance of each field interaction, and whether we recognize some other important field pairs which are neglected by the expert knowledge. We visualize the learnt field pair strength weights $R$ by heat maps on two datasets, shown in Fig. 5(a) and Fig. 5(b), where the x-axis and y-axis denote the fields from current sample (including item/user/context fields) and user behaviors, respectively.

We observe that the field pairs within item side fields indeed play an important role within the attention, such as (cate_id, brand), (brand, brand), (creative_id, ad_id), and (ams_second_industry_id, creative_id). In addition, we also observe that some field pairs from other sides are also important, such as (scenario_id, cate_id) and (user_grade, ad_id). These field pairs are usually neglected when manually selecting fields or field pairs, which can be identified in AutoAttention.

CTR Prediction is one of the most fundamental tasks in online advertising and recommendation systems, which aims at predicting the probability that a user clicks an item or ad. Pioneer works of CTR prediction are proposed mainly based on Logistic Regression (LR) [3, 1, 2], polynomial [25], collaborative filtering [26], tree models [27], Bayesian models [28], etc. In order to explicitly model the feature interactions, many factorization machine based methods are proposed for high-dimensional data, such as Factorization Machine (FM) [29], Field-aware Factorization Machine (FFM) [30], Field-weighted Factorization Machine (FwFM) [31, 32], and Field-matrixed Factorization Machine (FmFM) [33]. Besides, there are some works that aim at learning weight for different feature interactions, including Attentional Factorization Machines (AFM) [34], Dual-attentional Factorization Machines (DFM) [35], Dual Inputaware Factorization Machines (DIFM) [36].

Since the number of samples and the dimension of features have become larger and larger, many deep learning based models have been proposed, such as Wide&Deep [24], Deep Crossing [37], YouTube Recommendation [7], PNN [38], Deep&Cross [39]. There are some studies that combine FM with DNN, such as DeepFM [40], NFM [41], xDeepFM [42], InterAtt [43], DeepLight [15] and DCN V2 [44].

Traditional methods take a straightforward way to represent each behavior with an embedding vector, and then do a sum or mean pooling over all these embedding vectors to generate one embedding [7]. Then many works propose to assign a dynamic weight for each behavior and then conduct weighted sum pooling, such as Deep Interest Network (DIN) [4], Deep Interest Evolution Network (DIEN) [5], and Deep Session Interest Network (DSIN) [6]. There are also many works to use RNN or Transformer for behavior sequence modeling, including GRU4Rec [19], SAS4Rec [20], BERT4Rec [21], and Behavior Sequence Transformer (BST) [22].

Recently, there are also some works further consider long-term historical behavior sequences, such as Multi-channel user Interest Memory Network (MIMN) [45], Hierarchical Periodic Memory Network (HPMN) [46], Search-based Interest Model (SIM) [47], UBR4CTR [48], ETA [49] and LimaRec [50].