Disentangled Self-Attentive Neural Networks for Click-Through Rate Prediction

Suppose the training dataset $\mathbb{D}=\left\{\bm{x}_{i},y_{i}\right\}_{i=1}^{N}$ contains $N$ samples, where each sample $\bm{x}_{i}$ consists of $M$ fields of users’ and items’ features and its associated label $y_{i}\in\{0,1\}$ represents that user’s behavior (e.g., whether to click an item). The problem of click-through rate prediction is to predict $\hat{y}_{i}$, given a feature vector $\bm{x}_{i}$, for accurately estimating whether a user will interact with an item.

The proposed DESTINE consists of three key components: (a) the embedding layer, (b) the interaction layer, and (c) the output layer. At first, the input features are fed into the embedding layer, which transforms input features into dense, low-dimensional embedding vectors. Then, these feature embeddings are fed into several stacked interaction layers, which model high-order interaction. After that, we feed the embeddings from the last interaction layer into the output layer to estimate the click behavior.

For each sparse input feature $\bm{x}_{i}$, we transform it into dense embeddings $\bm{e}_{i}\in\mathbb{R}^{d}$ via embedding lookup. Once we obtained a compact representation for each feature, we use the scaled dot-product attention scheme to model high-order feature interaction among feature fields. Specifically, we formulate each feature interaction as a (key, value) pair and learn the importance of each feature interaction by multiplying each feature embedding, such that important key–value pairs get higher attention scores. Formally, we first transform each feature embedding into a new embedding space $\mathbb{R}^{d^{\prime}}$ as follows

where the query and key transformation are parameterized by two linear transformation matrices $\bm{W}_{\text{q}},\bm{W}_{\text{k}}\in\mathbb{R}^{d^{\prime}\times d}$, respectively.

Then, we compute the correlation $\alpha(\bm{e}_{m},\bm{e}_{n})$ between feature $m$ and feature $n$. Previous work (Yin et al., 2020) in visual representation learning demonstrates that the importance score of feature $m$ over feature $n$ could be decomposed into two terms: a pairwise term to model pure specific interaction and a unary term to model general impact over all feature fields. We take summation of these two terms:

For the pairwise term, we perform whitening (Friedman, 1987) on the key and the query vector to model pure interaction among features, which makes the two interacting features less correlated with each other:

where $\sigma(\cdot)$ is the softmax function; $\bm{\mu}_{\text{q}}=\frac{1}{M}\sum_{i=1}^{M}\bm{W}_{\text{q}}\bm{e}_{i}$ and $\bm{\mu}_{\text{k}}=\frac{1}{M}\sum_{j=1}^{M}\bm{W}_{\text{k}}\bm{e}_{j}$ takes average of the key and the query vectors, respectively.

Regarding the unary term, we introduce another query transformation matrix $\bm{W}_{\text{q}}^{\prime}\in\mathbb{R}^{d^{\prime}\times d}$ for modeling significant features:

where $\bm{\mu}_{\text{q}}^{\prime}$ is the mean vector from another transformation to model the general impact on key vectors, i.e. $\bm{\mu}_{\text{q}}^{\prime}=\frac{1}{M}\sum_{i=1}^{M}\bm{W}_{\text{q}}^{\prime}\bm{e}_{i}.$

After computing the attention score for each feature interaction $(m,n)$, we transform each candidate feature to a new embedding space by a value transformation, parameterized by a linear transformation matrix $\bm{W}_{\text{v}}\in\mathbb{R}^{d^{\prime}\times d}$, as given in the sequel,

At last, we update the final representation for feature field $m$ by linearly combining all features.

For the proposed model, we only introduce a new learnable parameter $\bm{W}_{\text{q}}^{\prime}\in\mathbb{R}^{d\times d^{\prime}}$ for each attention head, leading to additional space complexity of $O(dd^{\prime}H)$ for each interaction layer. In addition, the time complexity of DESTINE is $O(H(d^{\prime}dM+d^{\prime}M+d^{\prime}M^{2}+d^{\prime}M^{2}))=O(MHd^{\prime}(2M+d+1))$, compared to that of the original self-attention module of $O(MHd^{\prime}(M+d))$. Note that $d,d^{\prime}$, and $H$ are usually small, demonstrating that our model is memory-efficient and computation-friendly.

We use two large-scale datasets Avazu and Criteo for evaluation. Avazu contains click logs of mobile ads in 10 days, which is composed of 23 categorical features including domains, categories, connection types, etc. Criteo comprises traffic logs of display ads over 7 days. Each sample contains 13 numerical and 26 categorical feature fields. The detailed statistics is summarized in Table 1.

For fair comparison, following most existing studies (Song et al., 2019; Li et al., 2019; Zhu et al., 2020), we randomly sample 80% data as the training set, 10% as the test set, and the remaining 10% as the validation set. We report the performance in terms of two widely-used metrics AUC and logloss. Please kindly note that considering a large scale of user base, performance improvements of AUC at 1‰-level are considered as practically significant for industrial deployment (Cheng et al., 2016; Guo et al., 2017; Wang et al., 2017; Song et al., 2019; Zhu et al., 2020; Wu et al., 2020).

For diagnosing the proposed scheme, we further investigate four variants of the proposed disentangled self-attention module:

• •

  Pairwise (DESTINE-P). We only use the pairwise term in Eq. (4).

• •

  Unary (DESTINE-U). We only use the unary term in Eq. (5).

• •

  Multiplication (DESTINE-M). We use multiplication rather than addition for combining the pairwise and the unary term, as described below,

  (12)

  $$\alpha_{\text{d}}(\bm{e}_{m},\bm{e}_{n})=\sigma\left(\left(\bm{q}_{m}-\bm{\mu}_{\text{q}}\right)^{\top}\left(\bm{k}_{n}-\bm{\mu}_{\text{k}}\right)\right)\cdot\sigma\left((\bm{\mu}_{\text{q}}^{\prime})^{\top}\bm{k}_{n}\right).$$

• •

  Shared transformation (DESTINE-S). We consider preserving the shared key transformation $\bm{W}_{\text{k}}$ in the unary term, as formulated below,

  (13)

  $$\alpha_{\text{s}}(\bm{e}_{m},\bm{e}_{n})=\sigma\left(\left(\bm{q}_{m}-\bm{\mu}_{\text{q}}\right)^{\top}\left(\bm{k}_{n}-\bm{\mu}_{\text{k}}\right)\right)+\sigma\left(\bm{\mu}_{\text{q}}^{\top}\bm{k}_{n}\right).$$

The results are summarized in Figure 2. We can see that both the pairwise and the unary term benefits model performance. However, the way to combine the two terms plays an important role. DESTINE-M performs poorly despite its utilization of both terms and is even outperformed by DESTINE-U and DESTINE-P, where only one of the two terms is used. This verifies our justification that coupled gradient in DESTINE-M will deteriorate the performance. The performance DESTINE-S is also inferior to DESTINE, since the shared $\bm{W}_{\text{q}}$ transformation still leads to the coupling of gradients during learning. The outstanding performance of DESTINE compared to all other variants justifies the design of our model.