GRN: Generative Rerank Network for Context-wise Recommendation

As motivated, in the context-wise view, the underlying reasons driving a user to interact with a item in a list may depend on following two aspects: (1) the two-way evolution of user’s intent during browsing; (2) the mutual influence between items in the item list. Thus, by taking above factors into consideration, our evaluator $E(\mathbf{x}_{v}^{t}|u,\mathcal{V};\mathbf{\Theta}^{E})$, parameterized by $\mathbf{\Theta}^{E}$, estimates the contextual interaction probability between user $u$ and the $t$-th item $\mathbf{x}_{v}^{t}$ in the final ranking list $\mathcal{V}$. Specifically, our evaluator aims to capture following two aspects of information implicated in sequences.

• •

  Intent evolution. Naturally, user intent keeps bi-directionally evolving when browsing items sequentially, and thus, it is imperative to capture such dynamics of intent for modeling user preference. Here, we adopt the commonly used Bi-LSTM  (Hochreiter and Schmidhuber, 1997) to deal with sequences for long-term and short-term preference. Mathematically, the forward output state for the $t$-th item $x_{v}^{t}$ can be calculated as follows:

  (1)		$\displaystyle\textbf{i}_{t}$	$\displaystyle=\sigma(\textbf{W}_{xi}\mathbf{x}_{v}^{t}+\textbf{W}_{hi}\textbf{h}_{t-1}+\textbf{W}_{ci}\textbf{c}_{t-1}+\textbf{b}_{i})$
  		$\displaystyle\textbf{f}_{t}$	$\displaystyle=\sigma(\textbf{W}_{xf}\mathbf{x}_{v}^{t}+\textbf{W}_{hf}\textbf{h}_{t-1}+\textbf{W}_{cf}\textbf{c}_{t-1}+\textbf{b}_{f})$
  		$\displaystyle\textbf{c}_{t}$	$\displaystyle=\textbf{f}_{t}\mathbf{x}_{v}^{t}+\textbf{i}_{t}\tanh(\textbf{W}_{xc}\mathbf{x}_{v}^{t}+\textbf{W}_{hc}\textbf{h}_{t-1}+\textbf{b}_{c})$
  		$\displaystyle\textbf{o}_{t}$	$\displaystyle=\sigma(\textbf{W}_{xo}\mathbf{x}_{v}^{t}+\textbf{W}_{ho}\textbf{h}_{t-1}+\textbf{W}_{co}\textbf{c}_{t}+\textbf{b}_{o})$
  		$\displaystyle\overrightarrow{\textbf{h}_{t}}$	$\displaystyle=\textbf{o}_{t}\tanh(\textbf{c}_{t})$

  where $\sigma(\cdot)$ is the logistic function, and i, f, o and c are the input gate, forget gate, output gate and cell vectors which have the same size as $\mathbf{x}_{v}^{t}$. Shapes of weight matrices are indicated with the subscripts. Similarly, we can get the backward output state $\overleftarrow{\textbf{h}_{t}}$. Then we concatenate the two output states $\textbf{h}_{t}=\overrightarrow{\textbf{h}_{t}}\oplus\overleftarrow{\textbf{h}_{t}}$ and denote $\textbf{h}_{t}$ as the sequential representation of $x_{v}^{t}$.

• •

  Mutual influence. That is, any two items can affect each other, although they are far away from each other in the final ranking list. To achieve this goal, we apply the recently emerging self-attention mechanism (Vaswani et al., 2017) to directly model the mutual influences for any two items regardless the distances between them. Formally, we implement it as follows:

  (2)

  $$\begin{split}\mathbf{A}=\mbox{softmax}\left(\frac{\mathbf{V}\mathbf{V}^{T}}{\sqrt{d_{k}}}\right)\mathbf{V}\end{split}$$

  where $d_{k}$ is the dimension of each item representation in $\mathcal{V}$. Obviously, we can easily extend it to multi-head attention for the stable training process. We denote $\mathbf{a}_{t}$, the $t$-th representation in $\mathbf{A}$, as the mutual representation of $x_{v}^{t}$.

Due to the powerful ability in modeling complex interaction in the CTR prediction field Zhou et al. (2018); Feng et al. (2019); Pi et al. (2019); Zhou et al. (2019), we integrate the multi-layer perceptron (MLP) into the evaluator for better feature interaction. Hence, we formalize our evaluator as follows:

where $f(\textbf{x})=ReLU(\textbf{W}\textbf{x}+\textbf{b})$, $\sigma(\cdot)$ is the logistic function. The parameter set of the evaluator is thus $\mathbf{\Theta}^{E}=\{\mathbf{W}_{*},\mathbf{b}_{*}\}$, i.e., the union of parameters for Bi-LSTM and MLP.

Clearly, our evaluator can be optimized via binary cross-entropy loss function, which is defined as follows:

where $\mathbb{D}$ is the training dataset. For convenience, we refer $\hat{y}_{uv}^{t}$ as $E(\mathbf{x}_{v}^{t}|u,\mathcal{V};\mathbf{\Theta}^{E})$, i.e., $\hat{y}_{uv}^{t}=E(\mathbf{x}_{v}^{t}|u,\mathcal{V};\mathbf{\Theta}^{E})$, and $y_{uv}^{t}\in\{0,1\}$ is the ground truth. We can optimize the parameters $\mathbf{\Theta}^{E}$ through minimizing $\mathcal{L}^{E}$.

The goal of our generator $G(u,\mathcal{C};\mathbf{\Theta}^{G})$, parameterized by $\mathbf{\Theta}^{G}$, is to learn a reranking strategy for item selection from input ranking list $\mathcal{C}$, that are of the most interest to user $u$. In the following sections, we focus on how to select the appropriate item at each step by considering the user’s dynamic state during browsing, besides how to generate the final ranking list and train the generator.

Evolving Layer. As motivated, user’s intent or interest will change over time, which derive us to learn dynamic representation for the target user to adapt for the evolution of the browsing history. For this purpose, we aim to distill the sequential information into the state representation at each step for the target user $u$, which achieved by the time-efficient GRU module. Formally, at the $t$-th step, let $\mathcal{S}=[\mathbf{x}_{s}^{0},\mathbf{x}_{s}^{1},...,\mathbf{x}_{s}^{t-1}]$ be the $t-1$ items selected from $\mathcal{C}$ ²²2Specially, for the first step, $\mathbf{x}_{l}^{0}$ is a trainable vector which has the same embedding size with $\mathbf{x}_{l}^{t}$, the output of GRU module can be calculated as follows,

where $\textbf{W}_{z},\textbf{U}_{z},\textbf{W}_{t},\textbf{U}_{t},\textbf{W}$ and U are trainable variables. We denote $\mathbf{H}=[\mathbf{h}_{1},...,\mathbf{h}_{t}]$ as the sequential encoding of the selected list $\mathcal{S}$.

Activating Layer. Typically, the selected items have different influence towards the remaining items in the input ranking list. It is a common phenomenon that a user may get tired of the clothing category after browsing abundant different clothes. Hence, in this case, recommending other categories matching his/her interest could be a better choice. Inspired by vanilla attention mechanism (Bahdanau et al., 2015) , we learn the attention weights over remaining items in the input ranking list conditioned on the sequential representation of selected items. Given the representation of $j$-th remaining item $\mathbf{x}_{c}^{j}$ and the sequential representations of the selected list $\mathbf{H}$, we adopt weighted sum to generate the new representation for the $j$-th in the input ranking list with the corresponding attention weights as follows:

Selector Layer. After obtaining the representations for items (i.e., $\mathbf{a}^{j}_{c}$) in input ranking list, we now aim to select the most suitable item, which best matches the potential interest for the target user at the $t$-th step. Here, we choose pointer network (Vinyals et al., 2015) to achieve this goal. Formally, for the $j$-th item $\mathbf{x}_{c}^{j}$ in the input ranking list $\mathbf{C}$, we first apply MLP for better feature interaction at the $t$-th step:

Afterwards, we apply the softmax function to normalize the vector $\widetilde{s}_{c}^{j}$ as follows:

where $m$ is the number of items in $\mathbf{C}$. Then, at $t$-th step, our generator will select the item with the highest selection probability excluding the selected items, which can be formalized as follows:

The parameter set of the generator is $\mathbf{\Theta}^{G}=\{\mathbf{U}_{*},\mathbf{W}_{*},\mathbf{b}_{*}\}$, i.e., the union of parameters for GRU module and point network.

List Generation. Our generator is recurrently repeated until the generated final ranking list reaches the pre-defined length (i.e., $n$). After repeating calculating $G^{t}(u,\mathcal{C};\mathbf{\Theta}^{G})$ for $n$ times, we get the reranking strategy $\pi=[G^{1}(u,\mathcal{C};\mathbf{\Theta}^{G}),...,G^{n}(u,\mathcal{C};\mathbf{\Theta}^{G})]$ and the generated final ranking list $\mathcal{O}=[x_{o}^{1},...,x_{o}^{n}]$, correspondingly.

Advantage Reward. As illustrated earlier, we utilize the evaluator to guide the optimization of the generator for the context-wise reranking strategy through policy gradient. As mentioned above, at each step, our generator will select the most suitable item from $\mathcal{C}$ in the light of the probability distribution in Eq. 8. Besides, we propose the advantage reward to estimate the actual reward of each item in the final ranking list more comprehensively. Specifically, the advantage reward consists of two parts:

• •

  Self reward: As motivated, the interaction probability of itself in the final ranking list heavily affected by its context. Here we use the predicted contextual score by the evaluator to denote the its own reward in the list, which is calculated as follows:

  (10)

  $$\begin{split}r^{self}(x_{o}^{t}|u,\mathcal{O})&=\mbox{E}(x_{o}^{t}|u,\mathcal{O};\mathbf{\Theta}^{E})\\ \end{split}$$

• •

  Differential reward: Comprehensively, besides the self reward, each item brings differences in the interaction probability of other items in the list. For example, though selecting an unattractive item is generally not a good idea, it is appreciated if it can increase the interaction probability of other items. Hence, we propose the differential reward, with the assistance of the generator and calculated as follows:

  (11)

  $$\begin{split}r^{diff}(x_{o}^{t}|u,\mathcal{O})=&\sum_{x_{o}^{i}\in\mathcal{O}^{-}}\mbox{E}(x_{o}^{i}|u,\mathcal{O};\mathbf{\Theta}^{E})-\\ &\sum_{x_{o}^{i}\in\mathcal{O}^{-}}\mbox{E}(x_{o}^{i}|u,\mathcal{O}^{-};\mathbf{\Theta}^{E})\\ \end{split}$$

  where $\mathcal{O}^{-}$ is the item list which removes $x_{o}^{t}$ from $\mathcal{O}$.

By combining both above rewards, we define the advantage reward of $t$-th item $x_{o}^{t}$ in $\mathcal{O}$ as follows:

Finally, the loss function of the generator is defined as follows:

Here $s_{c}^{j}$ represents the sampling probability of $x_{o}^{t}$ from $\mathcal{C}$ in Eq. 8. We can optimize the parameters $\mathbf{\Theta}^{G}$ through minimizing $\mathcal{L}^{G}$.

we adopt the two-stage optimization strategy to train GRN, which is illustrated in Fig. 4. We first utilize the list interaction records to optimize $\mathbf{\Theta}^{E}$ and thus improve the evaluator until converge ((1-1) in Fig. 4). Next, we fix $\mathbf{\Theta}^{E}$, and optimize $\mathbf{\Theta}^{G}$ in order to produce the final ranking list and better meet user’s demands. Specifically, the generator will produce the final ranking list for the evaluator((2-1) in Fig. 4), and then update the paratemer $\mathbf{\Theta}^{G}$ via policy gradient with advantage reward based on the evaluator((2-2) in Fig. 4). The above process is repeated for more iterations until our generator converges. Overall, the training procedure for GRN is outlined in Algorithm 1.

We report the comparison results of the generator and evaluator module in GRN (i.e.,GRN_G and GRN_E) and other baselines on two datasets in Table 3 and Table 4. The major findings from the experimental results are summarized as follows:

• •

  On both datasets, point-wise methods (i.e., DNN and DeepFM) achieve relatively pool performance for the task of interaction probability prediction and reranking. It indicates that feature engineering of user and item profile is incapable of covering the list-wise information contained by the input ranking list. Generally, list-wise methods achieves remarkable improvements in most cases. By considering the mutual influence among the input ranking list, these methods learn a refined scoring function aware of the feature distributions of the input ranking list. Among these methods, DLCM and PRM consistently outperform MIDNN in all cases, proving that deep structures (i.e., RNN and self-attention) has superior abilities in extracting the feature distribution compared with handmade feature engineering. Moreover, compared with DLCM, adequate improvements are observed by PRM, which demonstrates the effectiveness of self-attention and personalization in the reranking task.

• •

  GRN_E consistently yields the best performance on the Loss, AUC and GAUC metrics of both datasets. In particular, GRN_E improves over the best baseline PRM by 0.073, 0.022 in AUC, and 0.005, 0.006 in GAUC on Rec and Ad dataset, respectively. By taking full advantage of the contextual information in the final ranking list, GRN_E shows the excellent ability to provide more precise contextual interaction probabilities. Different from extracting information from the disordered input ranking list in DLCM and PRM, with the help of well-designed Bi-LSTM and self-attention mechanism, GRN_E capture the sequential dependency and mutual influence in the final ranking list, which is another essential and effective factor to affect the contextual predictions.

• •

  GRN_G significantly and consistently outperforms state-of-the-arts methods by a relatively large margin on both datasets across all metrics. Overall, on Rec and Ad dataset, GRN_G achieves performance gains over the best baseline (i.e., Seq2Slate) by 0.008 and 0.003 for NDCG , 0.011 and 0.005 for LR, respectively, which evidences that the technically designed model architecture and training procedure of GRN_G have successfully transformed the context-wise ability of GRN_E into production. The evolving and activating layers model the information of the selected list, assisting the selector layer to make the most appropriate choice for each step by comparing in the input ranking list. Besides, under the guidance of GRN_E, policy gradient with the proposed advantage reward helps distill the reranking knowledge into GRN_G comprehensively.

• •

  Compared with the experimental results on the Rec dataset, the performance lift on the Ad dataset is relatively slight. One possible reason is that Ad dataset is published with random sampling, resulting in the inconsistent and incomplete list records as well as weaker intra-list relevance of user feedback.

In this section, we perform a series of experiments on the Rec dataset to better understand the traits of GRN, including well-designed components of evaluator, generator and training procedure. It is noteworthy that similar trends can also be observed on the Ad dataset, which are omitted due to the page limitation.

In order to clearly demonstrate how GRN addresses the limitations of the greedy reranking strategy existed in previous methods, we conduct one case study in the large-scale industrial Rec dataset. As shown in Fig. 5, the main findings are summarized as follows:

• •

  The rating scores below the item are estimated by a well-trained point-wise model. Intuitively, according to the score distribution, we can find that the target user is most interested in clothing category recently, followed by jewelry and pants.

• •

  The greedy reranking strategy choose to place four clothes in its top-5 results. After this user thoroughly browsing and engaging with the first clothe, he/she may be tired of the clothing category and eager for other categories he/she is also interested in. In this case, the greedy reranking strategy can not reach the best recommendation results and further decrease the user experience.

• •

  Different form the greedy reranking strategy, the generator selects the item by considering the contextual items. The recommendation results are more pluralistic, taking more chance to capture and activate the user’s latent interests. Hence, the generator shows its ability to generate a more contextually effective and attractive recommendation list, making users engage with the RS more.

To verify the effectiveness of our proposed framework GRN in the real-world settings, GRN has been fully deployed in homepage for the Guess You Like scenario, the most popular recommendation scenario of Taobao application ⁶⁶6In practice, we only deploy the GRN_G into production for serving reranking task.. In this waterfall scenario, users can browse and interact with items sequentially and endlessly. The fixed-length recommended results are displayed to the user when he/she reaches the bottom of the current page. Considering the potential issue of high latency to the user experience, it brings great challenge to deploy GRN into production.

To this end, we make the following improvements for efficient deployment:

• •

  Instead of the traditional cloud-to-edge framework, we directly deploy GRN on edge as introduced in EdgeRec (Gong et al., 2020). In this way, the network latency between the cloud server and user edge is saved. Nearly 30 milliseconds are saved, consisting of 10 requests and 3 milliseconds per request, which could be a bottleneck of deploying a context-wise algorithm.

• •

  Considering the long total time cost, we developed the advance trigger and delay completion mechanism. Specifically, GRN is requested a few places in advance of the last position of current page. Besides, GRN will return more results than expected at each request, to make up for the vacancy of the next request.

After successful deployment of the proposed GRN, we evaluate the performance from “2020-09-01” to “2020-09-07” and report the performance comparison with deployed baseline model PRM in the Fig. 6. Not surprisingly, we observe that GRN consistently and significantly outperforms PRM model across both PV and IPV metrics, which further verifies the effectiveness of our proposed framework GRN. Overall, GRN achieves performance improvement over the best baseline PRM of 5.2% for PV and 6.1% for IPV, with the average cost of 32 milliseconds for online inference.