A Generic Reinforced Explainable Framework with Knowledge Graph for Session-based Recommendation

As aforementioned, for each scenario, we firstly construct a KG $\mathcal{G}=\{(e_{i},r,e_{j})|e_{i},e_{j}\in\mathcal{E},r\in\mathcal{R}\}$. The entities in Amazon dataset²²2In this paper, for better elaboration, we take Amazon dataset as an example to showcase the KG construction process, which can be easily applied in other scenarios/domains. can be divided into five types: user, product, brand, category, and related_product. As shown in Figure 2, we connect every entity pair with a directed relation if there is a relation from a head entity to a tail entity. It is worth noting that, in SR, there exists dependent relationship between items (products) in a session. Therefore, we accordingly add the co_occurrence relation in the built KG. That is, with regard to a session $\mathcal{S}_{e}$, if item $v^{S}_{i}$ is firstly interacted followed by item $v^{S}_{i+1}$, we add the triple $(v^{S}_{i},co\_occurrence,v^{S}_{i+1})$ to $\mathcal{G}$, thus the KG has a edge from $v^{S}_{i}$ to $v^{S}_{i+1}$. For other types of relations, similar to [xian2019reinforcement, zhao2020leveraging], we add a bidirectional edge to the knowledge graph. For example, if there exits a triple $(product,belong\_to,category)$, we add two edges to the $\mathcal{G}$, i.e., $product\xrightarrow{belong\_to}category$ and $category\xrightarrow{belong\_to}product$. The statistics of total entities and relations on the three Amazon datasets are summarized in Tables LABEL:tb:relationsStatistics and LABEL:tb:entitiesStatistics.

After constructing the KG, we leverage Translating Embedding (TransE) technique [bordes2013translating] to obtain the initial representations for all entities and relations. Specifically, we first denote $\mathbf{X}^{0}\in\mathbb{R}^{(|\mathcal{E}|+|\mathcal{R}|)\times d_{0}}$ as initial embedding matrix:

where $d_{0}$ is the dimension of entity/relation representations, and $|\mathcal{E}|+|\mathcal{R}|$ is the number of total entities and relations.

We firstly elaborate some preliminaries of our Markov decision process (MDP). In general, we define the MDP environment with a quadruple $(S,A,T,R)$, where $S$ indicates the state space, $A$ refers to the action space, $T:S\times A\to T$ denotes the state transition function, and $R:S\times A\to R$ indicates the reward function.

State. Let $s$ ($s\in S$) denote a state in our MDP environment. The initial state $s_{0}$ at time $t_{0}$ is represented by the initial entity of the semantic path. Besides, for session-based recommendation, given a session $\mathcal{S}_{e}$, we use the last interacted product, i.e., $v_{l}^{S}$ in the session as the initial entity, considering that the last interacted product is the most recent behavior of this session and thus may involve more information for better next-item prediction [balloccu2022post]. Accordingly, we denote the state at time $t$ as $s_{t}=(\mathbf{e_{t}},\mathbf{h_{t}})$, where $\mathbf{e_{t}}$ is the representation of entity $e$ at time $t$, and $\mathbf{h_{t}}$ contains the history of all entities and relations in the semantic path from the beginning to the time $t$, i.e., $\mathbf{h_{t}}=(\mathbf{e_{0}},\mathbf{r_{1}},\mathbf{e_{1}},\cdots\mathbf{r_{t}})$, where $\mathbf{e_{0}}$ is the representation of entity $v_{l}^{S}$ in session $\mathcal{S}^{e}$.

Action. Based on the state $s_{t}=(\mathbf{e_{t}},\mathbf{h_{t}})$, we define, at time $t$, an action space $A_{t}=\{(r_{t+1},e_{t+1})|(e_{t},r_{t+1},e_{t+1})\in\mathcal{G}\}$, which indicates all possible edges of entity $e_{t}$ in $\mathcal{G}$ excluding those ones historically being visited in the semantic path. The action behavior is modeled by a policy network $\pi$, which outputs a probability distribution over actions in $A_{t}$ to decide the next action (happened at $t+1$). In our REKS model, the policy network is influenced by both session information and semantic path information.

In SR, we can obtain session representation by representative non-explainable SR models, like GRU4REC, NARM, SRGNN, BERT4REC, and GCSAN. For example, using NARM, session representation $\mathbf{S_{e}}\in\mathbb{R}^{d_{1}}$ is calculated as:

where $\mathbf{X}^{0}_{\mathcal{V}}$ is the initial embedding of items (measured by TransE in Equation 1), and together with $\mathcal{S}_{e}$, contribute to the inputs of NARM. Besides, the corresponding semantic path related information (denoted as $\mathbf{S_{p}}\in\mathbb{R}^{d_{0}}$) is modeled by the sum of the entity representation $\mathbf{x}^{0}_{e_{t}}$ and the relation representation $\mathbf{x}^{0}_{r_{t}}$ at time $t$, as the most related information to determine next action in this semantic path is the most recent information (i.e., $e_{t}$ and $r_{t}$). Thus, the total representation $\mathbf{s^{ep}_{t}}\in\mathbb{R}^{d_{2}}$ is calculated as:

where $\oplus$ denotes concatenation operation, and $MLP(*)$ denotes a multilayer perceptron.

Finally, we use a softmax function to compute the probability of selecting next product:

where $\mathbf{A_{t}}$ is a binary vector of $A_{t}$ and contains the probability of each action, and $\mathbf{W_{1}}$ is a trainable parameter matrix.

Reward. It is important to define an appropriate reward for reinforcement learning. For our research, we strive to simultaneously optimize the recommendation and explanation tasks, that is, to accurately recommend top $K$ products that might be probably preferred/liked, and provide suitable and reasonable explanations regarding recommended items. To fulfill the two objectives, we consider three components in our designed reward function: item-level reward, rank-level reward, and path-level reward:

where $R_{item}$, $R_{path}$, and $R_{rank}$ denote the item-level reward, path-level reward, and rank-level reward,respectively.

With regard to item-level reward, we consider the straightforward reward strategy, namely, the terminal reward, which inclines to give a higher reward if a semantic path ends with the target product (i.e. the ground-truth item that the target user will interact with). Here, we define $R_{item}$ with regard to the terminal state $s_{T}$ $(s_{T}=(e_{T},h_{T}))$:

where $v_{T}$ is the ground-truth product, $\sigma(\cdot)$ is the sigmoid function, and $e_{T}$ is the last entity on the semantic path, i.e., the predicted entity given the session. The reward function indicates that: if the predicted entity is the target product, $R_{item}=1$; if the predicted entity is not the target product but belongs to product entity, we set the item-level reward as the similarity between the target product and the predicted entity. Otherwise, we consider the item-level reward to be $0$.

For rank-level reward, we aim to further improve the recommendation accuracy of session-based scenario by pushing the target item to a higher position in the top-K ranking list. Towards this goal, we thus propose our rank-level reward, $R_{rank}$, which is particularly measured as:

where $r^{T}$ refers to the rank of entity $e_{T}$ in the predicted top $K$ product list.

Considering path-level reward, we want to generate a semantic path of high quality which is more relevant to the given session and thus can better explain our predicted item at the final step. Therefore, we define $R_{path}$ as:

where $\sigma(.)$ is the sigmoid function and $\mathbf{P}$ is the representation of this semantic path:

Transition. After obtaining each action $a_{t}$, the agent can get an intermediate reward, i.e., $r_{t+1}=R(s_{t},a_{t})$, which reflects the performance of our model. We then utilize the transition function $T:S\times A\to T$ to update the state. Given the current state $s_{t}$ and the action $a_{t}$, the transition function is defined as:

Optimization. Given the above MDP environment, we want to learn a stochastic policy $\pi$ that maximizes the expected cumulative reward, and minimizes the cross-entropy loss function. That is, our model’s loss function can be made up of two parts: reward loss function (cumulative reward maximization) $L_{r}$ and cross-entropy loss function $L_{ce}$.

$\beta$ is a hyper-parameter to balance the two parts of loss. The reward loss function is defined as:

where $Q(\theta)$ is the expected cumulative reward, $\theta$ denotes the set of model parameters, and $\gamma$ indicates the discount factor. In this work, we employ REINFORCE with a baseline algorithm [sutton2018reinforcement] to optimize parameters $\theta$.

Then the cross-entropy loss is calculated as:

where $y_{j}$ is the ground-truth score of item $v_{j}$ ($1$ or $0$). $\hat{y}_{j}$ is the predicted score, which is calculated by the policy network in Equation 4.

After training the model, we implement a probabilistic beam search algorithm [xian2019reinforcement, zhao2020leveraging] to find candidate products and explainable semantic paths given a session in SR. It should be noted that the start point for the semantic paths is the last interacted product in the session, which is the same as the training process.

We choose three real-world datasets, i.e. Beauty, Cellphones, and Baby from the Amazon e-commerce platform. Each dataset consists of both users’ behaviors and meta information. Firstly, we extract different types of entities and relations from Amazon metadata. The detailed information are summarized in Tables LABEL:tb:relationsStatistics and LABEL:tb:entitiesStatistics. Besides, we consider a user’s interactions that occurred in a day as a session, and filter out items with less than $5$ interactions and sessions with lengths smaller than $2$. The detailed information of three datasets regarding sessions are further reported in Table LABEL:tb:dataStatistics.

We apply our REKS model on five representative SR models, i.e., one RNN-based method (GRU4REC), two GNN-based models (SR-GNN and GCSAN), one attention-based method (BERT4REC), and one hybrid method that combines attention and RNN (NARM).

• •

  GRU4REC [hidasi2015session] applies GRU to model interaction sequences and designs a ranking-based loss function.

• •

  NARM [li2017neural] improves GRU4REC by utilizing vanilla attention to model the relationship of the last item in a session with other items of the session.

• •

  SR-GNN [wu2019session] is a solid GNN-based approach for SR, which employs a gated graph convolutional layer to obtain item embedding and applies self-attention mechanism to obtain session representation.

• •

  GCSAN [xu2019graph] utilizes self-attention mechanism to capture item dependencies via graph information aggregation.

• •

  BERT4REC [sun2019bert4rec] employs a deep bidirectional self-attention framework to model user behaviors and utilizes a Cloze objective loss for SR.

Noted that for fair comparisons, the inputs of these baselines are the same as our model, including both the knowledge graph $\mathcal{G}$ and session information. We also apply TransE method on $\mathcal{G}$ to obtain the initial item representations for these baselines without REKS.

Given a session, each method generates a top-$K$ recommendation list ($K$ is set to $5$, $10$, $20$ in our experiments). Towards exhaustive evaluations on recommendation task, we use two widely used accuracy metrics: HR@$K$ and NDCG@$K$. HR@$K$ indicates the hit ratio, i.e., the coverage rate of targeted predictions; and NDCG@$K$ (Normalized Discounted Cumulative Gain) rewards each hit based on its position in the ranked recommendation list. A larger value indicates better performance for the two metrics.

Regarding each method, we tune the best hyper-parameters according to validation data on each dataset. For all methods, the dimension $d_{0}$, $d_{1}$, and $d_{2}$ are set to $400$. We train all models with Adam optimizer and the path length is $2$. The settings of other parameters are shown in Table LABEL:tb:hyperParameter. Noted that for all these methods, we run each experiment five times and report the average as the performance in Table LABEL:tb:comparativeResults, where REKS_baseline denotes applying RKES with the corresponding baseline. We also conduct a paired t-test (regarding every baseline and REKS_baseline pair) to validate the significance of the performance difference (^∗ for p-value $\leq$.05 and ^∗∗ for p-value $\leq$.01). Besides, the results are reported in percentage (%).

We instantiate the REKS framework on five SOTA non-explainable SR methods, and their performance on three Amazon datasets is reported in Table LABEL:tb:comparativeResults, where every RKES next to a baseline (e.g., NARM) represents the baseline with RKES (e.g., REKS_NARM). From Table LABEL:tb:comparativeResults, we have some interesting observations as below: (1) With REKS framework, the performance of every baseline can be significantly improved in almost all cases, validating the effectiveness of our framework for non-explainable SR methods on recommendation task. For example, on Beauty, REKS_GRU4REC achieves 9.91% in terms of HR@5 with an improvement of 13.91% over vanilla GRU4REC (8.70%). (2) Among non-explainable SR models, NARM performs better than other methods on Cellphones and Baby, while BERT4REC performs the best on Beauty. Besides, surprisingly, vanilla GCSAN obtains the worst performance across all scenarios. However, RKES_GCSAN also makes the largest progress over GCSAN compared to other methods, indicating that our RL-based framework can well mitigate the performance gaps among non-explainable SR models. (3) REKS_baseline inclines to get better performance than other REKS equipped methods if the corresponding baseline performs better, implying that a better session representation learned from non-explainable SR methods can better guide our RL-based framework to find much more reasonable paths for recommendation task in SR. To conclude, our REKS framework can significantly improve the performance of non-explainable SR methods on recommendation task.

We conduct experiments to test the innovative designs in REKS: (1) loss function; (2) reward function; (3) start point of semantic paths; and (4) path length.

Impact of loss function. The loss function in REKS consists of two parts: reward-based loss and cross entropy loss. To validate the effectiveness of each part, we compare our model with two alternatives: REKS_R merely considers reward loss, while REKS_C only uses cross entropy loss. The results with different loss functions are shown in Figure LABEL:tb:differentlossfunction³³3We can get similar results regarding different methods with $K=5$ and $K=20$, which are not reported in detail due to space limitation (the same as following figures).. REKS_R and REKS_C perform consistently worse than REKS, which demonstrates that each loss function can improve recommendation accuracy. Besides, REKS_R performs better than REKS_C, which might be caused by that, cross entropy loss function is directly linked to recommendation accuracy for recommendation task, whereas in reward loss (reward maximization), we also consider to address explanation task (e.g., path-level reward design).

Impact of reward function. In REKS, we innovatively design a complex reward function for the recommendation and explanation tasks. It consists of three parts: item-level, path-level, and rank-level rewards. Thus, we compare our model with three variants: (1) REKS-rank does not consider rank-level reward; (2) REKS-path further removes the path-level reward from REKS-rank, i.e., only uses item-level reward; and (3) REKS_R1 merely employs the most basic reward, which equals $1$ if the predicted product is the ground-truth product. Figure LABEL:fig:rewardfunction shows their performance, where the x-label represents the corresponding non-explainable SR model. As we can see, each component contributes to the final performance, validating the effectiveness of our reward designs for making more accurate recommendation.

Impact of start points of semantic paths. As discussed in Section I, the start point of semantic paths can be a user or a product (the last interacted item of a session), while in our REKS, we choose to start from the last interacted product given a session. Hence, to verify the validity of our design, we compare our framework (REKS) with a variant, REKS_user, starting from the user of the session. Noted that for each REKS_user model, we re-tune the corresponding best hyper-parameters: path length is $3$, and the sampling size is set to $\{100,10,1\}$ for each step. From the results shown in Figure LABEL:fig:startingpoint, we observe that REKS consistently performs better than REKS_user, validating our hypothesis that, for SR, the last interacted product of a session can better reflect the corresponding user’s recent interests, and thus leads to more accurate next-item prediction.

Impact of path length. Here, we investigate the effect of path length on the model performance, while in our framework, we prefer the path length of $2$. We thus compare our model with two variants: REKS with path length $3$ (REKS_l3) and that of length $4$ (REKS_l4), where the sampling size is $\{100,1,1\}$, and $\{100,1,1,1\}$ for each step, respectively. Figure LABEL:fig:differentpathlength summarizes the comparative results, and we can see that when path length is $2$, our framework can obtain the best recommendation accuracy. This might be caused by that, though a larger path length can generate more patterns, it may also introduce noisy information. Besides, REKS_l4 consistently performs better than REKS_l3. This is because a path pattern is mostly “product $\to$ other entity $\to$ product $\to$ other entity” in the knowledge graph, we are thus expected to get better recommendation performance when the path length is even.

We further investigate the impact of learning rate $lr$, and discount factor $\beta$ in loss function on the model performance. Due to space limitation, we take REKS_NARM as an example, and Figure LABEL:fig:hyper shows the results with the learning rate in the range of $\{0.0001,0.0005,0.001,0.005\}$ and $\{0.2,0.4,0.6,0.8,1,1.2\}$ for $\beta$, respectively. In general, we can see that our framework is comparatively insensitive to these hyper-parameters.