Time-aware Hyperbolic Graph Attention Network for Session-based Recommendation

GNNs [22, 23] are designed to handle the structural graph data. In GNNs, aggregation is the core operation to extract structural knowledge. By aggregating neighboring information, the central node can gain knowledge from its neighbors passed through edges and learn the node embedding. GNNs have been demonstrated to be powerful in learning node embeddings, so they are widely used on many node-related tasks such as node classification[23], graph classification[24], and link prediction[22].

Based on the aggregation operation, the forward propagation of a GNN on graph $G=(\mathcal{V},\mathcal{E})$ is to learn the embedding of node $v_{i}\in\mathcal{V}$ via aggregating its neighboring nodes. We suppose that the initial node embedding of each node $i$ is $\mathbf{h}^{(0)}_{i}$, which generally is the feature of the node. In each hidden layer of a GNN, the embedding of the central node $\mathbf{h}^{(l)}_{i}$ is learned from the aggregated embedding of the neighboring nodes in the previous hidden layer $\mathbf{h}^{(l-1)}_{i}$. The process is described in math as follows:

where $\mathcal{N}_{i}$ represents the set of all neighbors of node $i$ in the graph, including the node $i$ itself. The aggregation function $\text{AGG}(\cdot)$ integrates the neighboring information together. A non-linear activation function $\sigma$, e.g., sigmoid or LeakyReLU, is applied to generate the embedding of node $i$ in the layer $l$.

Based on the vanilla GNN we mentioned above, GAT [25] is proposed to improve GNNs with self-attention mechanism [26]. Specifically, for all the neighbors of node $i$, we need to learn the attention coefficients for all its neighbors to calculate the importance of each neighbor node in the aggregation. Suppose the attention coefficient of the node pair $(i,j)$ is $\alpha_{ij}$, the process of learning $\alpha_{ij}$ is

where $\mathbf{d}_{ij}$ is the correlation between node $i$ and $j$. $\mathbf{d}_{ij}$ here can be the joint embeddings of node $i$ and $j$, e.g., concatenation of node embeddings or similarity of the node pair.

In definition, hyperbolic space is a homogeneous space with negative curvature. It is a smooth Riemannian manifold, which can be modeled in several hyperbolic geometric models, including Poincaré ball model[27], Klein model[28], Lorentz model[29], etc. In this paper, we choose the Poincaré ball model because the distance between two points grows exponentially, which fits well with the hierarchical structure of the session graph. Formally, the space of the $d$-dimensional Poincaré ball $\mathbb{P}_{c}^{d}$ is defined as

where $c$ is the radius of the ball and $\mathbf{x}$ is any point in manifold $\mathcal{P}$. If $c=0$, then $\mathbb{P}_{c}^{d}=\mathbb{R}^{d}$ and the ball is equal to the Euclidean surface. In this paper, we set $c=1$. The tangent space $\mathcal{T}_{\mathbf{x}}\mathcal{P}$ is a $d$-dimensional vector space approximating $\mathcal{P}$ around $\mathbf{x}$, which is isomorphic to the Euclidean space. With the exponential map, a vector in the Euclidean space can be mapped to the hyperbolic space. The logarithmic map is the inverse of the exponential map, which projects the vector back to the Euclidean space.

In hyperbolic spaces, the fundamental mathematical operations of neural networks (e.g., addition and multiplication) are different from those in Euclidean space. In this paper, we choose Möbius transformation as an algebraic operation for studying hyperbolic geometry. For a pair of random vectors $(\mathbf{a},\mathbf{b})$, we list the operations that will be used in our model as follows:

• •

  Möbius addition $\oplus$ [30] is to perform addition operation of $\mathbf{a}$ and $\mathbf{b}$.

  $$\mathbf{a}\oplus\mathbf{b}=\frac{(1+2\langle\mathbf{a},\mathbf{b}\rangle+\|\mathbf{b}\|^{2})\mathbf{a}+(1-\|\mathbf{a}\|^{2})\mathbf{b}}{1+2\langle\mathbf{a},\mathbf{b}\rangle+\|\mathbf{a}\|^{2}\|\mathbf{b}\|^{2}}.$$

  (4)

• •

  Möbius matrix-vector multiplication $\otimes$ [31] is employed to transform $\mathbf{a}$ with matrix $\mathbf{W}$.

  $$\mathbf{W}\otimes\mathbf{a}=\tanh(\frac{\|\mathbf{W}\mathbf{a}\|}{\|\mathbf{a}\|}\tanh^{-1}(\|\mathbf{a}\|)),$$

  (5)

• •

  Möbius scalar multiplication $\otimes$ is the multiplication of a scalar $\alpha$ with a vector $\mathbf{b}$.

  $$\alpha\otimes\mathbf{b}=\tanh(\alpha\tanh^{-1}(\|\mathbf{b}\|))\frac{\mathbf{b}}{\|\mathbf{b}\|}$$

  (6)

• •

  Exponential map transforms $\mathbf{a}$ from the Euclidean space to a chosen point $\mathbf{x}$ in a hyperbolic space.

  $$\exp_{\mathbf{x}}(\mathbf{a})=\mathbf{x}\oplus(\tanh(\frac{\lambda_{\mathbf{x}}\|\mathbf{a}\|}{2})\frac{\mathbf{a}}{\|\mathbf{a}\|}),$$

  (7)

• •

  Logarithmic map projects the vector $\mathbf{a}$ back to the Euclidean space.

  $$\log_{\mathbf{x}}(\mathbf{a})=\frac{2}{\lambda_{\mathbf{x}}}\operatorname{arctanh}(\|-\mathbf{x}\oplus\mathbf{a}\|)\frac{-\mathbf{x}\oplus\mathbf{a}}{\|-\mathbf{x}\oplus\mathbf{a}\|}$$

  (8)

• •

  $\lambda_{\mathbf{x}}$ is the conformal factor.

  $$\lambda_{\mathbf{x}}=\frac{2}{1-\|\mathbf{x}\|^{2}}.$$

  (9)

Session-based recommendation (SBR) is to predict the item a user will click next based on the user-item interaction sessions. Generally, it models the user’s short-term browsing session data to learn the user’s current interest. Here we formulate the SBR problem mathematically as below.

In the SBR problem, a session is denoted as $S=\{v_{1},v_{2},\cdot\cdot\cdot,v_{n}\}$ ordered by timestamps. Each $v$ in $S$ is an item, and the item set is $V_{s}$, which consists of all unique items in this session. To model the session into a directed graph, we take all items as nodes and the item-item sequential dependency as the edges to construct the session graph. The graph is denoted as $G_{s}=(V_{s},E_{s})$, where $V_{s},E_{s}$ are the node and edge sets, respectively. Each edge connects two consecutive items, which is formulated as $e=(v_{t-1},v_{t})$. Our target is to learn the embeddings of items and the session and generate the ranking of the items that the user may be interested in at the next timestamp.

In GNN, each node needs input as the initial embedding. Accommodated to SBR, the input of a GNN is the feature of items such as category or description. The initial embedding of item $i$ is $\mathbf{h}_{i}^{0}$. However, most feature embedding methods are based on the Euclidean space. To make the item features available in the hyperbolic space, we use the exponential map defined in Eq. 7 to project the initial item embeddings to the hyperbolic space. Specifically, the projection process is formulated as

where $\mathbf{m}_{i}$ is the mapped embedding in the hyperbolic space and $\mathbf{x}$ is the chosen point in the tangent space.

To achieve a high-level latent representation of the node features, we also add a linear transformation parameterized by a weight matrix $\mathbf{W}_{1}\in\mathbb{R}^{d^{\prime}\times d}$, where $d^{\prime}$ is the dimension of $\mathbf{m}_{i}$ and $d$ is the dimension of the node’s final embedding. Please note that $\mathbf{W}_{1}$ is a shared weight matrix for all nodes. Möbius matrix-vector multiplication defined in Eq. 4 is employed to transform $\mathbf{m}_{i}$ and the process is

where $\mathbf{h}_{i}^{1}$ is the transformed embedding, which is also used as the initial node embedding in the following steps.

According to [15, 20, 32, 33], embedding users and items in hyperbolic spaces is a significant improvement of graph-based recommender systems. However, none of these works model the time intervals and users’ current interests in hyperbolic spaces. Our proposed model TA-HGAT is the first attempt to solve the problem, in which time-aware hyperbolic attention is the core component. It is composed of two attention layers: 1) Hyperbolic self-attention in the aggregation process, which considers time intervals between items; 2) Hyperbolic soft-attention in the session embedding learning, which models the user’s current interest.

Here we introduce how to leverage evolutionary loss to provide recommendations given a specific timestamp. Unlike other works [3, 35], our evolutionary loss is also fully hyperbolic.

To demonstrate the effectiveness of TA-HGAT, we conduct experiments on two public datasets and compare the model with ten state-of-the-art baseline models.

In the TA-HGAT, we have two main modules in time-aware hyperbolic attention: hyperbolic self-attention with time intervals and hyperbolic soft-attention with users’ current interests. In this section, we evaluate their effectiveness separately to show the improvement compared with the ablation models without these two modules.

We set up four separate ablation models to compare the effectiveness of each attention layer. The first ablation model is no-att, in which we remove both the attention layers and only conduct the aggregation operations directly. The second and third ones are self-att and soft-att, and these two ablation models only include the self-attention and soft-attention layers, respectively. The fourth one is TA-HGAT, which is the complete model. The comparison results of the ablation models on datasets Diginetica and Yoochoose are illustrated in Figure 2.

From Figure 2, we observe the following results:

• •

  On both Diginetica and Yoochoose datasets, the non-att performs worst, and TA-HGAT performs best. The results show the effectiveness of the attention layers. This is because the TA-HGAT makes full use of the temporal information. Compared to the GNNs without temporal information, our model builds the relations between items with time intervals and also considers users’ current interests. Hence, the rich information helps the model to achieve better results.

• •

  Self-att performs better than soft-att, which means time intervals are relatively more meaningful than users’ current interests. This phenomenon may be due to the fact that users’ current interests are more complicated, so the last item cannot fully represent them. In contrast, the time interval is a more straightforward feature, so our proposed hyperbolic self-attention layer can handle this information effectively.

In this section, we compare our proposed hyperbolic evolutionary loss to conventional loss functions, i.e., BPR [36] and softmax loss [7]. Because the learned session embeddings in the output of our model are in the hyperbolic space, we need to use the logarithmic map to project the embeddings back to the Euclidean space before applying BPR and softmax loss.

The comparison results are shown in Table III. The hyperbolic evolutionary loss is denoted as TA-HGAT in the table. From this table, we can find that the performance of BPR and softmax loss is similar, but our proposed hyperbolic evolutionary loss has a clear improvement compared to the other losses. This observation demonstrates that considering the specific timestamp is effective for the SBR task models designed in hyperbolic space.

The embedding dimension is the hyperparameter in our proposed model, so we test the influence of different embedding dimensions in this section. The embedding dimensions range from 20 to 100. The results of the hyperparameter analysis are illustrated in Figure 3.

It is observed that a proper embedding dimension is essential for learning the item and session representations. From Figure 3, we can see that the Diginetica and Yoochoose 1/64 all achieve the best performance when the embedding dimension is 60, and the best result of Yoochoose 1/4 is 80. Because Yoochoose 1/4 is much larger than the other two datasets, it indicates that larger datasets need larger embedding space.

Recent research has shown that many types of complex data exhibit a highly non-Euclidean structure[19]. In many domains, e.g., natural language [42], computer vision[43], and healthcare[44], data usually has a tree-like structure or can be represented hierarchically. Since this type of data contains an underlying hierarchical structure, capturing such representations in Euclidean space is difficult. To solve this problem, current studies are increasingly attracted by the idea of building neural networks in Riemannian space, such as the hyperbolic space, which is a homogeneous space with constant negative curvature [27]. Compared with Euclidean space, hyperbolic space in which the volume of a ball grows exponentially with radius instead of growing polynomially. Because of its powerful representation ability, hyperbolic space has been applied in many areas. For instance, [45] learns word and sentence embeddings in hyperbolic space in an unsupervised manner from text corpora. [46] demonstrates that hyperbolic embeddings are beneficial for visual data. [47] proposes Hyperbolic Graph Convolutional Neural Networks, which combines the expressiveness of GCNs and hyperbolic geometry to learn graph representations. These works show the potential and advantages of hyperbolic space in learning hierarchical structures of complex data.

Based on the performance of hyperbolic space in these fields, it is natural for researchers to think of applying hyperbolic learning to recommender systems. [48] justifies the use of hyperbolic representations for neural recommender systems. [49] proposes HyperML to bridge the gap between Euclidean and hyperbolic geometry in recommender systems through a metric learning approach. [21] proposes a hyperbolic GCN model for collaborative filtering. [50] presents HyperSoRec, a novel graph neural network (GNN) framework with multiple-aspect learning for social recommendation.

Session-based Recommendation (SBR) has increasingly engaged attention in both industry and academia due to its effectiveness in modeling users’ current interests. In the recent SBR studies, there are mainly three types of methods that apply deep learning to SBR and have achieved state-of-the-art performance, which are sequence-based [4, 51], attention-based [5, 6] and graph-based models [7]. GRU4REC [4] is the most representative work in the sequence-based SBR models. It employs GRU, a variant of RNN, to model the item sequences and make the next-item prediction. Following GRU4REC, some other papers [51, 52, 39] improve it with data augmentation, hierarchical structure, and top-$k$ gains. Attention-based models aim to learn the importance of different items in the session and make the model focus on the important ones. NARM [5] utilizes an attention mechanism to model both local and global features of the session to learn users’ interests. STAMP [6] combines the attention model and memory network to learn the short-term priority of sessions. Graph-based models connect the items in a graph to learn their complex transitions. SR-GNN [7] is the first work that models the sessions into graphs. It leverages the Gated Graph Neural Network (GGNN) to model the session graphs and achieve state-of-the-art performance. Based on SR-GNN, [10, 12] improve SR-GNN with attention layers. [11, 8] consider the item order in the session graph in the model. [9, 13, 14] take additional information such as global item relationship, item categories, and user representations into account to design more extensive models. However, all these methods fail to consider the hierarchical geometry of the session graphs and the temporal information.

GNNs have proven to be useful in different research fields [53, 54, 55, 56, 57, 58]. There also exist many works considering the graph structures in data modeling of recommender systems. Generally, there are two main ways regarding the graph structure and embedding space. One way is to model the user-item interaction graph in Euclidean spaces. Among them, [59, 60, 61] perform graph convolution on the user-item graph to explore their interactions. [62, 63, 64] utilize layer-to-layer neighborhood aggregation in GNNs to capture the high-order connections. [65] pre-trains user-user and item-item graphs separately to learn the initial embeddings of the user-item interaction graph. [35] models the changes in user-item interactions with a dynamic graph and evolutionary loss. These works apply GNN to learn from high-dimensional graph data and generate low-dimensional node embeddings without feature engineering, but the learned embeddings are all in Euclidean spaces, while some graph data may be more suitable to other geometries in the representation learning.

The other way is to model the recommendation graph in hyperbolic space to learn the hierarchical geometry. HGCF [21] applies hyperbolic GCN to learn the node embeddings using a user-item graph. Wang et al. [20] propose a fully hyperbolic GCN where all operations are conducted in hyperbolic space. Xu et al. [32] model the product graph in a knowledge graph and learn the node embeddings in hyperbolic space. HCGR [15] is a novel hyperbolic contrastive graph representation learning method to make session-based recommendations. None of these models utilize the time-relevant information in the session graphs to improve the recommendation accuracy. In this paper, we propose a novel framework incorporating a time-aware graph attention mechanism in hyperbolic space, which is specifically devised for the session-based recommendation.