Deep Structural Point Process for Learning Temporal Interaction Networks

A series chronological interactions can be represented as a temporal interaction network. Formally, a temporal interaction network on the time window $[0,T)$ can be described as $\mathcal{G}(T)=(\mathcal{U},\mathcal{V},\mathcal{E})$, where $\mathcal{U}$, $\mathcal{V}$ and $\mathcal{E}$ denote the user set, item set and interaction set, respectively. Each element $(u_{i},v_{j},t)\in\mathcal{E}$ is an interaction, describing that the user $u_{i}\in\mathcal{U}$ conducts an action with the item $v_{j}\in\mathcal{V}$ at the concrete timestamp $t$.

TPPs are one of branches of stochastic processes for modeling the observed random discrete events, e.g. user-item interactions over time. Using conditional intensity function $\lambda(t)$ is a convenient way to describe TPPs. Given a interaction sequence that only has time information $\mathcal{T}:=\{t_{i}\}_{i=1}^{n}$ and an infinitesimal time interval $\mathrm{d}t$, where $t_{i}\in\mathbb{R}^{+}$ and $0<t_{1}<t_{2}...<t_{n}$, $\lambda(t)\mathrm{d}t$ is the conditional probability of happening an interaction in the infinitesimal interval $[t,t+\mathrm{d}t)$ based on $\mathcal{T}$. It can be also interpreted heuristically in the following way:

where $N([t,t+\mathrm{d}t))$ is used to count the number of interactions happened in the time interval $[t,t+\mathrm{d}t)$. A general assumption made here is that there is either zero or one interaction happened in this infinitesimal interval, i.e., $N([t,t+\mathrm{d}t))\in\{0,1\}$. Furthermore, given a future timestamp $t^{+}>t_{n}$ and $\mathcal{T}$, we can formulate the conditional probability of no interactions happened during $[t_{n},t^{+})$ as $S(t^{+})=\exp{(-\int_{t_{n}}^{t^{+}}\lambda(t)\text{d}t)}$. Therefore, the conditional probability density of the next interaction happened at the future timestamp $t^{+}$ can be defined as: $f(t^{+})=S(t^{+})\lambda(t^{+})$, which means that no interaction happens in time interval $[t_{n},t^{+})$ and an interaction happens in infinitesimal interval $[t^{+},t^{+}+\mathrm{d}t)$.

Multivariate Hawkes Process (MHP) is one of the most important TPPs for modeling interaction sequences [9]. We denote $\mathcal{S}_{u_{i}}(T)=(\mathcal{V},\mathcal{H}_{u_{i}})$ as an interaction sequence of user $u_{i}$ on the time window $[0,T]$, where $\mathcal{H}_{u_{i}}$ is the interaction sequence of user $u_{i}$. The $h$-th interaction of $\mathcal{H}_{u_{i}}$ is denoted as $(v^{h},t^{h})$, which describes that the user $u_{i}\in\mathcal{U}$ has interacted with the item $v^{h}\in\mathcal{V}$ at $t^{h}\leq T$. The intensity function of an arbitrary item $v_{j}$ in $\mathcal{S}_{u_{i}}(T)$ is defined as:

where $\mu_{v_{j}}$ (a.k.a base intensity) is a positive parameter which is independent of the interaction sequence $\mathcal{S}_{u_{i}}(T)$, $\alpha_{(v_{j},v^{h})}$ is also a positive parameter that estimates the influence between item pair $(v_{j},v^{h})$ and the $\kappa(t-t^{h})$ is a triggering kernel function. The intensity function explicit models dynamic influence among interaction sequence. However, most existing approaches ignore to model the topological structure between a series of interaction sequences. To fill this gap, DSPP includes a novel TFE module which provides a strong structure prior to enhance model expressiveness.

For a temporal interaction network $\mathcal{G}(T)=(\mathcal{U},\mathcal{V},\mathcal{E})$, the adjacency matrix would change over time, because the emerge of a new interaction would introduce a new edge in the temporal interaction network, causing the huge occupation of memory. To sidestep this problem, we exploit an ordered snapshot sequence $\{\mathcal{G}^{m}\}_{m=0}^{M-1}$ with the same time interval $d=\frac{T}{M}$ to simulate the temporal interaction network $\mathcal{G}(T)$. Each snapshot $\mathcal{G}^{m}$ equals to $\mathcal{G}(d\times m)$, and $M$ is a hype-parameter to control the number of snapshot.

Fig.1 shows the overview of our model. In DSPP, the TFE module aims to learn steady embeddings which represent the stable intentions of users and items in the corresponding time period $[d\times m,d\times(m+1))$ from $\mathcal{G}^{m}$. Here we denote the steady embeddings of an arbitrary user $u_{i}$ and an item $v_{j}$ as $\bm{u}^{m}_{i}$ and $\bm{v}^{m}_{j}$, respectively. The second module ASE aims to learn dynamic embeddings of users and items for describing their dynamic intentions at timestamp $t$.

The embedding layer is used to initialize node embeddings (all users and items) and time embedding. It consists of two parts: the node embedding layer and the time embedding layer.

In this section, we propose a novel topological fusion encoder (TFE) to learn the topological structure. The existing graph encoders [23, 6] learn node embeddings by aggregating the features of neighboring nodes directly. However, in temporal interaction networks, it would easily lead to improper feature aggregations due to the fact that the user neighbors are items. In this paper, we introduce a topological aggregation layer (TAL) into TFE to alleviate this issue.

In this section, we develop an attentive shift encoder (ASE) for temporal interaction networks for capturing the long-range dependency structure. Previous works employ different RNN variants which tend to forget the long history information, leading to the problem of long-range dependency structure missing. In contrast, our ASE module can explicitly learn the dependencies between each historical interaction and the new-coming interaction via the attention mechanism.

To explicitly capture dynamic influence between users and items, we devise a novel intensity function which is generated via the steady embeddings and dynamic embeddings.

To make a fair comparison, we evaluate DSPP on three pre-processed benchmark datasets [16], i.e., Reddit²²2http://snap.stanford.edu/jodie/reddit.csv, Wikipedia³³3http://snap.stanford.edu/jodie/wikipedia.csv and Last.FM⁴⁴4http://snap.stanford.edu/jodie/lastfm.csv. The concrete statistics of users, items, interactions and action repetition are listed in Table 1. Noticeably, these datasets are largely different in terms of action repetition rate, which can verify whether DSPP is able to capture the dynamic influence in various action repetition scenarios accurately.

Data Preprocessing: As used in JODIE [16] and DGCF [17], for each dataset, we first sort all interactions by chronological order. Then, we use the first 80% interactions to train, the next 10% interactions to valid, and the remaining 10% interactions for the test. In contrast with JODIE and DGCF, we generate a snapshot sequence $\{\mathcal{G}^{m}\}_{m=0}^{M-1}$. In our setting, the validation snapshots cover the training data, and the test snapshots also contain all training and validation data.
Evaluation Metrics: To evaluate our model performance, for each interaction, we first generate corresponding user and item steady and dynamic embeddings. Then, we rank all items by Eq.(10) and predict the future time by Eq.(11). Afterward, we evaluate the item prediction task with the following metrics: Mean Reciprocal Rank (MRR) and Recall@10. higher values for both metrics are better. For the time prediction task, we use Root Mean Square Error (RMSE) to measure model performance, and a lower value for RMSE is preferred.
Baselines: We compare DSPP with the following baselines.

• •

  Random walk model: CTDNE [21].

• •

  Recurrent network models: LSTM [11], RRN [27], LatentCross [2], Time-LSTM [30], JODIE [16] and DGCF [17].

• •

  Temporal point process model: DeepCoevolve [7].

Implementation Details: In our experiments, we use the official implementations⁵⁵5https://hanjun-dai.github.io/supp/torch_coevolve.tar.gz of DeepCoevolve. Except from it, we directly report the experimental results in the original papers [16, 17]. DSPP follows the same hyper-parameter setting with baselines: the embedding dimension $D$ is fixed as 128, the batch size $B$ is fixed as 128, the learning rate is fixed as 0.001, the model weight decay is fixed as 0.00001, the sampling number for Monte Carlo estimate is fixed as 64, the number of negative sampling is fixed as 10, the number of TAL $K$ is fixed as $2$, the Attention is stacked 8 layers, the GRUs are 1 layer, the number of snapshots $M$ is selected from $\{128,256,512,1024\}$ and the maximum length of interaction sequence $H$ is chosen from $\{20,40,60,80\}$. The Adam [14] optimizer is used to update all model parameters.

For item prediction, Table 2 shows the comparison results on the three datasets according to Recall@10 and MRR. From the experimental results, we have the following observations:

• •

  DSPP consistently yields the best performances on all datasets for both metrics. Compared with state-of-the-art baselines, the most obvious effects are that DSPP achieves the 6.8% improvement in terms of MRR on Last.FM, the 6.2% improvement in terms of Recall@10 on Wikipedia and the 2.6% improvement in terms of MRR on Reddit. It reveals that incorporating the topological structure and long-range dependency structure can bring good robustness in different action repetition scenarios.

• •

  DSPP outperforms DeepCoevolve significantly. This phenomenon demonstrates that DSPP has a more powerful intensity function that can better capture the dynamic influence between users and items.

• •

  DSPP and DGCF are superior to other baselines on Last.FM, which indicates that it is critical to model the topological structure information for learning temporal interaction networks.

Table 3 shows the prediction performances of DeepCoevolve and DSPP. From it, we can observe that DSPP achieves more accurately time prediction. Specifically, our model achieves the 19.1% improvement on Last.FM, the 20.3% improvement on Wikipedia and the 18.1% improvement on Reddit. We suppose that the improvements owe to the following reasons: 1) DeepCoevolve uses a linear intensity function to model dynamic influence over time, which would reduce the model flexibility of intensity function. 2) DeepCoevolve remains the same user and item embeddings until it involves a new-coming interaction, so it limits model expressiveness. In contrast, our intensity function can learn the nonlinear dynamic influence, since the ASE module can provide time-aware dynamic embeddings.

To investigate the effectiveness of our model components, we implement several variants of DSPP and conduct the experiment on Last.FM dataset for the task of item prediction. The experimental results are reported in Table 4. According to it, we can draw the following conclusions:

• •

  Remove TFE. To verify whether our proposed TFE module is useful to enhance the expressiveness of the intensity function, we first remove it and only remain the second term of Eq.(7) as our intensity function. Then, we directly predict the item that is most likely to interact with each user via Eq.(10). As shown in the results, both metrics Recall@10 and MRR sharply drop 14.6% and 26.2%, respectively. It demonstrates that modeling the topological structure of temporal interaction networks can provide a powerful structural prior for enhancing the expressiveness of intensity function.

• •

  Remove TFE. This variant can be also viewed as a non-graph based model, since it does not exploit the topological structure, and the remaining temporal attention shift encoder only provides the long-range dependency structure to model intensity function. Compared with DeepCoevolve, this variant yields 19.6% and 18.7% improvements on Recall@10 and MRR respectively. This observation shows that our proposed temporal attention shift encoder can further enhance the intensity function.

• •

  Replace TAL with GCN/GAT. To verify whether our proposed TAL is superior to other graph encoders for capturing the topological structure of temporal interaction networks. We replace TAL by GCN [15] and GAT [26]. For the GCN variant, both Recall@10 and MRR drop 5.8%. For GAT variant, Recall@10 and MRR drop 4.0% and 4.6%, respectively. So, We suppose that our proposed TAL can better capture the information of the same type entity.