HyperHawkes: Hypernetwork based Neural Temporal Point Process

Hawkes process (Hawkes 1971) is a point process (Valkeila 2008) with self-triggering property. i.e occurrence of previous events trigger occurrences of future events. Hawkes process has been used in earthquake modeling (Hainzl, Steacy, and Marsan 2010), crime forecasting (Mohler et al. 2011), social media (Rizoiu et al. 2017) finance (Bacry, Mastromatteo, and Muzy 2015; Embrechts, Liniger, and Lin 2011) and epidemic forecasting (Diggle, Rowlingson, and Su 2005; Chiang, Liu, and Mohler 2021). They provide a solid mathematical framework for modeling event sequences. Earlier works on point process modeling specify a parametric form for the intensity function characterizing the point process. However, parametric models may not be capable of capturing the complex event dynamics. To address this, several research works were proposed (Du et al. 2016; Mei and Eisner 2017; Omi, Ueda, and Aihara 2019; Zuo et al. 2020) where the intensity function is modeled using neural networks, which are better at learning complex event dynamics. Recently, (Zuo et al. 2020) and (Zhang et al. 2020) proposed to use positional encodings in transformer language models (Vaswani et al. 2017) to model point processes. There are some efforts for learning from small data for Hawkes process (Xie et al. 2019; Salehi et al. 2019). However, they are based on the statistical Hawkes process model (not the neural Hawkes process) and are not applicable to a zero shot learning setting.

• •

  Zero-shot Learning for time-to-event modeling: Assume we are given a collection of N seen sequences $\mathcal{D}^{S}=\{(\mathcal{T}^{1},\mathbf{d}^{1}),(\mathcal{T}^{2},\mathbf{d}^{2}),...,(\mathcal{T}^{N},\mathbf{d}^{N}\}$ where $\mathbf{d}^{i}$ represents the descriptor of the $i^{th}$ sequence or meta-information and $\mathcal{T}^{i}$ represents the times of occurrence of $n^{i}$ events in the $i^{th}$ sequence, i.e. $\mathcal{T}^{i}=\{t^{i}_{j}\}_{j=1}^{n^{i}}$. Our goal is to predict time of event occurrences for $\bar{N}$ unseen sequences $\mathcal{D}^{U}=\{(\mathcal{T}^{1},\mathbf{d}^{1}),(\mathcal{T}^{2},\mathbf{d}^{2}),...,(\mathcal{T}^{\bar{N}},\mathbf{d}^{\bar{N}}\}$ with the help of the sequence descriptor.

• •

  Continual learning for time-to-event modeling: Assume we are given a collection of N sequences $\mathcal{D}=\{(\mathcal{T}^{1},\mathbf{d}^{1}),(\mathcal{T}^{2},\mathbf{d}^{2}),...,(\mathcal{T}^{N},\mathbf{d}^{N}\}$ where $\mathbf{d}^{i}$ represents the sequence descriptor and $\mathcal{T}^{i}$ represents the times of occurrence of $n^{i}$ events in the $i^{th}$ sequence, i.e. $\mathcal{T}^{i}=\{t^{i}_{j}\}_{j=1}^{n^{i}}$ and we assume these sequences arrive one after the other in the order of their index. Our goal is to continually learn the sequences while avoiding catastrophic forgetting from the previous sequences. So, we aim to learn a NHP model which will be able to predict the future event occurrences in all the sequences $(\mathcal{T}^{i},\mathbf{d}^{i})$ where $i\leq j$ after training on the $j^{th}$ sequence $(\mathcal{T}^{j},\mathbf{d}^{j})$.

Point processes are useful to model the distribution of points over some space and are defined using an underlying intensity function. A Hawkes process (Hawkes 1971) is a point process with self-triggering property i.e occurrence of previous events trigger occurrences of future events. Conditional intensity function for univariate Hawkes process at time $t^{i}_{j}$ for the $i^{th}$ sequence is defined as

$$\lambda(t^{i}_{j})=\mu_{i}+\sum_{k=1}^{j-1}k(t^{i}_{j}-t^{i}_{k})$$

(1)

where $\mu_{i}$ is the base intensity function and $k(\cdot)$ is the triggering kernel function capturing the influence from previous events. The summation represents the effect of all events prior to time $t^{i}_{j}$ which will contribute to computing the intensity at time $t^{i}_{j}$. The probability density function at time $t^{i}_{j}$ given the past event times as $\{t^{i}_{1},t^{i}_{2},\ldots,t^{i}_{j-1}\}$, is obtained as follows:

$$p(t^{i}_{j}|t^{i}_{1},t^{i}_{2},\ldots,t^{i}_{j-1})=\lambda(t^{i}_{j})\exp\left\{-\int_{t^{i}_{j-1}}^{t^{i}_{j}}\lambda(t)dt\right\},$$

(2)

where the exponential term in the right-hand side represents the probability that no events occur in $[t^{i}_{j-1},t^{i}_{j})$.

Standard Hawkes process assumes a parametric form for the intensity function which is not generalizable to every event prediction problem. The influences between the events can be complex and need not be exponentially decaying. Various recent works introduced neural Hawkes processes (Du et al. 2016; Mei and Eisner 2016; Omi, Ueda, and Aihara 2019) which models the intensity function as a nonlinear function of history using a neural network. The central idea of these works is to use recurrent neural networks (RNN) to model intensity function which captures the influence of past events. So, conditional intensity function is modeled as:

$$\lambda(t^{i}_{j})=f(\mathbf{h}_{j}^{i})$$

where $\mathbf{h}_{j}^{i}$ represents a hidden state updated using RNN and $f(\cdot)$ is a positive valued function ensuring positivity of the intensity function.

In particular we employ neural Hawkes process (Omi, Ueda, and Aihara 2019) as base model for time-to-event modeling. It uses a combination of recurrent neural network and feedforward neural network to model the intensity function. We represent history by using hidden representations generated by recurrent neural networks (RNNs) at each time step. The hidden representation $\boldsymbol{h}^{i}_{j}$ at time $t^{i}_{j}$ is obtained as

$$\boldsymbol{h}^{i}_{j}=RNN(\tau^{i}_{j},\boldsymbol{h}^{i}_{j-1};W^{i}_{r})=\sigma(\tau^{i}_{j}V^{i}_{r}+\boldsymbol{h}^{i}_{j-1}U^{i}_{r}+b^{i}_{r})$$

(3)

where $\tau^{i}_{j}=t-t^{i}_{j}$ and $W^{i}_{r}$ represents the parameters associated with RNN for the $i^{th}$ sequence such as input weight matrix $V^{i}_{r}$, recurrent weight matrix $U^{i}_{r}$, and and bias $b^{i}_{r}$. $\boldsymbol{h}^{i}_{j}$ is obtained by repeated application of the RNN block on a sequence formed from previous $M$ inter-arrival times. This is used as input to a feedforward neural network to compute the intensity function (hazard function) and consequently the cumulative hazard function for computing the likelihood of event occurrences. In the proposed model, input to the feed-forward neural network is \Romannum1) the hidden representation generated from RNN \Romannum2) elapsed time from the most recent event. We model the conditional intensity as a function of the elapsed time from the most recent event as-

$$\lambda(t|H^{i}_{t})=\lambda(t-t^{i}_{j}|\boldsymbol{h}^{i}_{j})$$

(4)

where $\lambda(\cdot)$ is a non-negative function referred to as a hazard function. Therefore, we define cumulative hazard function in terms of inter-event interval $\tau^{i}_{j}=t-t^{i}_{j}$ as $\Phi(\tau^{i}_{j}|\boldsymbol{h}^{i}_{j})=\int_{0}^{\tau^{i}_{j}}\lambda(s|\boldsymbol{h}^{i}_{j})ds$. Cumulative hazard function is modeled using a feed-forward neural network (FNN)

$$\Phi(\tau^{i}_{j}|\boldsymbol{h}^{i}_{j})=FNN(\tau^{i}_{j},h^{i}_{j};W^{i}_{t}).$$

(5)

However, we need to fulfill two properties of cumulative hazard function. Firstly, it has to be a monotonically increasing function of $\tau^{i}_{j}$ and secondly, it has to be positive valued. We achieve these by maintaining positive weights and positive activation functions in the neural network (Chilinski and Silva 2020; Omi, Ueda, and Aihara 2019). The hazard function itself can be then obtained by differentiating the cumulative hazard function with respect to $\tau$ as

$$\lambda(\tau^{i}_{j}|\boldsymbol{h}^{i}_{j})=\frac{\partial}{\partial\tau^{i}_{j}}\Phi(\tau^{i}_{j}|\boldsymbol{h}^{i}_{j})$$

(6)

The log-likelihood of observing event times is defined as follows using the cumulative hazard function:

$$\begin{split}&\log p(\{t^{i}_{j}\}_{j=1}^{n^{i}};W^{i})=\sum_{j=1}^{n^{i}}\log p(t^{i}_{j}|\mathcal{H}^{i}_{j};W^{i})=\\ &\sum_{j=1}^{n^{i}}\big{(}\log(\frac{\partial}{\partial\tau^{i}_{j}}\Phi(\tau^{i}_{j}|\boldsymbol{h}^{i}_{j-1};W^{i}))-\Phi(\tau^{i}_{j}|\boldsymbol{h}^{i}_{j-1};W^{i})\big{)}\end{split}$$

(7)

where $\tau^{i}_{j}=t^{i}_{j}-t^{i}_{j-1}$ and $W^{i}=\{W^{i}_{r},W^{i}_{t}\}$ represents the combined weights associated with RNN and FNN. In NHP, the weights of the networks are learnt by maximizing the likelihood given by (7). The gradient of the log-likelihood function is calculated using backpropagation.

Hypernetwork is a meta-network which produces parameters used by other networks (Ha, Dai, and Le 2017). As discussed in the above section, the neural Hawkes process comprises of two building blocks - RNN and FNN. So, we use hypernetwork to produce weights for these two components. We use a feed-forward neural network (FNN) to produce parameters $W^{i}=\{W_{r}^{i},W_{t}^{i}\}$ associated with the NHP. Since the nature of the RNN parameters $W_{r}^{i}$ and FNN parameters $W_{t}^{i}$ are different, we use two different types of hypernetworks, $f_{r}(\cdot)$ producing $W_{r}^{i}$ and $f_{t}(\cdot)$ producing $W_{t}^{i}$. Given a sequence description $\mathbf{d}^{i}$, parameters for the RNN are generated as follows:

$$W_{r}^{i}=f_{r}(\mathbf{d}^{i};\theta_{fr})$$

(8)

where $\theta_{fr}$ denotes the parameters of the hypernetwork (weight vectors of a neural network). Note that the hypernetwork parameters are the same across the sequences. The descriptor $\mathbf{d}^{i}$ is used to generate the sequence specific parameters. As discussed above, the cumulative hazard function is a monotonically increasing function of $\tau^{i}_{j}$ and is positive-valued. The hypernetwork which will generate parameters for cumulative hazard function has to fulfill these properties. This can be achieved when hypernetwork generates only positive weights for which we use a positive activation function. Hypernetwork $f_{t}(\cdot)$ for FNN can be written as:

$$W_{t}^{i}=f_{t}(\mathbf{d}^{i};\theta_{ft})$$

(9)

where $\theta_{ft}$ denotes parameters of hypernetwork $f_{t}(\cdot)$. We propose the following variants of HyperHawkes -

• •

  HyperHawkes-FNN: Hypernetwork is considered only for the FNN modeling the cumulative hazard function.

• •

  HyperHawkes-FNN-RNN: This variant uses two separate hypernetworks, one to model the RNN modeling the history and the second to model the FNN.

Overview of the proposed architecture is shown in Fig 1.

In this section we discuss how we employ HyperHawkes for zero-shot event modeling. Our goal is to train the model on seen sequences $\mathcal{D}^{S}$ with event sequences $\mathcal{T}^{s}$ and task descriptors $\mathbf{d}^{s}$ and predict event times of unseen sequences in $\mathcal{D}^{U}$ given a task descriptor $\mathbf{d}^{u}$. We employ HyperHawkes for performing zero-shot learning on event sequences. The central idea of the proposed approach is to predict the parameters for the neural Hawkes process for the unseen task $\mathbf{d}^{u}$. This is achieved using hypernetwork which considers the sequence descriptor as input and parameters for neural Hawkes process as output as discussed in the previous section. Consequently, we can get parameters for RNN ($W_{r}^{u}$) using Equation 8 and FFN ($W_{t}^{u}$) using Equation 9 and use them to model the cumulative hazard function for an unseen sequence. These parameters can then be used for predicting events in the sequence $\mathcal{T}^{u}$.

Training We adopt a training procedure where we train hypernetwork using the maximum likelihood estimation for the NHP model. For each seen sequence $\mathcal{T}^{s}$ from $D^{S}$, we sample a mini-batch consisting of events. The sequence descriptor $\mathbf{d}^{s}$ of this sequence is used to generate parameters of the neural Hawkes process using the hyper-network and its parameters. These values are then used in Equation 7 to find the log-likelihood of the event times of the seen sequences. So, the log-likelihood described in Equation 7 will now be:

$$\begin{split}\sum_{i=1}^{N}\sum_{j=1}^{n^{i}}\log p(t^{i}_{j}|\mathcal{H}^{i}_{j};(f_{r}(\mathbf{d}^{i};\theta_{fr}),f_{t}(\mathbf{d}^{i};\theta_{ft})))\end{split}$$

(10)

The difference in training lies in the fact that by maximizing this log-likelihood, we will get weights of the hyperparameter network rather than the neural Hawkes process. So, we calculate $\theta_{fr}$ and $\theta_{ft}$ using the gradient of the log-likelihood function using backpropagation.

Prediction For prediction of events from unseen sequence $\mathcal{T}^{u}$ from $D^{U}$, we employ our trained hypernetwork to produce weights $\{W_{r}^{u},W_{t}^{u}\}$ for the neural Hawkes process using sequence descriptor $\mathbf{d}^{u}$. Neural Hawkes process uses the bisection method (Omi, Ueda, and Aihara 2019) to predict the time of the next event. Bisection method provides the median $t_{*}$ of the predictive distribution over next event time using the relation $\Phi(t_{*}-t_{j}^{i}|\boldsymbol{h}_{j}^{i};W_{r}^{u},W_{t}^{u})=\log(2)$.

In a more realistic setup of event time modeling, sequences appear one after the other and it is unrealistic to store the data and models associated with all the previous sequences. In spite of this, we need to predict correctly on these past sequences though we have only data from new sequences. We want the NHP models to retain information from the past sequences while learning from new sequences. The standard training of the NHP model adapts them to the new sequence data (by updating parameters to optimize the loss to new sequence data) and results in forgetting what it has learnt from past sequences. The inability of the neural network models to retain knowledge from past data is known as catastrophic forgetting and continual learning techniques have been proposed to address this. Though it is studied in the vision community, to the best of our knowledge we could not find any work on the time-to-event prediction problem and with NHP models.

We address the novel problem of learning the time-to-event sequences continually while retaining knowledge from past time-to-event sequences. Inspired by (Von Oswald et al. 2019), we use descriptor conditioned hypernetworks for continually learning from event sequences. Ideally, we want our model to remember the parameters of the neural Hawkes process for each sequence. A naive approach to achieve this is through storing and replaying over previous data, which is obviously memory expensive and unrealistic. However, HyperHawkes, being conditioned on the sequence descriptor, can be modified to handle this problem. The direct use of the HyperHawkes training through (10) would result in hypernetworks forgetting the generation of the NHP parameters corresponding to past event sequences. We overcome this by incorporating a regularization on the hypernetwork parameters such that it penalizes any change to the NHP parameters produced from old sequences.

Given a sequence description $\mathbf{d}^{s}$ for the descriptor $\mathcal{T}^{s}$, our descriptor conditioned hypernetwork $f_{r}(\cdot)$ can generate parameters $W_{r}^{s}$ and $f_{t}(\cdot)$ can generate parameters $W_{t}^{s}$. To perform continual learning, we use regularization to penalize changes in $\{W_{r}^{c},W_{t}^{c}\}$ generated for past sequences in order to retain information from those sequences and to learn continually. The regularization is applied to the hypernetwork parameters while learning a new event sequence, and this prevents adaptation of the hypernetworks parameters completely to the new event sequence. For a new event sequence $\mathcal{T}^{s}$ and its corresponding descriptor $\mathbf{d}^{s}$, the hypernetwork parameters are learnt by minimizing the following continual learning loss over events in the sequence:

$$\begin{split}&\sum_{j=1}^{n^{s}}-\log p(t^{s}_{j}|\mathcal{H}^{s}_{j};(f_{r}(\mathbf{d}^{s};\theta_{fr}),f_{t}(\mathbf{d}^{s};\theta_{ft})))\\ &+\frac{\beta}{s-1}\sum_{c=1}^{s-1}\biggl{(}\parallel f_{r}(\mathbf{d}^{c};\theta_{fr})-f_{r}(\mathbf{d}^{c};\bar{\theta}_{fr})\!\!\parallel^{2}\\ &+\parallel f_{t}(\mathbf{d}^{c};\theta_{ft})-f_{t}(\mathbf{d}^{c};\bar{\theta}_{ft})\parallel^{2}\biggr{)}\end{split}$$

(11)

where $\{\bar{\theta}_{fr},\bar{\theta}_{ft}\}$ represents the stored hypernetwork parameters after learning until sequence $s-1$ and $\{\theta_{fr},\theta_{ft}\}$ represent the hypernetwork parameters learnt considering the event sequence $s$ and regularization to avoid forgetting. The regularization term ensures that the newly learnt hyper-network parameters will be able to produce the required main network parameters from the past event sequences given the sequence descriptor without forgetting and the regularization constant $\beta$ captures the importance associated with it. So, in this way, we try to retain the information from previous sequences at a meta-level. By including a simple regularization term within the framework of HyperHawkes, our model is capable of learning sequences continually without forgetting knowledge learnt from previous sequences. We are able to achieve this because of the use of sequence-conditioned hypernetwork on the top of neural Hawkes process, emphasizing its usefulness for continual learning over event sequences in addition to zero shot learning.

Due to paucity of standard datasets for event modeling tasks which contain meta descriptions as well, we use the following two datasets: 1)Yelp:¹¹1https://www.kaggle.com/datasets/yelp-dataset/: This is a dataset comprising of business information and their check-in information. Each business is associated with 82 attributes like Wheelchair Accessible, Accepts Insurance, By Appointment Only, Business Category, Business Timings etc. Also, they are associated with latitude-longitude pairs. Moreover, these businesses are associated with a fine-grained category. For higher granularity, we convert them into 22 broad categories using the hierarchy mentioned in their website ²²2https://www.yelp.com/developers/documentation/v3/all_category_list. Using these attributes, we create a vector of length 1229 representing a business. This vector acts as a descriptor of the business. We select businesses with more than 5000 check-ins. For continual learning, we have considered another sample of dataset consisting of business with more than 10k checkins, hence considering 26 sequences of business. This is done to reduce the number of sequences for better visualization of performance of each sequence. 2) Meme:³³3https://snap.stanford.edu/data/memetracker9.html: This dataset(Leskovec, Backstrom, and Kleinberg 2009) tracks the popular phrases and quotes which appear appear most frequently over time in news media and blogs. Each meme is associated with the content and timestamps when they were quoted in the media. We select top 200 english phrases and doc2vec representation of the meme content is considered as the descriptor of length 100. We have considered memes from April, 2009 with an average number of events as 970.

To the best of our knowledge, the proposed problem statement is the first work along this direction. Therefore, we propose our own baselines as - 1) FNHP: This includes fully neural Hawkes process (Omi, Ueda, and Aihara 2019). This approach doesn’t incorporate the sequence descriptor. 2) FNHP-Descriptor: In this variant, we use concatenated descriptor and time as input to RNN and FNN. For Continual learning setup, we compare against HyperHawkes without any regularization as baseline.

For zero-shot setup, we perform a 60-20-20 split where 60% of sequences are considered as seen sequence and 20% for validation unseen sequence and rest 20% as test unseen sequence. We have used a single layer with 32 units for hypernetwork where we use softplus activation function for modeling cumulative hazard function. For the neural Hawkes process, we consider a recurrent neural network with one layer and 16 units and 2-layer feed-forward neural network with 16 units in each layer. We use Adam optimizer with learning rate, $\beta_{1}$ and $\beta_{2}$ as 0.0001, 0.90 and 0.99 for the reported results. We perform single step lookahead prediction where we use actual time of occurrence of events as past events for the historical information to predict future events. We have performed all the experiments on Intel(R) Xeon(R) Silver 4208 CPU @ 2.10GHz, GeForce RTX 2080 Ti GPU and 128 GB RAM. We have implemented our code in Tensorflow 2.2.0 (Abadi et al. 2015). For continual learning setup, we test on regularization parameters in range [0.001-0.9]. For Yelp, reported results are for $\beta$ is set as 0.5 for HyperHawkes-FNN, 0.6 for HyperHawkes-FNN-RNN and for Meme $\beta$ is 0.6 for HyperHawkes-FNN and 0.1 for HyperHawkes-FNN-RNN. More implementation details and hyperparameter settings are presented in the supplementary.

We consider these experimental setups to evaluate the performance of our model - 1) Zero-Shot: Training is done on seen sequences and testing is done on unseen sequences. 2) Generalized Zero-Shot: In this, testing is done by randomly sampling 20% of events from seen sequences and unseen sequences. 3) Standard Event Modeling: Training is done on the first 70% of the events from all sequences. Testing is done on the last 20% events for unseen sequences. Mean negative log-likelihood (MNLL) and mean absolute error (MAE) are considered as evaluation metrics for both zero-shot and continual learning setup. Lower MNLL and MAE indicates better performance.

Results for zero-shot setup are presented in Table 1. A better model is expected to have lower MNLL and MAE. We can observe that the proposed method HyperHawkes performs better than the baselines in terms of both evaluation metrics MNLL and MAE. We can observe that zero-shot setup, which consists of predictive performance for unseen sequences, exhibits significantly lower MNLL and MAE for HyperHawkes-FNN as compared to FNHP and FNHP-Descriptor. Also, HyperHawkes-FNN-RNN yields lower MAE for Meme dataset, however, MNLL for both the variants of the proposed methods is close. Comparing the results of generalized zero-shot setup where we consider instances from seen sequences as well as unseen sequences, we can observe that HyperHawkes-FNN-RNN performs better for both the datasets. The final section of the table discusses results for standard event modeling where we test on the last 20% of the events of unseen sequences. HyperHawkes-FNN-RNN performs well here as well except for MAE for Yelp dataset. We can also observe that HyperHawkes-FNN-RNN performs slightly better for generalized zero-shot and standard event modeling setup. Moreover, it is interesting to note that inclusion of sequence descriptors within FNHP in FNHP-Descriptor has not helped much in prediction of unseen sequences. In fact, in various cases, it has performed worse than FNHP itself. This confirms the necessity of an event sequence specific framework which can predict better for unseen sequences as well. Therefore, our results indicate that the proposed variants of HyperHawkes perform consistently and significantly better than the baselines for both the datasets.

Table 2 presents the averaged results over all tasks by enabling HyperHawkes for continual learning. We can observe that averaged performance for the proposed model is better than the case when no regularization is incorporated. Also, we can observe that both the proposed variants perform better than the model without using regularization (corresponds to the case when $\beta$ from Equation 11 is set to 0). Hence the use of regularization within the framework of HyperHawkes supports that proposed method can avoid catastrophic forgetting. Fig 2 displays sequence-wise performance for both the datasets for the proposed variants HyperHawkes-FNN and HyperHawkes-FNN-RNN.2(a)) displays average MNLL over previous sequences for both the models for Yelp. This shows that while training the model without regularization over new sequences, the network is unable to retain information learnt from previous sequences, hence MNLL increases as we train new sequences. However, with the use of regularization, we can avoid catastrophic forgetting, hence having lower MNLL for successive tasks. Similar behavior is observed by 2(b)) as well which displays average MAE over previous sequences for both the variants, with and without CL for Meme. So, this corroborates that use of regularization with HyperHawkes can help in backward transfer. 2(c)) shows MNLL for each sequence using the model HyperHawkes-FNN-RNN with regularization. MNLL for the model with regularization is having lower MNLL as compared to the model without any regularization. This essentially reflects that the proposed model is able to forward transfer the knowledge learnt from previous sequences as well. So, the model is able to perform forward and backward transfer, which are important continual learning desiderata. 2(d)) displays the effect of various regularization parameters for Yelp and Meme dataset for HyperHawkes-FNN. A possible explanation could be for Meme for $\beta$, the model is not able to learn from previous sequences and for large $\beta$ might not be able to learn from new sequences. To conclude, presented results suggest that proposed framework can aid in avoiding catastrophic forgetting while learning continually.