GRAINRec: Graph and Attention Integrated Approach for Real-Time Session-Based Item Recommendations

Session-based item recommendation models have garnered significant attention in recent years due to their ability to provide personalized recommendations based on item relationships. They model dynamic user preferences and provide recommendations that are sensitive to the evolution of session context. There have been many papers suggesting different session-based recommendation models applied to different use cases. Wang et. al. [1] performed a comprehensive survey in this regard. In the domain of news recommendations Song et. al. [2] proposed a session-based model to optimize for recommendations that are more focused on the freshness of the content while also modeling long term preferences with regards to topics. Shen et. al. [3] proposed a recency-regularized neural attentive framework that uses users’ sequential commenting behavior to recommend news. For music recommendation Wang et. al. [4] proposed a context-based model that mines knowledge from public opinion such as comments, music review, or social tags to recommend music. Bao et. al. [5] presented a session-based location recommendation system that leverages sparse geo-social data. It combines user preferences and geographic information to suggest locations. Wang et. al. [6] presented a dynamic time-aware attention model for recommending points of interest (POIs) by analyzing users’ check-in sessions, capturing temporal patterns and user preferences. For video recommendations Beutel et. al. [7] proposed a recurrent neural network-based approach on sequences of watches on YouTube to integrate the session context.

This subsection focuses only on papers proposed for item recommendations. Yap et. al. [8] proposed a sequential pattern mining based next-items recommendation algorithm to predict users’ next accesses by identifying frequent patterns. The paper introduces a personalized framework using a competence score measure to improve the accuracy of recommendations for individual users. Hu et. al. [9] proposed a KNN-based model that captures personalized item frequency information. Le et. al. [10] proposed a Markov chain-based model that introduces generative modeling that incorporates dynamic user-biased emission and context-biased transition, improving next-item prediction accuracy. Hidasi et. al.[11] introduced GRU4Rec that used Gated Recurrent Units (GRUs) for session-based recommendations. GRU4Rec’s ability to model complex user behavior sequences laid the foundation for subsequent research in session-based recommendations using deep learning. Li et. al.[12] proposed NARM, a neural attentive model that incorporated an attention mechanism to focus on relevant parts of the session history, enhancing the recommendation accuracy. Kang and McAuley [13] introduced SASRec, a self-attentive sequential recommendation model that employs the transformer architecture. By leveraging self-attention, SASRec captures long-range dependencies within a session more effectively than RNN-based approaches. Wang et. al. [14] proposed a novel approach using hypergraphs for session-based recommendations. The Sequential Hypergraph Attention Network (SHAN) models user sessions as hypergraphs, where hyperedges connect multiple items. This structure allows the model to capture higher-order item co-occurrences and complex dependencies within sessions. SHAN’s attention mechanism further enhances its ability to focus on the most relevant parts of the session, leading to improved recommendation performance. Feng et. al.[15] introduced Caser, a Convolutional Sequence Embedding Recommendation model, which applies convolutional neural networks (CNNs) to model user behavior sequences. Caser captures both point-wise and union-level sequential patterns by treating the embedding matrix as an "image" and applying horizontal and vertical convolutional filters. The model showed significant improvement over traditional sequential recommendation methods. Zhang et. al. [16] proposed HINRec, a session-based recommendation model using Heterogeneous Information Network (HIN) embeddings. This model captures various types of interactions and relationships among items by representing the session data as a heterogeneous graph. The embeddings generated from this network effectively incorporate both structural and semantic information, resulting in improved accuracy. Liu et. al.[17] proposed STAMP, a session-based recommendation model that integrates short-term attention and long-term memory. It utilizes an attention mechanism to capture users’ short-term interests within a session, while a memory network stores long-term preferences. This dual approach allows STAMP to dynamically balance between recent interactions and long-standing preferences, achieving superior performance.

This subsection highlights session based recommendation research on graph data. Gao et. al.[18] conducted a comprehensive review of the literature on graph neural network-based recommender systems. Wu et. al. [19] proposed SR-GNN which models sessions as graphs where items are nodes and interactions are edges. The model utilizes Graph Neural Networks (GNNs) to capture complex item transitions within sessions, significantly improving recommendation accuracy. Wang et. al.[20] proposed a gated GNN model for session-based recommendations. The gated mechanism helps the model selectively focus on important parts of the session graph, enhancing the overall recommendation performance by capturing more nuanced user behaviors. Wang et. al.[21] introduced a knowledge-aware GNN that incorporates external knowledge graphs into the recommendation process. By integrating knowledge with session data and applying label smoothness regularization, the model achieves improved recommendation accuracy. Zhou et. al.[22] presented Temporal Graph Neural Networks (TGNN) for session-based recommendation. TGNN captures temporal dynamics within sessions by modeling time-aware item transitions. This approach enhances the model’s ability to understand user behavior changes over time, leading to more accurate recommendations. Chen et. al.[23] proposed LESSR, a GNN and attention based approach this work borrows the model from. However their work does not propose a real-time inferencing solution.

In the data processing phase we generate a temporally-ordered sequence of items added to a cart. Each sequence can be represented as an ordered graph. For example, if a guest adds three items $a$, $b$ and $c$ in that order then it can be represented as a directed graph ($a\rightarrow b\rightarrow c$), where the edge $a\rightarrow b$ means item $b$ was added to cart after item $a$. The graph does not contain self-loops ($a\rightarrow a$), but the same item $a$ can be added more than once in the sequence. During model training the sequences are batched, which means all sequences in the batch are merged to generate a single graph representation. An illustration is shown in Figure 1, where two ordered sequences ($a\rightarrow b\rightarrow c$ and $c\rightarrow b\rightarrow d$) are merged to generate a single graph representation.

The proposed session-based recommendation architecture is a deep learning model that has two types of neural network layers: A Gated Recurrent Unit (GRU)-based graph neural network that captures short-range relationships in the sequences and an attention-based model that captures long-range relationships in the sequences. For simplicity we show in Figure 1 that item $c$ predominantly learns from item $b$ and items $a,d$ using GRU and attention mechanisms respectively. The model alternates between layer types; layer=1 is GNN, layer=2 is GNN-Attention, layer=3 is GNN-Attention-GNN, and so on. This helps propagate the features captured by each kind of layer more effectively. We also apply a readout layer that calculates a session embedding by concatenating the attention-based aggregate of all ordered items in the session (called the global embedding) and the embedding of the last session item (called the local embedding).

Motivation: Our original latency test was run on a microservice fetching real-time recommendations in the dev environment. This test used a model trained only on guest actions on frequent items (separate from the one we used for offline evaluation in this paper). Inference latency was found to be over 400 ms, which was unacceptable. This served as the motivation for implementing the nearest neighbor matrix solution for real-time inference.

Inference in our context means that as the guest session changes/evolves dynamically, the model calculates a real-time session embedding and uses it to generate a handful of relevant recommendations from a catalogue of millions of items. In order to meet the latency threshold, we used a heuristic approach of generating a nearest neighbor matrix as part of model training, which has a size of $no\textunderscore items$ x $no\textunderscore neighbors$. We generate this matrix during model training by finding the top-k nearest item embeddings to every item. Suppose we choose $no\textunderscore neighbors=100$, it means that while inferring recommendations for the session $d\rightarrow e\rightarrow f$, our lookup space for recommendations would not be the entire item catalog but $\sim$ 300 items (i.e combining 100 pre-calculated neighbors for each of the items $d,e,f$). We use this heuristic approach to address the latency and accuracy trade-off. Figure 2 shows how only the the nearest neighbors of session items are considered to generate the final recommendations list.

Approximate Nearest Neighbor (ANN) is a popular method for nearest neighbor generation; however, we opted for a Nearest Neighbor Matrix because ANN-based recommendations across the entire catalog often lacked relevance. This approach allows us to filter the item search space for each item more effectively. Additionally, ANN indexing introduces extra latency, which hindered our microservice from meeting SLA requirements during model inference.

We compared our proposed model against the following baseline models:
1. Item-KNN[25]: This model recommends similar items to the most recently added item in the session by calculating their cosine similarity scores. Embeddings for items are generated using word2vec.
2. FPMC[26]: This is a Markov chain-based model for next basket recommendation. For our case we customize it to predict the next item.
3. GRU4Rec[11]: This is a recurrent neural network model for session-based next item recommendations.
4. SR-GNN[19]: SR-GNN is a session-based recommendation model using graph neural networks to model session sequences as directed graph-structured data.
5. SASRec[13]: SASRec is a self-attention based sequential model for item recommendations.
6. LESSR[23]: LESSR model combines graph neural networks with attention.
7. GRAINRec: A graph- and attention-based model described above.

All models were trained using item sequences collected by Target, a large retail company. To mitigate data sparsity, we included only items that appeared in the sequence dataset at least 10 times during this period. We capped each sequence length at 20 items and filtered out consecutive duplicate items. The entire catalog was divided into two types: frequency (items usually purchased multiple times, such as groceries, cleaning products, etc) and discretionary (items typically purchased once, such as electronics and furniture). Table I summarizes item and sequence count statistics for both item types used for offline evaluation. While frequency items make up a little over 10% of the catalog, they account for just over 40% of the guest action sequences. Also for every sequence of length greater than 2 we also include its prefixes²²2prefixes are contiguous subsequences of a given sequence that start at the first element and extend to every other element in the sequence. Ex: For sequence $a\rightarrow b\rightarrow c\rightarrow d$, prefixes are $a\rightarrow b$, $a\rightarrow b\rightarrow c$ & $a\rightarrow b\rightarrow c\rightarrow d$. for model training.
Training parameters: Hyperparameter tuning was performed using the scikit-optimize python library. The model parameters were set as follows: an embedding dimension of 256, two layers (GNN-Attention), dropout rate of 0.146, learning rate of 0.00045, batch size of 1024, and decay rate of 0.0001. Weight matrix dimensions for outputs are as follows: Model Input : $no\_items*emb\_dim$, GNN layer output: $emb\_dim*emb\_dim$, Attention layer output: $1*emb\_dim$, ReadOut layer: $emb\_dim*(3xemb\_dim)$, Fully connected layer: $emb\_dim*batch\_size$. We selected a 2-layer model to leverage the strengths of both the GNN and attention mechanisms while maintaining a balance between model complexity and efficiency. Adding more layers would increase the model’s complexity, potentially extending training and inference times, which could hinder the model’s ability to meet the required service-level agreements (SLAs) for real-time inference. By using a 2-layer architecture, we ensure that the workflow runs more frequently. The model was trained using NVIDIA A100 3G 40GB (3 instances, 40 GB of GPU memory) multi-instance GPUs.
Inferencing parameters: For model inference we set an upper limit on the session length to 3.

For both training and inference, a filter was set on the session to ensure all items in the session belong to the same category as the most recently added item in the session. For example, if the session contains the items: $sugar\rightarrow iphone\rightarrow carrots\rightarrow milk$, the filtered session will be $sugar\rightarrow carrots\rightarrow milk$. $iphone$ belongs to a different category than milk and is excluded from the session, while $sugar$ and $carrot$ belongs to same category as $milk$ and are retained.

GRAINRec, like other deployed recommendation models in Target, is a guest-facing application built on an application layer called Kubernetes (k8s). When setting up the inference infrastructure, we faced a critical decision: whether to run inference on CPU or GPU. For our specific use case, CPU was chosen due to its simplicity and favorable cost economics. While GPUs are generally faster for model training, the simplicity of using CPUs for real-time request-response operations without the overhead of batching made it the preferred choice. Additionally, after optimizing our systems, we found that the cost-per-inference on CPU was significantly more economical.

It took less than 50ms (95th percentile latency) to generate GRAINRec recommendations for guests. We were able to achieve this low latency by implementing custom load balancing using gRPC as our RPC framework. We further reduced latency by building an in-memory customized local cache. Each instance of the GRAINRec PyTorch model is hosted within a Kubernetes microservice allocated with 2 CPUs and 10 GB of memory. This microservice has direct access to the real-time feature store, which contains comprehensive data on the purchase/browse/add-to-cart history of each guest. The throughput of each k8s pod is around 60 requests per second.

In our k8s environment, we faced challenges scaling GRAINRec PyTorch model inference on CPUs due to resource contention. PyTorch’s default behavior of using multiple CPU cores per inference caused performance issues when multiple pods competed for the same cores. To resolve this, we limited each pod to a single thread for PyTorch inference by setting $torch.set\_num\_threads(1)$, ensuring efficient concurrent inference across multiple pods.

The proposed model was A/B tested against a link prediction model already running in production. Increase in metrics is compared against the link prediction model. The evaluation was done against two metrics:
1. Click Through Rate (CTR): This metric quantifies how often guests click on recommended items compared to how often those items are displayed to guests.
2. Attributable Demand (AD): This metric quantifies the $ sales generated directly from the guests purchasing the recommended items.

Table II presents the impact of hyperparameter tuning on key performance metrics for an item recommendation system. Before hyperparameter tuning, the CTR increased by 6.4% and AD increased by 5.8%. When A/B testing was done after tuning, CTR rose to 10%, and AD increased to 9%. These results demonstrate that hyperparameter tuning significantly enhances the effectiveness of the recommendation system, leading to greater user engagement.

We evaluate our model against baselines listed in Section VI. A. using 3 popular metrics:
1. Hit rate at 10 (hit@10): It indicates the proportion of times a relevant item is found within the top 10 recommendations provided by the system.
2. Mean Reciprocal Rank at 10 (mrr@10): It measures the average reciprocal rank (average of inverse of the ranks) of the first relevant item across multiple test cases.
3. Normalized Discounted Cumulative Gain at 10 (ndcg@10): It combines the concepts of cumulative gain, discounting the gain of relevant items based on their position in the recommendation list, and normalizing the result to ensure comparability across different recommendation lists.

The performance analysis of various recommendation models across different item categories is encapsulated in Table III.

For frequency item categories, Item-KNN is the least performing model with scores of 0.134 (hit@10), 0.127 (mrr@10), and 0.131 (ndcg@10). As a word2vec based approach, its results are low due to its inability to capture any patterns in the item sequences. FPMC showed slight improvement over Item-KNN with scores of 0.151 (hit@10), 0.143 (mrr@10), and 0.149 (ndcg@10) as it captures position information by modelling sequential data through Markov chains. GRU4Rec and SR-GNN significantly outperformed the traditional models as they employ deep learning techniques. GRU4Rec, with 0.219 (hit@10), 0.103 (mrr@10), and 0.121 (ndcg@10) leverages gated recurrent units to capture sequential dependencies. Its improved performance can be attributed to its gated neural network model that stores information from previous item. A small drawback of the model is that information passing happens only in one dimension, which is overcome by graphical representation of sequences. SR-GNN with 0.222 (hit@10), 0.101 (mrr@10), and 0.123 (ndcg@10) further enhances this by incorporating graph neural networks to model complex item relationships better. SASRec exhibited stronger performance with 0.248 (hit@10), 0.119 (mrr@10), and 0.148 (ndcg@10). Its self-attentive mechanism effectively models sequence patterns that captures long range dependencies, providing a significant boost in recommendation accuracy. LESSR model shows performance of 0.255 (hit@10), 0.124 (mrr@10) and 0.151 (ndcg@10) which is better than the previous baselines as it incorporates both GNN and attention. Our proposed GRAINRec emerged as the best performer in this category with 0.258 (hit@10), 0.124 (mrr@10), and 0.153 (ndcg@10). In addition to the multi-layers attention and GNN based architecture, what contributes to the performance improvement is removal of self-loops, cross category dependency and nearest neighbor matrix.

In discretionary item categories, the trends observed is similar as in frequency item categories. Item-KNN and FPMC maintain their lower performance, with FPMC slightly better at 0.123 (hit@10), 0.116 (mrr@10), and 0.119 (ndcg@10) compared to Item-KNN’s 0.111 (hit@10), 0.105 (mrr@10), and 0.108 (ndcg@10). GRU4Rec (0.207 hit@10, 0.101 mrr@10, 0.123 ndcg@10) and SR-GNN (0.209 hit@10, 0.105 mrr@10, 0.122 ndcg@10) continue to show their strength. SASRec again show superior performance, achieving 0.228 (hit@10), 0.109 (mrr@10), and 0.139 (ndcg@10). LESSR shows even better performance with 0.234 (hit@10), 0.112 (mrr@10), and 0.141 (ndcg@10). GRAINRec leads with 0.238 (hit@10), 0.114 (mrr@10), and 0.144 (ndcg@10), indicating its robustness.

When considering the entire catalog, Item-KNN (0.122 hit@10, 0.118 mrr@10, 0.12 ndcg@10) and FPMC (0.145 hit@10, 0.133 mrr@10, 0.139 ndcg@10) again exhibit lower scores compared to more sophisticated models. GRU4Rec and SR-GNN continue their competitive performance, with GRU4Rec at 0.221 (hit@10), 0.123 (mrr@10), and 0.129 (ndcg@10) showing strong consistency across metrics. SR-GNN, however, shows a marked drop in mrr@10 (0.109), possibly indicating some limitations in its architecture for broader item catalogs despite having good hit@10 (0.231) and ndcg@10 (0.133) scores. SASRec remains effective with scores of 0.235 (hit@10), 0.134 (mrr@10), and 0.144 (ndcg@10). LESSR outperforms other baselines with 0.239 (hit@10), 0.136 (mrr@10), and 0.147 (ndcg@10), but GRAINRec maintains its leading position, achieving 0.245 (hit@10), 0.139 (mrr@10), and 0.149 (ndcg@10).

Our proposed model shows an average improvement of over 1.5% over LESSR for all metrics. Comparing the performances across frequency, discretionary, and entire catalog categories we observe that the model trained on frequency categories performs the best. This can be attributed to the lower data sparsity in the frequency dataset, which allows the model to learn item relationships more effectively. On the other hand, items in the discretionary category are more numerous but occur less frequently, leading to poorer learning of their representations.

Table IV compares the performance of GRAINRec under the single-class and multi-class classification settings. Single-class classification significantly outperforms multi-class classification across all evaluation metrics. Specifically, hit@10, mrr@10, and ndcg@10 values for multi-class classification show a drop of 36.33%, 34.53% and 32.21% respectively, compared to single-class classification. This could be attributed to the fact that as the number of classes grows (the catalog has roughly 600k items), the number of decision boundaries that a learning algorithm must address also increases. Experimental evidence[28] suggests that with more decision boundaries, the complexity of the problem rises which significantly reduces model performance.

Figure 3 summarizes the performance of the proposed model across different item categories using various configurations of graph neural network (G) and attention (A) layers. The configurations tested include a 4-layer GNN model (GGGG), a 4-layer attention model (AAAA), a mixed 2-layer GNN followed by 2-layer attention model (GGAA), the reverse order (AAGG), and an alternating layer model (GAGA).

The results show that GAGA, the proposed alternating layer model, outperforms all other configurations, demonstrating the effectiveness of alternating GNN and attention layers. This approach effectively leverages the strengths of both types of layers, with the GNN layers capturing the context of closer items and the attention layers capturing dependencies of more distant items. This layered alternation helps mitigate the over-squashing[27] issue seen in continuous GNN layers, where information propagation across long distances in the graph becomes inefficient. The AGAG configuration showed similar performance to GAGA, reinforcing the advantage of alternating the two types of layers for improving recommendation accuracy across all item categories.

We investigate the relationship between model performance and the size of the nearest neighbor matrix in Figure 4, evaluating the model using ndcg@10 across six different neighborhood sizes: 25, 50, 75, 100, 125, and 150. For the full catalog, the ndcg@10 value increases from 0.1316 to 0.1541. Discretionary items show a similar improvement, rising from 0.1251 to 0.1515, while frequency items exhibit the most significant growth, from 0.1370 to 0.1560. These results suggest that larger neighborhood sizes enhance recommendation accuracy across all categories, with frequency items achieving the highest ndcg@10 values. While the general trend shows increasing ndcg@10 values, with larger neighborhood sizes we observe diminishing marginal returns beyond size of 100. Specifically, the rate of increase in ndcg@10 values slows down past this point. Consequently, we select a neighborhood size of 100 for our model, as larger sizes lead to inferencing times that fail to meet platform’s SLA.

The experiment results are presented in Figure 5, which displays the performance of model across varying embedding dimensions: 32, 64, 128, 256, 512, and 1024. The models compared are a 2-layer Graph Neural Network with Attention (GNN-Attention), a 1-layer Attention model, and a 1-layer Graph Neural Network. Each subplot in the figure corresponds to a different item category: frequency items, discretionary items, and full catalog. All three subplots have a consistent y-axis range, making it easier to compare the results across different item categories.

In the first subplot, representing frequency items, the 2-layer GNN-Attention model consistently outperforms the other models across all embedding dimensions. Specifically, the ndcg@10 values for the GNN-Attention model range from 0.1421 to 0.1555. The 1-layer Attention model shows moderate performance, with ndcg@10 values ranging from 0.1357 to 0.1509. The 1-layer GNN model demonstrates the lowest performance, with ndcg@10 values between 0.1305 and 0.1449. This indicates that incorporating both GNN and Attention mechanisms yields the best results for frequency item recommendations.

The second subplot, which depicts discretionary items, shows a similar trend. The 2-layer GNN-Attention model again leads in performance, with ndcg@10 values between 0.1317 and 0.1450. The 1-layer Attention model follows, with values ranging from 0.1254 to 0.1404. The 1-layer GNN model has the lowest ndcg@10 values, ranging from 0.1202 to 0.1346.

In the third subplot, which represents the full catalog, the performance of the models is evaluated on all items. The 2-layer GNN-Attention model maintains its superior performance, with ndcg@10 values ranging from 0.1357 to 0.1510. The 1-layer Attention model achieves ndcg@10 values between 0.13024 and 0.14549. The 1-layer GNN model shows the lowest performance with values ranging from 0.1239 to 0.1392. The consistent outperformance of the 2-layer GNN-Attention model across different item categories and embedding dimensions highlights its robustness and suitability for diverse recommendation scenarios.

Overall, the experiment results clearly demonstrate the effectiveness of the proposed 2-layer GNN-Attention model in achieving higher ndcg@10 values compared to the 1-layer Attention and 1-layer GNN models. The enhanced performance is observed across all embedding dimensions and item categories, indicating the model’s capability to capture complex interactions and improve recommendation accuracy in real-time session-based systems. While GNNs are adept at capturing short-range dependencies, attention mechanisms additionally captures long-range dependencies, thereby providing a more comprehensive representation of the data. Furthermore, we observed that model performance improves with an increase in embedding dimensions, as larger dimensions enable the model to capture a greater amount of information. However, this improvement follows a trend of diminishing returns, where the incremental gains in performance decrease as the embedding dimension continues to increase.