MobileRec: A Large-Scale Dataset for Mobile Apps Recommendation

Mobile apps have seen exponential growth in the last decade and over $5$ billion users (number-of-smartphone-users, ) utilize them for a variety of reasons, including social media, entertainment, news, productivity, and ride-sharing, among others. As a result of this boom, Google Play (googleplay, ) and Apple App store (appstore, ) host more than $3.5$ and $2.2$ million apps, respectively (number-of-apps-on-stores, ). The increasingly crowded app marketplaces pose a significant challenge for users to discover apps that align with their preferences effectively. Personalized app recommendations can relieve users’ cognitive overload and improve the app installation experience. As illustrated in Figure 1, an app recommendation system has the capability to suggest new applications to users based on their previous app installations and interactions. Although Google Play and the App Store employ app recommendation techniques for suggesting apps to their users potentially leveraging user data collected internally, the research in app recommendation is almost nonexistent.

Recommendation systems have demonstrated remarkable effectiveness in a wide range of domains, such as news (wu2020mind, ), movies (diao2014jointly, ), products (yan22personalized, ; tanjim20attentive, ), fashion (iwata2011fashion, ), fitness (ni19modeling, ), toys, beauty, CDs (mcauley2015image, ; he2016ups, ; ni2019justifying, ), to name a few (rappaz21recommendation, ). The availability of datasets for specific domains that contain user-item interactions (amazon-dataset, ; koren2008factorization, ; movielens, ) have played a critical role in the development and improvement of recommendation systems. In earlier research items and users were typically represented using their unique identifiers, and their respective interactions, such as rating scores, were used to learn useful relationships that could aid in recommendation tasks. These relationships were typically learned using collaborative filtering techniques or similar methods (koren2008factorization, ). Recently, incorporating sequential user interactions into the development of recommendation systems has led to significant improvements in their performance (kang2018self, ). By taking into account the sequential patterns of user interactions with items (ni2019justifying, ) (e.g., the order in which they viewed or purchased items), these methods can better capture the user’s preferences and provide more accurate recommendations (kang2018self, ). Furthermore, the development of novel techniques such as recurrent neural networks (RNNs) (elman1990finding, ; hochreiter1997long, ) and attention-based models (vaswani2017attention, ) that effectively handle sequential data has shown exceptional performance in recommendation tasks (yan22personalized, ; tanjim20attentive, ).

There are several notable datasets that focus on mobile applications, such as the Top 20 apps dataset (top20-dataset, ), RRGen (rrgen, ), AARSyth (aarsynth, ), Srisopha et al.(srisopha-how, ), and PPrior(pprior, ), among others. However, Top 20 Apps and RRGen only contain a limited number of reviews, typically in the hundreds of thousands. Moreover, they only cover fewer than 100 apps from less than 20 categories. Despite providing larger datasets with millions of reviews from thousands of apps, datasets such as AARSynth, Srisopha et al., and PPrior are not amenable to building effective app recommendation systems, since they lack unique user identifiers. Furthermore, the dataset from Srisopha et al. (srisopha-how, ) is not publicly available and PPrior only provides negative user reviews. A comparison of mobile app datasets is presented in Table 1. This work attempts to fill this research gap by providing a large-scale, rich, and diverse benchmark dataset, we call $\mathsf{MobileRec}$, to facilitate researchers in developing app recommendation systems. To the best of our knowledge, this is the only mobile app dataset that offers sequential and multiple interactions per user with apps. We also present a comparison of our proposed dataset with the latest versions of the well-established recommendation datasets in various domains in Table 2.

$\mathsf{MobileRec}$ is a large-scale dataset constructed from users’ activity on the Google Play Store, containing $19.3$ million user interactions (i.e., user reviews along with rating) across $10$K unique apps in $48$ categories. It records the sequential activity of $0.7$ million unique users, each of whom interacted with at least five apps. Every user interaction within our dataset consists of essential information, such as the user’s rating, textual review, and the review date, pertaining to installed apps. Moreover, every individual app in $\mathsf{MobileRec}$ is equipped with abundant metadata, such as the app’s title, category, long-form textual description, overall average rating, developer information, content rating, and additional pertinent details. Table 3 presents important features of the dataset. To stimulate research in app recommendation systems, $\mathsf{MobileRec}$ is on par with well-established and widely recognized datasets from various other domains. On top of app recommendation, our comprehensive dataset can offer significant insights and serve as a valuable resource for diverse research and data analysis pursuits, such as sentiment analysis, app development and optimization, market research, and fraud detection, among others.

We also show that $\mathsf{MobileRec}$ can serve as a valuable experimental testbed for app recommendation research through a comparative evaluation of several cutting-edge recommendation techniques. Specifically, we employed several existing general and sequential recommendation systems, including Pop (i.e., the popularity of items), SASRec (kang2018self, ), ELECRec (chen2022elecrec, ), BERT4Rec (sun2019bert4rec, ), HGN (ma2019hierarchical, ), SINE (tan2021sparse, ), LightSANs (fan2021lighter, ), GRU4Rec (tan2016improved, ), and GCSAN (xu2019graph, ) using $\mathsf{MobileRec}$. Our analysis of the results obtained by these methods on the $\mathsf{MobileRec}$ has revealed some intriguing trends. Notably, we observed that ELECRec (chen2022elecrec, ) had performed significantly better than GRU4Rec (tan2016improved, ) on the $\mathtt{Amazon}$-$\mathtt{Beauty}$ dataset, achieving over $300\%$ improvement in the $\mathtt{Hit@5}$ metric. However, when tested on $\mathsf{MobileRec}$, ELECRec exhibited a considerable drop in performance compared to GRU4Rec on the same metric. Similarly, we observe that LightSANs (fan2021lighter, ) had achieved a $3.91\%$ and $2.65\%$ increase in $\mathtt{Hit@10}$ and $\mathtt{NDCG@10}$ metrics, respectively, compared to SASRec (kang2018self, ). However, on $\mathsf{MobileRec}$, SASRec (kang2018self, ) outperforms LightSANs (fan2021lighter, ). This signifies the inherent dynamism and fleeting nature of user-item interactions and user interests in the $\mathsf{MobileRec}$ dataset, which gives rise to complex user interaction history. This observation has important implications for state-of-the-art recommendation models. The numerical results presented in this work can also serve as a baseline for fellow researchers to evaluate their own results and objectively assess the quality and effectiveness of different approaches. Moving forward, numerous natural language processing techniques, such as advanced text representation methods and pre-trained language models (devlin2018bert, ; radford2019language, ), have the potential to unlock new research avenues for app recommendations using $\mathsf{MobileRec}$. These techniques hold great promise for enhancing the quality and effectiveness of app recommendation systems, and as such, are of particular interest to the research community.

Specifically, this work makes the following contributions:

• •

  We present $\mathsf{MobileRec}$, the most extensive collection of sequential user-app interactions to date, comprising over $19$ million interactions across a diverse range of over $10$ thousand distinct apps from Google Play, spanning all categories. Notably, this is the only mobile app dataset that features multiple interactions per user.

• •

  Our experimental study investigates the dynamics of user-app interactions and demonstrates the practical utility of $\mathsf{MobileRec}$ by employing several state-of-the-art recommendation systems and establishing baseline results for the research community.

We consider a wide range of recommender systems for benchmarking and establish baseline results for the $\mathsf{MobileRec}$ dataset.

Table 4 presents the performance analysis of various baselines on $\mathsf{MobileRec}$. Next, we discuss these results and provide further details. As discussed earlier, Pop is a popularity-based model which relies on the popularity of items. Pop is the most naive model and, as expected, its performance is the worst compared to almost all of the other baselines. SASRec had previously reported better results than Pop according to Hit@10 and NDCG@10 performance metrics. In SASRec, 100 negative items were randomly sampled, and Hit@10 and NDCG@10 was calculated against these 101 items, including the ground truth. Using $\mathsf{MobileRec}$, we employ a full item set for evaluation, which is a stricter evaluation criterion for recommendation systems. We observe that on all the performance metrics, SASRec outperforms Pop except Hit@1. For example, SASRec manages to get $19.86\%$ improved Hit@10 in comparison with pop. On the same lines, a $16.34\%$ performance improvement is observed in Hit@15. Similarly, SASRec achieves $15.23\%$ improved Hit@20 in comparison with Pop on $\mathsf{MobileRec}$.

Considering NDCG metric, SASRec had reported $41.37\%$ improved NDCG@10 compared to pop, when evaluation is done on $\mathtt{Beauty}$. On $\mathsf{MobileRec}$, we observe only $14.28\%$ improvement in NDCG@10 compared to pop. We believe that this difference in the improvement on NDCG@10 between SASRec and Pop on these two datasets ($\mathtt{Beauty}$ and $\mathsf{MobileRec}$) can be explained by keeping in view the comparative dataset statistics of the $\mathtt{Beauty}$ and $\mathsf{MobileRec}$. There are $52,024$ users in $\mathtt{Beauty}$ while $\mathsf{MobileRec}$has $700,111$ users. Secondly, $\mathtt{Beauty}$ dataset has $57,289$ items compared to $\mathsf{MobileRec}$ which has $10,173$ apps. $\mathtt{Beauty}$ has $0.4M$ interactions, with $7.6$ average interaction per user and $6.9$ average interaction per item. Whereas $\mathsf{MobileRec}$ has $19.3M$ interactions with $27.56$ interactions per user and $1,896.88$ interactions per item (i.e., app). For example, $\mathsf{MobileRec}$ has $27.56$ interaction per user, which is more than 200 times more interactions per user than $\mathtt{Beauty}$. Likewise, $\mathsf{MobileRec}$ also has a considerably larger number of average interactions per item in comparison with $\mathtt{Beauty}$. Keeping these factors in view, $\mathsf{MobileRec}$ presents a more dynamic recommendation scenario. The high degree of user and item interactions in $\mathsf{MobileRec}$ can be attributed to being the reason for the comparatively smaller improvement in NDCG@10 by SASRec compared to the improvement SASRec had reported over Pop on $\mathtt{Beauty}$. Secondly, since we consider a full item set for evaluation compared to the ranking strategy by SASRec, this may also be a potential reason for smaller performance gains by SASRec in comparison with pop. On $\mathtt{Beauty}$, SASRec reported better results than GRU4Rec. We observe the same pattern of SASRec outclassing the GRU4Rec when training and evaluation are performed on $\mathsf{MobileRec}$.

On $\mathtt{Beauty}$ dataset, ELECRec reports better Hit@5, Hit@10, NDCG@5 and NDCG@10 than pop. We notice the same pattern when training and evaluation are done on $\mathsf{MobileRec}$ with Hit@1 to be the only exception, where Pop performs better than ELECRec. We assume that a very dynamic user-item interaction history might be the reason. Similarly, ELECRec had reported $329.87\%$ improved Hit@5 and $242.04\%$ improvement in Hit@10 compared to GRU4Rec on $\mathtt{Beauty}$. Moreover, for NDCG@5 and NDCG@10 metrics, ELECRec achieved $412.12\%$ and $331.38\%$ improvements over GRU4Rec, respectively. Focusing on the performance improvements obtained by ELECRec over GRU4Rec when training and evaluation are done on $\mathsf{MobileRec}$, ELECRec manages to get a $9.30\%$ and $13.72\%$ improvement in Hit@5 and Hit@10, respectively. Similarly, $1.88\%$ and $10.81\%$ improvement is observed in NDCG@5 and NDCG@10, respectively. This observation also points toward the highly dynamic user-item interaction involved in $\mathsf{MobileRec}$. Both models struggle to achieve high performance, which results in diminished performance gaps between the two approaches as compared to the performance gaps reported in ELECRec. There are over 19 million interactions in $\mathsf{MobileRec}$ while only 0.4M user-item interactions are there in $\mathtt{Beauty}$. This high level of dynamism in user-item interactions pushes the models towards lower performance gains. Considering this analysis, $\mathsf{MobileRec}$ should be a valuable addition to the existing volume of recommendation datasets.

ELECRec also shows the comparative gains over SASRec on $\mathtt{Beauty}$. First, we will cover the relative performance gains ELECRec manages to get over SASRec with respect to Hit@5, Hit@10, NDCG@5 and NDCG@10 on $\mathtt{Beauty}$ dataset. After that, we will discuss the performance improvements that ELECRec achieves over SASRec when the training and evaluation dataset is $\mathsf{MobileRec}$. Starting off with Hit@5 and Hit@10, ELECRec reports $83.59\%$ and $59.47\%$ improvement over SASRec. Similarly, $103.61\%$ and $84.11\%$ improvement is reported by ELECRec over SASRec with respect to NDCG@5 and NDCG@10, respectively. Now, let us discuss the performance comparison between ELECRec and SASRec when training and evaluation are performed on $\mathsf{MobileRec}$. It can be observed that SASRec performs better than ELECRec on $\mathsf{MobileRec}$ in Hit@5, Hit@10, NDCG@5, and NDCG@10 metrics. The quantification of the performance improvement of SASRec over ELECRec in Hit@5, Hit@10, NDCG@5, and NDCG@10 yields $4.25\%$, $4.02\%$, $8.92\%$, and $7.31\%$, respectively. We believe that ELECRec employs a generator that is trained with an NLP task, while the discriminator is responsible for discerning if the item is the next rightful (real) item in the sequence or a fake next item. The discriminators discerning ability depends upon the quality of samples generated by the generator. Since $\mathsf{MobileRec}$ presents as highly dynamic user-item interaction sequences with fleeting interests shown by users, the generator might have difficulty generating high-quality training samples. Since the discriminators’ ability to capture the true item correlation depends upon the quality of samples generated by the generator, low-quality samples may lead to diminished performance gains on the discriminator’s part.

It is worth recalling Figure 2 that provides an interesting snapshot of the dynamic user-item interaction sequences. For example, it can be noticed from Figure 2(b) that a considerable percentage of users migrate from paid applications to free applications. A similar dynamic migration pattern is evident from Figure 2(d) which depicts the user migration among top-5 categories. We think that with 19 million user-item interactions having such a dynamic temporal interaction pattern may present a challenge for the generator to learn high-quality sample generation. Moreover, ELECRec reported its comparative performance to BERT4Rec. ELECRec outperformed BERT4Rec in the original paper on all the metrics on $\mathtt{Beauty}$ (chen2022elecrec, ). Specifically, ELECRec had reported a $100.85\%$ and $61.06\%$ performance gain over BERT4Rec in Hit@5 and Hit@10, respectively, and $131.50\%$ and $97\%$ gains were reported in NDCG@5 and NDCG@10 metrics. We observe the same pattern of ELECRec outperforming BERT4Rec in most of the evaluation metrics when $\mathsf{MobileRec}$ is employed for training and evaluation except for Hit@1 where BERT4Rec does better than ELECRec. Considering the fact that Hit@1 is a stricter criterion, we trust that the training objective of predicting the masked items in the sequence by joint conditioning on bidirectional context, employed by BERT4Rec might be a better training objective for stricter evaluation metrics like Hit@1. Especially, in the context of fleeting user interests like in $\mathsf{MobileRec}$, masked item prediction objective with joint conditioning on bidirectional context seems to be effective for encoding the user’s dynamic behavior.

Graph contextualized self-attention model GCSAN employs both graph neural networks and self-attention networks for the session-based recommendation. In the original paper, several comparative evaluations are reported between GCSAN and competing methods. We are primarily interested in the quantification of the comparative representation learning ability of GRU4Rec and Pop versus GCSAN since we also include Pop and GRU4Rec in our baselines along with GCSAN. In the original paper, using Amazon-Books dataset, GCSAN reports improvement over Pop and GRU4Rec in NDCG@5 and NDCG@10. Similarly, improved results are reported as compared to Pop and GRU4Rec in Hit@5 and Hit@10 on different benchmark datasets. We observe similar pattern of GCSAN outperforming Pop and GRU4Rec in NDCG@5, NDCG@10, Hit@5, and Hit@10 consistently on $\mathsf{MobileRec}$. We notice $9.30\%$ and $6.62\%$ improvement over Pop in NDCG@5 and NDCG@10, respectively. Similarly, improvement is observed considering Hit@5, Hit@10, NDCG@5, and NDCG@10 for GCSAN over GRU4Rec.

LightSANs is a transformer variant for the next-item recommendation task which employs a low-rank decomposed self-attention for projecting the user’s historical interests into a few latent interests. LightSANs reported Hit@10 and NDCG@10 results on several competing baselines. We will focus on Pop, GRU4Rec, BERT4Rec, and SASRec for comparing and analyzing the LightSANs performance on Amazon-Books dataset and $\mathsf{MobileRec}$. Before further going into the comparative analysis of the performance of LightSANs on Amazon-Beauty versus $\mathsf{MobileRec}$, let us first look at the Amazon-Books datasets. Amazon-Books dataset has $19$K users, $60$K items, and $1.7$ million interactions. On Amazon-Books dataset, LightSANs reports a $121.77\%$ and $172.43\%$ improvement in Hit@10 and NDCG@10 over Pop. Similarly, $3.91\%$ and $2.65\%$ of improvement is reported in Hit@10 and NDCG@10, respectively, over SASRec. $8.41\%$ and $5.72\%$ improved Hit@10 and NDCG@10 are reported compared to those of GRU4Rec. Finally, an improvement of $8.76\%$ and $4.03\%$ is reported over BERT4Rec. When dataset is $\mathsf{MobileRec}$, we observe $13.90\%$ and $10.38\%$ improvement over Pop in Hit@10 and NDCG@10. LightSANs manages to get $12.41\%$ and $14.86\%$ improvement over GRU4Rec for the same metrics.

Similar to ELECRec, LightSANs also shows $22.85\%$ and $18.05\%$ better results than BERT4Rec for Hit@10 and NDCG@10 metrics. We notice that LightSANs maintains the pattern of outperforming competing baselines on $\mathsf{MobileRec}$ as it does on the Amazon-Books dataset, with SASRec being an exception that outperforms LightSANs in Hit@10 and NDCG@10 metrics. The consistent performance exhibited by LightSANs against BERT4Rec, Pop, and GRU4Rec can be attributed to better design choices in LightSANs. For example, LightSANs proposes a decoupled position encoding in place of implicit position encoding. We believe that this implicit position encoding helps LightSANs to model the historical user interests and user-item interaction more precisely, which steers the model towards better Hit@10 and NDCG@10 on $\mathsf{MobileRec}$. Nevertheless, LightSANs is outperformed by SASRec. The superior performance of SASRec might be due to the learnable position embeddings introduced by SASRec.

SINE investigates the idea of encoding a user’s interests with multiple embedding vectors, building upon their empirical findings that a user’s behavior sequence exhibits multiple distinct interests. SINE employs several datasets for benchmarking their results. We consider the performance reported by SINE on the Amazon Product Review dataset using Hit@10 and NDCG@10 metrics. We restrict our comparative analysis to SASRec and GRU4Rec. SINE reports improvement over SASRec in Hit@50, Hit@100, NDCG@50, and NDCG@100, respectively. Better performance is also reported against GRU4Rec in Hit@50, Hit@100, NDCG@50, and NDCG@100. Given $\mathsf{MobileRec}$ for training and evaluation, SINE manages to outclass GRU4Rec by $7.27\%$ and $4.90\%$ in Hit@20 and NDCG@20, respectively; but struggles to perform better than SASRec on Hit@20 and NDCG@20. First, it might be because Hit@20 and NDCG@20 are stricter criteria as compared to Hit@50, Hit@100, NDCG@50, and NDCG@100 opted by SINE. Secondly, keeping in view that SINE strives to capture the user’s distinct interests from the interaction history of the user, we think that converging to distinct user interests in $\mathsf{MobileRec}$ presents a challenge because of fluctuating user interests, as depicted in Figure 2. This inability of learning and embedding the distinct user interests from the interaction sequence may lead SINE to show worse performance than SASRec.

In this paper, we have introduced $\mathsf{MobileRec}$, a large-scale dataset of sequential user-app interactions. The unique feature of $\mathsf{MobileRec}$ is that it captures multiple interactions per user, providing a more comprehensive view of user behavior. With a total of $19.3$ million user-app interactions, spanning across more than 10 thousand distinct apps from 48 categories, and involving $0.7$ million unique users, each with at least five distinct app interactions, $\mathsf{MobileRec}$ offers unprecedented granularity in understanding user engagement across different app categories. Moreover, every user-app interaction in $\mathsf{MobileRec}$ contains rich contextual information, such as the user’s rating, textual review, and review date. Last but not least, each app in $\mathsf{MobileRec}$ carries extensive metadata, such as the app’s title, category, long-form textual description, overall average rating, developer information, and content rating, among others. We also show the usefulness of the $\mathsf{MobileRec}$ dataset as an experimental testbed for research in app recommendation by conducting a comparative evaluation of various state-of-the-art recommendation techniques. This evaluation also establishes baseline results that will benefit the research community. We hope that our dataset will inspire further research, enable novel insights, and pave the way for future mobile app recommendation systems.