Emerging App Issue Identification via Online Joint Sentiment-Topic Tracing

In mainstream topic detection studies [24, 25, 26, 27], an event/issue is considered emerging if it is (heavily) discussed in current time slice but not previously. The application scenario of these studies is generally targeted at social media platforms, e.g., Twitter and Sina Weibo. However, user discussion on social media and app stores has significant differences. One difference is that the app reviews usually associate with specific app versions, while typical social media contents do not concern with the version concept. Another big difference is that emerging event detection for social media is simply dependent on volume of user posted content, regardless of the sentiment associated with it; while for app reviews, user sentiment is one indicator to emerging issues [28, 29]. Thus, simply applying standard emerging issue detection methods from the social media field is not optimal for our scenario. In this paper, we define an emerging app issue as follows.

Definition (Emerging App Issue) An issue reported in user review(s) at a particular time slice is defined as an emerging issue if its distributions presented marginal fluctuations in previous time slices in previous time slices, corresponds to a significant increase in terms of the percentage of reviews reporting it, and is negatively reported by users in the current time slice.

For example, Fig. 1 illustrates the changes of the number of reviews and user rating distributions over time for Facebook. As can be seen, the number of reviews received on July 14, 2019 is significantly larger than the number of reviews received on July 12, 2019; this is true especially for the one-star reviews (represented by the red bar), which means that an emerging issue may exist for the recent update. By checking the detailed user reviews, we find that it was related to a huge redesign of the app in July [30]. With continuous monitoring and accurate identification of emerging issues, such problem can be detected in a timely manner. Developers can then be alerted of the need to perform further maintenance activities to ensure good user experience. We also discover that both number of reviews reporting the issue and user rating can indicate the emergence of an issue. Involving review ratings in the analysis can help our tool avoid false positives, i.e., topics that are mentioned in many reviews but do not correspond to important problems that need to be rectified urgently [27].

Topic models [32, 13] have been proven useful for discovering latent structures in a collection of documents [33, 34]. The models capture the co-occurrence of words in the collection under a probabilistic framework by assuming that each topic can be represented by a set of word clusters. In this way, the models can uncover latent semantic structures (i.e., topics) in the documents. Due to the unsupervised nature, the models such as Probabilistic Latent Semantic Analysis (PLSA) [32] and Latent Dirichlet Allocation (LDA) [13] have been widely applied in mining software repositories [29, 35, 11] where labeled datasets are not available in practice. The outputs of the topic models are two matrices: (1) Document-topic matrix, denoted as $\Theta\in\mathbb{R}^{D\times K}$, where $D$ is the number of documents and $K$ is the number of topics. The $i$-th row of the matrix, i.e., $\theta_{i}\in\mathbb{R}^{K}$, is a topic distribution vector for the $i$-th document; (2) Topic-word matrix, denoted as $\Phi\in\mathbb{R}^{K\times V}$, where $V$ is the total number of unique words (i.e., vocabulary). The $i$-th row of the matrix, that is, $\phi_{i}\in\mathbb{R}^{V}$, is a word distribution vector for the $i$-th topic. Table I shows an example of the output of topic models, with top five words and corresponding probabilities presented for each topic.

Latent Dirichlet Allocation (LDA) [13] assumes that each document consists of a mixture of topics, and each word in documents belongs to one topic.

Biterm Topic Model (BTM) [14] is designed specifically for modeling topics of short texts. BTM extends a document into a set of biterms (i.e., two terms) which includes all combinations of any two distinct words appearing in one document. In this way, BTM can enrich the short texts by explicitly modeling the word co-occurrence patterns. In BTM, instead of assuming that each word belongs to one topic, it assumes that each biterm relates to one topic. BTM has demonstrated better performance than LDA in modeling short texts [36, 37].

Joint Sentiment/Topic Model (JST) [38] is also a variant of LDA, but involves the sentiment of each topic. JST assumes that each word should be associated with one sentiment polarity, such as positive, neutral, and negative. The output of JST is also two matrices but with three dimensions: (1) Document-sentiment-topic matrix, denoted as $\Theta\in\mathbb{R}^{D\times 3\times K}$ where $3$ is the number of sentiment polarities (positive, negative, and neutral); (2) Sentiment-topic-word matrix, i.e., $\Phi\in\mathbb{R}^{3\times K\times V}$.

Figure 2 illustrates the differences among the above topic models in terms of their input and output. However, documents mostly come as ephemeral streams in most scenarios, such as scientific articles and Twitter messages, and thus the topics and subordinate attributes (e.g., word distributions) in the documents are time-evolving [33]. To capture such topic variations, online topic models, including Online LDA (OLDA) [38], have been proposed. The output of online topic models is topic distributions along with consecutive time slices, and Table I can be regarded as the topic distributions of the documents in one time slice.

Word embedding (also known as distributed representation [39, 22]) is one of the most popular techniques that represent document vocabulary by training on a large text corpus. They map each word to a low-dimension real-valued vector and are capable of capturing the context of a word based on the semantic similarity relations with other words. The words that exhibit the same semantics have similar vector representations. For example, suppose the word “photo” is represented as [0.53, -0.21, 0.02] and the word “image” is represented as [0.49, -0.35, 0.01]. From their vectors, we can estimate their distance and identify their semantic relation. Word embedding is usually implemented through training a machine learning model such as CBOW and Skip-Gram [39] on large datasets. Phrases, sentences, and documents can also be embedded as vectors based on word embedding techniques. For example, a simple way of sentence embedding is to compute the average word embeddings in the sentence [40].

Since user reviews are usually written on mobile phones with limited keyboards on mobile screens, they often contain a large number of noisy words, such as misspelled words and abbreviations (e.g., “asap”). In the following, we elaborate on the preprocessing method and also the method to prepare polarity-carrying words (i.e., the words that carry sentiment polarities, e.g., positive or negative) for the subsequent modeling process.

Inspired by existing topic modeling techniques [38, 14], we propose a novel unsupervised model named AOBST (Adaptive Online Biterm Sentiment-Topic Model) for jointly modeling the topics and sentiment of app reviews. We will first illustrate the proposed biterm sentiment-topic model for building connections between topics and sentiment, and then elaborate on its online adaption.

In this section, we describe how we determine the emerging app issues based on the evolution of the topics belonging to negative sentiment along with app versions.

For facilitating developers to efficiently understand the identified emerging app issues, we visualize the evolution of app issues along with versions based on issue river [11]. Figure 6 (Left) shows an example for Swiftkey for Android. The whole river represents all the app issues, and different branches indicate different topics. The width of each branch $k$ presents the user-concern degree of the issue for the corresponding version $t$, defined as:

We employ the same dataset by Gao et al. [11] for evaluation. Details of the dataset are shown in Table III. The dataset includes 164,026 reviews (from August 2016 to April 2017) for six apps, from 89 versions in total. The apps are distributed in different categories, with two of them from the App Store and the others from Google Play.

We use the keywords in changelogs as ground truth (one example shown on the right of Figure 6) and employ the three metrics as used by Gao et al. [11] for verifying the effectiveness of MERIT. We define an app issue to be successfully identified by MERIT and its baselines if its corresponding description in the changelog of the immediate version present a high similarity⁷⁷7The similarity is measured as the cosine score between the two vector representations in the word embedding space, and it is high if the cosine score is larger than 0.6 [54]. with the identified issues (either at phrase level or sentence level). The evaluation method is based on the hypothesis that emerging issues need to be quickly solved in an updated version and thus are typically reflected in the changelog of the immediate version. Here we use three performance metrics as used by Gao et al. [11] for verifying the effectiveness of MERIT. The first metric is for measuring the accuracy in detecting emerging issues, defined as $Precision_{E}$. The second is to evaluate whether our prioritized app issues (including both emerging and non-emerging issues) reflect the changes mentioned in the changelogs, defined as $Recall_{L}$. The last metric $F_{hybrid}$ is for measuring the balance between $Precision_{E}$ and $Recall_{L}$. Higher values of $F_{hybrid}$ indicate that changelogs are more precisely covered by detected emerging issues, and more changelogs are reflected in the prioritized issues.

We compare the effectiveness of our proposed framework with a popular emerging event detection approaches on social networks, OLDA [55] and the state-of-the-art emerging app issue identification approach, IDEA [11].

On-line Latent Dirichlet Allocation (OLDA) is an online version of Latent Dirichlet Allocation (LDA) [13] that manually captures the topic patterns and identifies topics of text streams and their changes over time. It generates an evolutionary word distribution matrix for each topic. In this way, it incrementally builds an up-to-date model when new documents appear. The emerging topics in the current app version are determined by comparison with the topic distributions in the previous version.

IDEA is a state-of-the-art emerging app issue identification approach proposed recently. It improves OLDA by considering the topic distributions in previous versions within a version window during emerging topic detection. The improved method is named as Adaptive OLDA (AOLDA). It also includes an automatic topic interpretation method for labeling each topic with the most representative phrases and sentences.

This research question relates to the capability of MERIT in identifying accurate and complete emerging app issues in comparison with IDEA [11] and OLDA [55]. Having too many false positives would end up being counterproductive, whereas having too many false negatives would mean that the proposed framework is not able to alert emerging issues in many cases where those are important. Table IV displays the comparison results.

As seen in Table IV, the proposed MERIT approach outperforms the baseline approaches on all the metrics. We discuss the performance of MERIT from two aspects as below.

Result 1: Interpreting Topics with Phrases v.s. Sentences. As mentioned in Section 3.3.2, there are two ways to represent an app issue: by phrases and by sentences. For example, the “Label” and “Sentences” in the top boxes of Figure 6 are the phrase and sentence representations respectively. As shown in Table IV, considering all the three methods, issues in sentences present better performance than those in phrases with 9.5%, 29.1%, and 15.6% increase in $Precision_{E}$, $Recall_{L}$, and $F_{hybrid}$ on average respectively. This result may be attributed to the fact that sentences can convey more details than phrases and thereby cover more key terms mentioned in changelogs, which is also in line with findings of our previous study [11]. Specifically, the sentences identified by MERIT can enhance the performance of phrases by 8.1%, 22.6%, and 16.3% wrt. three metrics, respectively. We then use Wilcoxon signed-rank test [56] for statistical significance test, and Cliff’s Delta (or $d$) to measure the effect size [57]. The significance test result ($p-value<0.05$) and large effect size ($d=2.76$) on the difference in the mean of the $F_{hybrid}$ scores of phrase-level issues and sentence-level issues confirm the better performance of sentence representations over phrase representations.

Result 2: MERIT v.s. Baselines. Comparing MERIT with baseline approaches, we find that MERIT can outperform both baselines in all the three metrics with respect to sentence-level issues. For phrase-level issues, although MERIT shows a slightly lower $Recall_{L}$ than IDEA for the SwiftKey app, it exhibits better performance in both $Precision_{E}$ and $F_{hybrid}$. On average, MERIT can achieve precision, recall, and f-score of 81.4%, 81.2%, and 80.9% respectively, and outperform OLDA by 37.8% and IDEA by 22.3% for $F_{hybrid}$, which indicates that MERIT can better balance the precision and recall in emerging issue detection. Besides, the significant statistical test results ($p-value<0.01$) and large effect sizes ($d>2$) on the $F_{hybrid}$ scores for both phrase and sentence -level issues of MERIT and IDEA/OLDA confirm the superiority of MERIT over IDEA/OLDA.

MERIT extends IDEA by (1) adopting BTM for topic modeling instead of LDA, (2) jointly modeling sentiment and topics, and (3) employ the proposed word-embedding-based topic labeling (NTL) approach. We perform ablation experiments by considering each of the 3 extensions one-at-a-time, which we refer to as “+BTM”, “+Sentiment”, and “+NTL” respectively. Table V shows the results of comparing each of these 3 approaches with the baselines.

Unsurprisingly, the combination of all extensions gives the greatest improvements in terms of $F_{hybrid}$, and all the components are beneficial on their own. Similar to the answer to RQ1, we also observe that sentence-level issues generally present better performance than the phrase-level issues.

Specifically, with respect to each extension considered independently using BTM instead of LDA for topic modeling can enhance the average performance by 8.8% and 16.9% for the phrase-level and sentence-level $F_{hybrid}$ scores respectively. In terms of $Precision_{E}$ and $Recall_{L}$, with BTM involved, the performance increases by 11.5% and 19.0%, respectively. When jointly modeling topics with the sentiment, the $F_{hybrid}$ scores are increased by 5.8% and 19.4% in terms of phrase and sentence representations, respectively. On average, both precision and recall show an increasing trend, +9.6% and +13.4%, respectively. The results indicate that by the considerations of sentiments, overall results including both precision and recall have been improved. But for some apps, such as YouTube and Clean Master, although the recall is increased (+20.2% and +31.4% respectively), the precision is slightly dropped (-7.8% and -4.2% respectively). This may be because with the sentiment involved, the topics predicted as negative sentiment tend to be identified as emerging issues, which is helpful for enhancing the recall. But the negative topics might not always be emerging, such as some constantly recurring topics (e.g., the “screen” topic for YouTube and the “battery” topic for Clean Master), so the precision is slightly weakened. Besides, involving word embeddings during topic interpretation gives us a 7.3% increase for phrase-level issues and a 5.9% increase for sentence-level issues with respect to $F_{hybrid}$. We also observe that although the YouTube app (with 1,143 reviews per version) shows a slightly decrease (-2.4%) on the $F_{hybrid}$ score, all the other apps, especially NOAA Radar which has only 523 reviews per version, enjoy an increase. Thus, the experiment results demonstrate that the novel topic labeling method can work well even for apps with few reviews. Moreover, the gain from different extensions is not fully cumulative since the information delivered by these components overlaps. For instance, both the topic modeling and topic labeling steps help capture the semantics of the words in app reviews to generate accurate emerging issues.

We have identified three advantages of MERIT that may explain its effectiveness in detecting emerging app issues.

Observation 1: MERIT can better model the topics of short texts. In this work, we propose to use the biterm topic model (BTM) for short text mining instead of LDA. Since LDA learns review-level word co-occurrence patterns to reveal topics, it suffers from the severe data sparsity in short review texts. Instead, BTM learns the topics from word co-occurrence patterns directly and thus alleviate the data sparsity problem. Table VI shows the top eight terms of three example topics obtained from LDA and BTM. We discover that BTM can generate more semantically-coherent terms for each topic. For example, the terms “also”, “there$\_$be”, and “great$\_$app” terms are not related to Topic 2 “play button”. The semantic inconsistency of the top terms in one emerging issue would confuse developers or influence the performance of subsequent automatic topic interpretation step. By using BTM instead of LDA for review modeling, the semantics of top terms belonging to one topic can be more coherent.

Observation 2: MERIT can focus on negative topics. As shown in Figure 5, the topics extracted from reviews are usually mingled with various polarities. Even for the same topics, users may express totally different opinions. Motivated by the intuition that developers are more concerned about the negative app aspects [29, 28], MERIT focuses on the topics inferred as negative instead of incorporating topics in all sentiment polarities. Thus, MERIT can expose the topics likely corresponding to app issues.

Observation 3: MERIT can interpret topics with more representative and coherent labels. For accurate topic labeling, we combine word embeddings to prioritize semantically-representative phrase/sentence candidates. Table VII shows the ranked phrases with and without word embeddings involved, respectively. We can discover that the proposed word-embedding-enhanced topic labeling (NTL) approach can better interpret the topic meanings in terms of coherence and semantic accuracy. For example, the original topic labeling approach selects “playback error galore” and “playback error” as the most representative phrases of Topic 1 and Topic 2 respectively, which are intuitively different from the general meaning of the topics (i.e., split view and subscription box respectively) and can cause confusion to developers in understanding the detected emerging issues. Instead, the top three phrases of these two topics obtained by MERIT are semantically coherent, all about split view or subscription.

We have also summarized two main scenarios that may lead to inaccurate emerging issue prediction.

Observation 1: MERIT may miss the emerging app issues only mentioned in few user reviews. For the emerging issues only expressed in few (e.g., three or four) reviews, they are difficult to be exposed through topic modeling approaches [26, 27]. For example, one major modification claimed by the version 5.9.3 of the Clean Master app, i.e., “Added Cloud Recycle Bin - Recover misdeleted photos from the cloud up to 30 days after deleting”, MERIT misses capturing any emerging issue related to “recycle bin”. After inspecting the collected corpus, we find that only three reviews received in the previous version are describing the recycle bin, which possibly leading to the omission. We discover similar failure scenarios for other apps. For instance, for the NOAA Radar iOS app, it made a major change about its widget in its version 2.0 that fixes an issue, i.e., “Tap on Today tab, scroll to the bottom and tap Edit” as written in the changelog. MERIT fails to identify the issue since it is only discussed by three pieces of reviews in the corpus of the previous version.

Observation 2: Official changelogs may not cover all the app issues of the previous version that are fixed in the current version. Although app markets such as the App Store encourages app developers to write what is actually happening to the apps in the changelog [58], app developers tend to write sketchy and vague bullet points for the changes, such as “Bug fixes” and “We’re always trying to improve your experience”. Although we already filter such changelogs out during validation, the release notes may not cover all the major changes made to current versions and could lead to false negatives.

First, the app issues may not be fixed instantly in the next updated version. For example, MERIT detects an emerging issue associated with video orientation modes, described as “portrait mode” and “full screen mode”, for version 11.39 of the YouTube iOS app. One user complained that “Sometimes when I’m on full screen mode, I click the minimise screen button and it doesn’t work. I try to flip my phone and it doesn’t minimise.”, and gave a two-star rating. The issue also aroused heated discussion on the YouTube online forum [59]. We discover that the issue was fixed in a later version 12.05 instead of the immediate next version, as indicated in the changelog “Fixed delay when pressing the full screen and minimize buttons in the player。”. Such postponement of issue fixing is reasonable since not all bugs in apps would be addressed right away [60, 61]. Second, changelogs may describe modifications in general terms. For example, MERIT alerts an emerging issue related to “Samsung keyboard” and “force close” for version 5.0.4.93 of the SwiftKey app. Although the issue is greatly relevant to the corresponding changelog “Fixed issues causing repeated crashes on some devices when loading the keyboard”, the evaluation of MERIT regards “device” and “Samsung” as mismatched. Finally, changelogs may not cover all the major app changes. For instance, we find that the voice dictation issue identified by MERIT for version 5.1.0.60 of the SwiftKey app is a change (i.e., forcing install additional app for voice dictation) made in the app version but not described in the changelog. For example, one user commented that “After update, force install additional app ‘google voice search’ for voice dictation, which was not previously required. …. I wish I knew so I would not update the app.”.

We evaluate whether MERIT can output emerging app issues within reasonable time, by comparing the execution time of MERIT on the subject apps with IDEA, and also with different extensions of IDEA (i.e., “+BTM”, “+Sentiment”, and “+NTL”). In this experiment, we randomly select subsets of the 5,000 reviews from the YouTube dataset (of different sizes) and run all the models. We run our experiments on a PC with Intel(R) Xeon E5-2620v2 CPU (2.10 GHz, 6 cores) and 16GB RAM. Figure 7 displays the comparison results of time consumed on different dataset sizes. As can be seen from Figure 7, all the models spend more time as the amount of data increases. We also find that the “IDEA+BTM”, “IDEA+Sentiment” and ”IDEA+NTL” models cost 16.0%, 25.8%, and 58.4% more time than the IDEA model when handling 5,000 reviews, respectively. Undoubtedly, MERIT incurs the highest time cost among all the models due to its higher complexity, which can cost 1.3 times more time than IDEA when processing the 5,000 reviews. In spite of the higher time cost, MERIT can deal with 1,000 reviews within eight seconds and 5,000 reviews within three minutes, which we believe to be still acceptable. Therefore, our experiments demonstrate that MERIT can detect emerging app issues more accurately while preserving reasonable time costs.

We also quantitatively compare the performance of MERIT in different parameter settings. We analyze three parameters, that is, the number of topics $K$, the window size $\omega$, the penalty factor $\mu$ (in Equ. 10), and balance parameter $m$ (in Equ. 11). We vary the values of these four parameters and evaluate their impact on the performance of MERIT. The results are shown in Figure 8.

In this paper, to automate the parameter tuning and result verification processes, we adopt a common semantic measurement metric [62], i.e., cosine similarity. However, the automatic evaluation results might not exactly reflect practical performance. Manual evaluation is therefore needed to comprehensively evaluate the consistency between detected emerging issues and the ground truth. Since validating all the emerging issues would consume huge human effort, we choose the YouTube app, which contains the most versions among the studied apps (accounting for 37.1% of all the app versions), for manual verification. The first two authors examined the detected emerging issues version by version with extensive discussions to reach a unanimous score. The results are shown in Table VIII. As can be seen, comparing manual inspection results with the scores computed by cosine similarity, the differences range from -5.2% to +4.1%. Also, in terms of the $F_{hybrid}$ scores, there is only a small disparity (around -1%). Thus, using cosine similarity could be regarded as a reliable way to alleviate the labor burden and time consumed in both parameter tuning and verification. The effectiveness of MERIT is consequently confirmed.

To further demonstrate the effectiveness and practicability of our work, we conduct a user study among 44 industry practitioners in large IT companies including Alibaba and Tencent, with 19 developers (43.2%), nine data analysts (20.5%), six test engineer (13.6%), two product managers (4.5%), and 10 from other positions (22.7%). Around 80% of the participants have more than one year software engineering experience. The developer survey is conducted through an online questionnaire, which consists of five questions: two questions on participants’ background and three questions for understanding their attitude towards the practicability of MERIT. During the survey, we validate the practicability of MERIT in terms of three aspects: acceptability of the provided emerging issues, preference of higher accuracy but with more time consumption, and willingness of applying such a tool into their development pipeline. Each aspect is rated on a 1-4 Likert scale (4 for agreement, 3 for mild agreement, 2 for mild disagreement, and 1 for disagreement).

First, our model evaluation is based on the six subject apps in [11], which may not guarantee the generalization of the findings. We pick the dataset used in [11] to allow for fair comparison. Second, app versions with few user reviews can impact the performance of MERIT. Since small datasets can be easily analyzed manually, MERIT is targeted for automatic analysis of large review datasets. Also, MERIT”s good performance on different quantities of user reviews (on average 523$\sim$6,332 reviews per version) show that MERIT would well adapt to different review sizes. Third, the 500 opinion words manually labeled during polarity word preparation procedure may not be the optimal opinion words for inferring the sentiment associated with each topic. To mitigate the threat, we randomly selected the words weighted by their frequencies, so the words with higher frequencies are more likely to be selected and labeled. Also, we ensure the sample corresponds to a statistically significant proportion of the whole opinion lexicons. In practice, app developers can choose a different opinion word set for emerging issue detection. How to select an optimal set of opinion words for better sentiment inference can be future work. Fourth, the changelogs and cosine similarity measurement that we adopt for evaluation may not accurately reflect the practical performance. We mitigate this threat by manually validating the results on a sample of reviews and demonstrating that the results reported using cosine similarity are consistent with those obtained via manual inspection.

Moreover, in this study, we only combine the sentiment characteristic of app reviews into MERIT while other factors such as device types that may be helpful for emerging issue detection are not involved. Future studies should broaden the set of features used to characterize app reviews in our study and investigate the impact of different characteristics on the performance of emerging issue detection. In addition, there may be alternative approaches to combine topics and sentiment for emerging issue detection, e.g., implementing a two-step pipeline where extracting negative review sentences or paragraphs is the first step, and modeling topics of the negative texts is the second step. We leave implementation and evaluation of these alternatives to future work. Finally, MERIT shares the same limitation with the adopted topic modeling approach [14], i.e., the number of topics should be determined initially. This limitation is brought by the unsupervised nature of the approach. There are studies [63, 64, 65] on automatically identifying the optimal topic number, but they are not easy to be adapted to online topic modeling approaches, which is the core of our proposed framework. How to efficiently discover the optimum topic numbers for online topic models can be regarded as a challenging and interesting work for future research.

Since app reviews serve as an essential channel between users and developers, and provide rich information about app usage, the number of studies on user review analysis is on the rise [66]. Recent research has leveraged Natural Language Processing and Machine Learning techniques to extract useful information from online app products to help developers realize, test, optimize, maintain, and release apps (see e.g., [67, 68, 69, 70]). The major goal of these studies is to alleviate the burden of summarizing useful knowledge from a relatively huge quantity of unstructured texts. Here, we focus on the research that exploiting app reviews to facilitate the process of app maintenance and release.

A number of studies [71, 72, 73, 74] categorize user reviews based on their sentiment (e.g., either praise or complaint) and general topics (e.g., bug report or feature request). Di Sorbo et al. [75] presented an approach called SURF to further classify reviews into fine-grained topics (e.g., GUI and security). Based on the categorized reviews, Gu and Kim [76] applied aspect opinion mining and sentiment analysis to find the most popular features of an app. Although they can present the rating changes of one app feature over time, the ratings are tracked based on feature words instead of topics. Moreover, their work does not establish the relation of features with star-ratings [77]. Besides, Islam and Zibran [78] and Calefato et al. [79] design sentiment analysis tools specific to software development.

Topic modeling is widely used in different domains, and interesting results have been inferred [80, 81]. Consequently, some researchers rely on topic modeling technique [13, 14] to analyze user reviews. Iacob and Harrison [82] and Guzman and Maalej [28] applied LDA to extract app features. Chen et al. [29] adopted LDA to capture the topic distribution of each user review, based on which they prioritized useful user reviews to developers. Fu et al. [83] analyzed the changes in the review number associated with each topic over time. Noei et al. [77] used LDA to determine the key topics of user reviews for different app categories. Gao et al. [81, 16] resorted to topic modeling methods for prioritizing app issues. None of the papers mentioned above have considered the sentiment changes of topics along with time or exploited the changes to detect emerging app issues.

Our previous work [11] is the most recent study focusing on tracking app issues along with release versions. Specifically, the IDEA proposed in [11] analyzed issue changes along with app versions using online topic modeling during which the emerging app issues are identified. Another of our recent work [84] also aimed at detecting emerging app issues but mainly during the beta testing periods. Although the IDEA model performs well on the studied apps, the proposed model still meets several limitations as discussed in Section 1.

Nayebi et al. investigate app updating frequency and its impact [85, 86]. They find that users prefer to install apps that were updated more recently and less frequently. Thus, frequent updates are not always considered positively in practice. Updating frequencies should also be carefully determined. Determining the sweet spot for app update frequency is an important and an interesting research topic. As this is beyond the scope of the current paper, we will leave this for future work.

An event⁸⁸8An event in social media corresponds to an app issue in the context of app reviews., in the context of social media, is an occurrence of interest in the real world which initiates a discussion on the event-associated topic on social media platforms, either soon after the occurrence or, sometimes, in anticipation of it. The emerging event detection approaches can be based on term interestingness [87], incremental clustering [88], or topic modeling [89], etc. For example, Li et al. [90] identified emerging events from Twitter by first selecting top bursty words and then conduct word clustering, which is a term-interestingness-based approach. A comprehensive survey on emerging topic detection approaches can be found in Hasan et al.’s work [91]. Different from texts on social media, each review is specific to one app version, and generally shorter in length [92, 12], which renders app review mining a more challenging task, specialized to software engineering context. However, the existing studies [91] in machine learning field either do not involve automatic labeling of topics or do not consider the short length nature of input texts, so directly applying them into our app review scenario will not be optimal.

Thus far in the app review analysis literature, term interestingness [84, 93, 41] and topic modeling methods [11] have been widely used.

The term-interestingness-based methods rely on tracking the terms likely to be related to being an event, and are usually followed with clustering methods. Various approaches are proposed to determine the interestingness score⁹⁹9The interestingness score refers to the possibility of a term to be related to an emerging event. of each term. For example, Minh Vu et al. [93, 41] first grouped the keywords using clustering algorithms and then determine the emergent clusters based on the occurrence frequencies of the keywords in each cluster.

The topic-modeling-based methods associate each document with a probability distribution over various latent topics and track the topic distributions over time. For example, our previous work [11] proposed an online topic modeling approach to infer the topic distributions of user reviews along with app releases. The emerging topics are identified based on their differences with the corresponding topics in previous time slices.

The term-interestingness-based methods can be regarded as a down-top model (i.e., from word to topic), while the topic-modeling-based methods are top-down (i.e., from topic to word). Since the abruptness of one topic does not indicate that all the words belonging to the topic show bursty trends, term-interestingness-based models may generate true negatives due to missing bursty words. Thus, our proposed model is based on the topic modeling approach.