Pattern Learning for Detecting Defect Reports and Improvement Requests in App Reviews

In the two last decades, the growth of user-generated content on the Web has accelerated enormously. This acceleration is bolstered by parallel developments, such as increased Internet access, increased Internet speeds, technological advancements in mobile devices, the growth of e-commerce and online payment methods, and many more. An important source of user-generated content with respect to customer feedback are online reviews, usually interpreted using Sentiment Analysis (SA) methods. The aim of SA is to automatically detect positive, neutral, and negative sentiments [18].

A major downside of SA is that it measures satisfaction at a certain point in time. In this light, we argue that in addition to SA, it is important to focus on detecting specific types of feedback that indicate potential causes and influence factors of satisfaction. We consider such specific customer feedback as actionable, since it suggests a clear course of action for addressing the feedback, and thus directly help to modify and hopefully improve products.

In this paper, we focus on customer feedback related to mobile software applications which we will refer to as “apps”. We argue that software reviews are very important for aggregating valuable feedback. Firstly, because many companies have come to realise that all the technology required to transform industries through software is available on a global scale [1]. Secondly, the field of software engineering has the well-accepted notions of bugs and feature requests, which we argue, are actionable types of feedback.

We approach feedback detection as a multi-label classification problem based on knowledge-base rules, in which our goal is to automatically determine if a given review is an example of actionable feedback. In general, managing and adapting a knowledge-based method is very tedious and labour intensive, especially because the considered domain that might have different (actionable) types of feedback. Therefore the developers of extraction rules or patterns need to have a proficient amount of domain knowledge, which might be rare. As a result, making a knowledge base of patterns is impractical to manage over time and across different domains. In this light, we suggest a system that is capable of performing pattern construction in an automated manner using genetic programming. Keeping in mind the importance of reduction of the human control over the system’s design, we also tackle the problem of having a small number of labeled reviews (gold labels) using noisy labels generated based on patterns in a distantly-supervised way [22, 14]. The employed dataset and the proposed framework implemented in Scala are available at https://github.com/mtrusca/PatternLearning.

The remaining parts of the paper are structured as follows. In Sect. 2 we examine the most relevant works. Section 3 presents a detailed overview of the proposed framework in this study. In Sect 4 we evaluate our framework through a series of experiments. Finally, in Sect. 5 we present our conclusions and suggest future work.

SA measures are not actionable, which means that further investigation is required to discover causes and opportunities for making improvements. While user-generated content is clearly an important source of customer feedback for businesses, only a few works are focused on exploring such data, beyond sentiment analysis.

There are very few works [5, 9, 19, 21, 20] that aim to detect specific information in customer feedback. The majority of these works only aggregate broad insights, mainly for the purpose of further exploration. In that sense, the information that is extracted through such approaches, can barely be considered actionable, as is also the case for the sentiment scores.

Among the aforementioned works, only the method proposed in [19] is more refined and suggests classifying actionable feedback, through a business-oriented approach. Namely, in [19] lexical patterns are used to train a supervised classifier, rather than directly employing patterns for information extraction, which makes the extraction mechanism more adaptive to the various representations of feedback. Further on, this system summarizes the extracted feedback by means of a Topic Model, generated using a clustering technique called Latent Dirichlet Allocation (LDA) [2]. However, while the objective is very relevant, the suggested methods require a vast amount of manual labour to create useful feedback patterns. We argue this to be a great limitation since analysing customer feedback is an important process that should ideally be performed in a continuous fashion. Nevertheless, the study conveys a promising direction for future research in opinion mining, and clear feedback types to focus on, which we adopt in this work.

Pattern-based information extraction methods achieve machine comprehension of texts by transforming the unstructured texts to structured data. Some of the first methods that define pattern-matching engines to detect regular expressions in annotations on documents were proposed in [8, 11]. Later on, in [13] it was proposed a pattern language called Hermes Information Extraction Language (HIEL), that creates information extraction patterns in a simpler (more compact) fashion compared to the aforementioned works, by including semantic information.

Usually pattern-based methods use diverse sets of rules that hone the process of information extraction. While the common rule learning methods like SEER [10], TANGO [15], or UIMA Ruta [16] have proven to be effective for information extraction, they are not directly helpful for our goal to label documents that include a particular type of feedback. Since we are not sure which forms the feedback type might take, we require different patterns per target feedback. A more promising approach, would be to focus on discovering patterns, for which metaheuristics (e.g., evolutionary algorithms, simulated annealing, or particle swarm optimization) are more suited. As example, in [4] a Genetic Programming approach is employed to learn grammatical rules helpful in discriminating between definitions and non-definitions of terms. However, in [6] a Genetic Algorithm [3] is used to learn the most relevant combinations of prefixes and suffixes for an entity type of interest, extracted from the context they are in. This approach is similar to our work, since we also aggregate frequent combinations of words and tags, in the context of a certain feedback type, to include in patterns. However, we also include entity types as building blocks for generating patterns, which gives to our patterns extra flexibility.

In this research, our goal is to automatically detect actionable feedback in reviews. More specifically, we aim to detect two specific types of feedback: defect reports and improvement requests. We approach this task as a binary document classification problem, meaning that each review is considered a document that requires two classifications, one for each feedback type. Using this setup it is possible to classify some reviews as both defect report and improvement request.

In terms of functionality, a clear distinction can be made between two primary subsystems of our framework, namely the Prediction Engine, and the Learning Engine. Even though both subsystems share numerous services from the framework core module, their purpose is very different. In Sect. 3.1 we present the Prediction Engine, and in Sect. 3.2 we describe the Learning Engine.

In order to evaluate the approach suggested in our framework, we performed experiments on a real-life dataset. The dataset contains 4470 reviews about Evernote, a mobile app for the Android platform. We automatically extracted the review dataset from the Google Play Store, through Web scraping techniques. We selected Evernote because it is a widely used app with a large user base, that publicly share their feedback on the Web, and therefore serves as a great example for our examined research problem.

We have annotations for 46% of the total review dataset. We hold out 20% of all reviews for testing purposes in all methods. Therefore, we have the remaining 26% of reviews available for training purposes. However, for the experiments that employ distant supervision, we generate noisy labels, hence, have 80% of the full review dataset available for training. In Fig. 1 we depict the distribution of both defects and improvements in the labeled collection of reviews. The terms “Positive” and “Negative” refer to the classification labels that were assigned to every review per feedback type by human annotators. On average 12.6% of our labeled set of reviews contains one or more actionable types of feedback, in which there are 8.4% more requests for improvement than defect reports. Finally, only 1.3% of our annotated reviews is labeled as both a defect report and an improvement request.

We collected annotations for both feedback types through CrowdFlower (recently renamed Figure Eight), an online data enrichment platform. The instructed task is to label every individual review for both defect reports and improvement requests. Every review was annotated by at least 3 annotators, and in some cases even 5 or 7 (when it is recorded a low accuracy of the test questions that inspect the quality of the annotator). In order to estimate the reliability of the aggregated annotations, we computed the inter-annotator agreement (IAA) for each question, using Fleiss’ Kappa measure. As can be observed in Table 2, the inter-annotator agreement for both questions is very high, which indicates that different contributors performed similar judgements consistently. We use the majority voting for building our gold standard.

The employed patterns are constructed both manually and automatically. In the Evernote dataset, there are proposed five manual and two generated patterns for defects, and eight manual and ten generated patterns for improvements. The most likely reason for this contrast is the variation in distribution of feedback types in our dataset, as a result of the fact that Evernote is a popular app, well tested, and optimised. Furthermore, we noticed that the most effective patterns only use function nodes of type Sequence and OR. Also, many examples of feedback can be recognized with a single terminal, such as the Entity Type “software update” for defect reports or the Literal “stars” for improvement requests, which indicates that the level of specificity does not necessarily have to be high. In that light, patterns that include the NOT node, which requires feedback examples in which a very specific word is not mentioned are often not necessary. While NOT functions can be useful to make a pattern very expressive and precise, it becomes obsolete when that level of selectivity is not required, as in our case. A similar line of reasoning can be applied to the AND functions. Table 3 lists two example of patterns for each pair (feedback type, pattern type).

To classify defect reports and improvement requests we test the following methods:

Table 4 displays an overview of performance measures of all proposed methods. We can notice that the Distant Supervision methods are not far behind the direct classification through patterns, in terms of $F_{1}$-scores. Nevertheless, given that the results are obtained with noisy labels shows the usefulness of this approach for datasets where the annotated data is limited.

As regards, the comparison between the two types of patterns, it is obvious that the automatically generated patterns perform better than the human ones. In order to have a complete insight over the pattern creation process (manual versus automated) we additionally explore the patterns’ efficiency besides their effectiveness. Table 5 displays the running time for creating patterns both manually and automatically. We can observe that it takes 70% less time to generate the automatic patterns than the manual ones.

In this study we presented a framework for automatically learning lexico-semantic patterns helpful for detecting specific types of feedback expressed in conversational customer feedback (defect reports and improvement requests). Using a custom dataset, we showed that the automatically generated patterns perform slightly better than the manual ones and there is a 70% reduction in construction time. Further on, we demonstrated that the distantly-supervised SVM with noisy labels is not far behind the pattern-based classification. The results reveals the applicability of this approach when the amount of available labels is limited.

As our future work, we would like to increase the flexibility of our patterns by considering more complex terminal structures. Using techniques from entity-learning we would like to explore the automatic generation of our domain-specific gazetteers lists to increase coverage and the framework applicability in other domains.