On the Identification of the Energy related Issues from the App Reviews
^††thanks: This study is sponsored by **** Lab of ***** ***** University, **********

A. Text Preprocessing
To avoid ambiguity, we have converted all the letters of the reviews into lower cases. We have applied this technique at the very beginning as lowercasing all the reviews (textual data) is one of the simplest and most effective form of text pre-processing. Although commonly overlooked, it helps significantly when it comes to consistency of expected output. If our dataset contains mixed-case occurrences of any given word (such as ”Battery”, ”battery”, ”BATTERY”), which it does, it is most likely to effect the features that we are planning to extract from the text reviews if we do not apply lowercasing. It is a great way to deal with sparsity issues when the dataset we are working on is rather small like ours. An example of mapping same words with different cases to the same lowercase form is provided: the following words ”POWER”, ”power”, ”Power” which appear in the raw review text are all converted into ”power” using lowercasing.

We have also masked account names, links, and hashtags using the following manner: whenever an account is addressed within the review using the “@” symbol, we took the adjacent word and replace the account name with “account”. We followed the same procedure for hashtags: any word adjacent to ”#” symbols was replaced by word ”hashtag”. For the links we have used a regular expression filter to find the links and replace it with the word ”link”.

We did not remove meaningless words such as ”lol”, ”fugly” since many of them are really popular among mainstream users, yet can not be found in traditional dictionary. These abbreviations, acronyms, and word combinations in reviews appears to be very important as written communication in instant messaging, text messaging, chat, and other forms of electronic communication such as user feedback seemed to have generated a “new language” and contain great weight when it comes to sentiment. [7]

After that, we have removed punctuation and stop words from the reviews. Stop words are a set of most frequently used words in a language. Examples of stop words that were removed from our reviews are ”i”, ”me”, ”my”, ”myself”, ”we”, ”our” etc. These words contain low information about the text and so by removing them we are enabling our features to focus more on the important words. Let’s consider an example in the context of one of our reviews: “It is frustrating… the app is always running in the background… draining my battery really fast”; we would want our features to focus on the following words ”frustrating app running background draining battery”. If we let our feature extraction procedure to analyze every word in the sentence, words with low information might through the generated features off of its true goal which is detecting energy related reviews. On top of that, preventing all stop words from being analyzed can help us minimize the number of features in consideration while keeping our models decently sized. Instead of replacing the stop words with a dummy character, we ennuled them entirely.

In the next step, we used Lemmatization which necessarily means mapping common words into their base forms. The objective of Lemmatization is to remove inflections and map a word to its root form. There is another approach for removing inflection: stemming which also reduces the word into its; root form but the “root” in this case may not be a base word, but just a canonical form of the original word. The only difference between stemming and lemmatization is the latter one tries to accomplish the task in the proper way. lemmatization does not just chop prefix/ suffix off, it actually transforms words to the actual root. That is why we have used only lemmatization for out purpose. For instance, the words ”good”, “better”, ”best” in the reviews would map to “good” using a dictionary, namely WordNet for mappings.

B. Feature Engineering
Text representation is a fundamental process for information processing which includes the tasks of determining the index terms for documents and producing the numeric vectors corresponding to the documents. Thus, we have extracted flat feature vectors from text reviews. We implemented Count Vectors, TF-IDF (Word level, N-Gram level, Character level) Vectors to obtain relevant features from our dataset.

Count Vectors counts the number of occurrences of each word appeared in a review. It is a matrix notation of the dataset in which every row represents a review from the dataset, every column represents a word from the dataset, and every cell represents the frequency count of a particular word in a particular review.

On the other hand, TF-IDF score represents the relative importance of a word in the review and the entire review- set. TF-IDF(Term Frequency- Inverse Document Frequency) score is composed by two terms: Term Frequency computes the normalized frequency of each word, and Inverse Document Frequency computes as the logarithm of the number of the reviews in the corpus divided by the number of reviews where the specific term appears.

TF(t) = (Number of times word t appears in a review) / (Total number of words in the document)

IDF(t) = ln(Total number of documents / Number of documents with term t in it)

These TF-IDF Vectors can be generated at different levels of input tokens such as words, characters, and n-grams

• •

  Word Level TF-IDF: Matrix representing tf-idf scores of every word in different reviews

• •

  N-gram Level TF-IDF: N-grams are the combination of N number of words together. This Matrix is representing tf-idf scores of N-grams. ngram_range is set as (2,3) for our experiment.

• •

  Character Level TF-IDF: Matrix representing tf-idf scores of character level n-grams in the review- set

We have adopted different variations of TF-IDF in our review classification based on the study that Zhang, Yoshida, and Tang presented on the performances of TF*IDF, LSI and multi-word in 2009 (performance were examined on the tasks of text classification). They have investigated the traditional indexing methods as TF-IDF, LSI (latent semantic indexing) with multi-word (which contains more contextual semantics than individual words and possesses favorable statistical characteristics). The performances of TF*IDF, LSI and multi-word were examined on the tasks of text classification, much like ours. Their experimental results demonstrated that TF*IDF and multi-word are comparable when they are applied for Text Classification and LSI was the poorest one of them. Even the rescaling factor of LSI had an insignificant influence on its effectiveness on text classification. [8]

In Table II, III and IV, we have denoted Count Vector, WordLevel TF-IDF, N-Gram TF-IDF, and CharLevel TF-IDF as (i), (ii), (iii), (iv) respectively. We have combined these features using horizontal stacking and denoted their combination with ’+’ sign in the tables.

C. Experiment Configuration
Each participating developer were instructed to train 3 traditional machine learning models: Naive Bayes, Linear Classifier (Logistic Regression), Support Vector Machine using their designated set of reviews. For each model, developers have tried different feature combinations (total 15 of them) as described in the previous section. We have used 4000 max_iter and Limited-memory Broyden–Fletcher–Goldfarb–Shanno algorithm (L-BFGS) as the solver for Logistic Regression. For SVM, gamma scaling was set to true. Python library Keras [9] was used for composing, training, and evaluating the mentioned models.

In these tables, we have reported the performance of 30 traditional machine learning models with engineered feature combinations using the mean of the following metrics: accuracy, F1-score, and run-time. Once the developers reported back their respective accuracy, F1-score, and run-time, we took the mean value for each case in the following manner. If reported accuracy, F1-score and run time for a specific model+feature is x1, y1, z1 (for developer-1); x2, y2, z2 (for developer-2); x3, y3, z3 (for developer-3); and x4, y4, z4 (for developer-4); then mean or average accuracy = (x1+x2+x3+x4)/4, mean F1-score = (y1+y2+y3+y4)/4, and mean Runtime = (z1+z2+z3+z4)/4.

We have investigated the deviation of each reported values from the average accuracy, F1-score, Run-time and found that for each case maximum deviation calculated were $\pm 0.02$, $\pm 0.021$, and $\pm 1.4s$ respectively. For every model, different combination of TF-IDF vectors prevailed yielding better performance with each of the traditional machine learning models.

Extreme Gradient Boosting: In traditional machine learning, ensemble methods are quite popular as they use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone [10, 11, 12]. A machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.

Boosting is a popular ensemble method which involves incrementally building an ensemble by training each new model instance to emphasize the training instances that previous models misclassified. In most cases of text categorization (like ours), boosting has been shown to yield better accuracy than bagging, but it also tends to be more likely to over-fit the training data. [13] To include ensemble method into our tool we have our developers train an Extreme Gradient Boosting Model (XGB) with all 15 feature combinations and presented the data for XGB in Table III.

We have tried better feature engineering for getting improved performance out of traditional machine learning models. For deep learning, we will try a text enrichment approaches such as using word embeddings.

A. Word Embedding
We leveraged pre-trained word embedding (an approach for representing words and documents using dense vector representation) to work with our Neural Network based models. Traditional bag-of-word model encoding schemes were the most popular and largely used before word embedding came into light. For a predefined vocabulary of fixed length (derived from a corpus of text such as our review-set), word embedding methods learn a real-valued vector representation where dense vectors, vectors represent words and the projection of the word into a continuous vector space respectively. Within the vector space, the position of a word is determined from text (specifically from the words that surround the word when it is being used). A word’s embedding is referred to as the position of a word in the learned vector space[14]. Three most popular examples of carefully designed methods of learning word embeddings from text include:

• •

  fastText[15]

• •

  GloVe [16], and

• •

  Word2Vec[17].

We could train word embeddings using the input corpus itself. Two most popular way to train our own word embedding are Learn it Standalone and Learn Jointly. The first one is a good approach for using the same embedding in multiple models where a model is trained to learn the embedding, saved and later used as a part of another model. To use the embedding on one task, second approach may be used where the embedding is learned as part of a large task-specific model. But learning a word embedding from scratch for our problem would require a large amount of text data (millions of documents containing large number of words) to ensure that useful embeddings are learned.

Word Embeddings could also be downloaded from previously trained models as it is common for researchers to make pre-trained word embeddings available (for free; often under permissive license). For instance, FastText, Word2Vec and GloVe word embeddings are available for free download. These word embeddings (trained on the same corpus: 1M wiki-news documents) were used on our project instead of training our own embeddings from scratch.

There are two popular options: Static and Updated when it comes to using pre-trained embeddings. For the first option the embedding is kept static and can be used as a component of our model and for the second option the pre-trained embedding is used to seed the model, and then the embedding is jointly updated during the training of our model. The second option suits our problem most as we want get the most out of our model and embedding to accomplish the task. [14]

Brief discussion about the models (that have been used to train our word embeddings) are given below.

Word2Vec[17] takes a large corpus as its input and produces a vector space (typically of several hundred dimensions) with each unique word in the corpus being assigned a corresponding vector in the space. Word vectors are positioned in the vector space such that words that share common contexts in the corpus are located close to one another in the space. In 2013, after Mikolov, Chen, Corrrado, and Dean published [18], the avalanche of word embeddings began. The proposed approach (Word2Vec) uses small neural networks to calculate word embeddings based on the context of the words. The authors of the paper published two approaches to implement this: Continuous Bag of Words (CBOW) and skip-gram. In the first approach, for the given context, the network tries to predict which word is most likely to occur/appear. The network predicts a similar probability for words that are equally likely to appear (which can be interpreted as having a shared dimension). For the second approach, the network works the other way around, that is, uses the target word to predict its context. The results of Word2Vec were unprecedented but also hard to explain from a theoretical point of view.

GloVe [16] is an unsupervised learning algorithm which obtains vector representations for words by mapping words into a meaningful space where the distance between words is related to semantic similarity. One year after the publication of Word2Vec[18], researchers of Stanford published GloVe[19]. Glove is deemed as a slight variation of Word2Vec which attempts to benefit from a less obvious aspect of Word2Vec. Word2Vec learns embeddings by relating target words to their context, ignoring co-occurrences of some context words as if the frequent co-occurrence of words only helps it creating more training examples, but carries no additional information. On the other hand, GloVe accentuates on the frequency of co-occurrences and takes it as vital information rather than wasting it as mere additional training examples. So, GloVe focuses on building word embeddings in such a way that the probability of words’ co-occurrence in the corpus relates directly to a combination of corresponding word vectors. In brief, its embeddings can be interpreted as a summary of the training corpus with lower rank in dimensions that associates co-occurrences.

fastText[15] is formally a library for learning of word embeddings and text classification created by Facebook’s AI Research (FAIR) lab[20, 21, 22, 23]. The model allows to create an unsupervised learning or supervised learning algorithm for obtaining vector representations for words. Facebook makes available pretrained models for 294 languages.[24] fastText uses a neural network for word embedding.is an unsupervised approach to learn high dimensional vector representations for words from a large training corpus where the vectors of words that occur in a similar context are close in this space.[25] fastText’s kept its’ base idea pretty similar to Word2Vec where instead of using words to build word embeddings, it goes one level deeper and takes characters, part of words as the building block for the embedding so that a word becomes its own context. Word embeddings generated by FastText and Word2Vec are almost similar although they are not calculated directly. Instead, they are generated as a combination of lower-level embeddings. Two main advantages to this approach are generalization and necessity of less training data. Generalization comes to play when there are new words which have the same characters as known ones. Additionally, we need less training data as much more information can be extracted from each piece of document.

For our study purpose, we downloaded three word embeddings; all of three embeddings were generated through training same corpus of 1 million wiki-news with three aforementioned models: fastText, Glove. Word2Vec. To make each of these word embeddings applicable with Neural Network based models, we implemented following procedure: first we loaded a pre-trained word-embedding vectors in order create a tokenizer, then converted text to sequence of tokens, padded them to ensure equal length vectors, and at last created token-embedding mapping in the form of embedding matrix.

B. Neural Network Architectures
Traditional machine learning models depends on a data representation relying upon hand-crafted features, chosen by users or domain experts, based on its’ usefulness for the classification problem at hand. On the other hand, neural networks (used in deep learning approaches) learn high-level feature representations automatically by using raw textual data as input. Researchers have previously applied Convolutional Neural Network (CNN) successfully to natural language processing problems such as text classification[26]. Usually, deep learning approaches require a large amount of training data to outperform traditional machine learning approaches[27]. As our dataset is rather small for conventional deep learning approaches, we are trying to leverage pretrained word embeddings as discussed in the previous section.

Figure 2 shows a simple architecture of neural network which was used by Haring et al. [28] to identify user comments on online news sites that address three different classes: media house, journalist, or forum moderator. We are going to use this architecture and its’ different variation for empirical study purpose.

The input layer for this architecture requires its’ textual inputs to be of fixed size; so, we searched for the longest review in our dataset and assigned its’ length as the fixed size of input for the specified layer. To make the other reviews reach the required length for the input layer, we have padded them. Right after the input layer, our network consists of an embedding layer which contains the pre-trained word embeddings: we have assigned the embedding matrix (discussed broadly in the previous section) as the weights of this embedding layer and set the trainable parameter for this layer to be false so that the weights of the embedding layer stays frozen during training. After this embedding layer, our network consists of a 1D convolution layer, a 1D global max pooling layer, a dense layer with a concluding output layer. We have used ReLU (Rectified Linear Unit) as the activation function for convolutional layer and sigmoid function for the dense output layer.

In Table IV, output of this architecture is denoted by CNN. We have swapped the convolution and max pooling layer of described architecture with layers of RNN- LSTM (Recurrent Neural Network– Long Short Term Memory), RNN- GRU (RNN- Gated Recurrent Units), Bidirectional RNN, and RCNN (Recurrent CNN) to try out different variants of this architecture. Participating developers trained the models with a batch size of 32 and 10 epochs. The developers experimented with three mentioned word embeddings and reported the results. For comparison purpose, we also reported the performance for S- NN (Shallow Neural Network) trained with different feature combinations engineered for traditional machine learning models.

The detailed evaluation is presented in Table IV. Similar as before, the average accuracy, F1-score and Run-time for 4 of our participating developer is reported for each case.

C. Data Visualization
We developed an interactive dynamic tree where each node in the first, second and third layer represent different models, feature, and result-data (accuracy, F1-score, run time) respectively. Each node can be dragged, zoomed and panned. This part of the tool was developed so that developers may traverse through the resultant data set with ease. In figure 3, we have only shown up to the first level of model selection where each node can expand into different number of second level nodes to accommodate user to select different features and generate the results.

D. Runtime- Performance Trade off
Figure 4 shows the average run time for all 90 cases described in the previous traditional Machine Learning and Deep Learning sections. In the x- axis, all the model feature combinations are presented in the ascending order (left-to-right) of their F1-score performance. Here, we observed that although Neural Network based models are outperforming traditional machine learning models accuracy-wise, these NN based models require significantly greater run-time than those of traditional ML models. Run time is almost perceived as an opportunity cost for accuracy or F1-score. When it comes to industrial implementation, developers may choose to use a model+feature combination that might yield sub-optimal accuracy but run faster than other choices.

E. Validity of the approach
To ensure that the designed approaches and the result is valid and analogous to other binary classification cases, authors performed a similar experiment to compare the outcomes without developers involvement. We selected 600 security related application reviews and 600 non- security related reviews and applied the same approach developers used for automatic extraction of the energy related reviews. All the 90 instances (60 traditional machine learning approach, 30 Neural Network based approach) were observed but no significant difference in the accuracy, F1-score and run time. Maximum difference reported is $\pm.03$ for both accuracy and F1-scores. For run-time maximum difference recorded was $\pm 3.9s$. For brevity, we have reported 32 most prominent cases out of 90 in Table V.

Three techniques used for identification of emerging issues from already separated energy related reviews are briefly discussed here.

A. String Matching

The most used technique to categorize user reviews is to check whether it contains a certain key- word. Developers initially predict what issues might emerge and define a list of keywords to search the reviews using ”LIKE” in SQL query. The query in turn return the reviews containing the defined key- words. We compiled the keywords from the literature [29] and used for the string matching classifier. Some of the keywords are as following: battery, energy, power, charge, affect, consume, hog, deplete, discharge, drain, kill, devour, hot, heat, slow, consume, leak and 24 more. This technique is widely used but it cannot identify any energy related issues unforeseen by developers.

B. Adaptive Online Latent Dirichlet Allocation [6]

This is an online Variational Bayes algorithm developed for LDA that can be trained in small batches. While implementing Online LDA, we only need to hold a very small subset of the dataset in memory at a given time.[30]. According to [6], Adaptive Online LDA finds topic models better and faster than those found with batch VB. The batch VB algorithm assumes that user (in our case, developers) would have the entire training set at the start of training, rendering the approach ineffectual for our experiment as app developers would want to train the model incrementally. AOLDA supports incremental training so that developers can train on a chunk of reviews, then resume training if they receive more reviews from the user (practical for our purpose) without have to retrain on the original chunk of reviews. As it also works good with short text data, we have integrated it in our experiment.

C. Online Biterm Topic Modeling Algorithm

OBTM is inspired by the online LDA algorithm proposed in [30], which assumes documents are divided by time slices; and within these time slices, the documents are exchangeable. OBTM tries to fit a BTM model over the data in a time slice and use the counts in current time slice to adjust the hyper parameters for the next time slice. Although we could not use it with the streaming data as conveniently as AOLDA, we have integrated it into our experiment as according to [5], OBTM out performs LDA, iLDA, and iBTM for topic modeling from a set of short text documents.

For Topic modeling part, each developer worked on their unique dataset but only for energy related reviews (each developer originally had 600 exclusive energy related reviews). At first, each of them defines some keywords based on the prediction about probable issues and uses them for string matching (process defined in String Matching section); number of identified issues from this process is reported in the 3rd column of Table VI. In the second phase, we manually investigate the true number of issues appeared in reviews and for each developer’s review set, the number of true reviews discovered is specified in column 2. Let’s assume, k number of issues were discovered for developer-A’s review- set. Then for the third phase, developer- A deploys OBTM, AOLDA with an instruction to generate k number of topics. ’k’ is chosen as threshold since we wanted to find out how many (out of the total number of actual issues=k) issues we can identify from the k number of topics generated by OBTM and AOLDA. After that developer- A will define/label the generated topics and try to match them with true topics identified in second phase. Recorded number of matched issues are reported in 4th and 5th column of Table VI.

Here, Developer-1 had identified the following 4 issues: (i) User Interface, (ii) Extraneous work, (iii) Defective Task Allocation, (iv) App Dependency. String matching could discover only (ii); aoLDA discovered (i), (iii), (iv); and oBTM also discovered (i), (iii), (iv).

Developer-2 had identified the following 3 issues: (i) Power Save mode (ii) Internet Connection, (iii) Phone Heat up. String matching could discover only (iii); aoLDA discovered (i), (iii); and oBTM discovered (i), (ii).

Developer-3 had identified the following 5 issues: (i) Slow Animation, (ii) High resolution Videos, (iii) Defective Task Allocation, (iv) No power awareness, (v) Cache mismanagement. String matching could discover only (i), (ii); aoLDA discovered (i), (ii), (iv); and oBTM discovered (i), (ii), (iii), (iv).

Developer-4 had identified the following 2 issues: (i) Internet Connection , (ii) App dependency. String matching could discover only (i); aoLDA discovered (i); oBTM discovered all of them to some extent.