Evaluating Software User Feedback Classifiers on Unseen Apps, Datasets, and Metadata

The benefits gained from RE has been shown to be integral to software quality. Many models exist within the literature that include RE to improve software quality berki2004requirements ; broy2006requirements . The practice of RE has also been shown to be beneficial, with Damian and Chisan demonstrating that productivity, quality, and risk management were all improved when effective RE was done within a commercial software project damian2006empirical . Similarly, Radliński showed that multiple RE factors had a positive, statistically significant effect on software quality factors within a literature dataset of thousands of software projects radlinski2012empirical . A survey of developers also showed that teams that used RE approaches were much more likely to say that their product’s capabilities fit their customers’ needs well and that end users found their products easy to use than those which did not kassab2014state . Other RE-adjacent concepts such as requirements traceability have also been shown to positively impact software quality rempel2016preventing .

Machine learning has become a common tool used within the requirements engineering literature for supporting the creation of requirements. Approaches like the one proposed by Cleland-Huang et al. cleland2007automated have been made to integrate automated text classifiers into the requirements engineering process. The MARA model, developed by Iacob et al. iacob2013online focuses on developing requirements from the ingestion of online user feedback using such a text classifier. These classifiers have become prevalent throughout the literature, with a systematic literature review by Lim et al. showing that 38 out of 63 studies which did user feedback analysis based on manually labelled data used machine learning to classify this feedback lim2021data . One of the potential reasons for this popularity is the reported high classification performance of some of these machine learning models.

Work from Maalej et al. maalej2016automatic , Panichella et al. panichella2016ardoc , and Stanik et al. stanik2019classifying all report bug report classification F1 scores of app reviews higher than 0.75, with Maalej et al. reporting as high as 0.9. However, Stanik et al. also report a bug report classification score of 0.59 in Tweets, with Nayebi et al. also reporting lower feature request classification F1 scores of 0.67 nayebi2018app . These diverse values highlight that the expected classification performance can vary dramatically depending on data source, classification method, and evaluation method. This underscores the need to rigorously compare and standardise techniques for both training and evaluating classification models.

There are several studies within the literature that compare techniques used for classifying user feedback. Work by Aurajo et al. evaluated the performance of four classical machine learning classifiers on classifying user feedback from one dataset using both term frequency derived features and features from deep pre-trained language models, showing that deep pre-trained language models generate superior text embedding features compared to frequency-based features for classification araujo2020bag . Similarly, Henao et al. demonstrated the increase in performance in user feedback classification when using pre-trained language models over both classical models as well as other deep models henao2021transfer . Hadi and Fard proposed a study where the classification accuracy of pre-trained language models is compared against that of previously constructed classifiers from the literature as well as exploring the effect of self-supervised pre-training, binary classification, multi-class classification, and zero-shot settings on classification performance hadi2021evaluating . Dhinakaran et al. showed that models trained on training data that was chosen randomly were found to consistently underperform more sophisticated training data selection techniques, such as active learning dhinakaran2018app . Di Sorbo et al. investigated the correlation between app review rating and feedback type classifier prediction, finding that predictions of “problem discovery” from the ARDOC classifier were negatively correlated with the app rating, whereas predictions of “feature request” were uncorrelated di2021investigating . As can be seen, there has been extensive work evaluating which text-based features and machine learning models are best to use when classifying user feedback. Some work has also been done to improve the data-efficiency of training a classifier. What remains unclear is how different training and evaluation methods affect the evaluation result of these classifiers (particularly on out-of-domain data), and how features apart from text affect classification performance.

This study adds to the literature by exploring the effect of several machine learning techniques to highlight where and when user feedback classifiers can and cannot be used in the real world. Firstly, evaluating on seen and unseen app reviews is evaluated, in order to determine how well user feedback classifiers perform in classifying feedback for an unseen app. Secondly, classifiers trained and tested on separate datasets are evaluated so as to determine how well classifiers can be applied to similar data. Finally, the use of metadata for classification across multiple domains is evaluated to determine its effect on user feedback performance.

Within this study, the performance of classifiers on feedback from apps that it had not been trained on is analysed, and as such, the information as to which app a piece of feedback came from is needed for each dataset. For this reason each dataset was cleaned so that any rows which did not have an app identifier (i.e. are null) were dropped. This only affected dataset C, where some rows did not contain app identification data, and accounted for 1,821 out of 3,259 (55.9%) rows within that dataset, leaving 1,438 pieces of feedback from that dataset to use in our experiments.

Two different configurations of generating train, validation and test data splits were then used for each dataset, and these were named “mixed” and “separated”. The “mixed” data splits were created by randomly splitting all feedback across the dataset into a 64:16:20 (train : validation : test) split. These ratios were chosen to be an initial 80:20 (train+validation : test) split before splitting the train + validation set with a further 80:20 split. This random splitting of data into train, validation, and test sets is current standard practice throughout the user feedback classification literature.

The “separated” data splits were generating by randomly sampling apps within a dataset and assigning them to either the train, validation, or test split. All feedback for a given app was put into the same data split. Again, we aimed for a 64:16:20 (train : validation : test) ratio while ensuring each of these splits contained data from different apps. “Separated” data splits were not created for Dataset F because it included only two apps, and so contained too few distinct apps to split into train, validation, and test sets. A visual depiction of how these two configurations were created can be seen in Fig. 1.

The “mixed” and “separated” configurations were achieved by using Sci Kit Learn’s ShuffleSplit and GroupShuffleSplit respectively. Cross validation was used to generate 5 distinct data folds for each dataset, and reported metrics in our results are the mean over these 5 folds.

Each dataset was evaluated using the F1 metric. The equation for this metric can be found in equations 1, 2, and 3. This metric provides a good measure of how well a class is being correctly labelled due to it balancing recall and precision, and is less sensitive to class imbalances in the data compared to the accuracy metric.

Statistical significance between the performances of different classifier types were determined by using an independent two-sample t-test on the F1 metrics all folds of cross validation for one given test dataset.

Table 4 details the mean F1 scores of classifiers from RQ1 for classifying bug reports and feature requests.

For the models which were trained on only one dataset, it can be seen that app-separated splits have a lower F1 score than mixed-app splits for 5 out of the 6 bug report datasets and 4 out of the 5 feature request datasets. For bug reports, two of these differences (B and E) were statistically significant to p<0.05, while for feature requests, two (E and G) were significant. Moreover, we see that only one of these differences is significant to p<0.01 over these 11 differences.

Answer to RQ1 - How does training and testing on feedback from separate apps affect classification performance? In aggregate, a minority (4/11) of datasets exhibited statistically significant difference between being split along app lines compared to being split randomly. Training and testing a user feedback classifier on feedback from the same apps does not result in meaningfully different performance compared to training and testing on feedback from separate apps in a majority of cases.

Table 5 and Table 6 details the mean F1 scores of separate-dataset classifiers from RQ2 for classifying bug reports and feature requests, respectively.

As can be seen for both bug report and feature request classification, F1 score for any one given test dataset can vary wildly depending on the dataset of the training data. For bug report classifiers, we observe that the classifier trained only on dataset G training data (tweet user feedback) performs best on 4 of the 6 test datasets it is applied to. For feature request classifiers, the classifier trained only on dataset B (app review data) performs best on 4 of the 5 datasets it is applied to. The classifier trained in dataset F (forum data) performs worst on all test datasets it is applied to compared to other classifiers for bug reports. The classifier trained in dataset E (app reviews) performs worst on all test datasets for feature requests. Overall, every classifier trained on one dataset and evaluated on a separate dataset achieves lower classification performance compared to models trained and tested on the same dataset.

In comparison to the models trained on only one (different) dataset, the leave-one-out classifier performs best on 6 of the 7 bug report datasets and 5 out of 6 feature report datasets that it is applied to. Compared to the model trained and tested on the same dataset, the leave-one-out classifier performs slightly worse on all but one dataset.

The zero-shot classifier performs better than any of the single-dataset out-of-dataset for 5 out of the 7 bug report datasets and 4 out of the 6 feature request datasets. The zero shot model exceeds the performance of the leave-one-out models for the test sets of Dataset F and G (forum posts and tweets) for bug reports, and 4 out of 6 datasets (C, E, F, and G) for feature requests. Therefore, zero shot models perform best relative to other models on datasets of distinct feedback platforms.

Answer to RQ2 - How does training and testing on data from separate datasets affect classification performance? Training and testing a user feedback classifier on feedback from separate datasets results in overall lower performance than training and testing on the same dataset. However, this lower performance can be improved upon by models trained on multiple datasets or by zero-shot text classification models.

Table 7 details the mean F1 scores of RQ3 classifiers both including and excluding metadata features to classify feedback into bug reports and feature requests.

For bug reports, we find all datasets have higher F1 scores when metadata and text is used to classify compared to when only text is used. However, only one of these differences (dataset B) increases between models were statistically significant with a p-value of <0.05.

Similarly for feature requests, we find that for 5 out of the 6 datasets studied, models which use metadata and text perform better than just using text. Again, only one (datasets F) of these differences were statistically significant to a p-value of 0.05.

Overall, we can see either a slight increase or no change in the performance of classifiers when metadata and text are used together compared to when text alone.

Answer to RQ3 - How does training and testing with metadata affect classification performance? Training on metadata results does not result in a statistically significant increase in classification performance on the majority of datasets tested.

From our results, we found little difference between evaluating a classification model on feedback from unseen apps compared to evaluating on feedback from the same apps that it was trained on. This finding is made across both models classifying bug reports and feature requests. This result suggests that model evaluation as is currently carried out within the literature (i.e. not specifying that train, validation, and test splits must contain feedback from separate apps) can be seen to be a good predictor of performance of a classifier on unseen apps from within a dataset. This hints at potential real-world applicability of these models in that they could be used on feedback from unseen new apps (but crucially from the same platform and data-gathering process) without an expected drop in performance.

Another outcome of these experiments is that the classification F1 score can range from high (greater than 0.8) to low (less than 0.5). When a classifier has low absolute classification performance, their utility in finding requirements is limited. This finding highlights the fact that automatic classification using current technology is not universally useful across all feedback datasets. The fact that many of these values are slightly lower than their literature quoted values could possibly be due to the fact that we decided against doing extensive hyperparameter tuning when training our classifiers. Reasoning and discussion of this is given in Section 7. Finally, the lower classification performance across most of the datasets for classifying feature requests compared to bug reports is a trend that can be broadly seen throughout the literature, and calls into question exactly why a bug report is so much easier to identify (from a machine learning perspective) compared to a feature request.

In RQ2, we found that a model trained on one dataset and then applied to another dataset achieves worse performance than a model trained and tested on the same dataset. This is not a surprising finding, given that class balance and labelling methods vary slightly between datasets. However, it raises an important question: How informative are the predictions of these models when used in the real world? A dataset of user feedback for a given software project is not guaranteed to have a certain class balance, and a given researcher or developer is not guaranteed to consider a piece of feedback to contain a bug or feature request in the same way that the training data labellers did.

Our results with the leave-one-out models show better performance, in contrast. While the leave-one-out models perform worse than models trained and tested on the same data, they perform better and are more consistent compared to models trained on one dataset. The leave-one-out models also perform better compared to the zero shot classifier except for feedback from an unseen platform (tweets and forum posts) at training time. This indicates that while leave-one-out classifiers are useful, zero shot classifiers are more appropriate for classifying feedback from unseen feedback platforms.

These results suggest that user feedback analysis tools will achieve highest performance if before use they first require a sample of labelled user feedback from the developer who intends to use the tool (i.e. use in-domain training data). This could be done through an active learning approach, such as was explored by Magalhães et al. magal , in order to limit the amount of labelled data required. However, a tool that requires no further labelled data before being used can still achieve good performance if it is either: trained using a labelled feedback dataset from the same feedback platform; or a zero-shot text classifier if no such dataset exists.

With these results, we recommend that future creators of user feedback analysis tools train a classification model using as much labelled user feedback as possible, especially using data from the same platform as its intended use-case. If such a dataset does not exist and is prohibitively expensive to create, then we recommend using zero-shot classification models instead.

In order to aid future user feedback analysis tools, we make bug report and feature request classifiers available for use on the Huggingface platform. We aim to make these available with a link upon publication.

Our findings for RQ3 are that classification performance is modestly, but not significantly, improved when using metadata. This finding is in contrast to previous findings which reported the use of metadata on feedback classification, in which metadata was shown to have a positive impact on classification performance maalej2016automatic ; tizard2019can . We theorise that this may be due to the fact that the state-of-the-art classifiers that we used contain millions of parameters devlin-etal-2019-bert , compared to very few parameters available in the classical machine learning models used in these earlier works. With this increased capacity, our model may be better able to infer metadata from the text itself (for example low review ratings would also be associated with more negative sentiment text), which means that having this information explicitly provided would not have much of an effect on the final prediction. We therefore recommend that the use of metadata as a feature should be reviewed within text-based software engineering machine learning tasks with the advent of new, very capacious language models such as BERT. Without the use of what appears to be largely superfluous metadata, these models are better able to be applied to different feedback and to new feedback platforms, where metadata may differ.

We have shown that metadata does not affect performance significantly across a majority of datasets in the classification of bug reports and feature requests, but it is an open question as to how metadata would affect other classes of feedback. It is for future work to investigate a fuller picture as to which classes benefit most and least from use of metadata in their prediction.