Prioritize Crowdsourced Test Reports
via Deep Screenshot Understanding

Nowadays, apps support different themes, making it possible for users to customize the app appearance according to their preferences (Fig. 1). Moreover, the supported “dark mode” makes the color scheme more complex. Image feature extraction algorithms can hardly handle such complexity and will make mistakes. From the examples, we can find that the app screenshots in three reports are of blue, white, and green themes. All these three reports are reporting the loading failure of the music resource files. However, according to [2], the image color feature is one vital component of the report surrogate. App screenshots with different colors will be recognized as different screenshots.

Report #1135: Choose a directory containing music files and “mark as music dir”, but music files do not show when returning to main page. Report #1128: The song list cannot show the music files. Report #1163: The page shows “No media found” after choosing the music library.

As shown in Fig. 2, the two reports use the screenshots of the same app activity, and the image feature extraction algorithm will assign a high similarity between these two screenshots. However, according to the bug description, the two reports are describing completely different bugs. In DeepPrior, for Report #1128, we can extract the text “no media found”; for Report #1127, we can extract the volume widget besides the prompt information, and DeepPrior can identify the different problems.

Report #1127: When the headphones are plugged in, the volume is automatically increased while playing. Report #1128: The song list cannot show the music files.

As shown in Fig. 3, the ImageView widget on the top is of different contents, and it occupies a large proportion of the entire page. Also, the comments are different due to different testing time. Therefore, existing approaches will consider the two screenshots are of low similarity which will pull down the whole similarity even if the textual descriptions are with high similarity. With DeepPrior, we can extract the pop-up information on the bottom, saying “comment failed”, and assign a high similarity to the two reports. Such pop-ups are considered as a quite significant widget that contains the bug.

Report #25: The page reminds failure after submitting comment, but the submission shows in the list. Report #146: When inputing the comment, the page reminds submit failure, but when reentering the page, the comment has been in the list.

The first and most important step is feature extraction. In this step, we analyze the app screenshots and textual descriptions in the crowdsourced test reports separately.

After acquiring all the features both from app screenshots and textual descriptions, we aggregate them into two feature categories: Bug Feature (BFT) and Context Feature (CFT). Bug Feature refers to the features that directly reflect or describe the bug in the crowdsourced test report, while Context Feature is assembled by the features that provide a depiction for the environment of the bug occurrence.

To prioritize the crowdsourced test reports, one significant step is to calculate the similarity among all the reports. Because we are the first to introduce the deep screenshot understanding into report prioritization, we name the similarity as DeepSimilarity. As the common practice to merge different features in previous studies [2][3], we calculate the DeepSimilarity of different features separately, and allocate different weights for the results of different features. The formal expression is as follows ($Sim$ is short for similarity):

After aggregating the DeepFeature, and defining the DeepSimilarity calculation rule, we start to prioritize the crowdsourced test reports. First, we construct two null report pools: the unprioritized report pool and the prioritized report pool. All the crowdsourced test reports are put into the unprioritized report pool initially.

Different from the strategy adopted in [3], where a report is randomly chosen as the initial report, we think all reports should be treated equally, and the randomly chosen report is likely to affect the final prioritization. Therefore, to formalize and unify the prioritization algorithm, we introduce the concept of NULL Report, which also contains four features.

• •

  Problem Widget: the screenshot of the problem widget is essentially a 3-dimensional matrix representing the width, the height, and three color channels. Therefore, we construct the problem widget as a zero matrix. The width and the height of the zero matrix are set as the average size of all the actual crowdsourced test reports. Intuitively speaking, it is an all-black image.

• •

  Bug Description: the bug description of the NULL Report is directly set as an empty string, and since the string length is 0, obviously it contains no words, after the Word2Vec processing, the feature vector will be a 100-dimensional vector of all ‘0’s.

• •

  Context Widget: for the context widget of the NULL Report, we directly construct the vector representing the numbers of the 14 different types of widgets, and all elements are 0. This represents that there are “no” widgets on the app screenshot of the crowdsourced test report.

• •

  Reproduction Step: the reproduction step of the NULL Report is also set as an empty string, and the “action-object” sequence is also with a length of 0.

The primary consensus for prioritization is to reveal all the bugs as early as possible under the circumstances when some reports would describe the problems repetitiously [3][13][14].

Therefore, it is important to provide as many reports describing different bugs as possible for the developers early. Based on this idea, we design our prioritization strategy as follows, and the formal expression is presented in Algorithm 1.

First, we construct the NULL Report according to the rules. mentioned above and append the NULL Report to the null prioritized report pool. Following is a iterative process. We calculate the DeepSimilarity of each unprioritized report to the whole prioritized report pool, which is defined as the minimum DeepSimilarity of the unprioritized report to all the reports in the prioritized report pool. The report with the lowest DeepSimilarity with the prioritized report pool will be moved to the prioritized report pool.

To evaluate the proposed DeepPrior, we design an empirical experiment. To complete the experiment, we collect 536 crowdsourced test reports from 10 different mobile apps (details in TABLE I). The apps are labeled from A1 to A10, and the number of test reports of different apps ranges from 10 to 152. We also invite software testing experts to manually classify the test reports according to the bugs they describe, and the average number of reports in a bug category is 8.06.

Based on the crowdsourced test report dataset, we build three specific datasets to better support the evaluation, including 1) a large-scale widget image dataset, 2) a large-scale test report keyword set, and 3) a large-scale text classification dataset. All the 4 dataset build up the integrated dataset group.

In total, we design three research questions (RQ) to evaluate the proposed test report prioritization approach, DeepPrior.

• •

  RQ1: How effective can DeepPrior identify the widget type extracted from the app screenshots?

• •

  RQ2: How effective can DeepPrior classify the textual descriptions from the crowdsourced test reports?

• •

  RQ3: How effective can DeepPrior prioritize the crowdsourced test reports?

The first research question is set to evaluate the effectiveness of our processing to the app screenshots. The most important component of app screenshot processing is the widget extraction and classification. Therefore, we evaluate the accuracy of the widget type classification CNN. 36,573 different widget images are collected from real-world apps, and the images have an even distribution in 14 categories.

The details of the CNN is presented in Section III-A1. The dataset is divided by the ratio 7:2:1 into the training set, the validation set and the test set, as the common practice. After the CNN model is trained, we evaluate the accuracy on the test set. The overall accuracy of widget type classification reaches 89.98%. Specifically speaking, we use precision, recall and F-Measure values to evaluate the network. The calculating formula can be seen as follows, where TP means true positive samples, FP means false-positive samples, TN means true negative samples, FN means false negative samples.

The evaluation results can be seen in TABLE II. The precision value reaches an average of 90.05%, and the lowest precision is 74.36% and the highest is 99.81%. For recall, which measures the total amount of relevant instances that are actually retrieved, the average values is 89.98%, and the recall values range from 70.83% to 100.00%. F-Measure is a harmonic mean of the precision and recall, and it reaches an average of 89.92%. The above results reflect the outstanding capability of the proposed classifier.

We also have an in-depth insight into the results. We find that there are two groups of widgets that are easy to be confused. The first group includes ImageButton and ImageView. It is easy to understand that from a visual perspective, these two types can hardly be identified. The only difference between these 2 types is that ImageButton can trigger an action while ImageView is a simple image. However, one important thing to mention is that the in app design, developers can add a hyperlink to the ImageView widget to realize the equivalent effect. The second group includes Button, EditText and TextView. These three widgets are all a fixed area containing a text fragment, which is also visually similar and hard to identify even for humans. Moreover, some special renderings make the widgets even harder to identify. According to our survey, we find that these two confusing groups will not affect much, and the widgets can be treated as equivalent from both visual perspective and function perspective.

Results for RQ1: The overall accuracy of the CNN to classify the widget types reaches 89.98%. For each specific type, the average precision is 90.05%, the precision is 74.36% and the F-Measure is of 89.92%. Also, according to our survey on real test reports, even if some types that are with low precision, their visual and functional features will not negatively affect DeepPrior.

In the processing of the textual description, we classify into two categories: bug description and reproduction step. Different textual descriptions are considered as different report features. To classify the textual descriptions, we segment the textual descriptions into sentences. Then, we feed the sentences into a TextCNN model to complete the task, and the details are presented in Section III-A2. Also, to better train and evaluate the network, we build a large-scale text classification dataset. The dataset contains 4,340 pieces of labeled textual segments, including 2,252 pieces of bug descriptions and 2,088 pieces of reproduction steps. The dataset is divided into training set, validation set and test set at the ratio of 7:2:1.

The results on the test set can be seen from TABLE III, and the overall accuracy of the model reaches 96.65%. More specifically, we use the same measurements like the evaluation in RQ1, including precision, recall and F-Measure. The average precision of 2 types of textual description reaches 98.21%, the average recall reaches 98.17%, and the he average F-Measure reaches 98.19%. The result is quite promising, and we also manually check the textual descriptions. We find that compared with reproduction steps, bug descriptions tend to contain bug-related words, such as “crash”, “flashback”, “missing element”, “wrong”, “fail”, “no response”, etc. While the reproduction steps contain just the operations, target widgets and the corresponding responses.

Results for RQ2: The overall accuracy of text classification reaches 96.65%, and the precision, recall and F-Measure are all over 98%. Such results show DeepPrior’s excellent capability to analyze textual descriptions, which also lays a solid foundation for the crowdsourced test report prioritization.

In this research question, we evaluate the test report prioritization effectiveness of DeepPrior. The metric we use is the APFD (Average Percentage of Fault Detected) metric [15], which is also used by Feng et al. to prioritize crowdsourced test reports [3]. In the formula, the $T_{fi}$ means the index of the report that first finds the bug $i$, the $n$ is the total report number, and the $M$ is the total number of revealed bugs.

To better illustrate the advantage of DeepPrior, we compare DeepPrior with the following prioritizing strategies:

• •

  Ideal: This strategy is the best prioritization on theory, which means that developers can review all the bugs revealed by the reports in the shortest time.

• •

  Image: This strategy uses only the results of deep image understanding of DeepPrior to rank the test reports because deep image understanding is a significant part of our research.

• •

  BDDiv: This strategy refers to the algorithm in Feng et al.’s work [3], which is also the state-of-the-art approach for crowdsourced test report prioritization.

• •

  Random: The Random strategy refers to the situation without any prioritization strategy.

For DeepPrior and Image strategy, we run once because of the stability of our approach, and the trained model will not produce different results for various attempts; for Ideal strategy, we manually calculate the APFD because for a fixed report cluster, it is a fixed value; for BDDiv strategy, we run 30 times and calculate the average value as the original paper [3]; and for Random strategy, we run 100 times to eliminate the effect of the occasional circumstances.

First, we compare DeepPrior with Random strategy. As shown in the TABLE IV, we find DeepPrior outperforms Random strategy much, ranging from 15.15% to 38.93%, and the average improvement reaches 27.04%. This shows a superiority of DeepPrior.

Then we compare the results of DeepPrior with the single Image strategy. The average improvement of DeepPrior is 4.54%, and in 2 apps (A3 and A4), DeepPrior outperforms much. For A8, DeepPrior is slightly weaker than Image strategy. We review the reports of A8 and find that the textual descriptions are not well written, and cannot positively help the prioritization of the reports. Generally speaking, the results prove the necessity of combining both text analysis and deep image understanding, and a single strategy will compensate each other’s drawbacks and improve the prioritization accuracy.

Also, we make a comparison between DeepPrior and BDDiv strategy, which is the state-of-the-art approach. According to the experiment results, DeepPrior outperforms BDDiv with an average improvement of 3.81%. The improvements in some apps are especially obvious. Moreover, we record the total time overhead from reading the report cluster to the output of the prioritized reports. It shows that DeepPrior uses less than half of the time of BDDiv, which shows great performance superiority.

Another advantage of DeepPrior over BDDiv is that the DeepPrior can output stable results while BDDiv’s results will float. According to the detailed results of BDDiv strategy (in the online package), we find that BDDiv is quite volatile.

The improvements over baseline strategies vary among different apps, and some reasons account for this. First, the “report/category” rate for each app is different, so in the limited activity set of an app, the recurring of the same activities become frequent. Second, different apps have various contents. For example, A1 is a kids’ education app, and it consists of a large number of pictures, videos, variant texts. Such a situation makes it much more complex to extract useful text information and have a thorough understanding of the app screenshot. As a result, the matric would decrease.

Results for RQ3: DeepPrior’s capability of prioritizing crowdsourced test reports is excellent, it outperforms the state-of-the-art approach, BDDiv, with less than half the overhead. Also, the specific experiment of the Image strategy shows the effectiveness of our deep screenshot understanding algorithm. Compared with the state-of-the-art approach, DeepPrior performs much more stable.

The categories of the apps in this experiment are limited. Our 15 experimental apps cover eight different categories (according to app store taxonomy). The coverage is limited. However, we want to emphasize that due to our deep screenshot understanding involves the layout characterization to the app activity, the DeepPrior is only suitable for analyzing the apps with a grid layout or a list layout. We also limit our claim within apps of such layouts.

The enrollment of the crowdworkers is also uncontrolled. The crowdworkers’ capability is uncontrolled, and low-quality reports may occur. However, even if the quality of some reports is low, DeepPrior can identify the bug it is describing if it actually contains one. If not, DeepPrior will categorize the report as a single category, and will not affect the prioritization of other reports.

The datasets we construct are of Chinese language. The language of the datasets may be another threat, but NLP and OCR technologies are quite strong. If we replace the text processing engine with that of another language, the text processing will also be completed well, and will not have a negative impact on DeepPrior. Moreover, the maturity of machine translation [16] also makes it robust to process cross-language textual information.

Crowdsourced testing has been a mainstream testing strategy. It is significantly different from traditional testing. Testing tasks are distributed to a large group of crowdworkers of different locations and have widely varying abilities. The most notable advantage of crowdsourced testing is the capability of simulating different using conditions and the relatively low economic cost [17][18]. However, the openness of crowdsourced testing leads to a large amount redundant reports. The key problem is to improve the developers’ efficiency to review the test reports. some researches start from selecting skillful crowdworkers to complete the tasks [19][20][21]. Such a strategy is effective, while it is still hard to control because even skillful crowdworkers would loaf on the task. Therefore, we think it is more important to process the test reports instead of other factors in crowdsourced testing. Liu et al. [22] and Yu [6] proposed approaches respectively to automatically generate descriptions from screenshots for test reports based on the consensus that app screenshots are easy to acquire while the textual descriptions are hard to write for all the crowdworkers. This idea inspired us to have a deep screenshot understanding of to help better prioritize the test reports.

Many researches have been done to process crowdsourced test reports to better help developers review the reports and fix bugs. Basic strategies include report classification, duplicate detection, and report prioritization. In this section, we will present the related works of different basic strategies.

Banerjee et al. proposed FactorLCS [23], utilizing the common sequence matching, and the approach is effective on open bug tracking repositories. They also proposed a method [24] with a multi-label classifier to find the “primary” report of a cluster of reports with high similarity. Similarly, Jiang et al. proposed TERFUR [14], a tool that clusters the test reports with NLP technologies, and they also filter out the low-quality reports. Wang et al. [25] takes the features of the crowdworkers into consideration as a feature of the test reports, and then make the clustering. Wang et al. propose the LOAF [26], which is the first considering the operation steps and result descriptions separately for report feature extraction.

More researches are done to detect the duplication of the test reports. Sun et al. [27] use information retrieval models to detect duplicate bug reports more accurately. Sureka et al. [28] adopted a character n-gram based model to complete the duplicate detection task. Prifti et al. [29] conducted a survey on large-scale open-sourced project test reports and proposed a method that can concentrate the search for duplicate reports on specific portions of the whole repository. Sun et al. proposed a retrieval function, REP [30], to measure the similarity, and the function includes the similarity of non-textual fields like component, version, etc. Nguyen et al. introduced DBTM [31], a tool that utilizes both IR-based features and topic-based features, and detects the duplications bug reports according to technical issues. Alipour et al. [32] had a more comprehensive analysis of the test report context and improved the detection accuracy. Hindle [33] makes improvements by combining contextual quality attributes, architecture terms, and system-development topics to improve bug deduplicate detection.

The above approaches, including the report classification and the duplicate detection, choose part of the test reports to represent all the test reports. However, we hold that all the reports contain valuable information even if duplicates exist. Moreover, After duplicates are detected, developers still need to review the reports to carry forward the bug processing. Therefore, we think report prioritization is a better choice.

There are also many researches on report prioritization. Zhou et al. introduced BugSim [34], considering both textual and statistical features, to rank the test reports. DRONE, proposed by Tian et al. [35], is a machine learning-based approach to predict the priority of the test reports, considering different factors of the test reports. Feng et al. proposed a series of approaches, DivRisk [36] and BDDiv [3], to prioritize the test reports, and they first consider the screenshots of the test reports. Subsequently, Wang et al. [2] work further and explore a more sound approach to prioritize test reports with much more attention paid to the screenshots.

Among all the above researches, only a few of them, like [2] and [3], consider the app screenshots, which we think is a valuable factor for extracting the features to process the test reports. But such researches only treat the screenshots as simple matrixes instead of meaningful content.

Image understanding is a hotspot issue in the Computer Vision (CV) field. In this section, we mainly focus on the researches utilizing image understanding in software testing.

Lowe [10] proposed the SIFT algorithm, which is used to match the feature points on the target images and calculate the similarity, making use of a kind of new image local features, which are invariant to image changing, including translation, scaling, and rotation. Optical character recognition (OCR) is a widely-used tool to recognize the texts, which is helpful to better understand the images based on the rich textual information. Nguyen et al. [37] proposed REMAUI, making use of CV technologies to identify the widgets, texts, images, and even containers of the app screenshot. Moran et al. [38] proposed REDRAW based on REMAUI, which is more precise to identify the widgets and can automatically generate codes for the app UI. Similarly, Chen et al. [39] also proposed a tool to generate GUI skeletons from app screenshots with the combination of CV technologies and machine learning. Yu et al. [1] proposed a tool named LIRAT to record and replay mobile app test scripts among different platforms with a thorough understanding of the app screenshot.