Towards Understanding the Impacts of Textual Dissimilarity on Duplicate Bug Report Detection

Dataset collection. We collect duplicate bug reports from three large-scale, open-source software systems – Eclipse, Firefox, and Mobile – using a popular bug tracking system, namely Bugzilla. Existing studies [27, 4, 59, 37] have frequently used these systems, which makes them suitable for our research. Besides, these systems have stemmed from diverse application domains. Eclipse is a popular open-source Integrated Development Environment (IDE) written in Java. Firefox is a popular open-source web browser. While the above two systems are desktop-based applications, the remaining systems are mobile-based (e.g., Firefox for iOS, Focus for iOS, and GeckoView for Android). For the sake of brevity, we combine these three small systems and call them Mobile in the rest of the paper. Almost all the existing studies on duplicate bug report detection used the dataset dated before 2017 [27], which might not be an ideal representation of recent software bugs and issues [69]. In order to avoid the issue of concept drift [43], we choose the bug reports from the last five years. We thus collected a total of 92,854 bug reports from three systems that were submitted within the last five years (01-01-2017 to 01-01-2022).

For many duplicate bug reports, the master reports were created before the last five years (Eclipse: 766, Firefox: 3514, Mobile: 214). In order to make a complete dataset with all the duplicate bug reports along with their master bug reports, we also retrieved them separately. Table I summarizes our study dataset. We see that about 6.60%, 20.53%, and 10.56% of the submitted bug reports were duplicates in Eclipse, Firefox, and Mobile systems, respectively.

Data cleaning and preprocessing. We capture four key fields from each bug report for our study: (a) bug id, which is unique for each bug report, (b) duplicate bug id, which points to the duplicate bug report, (c) title and description of the bug report, and (d) resolution, which indicates the duplicate status of a bug report.

We collect title and description from each bug report since they capture pertinent information for detecting duplicate bug reports. We clean and preprocess the title and description from each bug report using several steps as follows.

We apply standard natural language preprocessing to the title and description texts. First, we remove stopwords using a standard set of stopwords, which have little to no significance in capturing semantics. Then we perform token splitting along with the removal of punctuation marks, non-alphanumeric characters, numbers, HTML meta tags, and URLs. We also replace any non-alphanumeric characters with spaces and transform the text into lowercase [52]. Lastly, to transform each term into its base form, we use lemmatization using the NLTK library in Python. As performed by an earlier work [52], we discard any description with fewer than 50 characters since they do not contain enough information to be meaningful. On the other hand, bug reports might have long description text containing source code, lengthy stack traces, and log files, which could be noisy [52]. Hence, several previous studies [20, 52] selected bug reports containing at most 350 to 500 tokens. We experimented with 350, 500, and 1000 tokens. Using 500 tokens, the model delivers the best performance. Thus, we chose 500 as our token limit for the bug reports.

Construction of triplets. After we clean and preprocess the dataset, we construct triplets (b, b+, b-) where the existing techniques are supposed to detect b+. Here, b means the query bug report, b+ means duplicate bug report, and b- means non-duplicate bug reports. We created these triplets inspired by an existing work [20]. In the existing work, the duplicates (b,b+) were determined based on bug-tracking systems, whereas non-duplicates (b,b-) were randomly selected from the dataset. We also follow the same process in our ground truth construction. Then (b, b+) pairs were used as the ground truth for evaluating the existing techniques in duplicate bug report detection.
Dataset preprocessing for ML-based approach. To design an ML-based model (e.g., Siamese CNN) for duplicate bug report detection, pairwise bug reports containing texts and ground truth are required. Hence, we extract (b, b+) and (b, b-) pairs from the triplets above to generate our positive and negative samples, respectively. A similar process was adopted by the existing literature [20]. For training and testing, we split the whole dataset into an 80:20 ratio with random shuffling. Table I (bottom section) shows our Machine Learning models’ training and testing datasets from all three systems.

First, we store all duplicate bug reports as pairs in a separate dataset. Then, we collect N-grams (n = 1, 2, and 3) to compute the textual similarity [13] of each duplicate bug report pair. We use the cosine similarity metric [56] to calculate the textual similarity between two bug reports. After getting the similarity score between the duplicate pairs, we analyze their descriptive statistics to determine their median similarity score, lower quartile (25th percentile) value, and upper quartile (75th percentile) value. For duplicate pairs that have a similarity score less than the lower quartile value, we denote them as textually dissimilar duplicate bug reports.
On the other hand, for duplicate pairs that have a similarity score more than the upper quartile value, we denote them as textually similar duplicate bug reports. We repeat all these steps using Unigram, Bigram, and Trigram to collect the common set of textually similar and textually dissimilar duplicate bug reports across three trials for our experiment. Table II shows similarity scores used for various N-grams to construct our textually similar and dissimilar duplicate bug reports. Finally, we got two subsets for textually similar and textually dissimilar duplicate bug reports, respectively (Eclipse: 679 & 662, Firefox: 1414 & 1455, Mobile: 122 & 131).

To answer our first research question, we needed to replicate existing techniques on duplicate bug report detection. We thus select suitable representatives from the frequently used methodologies in duplicate bug report detection. In particular, we choose baseline methods from three frequently used methodologies – Information Retrieval, Topic Modeling, and Machine Learning. We select BM25 [67] from Information Retrieval, LDA+GloVe [3] from Topic-Modeling, and Siamese CNN [20] from Deep Learning for our experiment. Most of the recent models are based on these three primary approaches with incremental improvements [27, 8, 44]. We chose these baseline methods to determine the impact of textual dissimilarity on duplicate bug report detection without the effect of compounding factors (e.g., severity, priority, components, products, multimedia attachments).

We used the authors’ replication package for the LDA+GloVe model [3]. On the other hand, the replication packages of BM25 [67], and Siamese CNN [20] were not publicly available, and thus those techniques were carefully re-implemented by us based on the corresponding papers.

Information Retrieval (IR) relies on keyword overlaps between any two documents. We select a representative of IR, namely BM25, with default parameters (k1=1.5, b=0.75) for retrieving the duplicate bug reports. Yang et al. [67] first use BM25 for duplicate bug report detection. While BM25 is an established technique, it suffers from Vocabulary Mismatch Problem (VMP) [15, 23]. Several studies [3, 45, 31] adopt Topic Modeling in duplicate bug report detection to overcome this challenge. Latent Dirichlet Allocation (LDA) is a Topic Modeling approach that has the potential to overcome the vocabulary mismatch problem [28]. We call this approach LDA+GloVe in our research. As done by the original work [3], we use LDA for topic-based clustering (topic number=20), GloVe for the pre-trained word embedding (embedding dim = 100), and a unified text similarity measure (cosine similarity and euclidean metrics) for ranking the topmost, similar bug reports against a given bug report.

Unlike the above two techniques, Machine Learning based approaches might be able to find non-linear relationships between dependent and independent variables [47, 5, 10]. We thus implement the Siamese Convolutional Neural Network (CNN) for duplicate bug report detection by adapting an earlier research [20]. We train our Siamese CNN model using K-fold cross-validation (e.g., K=10) and batch gradient descent, where original authors’ batch sizes (e.g., 512 for Eclipse and Firefox, 256 for Mobile), learning rate (e.g., 0.001) and epoch number (e.g., 12) were chosen during the training phase to avoid model overfitting. Since the Mobile system has a small number of bug reports, a smaller batch size was chosen.

As we detected duplicate bug reports using Information Retrieval, Topic Modeling, and Machine Learning techniques in our experiments, they were evaluated using appropriate performance metrics from these domains. In the case of BM25 and LDA+GloVe, we find all the duplicate bug reports for a given query bug report. Then we used Recall-rate@K [27, 3, 41, 63], one of the most popular performance metrics, to evaluate the IR and Topic Modeling approaches. On the other hand, for the deep learning-based model (Siamese CNN), we used traditional metrics such as F1 score, AUC, recall, and precision [27]. We used different evaluation metrics based on the original works [67, 3, 20]. It should be noted that our main goal was to contrast the performance of existing techniques between two sets of duplicate bug reports rather than to compare the techniques.
Recall-rate@K: Recall-rate@K determines the percentage of bug reports for each of which the duplicate bug report is found within the top K positions [57].

After conducting the experiments, we evaluated our BM25 and LDA+GloVe models using Recall-rate@K, as used by existing work [67, 3]. On the other hand, we have evaluated the Siamese CNN model, as was done by the original work [20], using the remaining performance metrics.

We first evaluate the existing techniques against our whole dataset. Tables III and VIII summarize their performances.

From Table III, we see that the BM25 approach, on average, performs higher with the Eclipse system than with Firefox and Mobile systems. On the other hand, the performance of the LDA+GloVe model is approximately 38.36% lower than that of BM25 for Recall-rate@100. LDA is limited in modeling topic correlations [39], which could be crucial to duplicate bug report detection. Table VIII (top section) shows that the performance of our ML-based technique in detecting duplicate bug reports ranges from 61.41% – 84.80% in terms of AUC. Precision, Recall, and F1-measure are slightly higher for Eclipse than for Firefox. Firefox has a better AUC score than Eclipse, as the AUC metric is robust to imbalanced data [68]. Precision, Recall, F1-measure, and AUC scores are slightly higher for the Mobile dataset than for the other two systems. Overall, the ML-based approach delivers the highest performance in duplicate bug report detection.

While the above analysis focuses on the whole dataset, we also determine the performance gap of existing techniques between textually similar and textually dissimilar duplicate bug reports. Table IV shows that BM25 delivers a higher Recall-rate@K for textually similar duplicate bug reports than for dissimilar ones across all three systems - Eclipse, Firefox, and Mobile. For instance, with K=100, the difference between these two sets ranges from 10.20% to 18.45%. Fig. 2 further shows the performance difference between these two sets of bug reports for various K values. We see that the difference is noticeably higher for Firefox and Mobile systems.

On the other hand, for the LDA+GloVe model, the performance differences between textually similar and textually dissimilar duplicate bug reports are 2.00% for Eclipse, 6.49% for Firefox, and 5.5% for the Mobile system with K=100 (Table IV). The performance gap is higher for Firefox and Mobile systems in the same manner as BM25. One possible reason behind this could be the higher duplicate ratios in Firefox and Mobile systems (see Table I).

Table VIII shows the performance of our Machine Learning model - Siamese CNN - for both sets of duplicate bug reports across three systems. Here, the performance difference between textually similar and textually dissimilar duplicates is less apparent than that of the traditional methods above. Machine Learning models, especially deep learning models, can capture more contextual information beyond the syntax [40], which might explain the phenomenon. From Table VIII, we see that the performance difference is smaller for Eclipse than for the other two datasets. For instance, the AUC differences between textually similar and textually dissimilar duplicate bug reports are 9.57% for Eclipse, 9.38% for Firefox, and 2.97% for the Mobile dataset. In the case of the F1-measure, the performance difference for the Eclipse dataset is 10.92%, whereas, for the Firefox dataset, it is 8.48%. For Mobile data, the performance difference is 10.00%. We also note that the performance of our ML-based technique is lower for the two subsets of bug reports than for the whole dataset. As shown in Table I, both subsets contain a smaller number of duplicate bugs than the whole dataset. That is, the two experiments use the same trained model but are tested with different numbers of test samples, which might explain the finding.
We also perform statistical tests to determine the significance of the performance gap between textually similar and dissimilar duplicate bug reports (Table V). For each of the three systems, we first evaluate BM25 and LDA+GloVe using Recall-rate@K measures against textually similar and dissimilar duplicate bug reports where we consider various K values (K = 1, 5, 10, 20, 25, 30, 50, 75, 100). Then, we performed Shapiro-Wilk normality test [55] to determine the distribution of each set. We got two non-normal distribution pairs (LDA+GloVe for Eclipse and Firefox) out of six sets of pairs. Then we used appropriate parametric, non-parametric tests and effect size tests to compare the two sets of Recall-rate@k values from textually similar and dissimilar duplicate bug reports. For the normal distribution, we used paired t-test as the parametric test [36], and for the non-parametric test, we used the Wilcoxon Signed‐Rank test [64]. In both types of significance tests, the p-values were less than the threshold (0.05) except for two sets (BM25 in the Eclipse system & LDA+GloVe in the Mobile system). Thus, the null hypothesis can be rejected for all comparisons except for the case of BM25 in Eclipse and LDA+GloVE in the Mobile system. In other words, the performances of BM25 and LDA+GloVE techniques are significantly different between textually similar and dissimilar duplicate bug reports.
While the significance of a result indicates how probable it is that it is due to chance, the effect size indicates the extent of the difference [53]. Our experiments found different effect sizes ranging from medium to large (Table V). We see that the effect size of BM25 is large for Firefox and Mobile systems and medium for Eclipse. On the other hand, the LDA+GloVe model has a medium to large effect size across the three systems. Thus, our results from effect size tests reinforce the above finding from significance tests. In other words, the existing techniques perform significantly poorly detecting textually dissimilar duplicate bug reports. Even though our findings above mostly match natural intuition, we performed extensive experiments on three different systems using three different methodologies, which resulted in strong empirical evidence. Thus, we not only reinforce the existing belief about the existing techniques on duplicate bug detection but also substantiate it with solid empirical evidence.

In this research question, we investigate how textually dissimilar duplicate bug reports might be different from textually similar duplicate bug reports. We answer this question using three different analyses – descriptive analysis, embedding analysis, and manual analysis – as follows:

Descriptive analysis. Descriptive analysis involves examining the data that helps describe, show, or summarize data points. It helps determine patterns or outliers that might emerge, which could lead to further statistical analyses. After cleaning and preprocessing the bug reports from each subject system, we calculate the cosine similarity score between each pair of duplicate bug reports using their TF-IDF measures from the textually similar and dissimilar subsets. Then we perform descriptive analysis on these similarity scores and capture five different statistics: Skewness, Kurtosis, Mean, Median, and Standard Deviation. Fig. 3 and Table VI summarize our descriptive analysis for Eclipse, Firefox, and Mobile systems.

Skewness is a measure of symmetry [26]. Distribution is symmetric if it looks the same on the left and right of the center point [26]. From Fig. 3, we find the scores of textually similar duplicate bug reports to be positively skewed for Eclipse and Firefox. The positive skewness indicates that a significant number of duplicate pairs are highly similar [26]. On the other hand, for the textually dissimilar dataset, we found negative skewness for all three datasets, indicating that the cosine similarity measures are very low for most of the textually dissimilar duplicate pairs [26].

Kurtosis is a measure of whether the distribution is heavy-tailed or light-tailed relative to a normal distribution [26]. Distributions with positive kurtosis tend to have heavy tails or outliers, whereas distributions with negative kurtosis tend to have light tails [18]. A uniform distribution would be the extreme case. From Fig. 3 and Table VI, we note that textually dissimilar duplicate pairs have a negative kurtosis for Firefox and Mobile systems. The kurtosis for the Eclipse system is also close to zero. That is, the similarity scores from textually dissimilar duplicate pairs have a lighter tail than the normal distribution. In other words, their similarity scores are mostly centered around the mean value, which is also low.

On the other hand, a similar conclusion can be made for the textually similar duplicate pairs with the Eclipse and Mobile systems. However, it should be noted that the Firefox system contains several times more bug reports than the Eclipse and Mobile systems. The remaining two of three statistics - mean and median - are also several times lower for textually dissimilar duplicate bug reports than their counterparts.

Embedding analysis. Word embedding is a frequently used mechanism for detecting duplicate bug reports, representing words as semantically relevant dense real-valued vectors [38]. While our descriptive analysis above focuses on text-level similarity, we now perform embedding analysis to visualize the semantic differences between textually similar and textually dissimilar duplicate bug reports. We employ t-SNE to visualize high-dimensional data representing embedding vectors in lower dimensions [61]. It illustrates high-dimensional data (e.g., bug report embeddings) by projecting them into a two-dimensional space. The intra-cluster detail can be observed by measuring the pairwise distances in the higher and lower dimensions spaces [61]. We collect 100 random bug report pairs from both textually similar and dissimilar datasets. We generated each pair’s GloVe embeddings and performed t-SNE to visualize the embedding of each pair. We use the optimal perplexity and iterations (e.g., perplexity = 40, iterations = 7000) and default similarity metric (i.e., cosine similarity) for our visualization.

From Fig. 4, we see the difference in embedding visualization of textually similar and dissimilar bug reports for all three datasets. In embedding space, textually dissimilar duplicate pairs (orange dots) are clustered in a different dimensional area than the textually similar pairs (blue dots). In all three systems, the embedding positions of textually dissimilar bug reports are in lower coordinates than that of textually similar duplicate bug reports. In particular, the changes are noticeable across the vertical dimension between textually similar and dissimilar duplicate bug reports for Firefox and Mobile. Interestingly, for Eclipse, we see that textually dissimilar duplicate bug reports are at different locations, even across the horizontal dimension. All these differences above suggest that the word semantics of duplicate pairs within the same dataset could be similar but very different from that of other datasets. In other words, the semantics of textually dissimilar duplicate bug reports are noticeably different from that of textually similar bug reports.

Manual Analysis. We chose 100 randomly selected duplicate bug report pairs (50 textually similar + 50 textually dissimilar) from each subject system. Then, we manually analyze 150 pairs of textually similar and 150 pairs of textually dissimilar duplicate bug reports. Ideally, each bug report should have three components – Expected Behaviour (EB), Observed Behaviour (OB), and Steps to Reproduce (S2R) [9]. These components have been used by previous research to reformulate queries during duplicate bug report detection using Information Retrieval [14]. Similarly, we make use of these components from each duplicate pair to understand how textually similar duplicate bug reports and textually dissimilar duplicate bug reports might differ from each other.

First, we go through the title and description of each bug report and detect the presence of EB, OB, and S2R components in each bug report. Then we analyze the prevalence ratios of these components in both textually similar and textually dissimilar duplicate bug reports. We also calculate the textual similarity between the two bug reports from each pair for each of the three components separately. As a part of manual analysis, we also look for shared terms, keywords, technologies, and overall literary analogies between a duplicate bug report and a master bug report. We spent a total of $\approx$25 hours on our manual analysis.

Table VII shows the prevalence ratios of all three components from both textually similar and textually dissimilar duplicate bug report pairs. We see that textually dissimilar duplicate bug reports have a higher percentage of missing components. For example, as shown in Table VII, 44.61% and 13% of textually dissimilar pairs do not contain any steps to reproduce (S2R) and expected behaviors (EB) in their bug reports, whereas such statistics are 13% and 1% respectively for the textually similar duplicate bug reports. Such higher ratios of missing components might explain the lower textual similarity between each duplicate pair of their bug reports.

Table VII also shows the component-level similarity between two bug reports from each duplicate pair. We see a lower component-level similarity for textually dissimilar duplicate bug reports. For example, on average, two bug reports from each of their pairs are only 38% and 57% similar when OB and EB components are considered, whereas such statistics are 83% and 78%, respectively, for the textually similar duplicate pairs. Thus, even at the component level, textually dissimilar duplicate bug reports displayed lower similarity ratios.

In other words, missing components and component-level differences might have led to their overall textual dissimilarity. We also record several qualitative insights during our manual analysis of textually similar and textually dissimilar duplicate bug reports. They are outlined as follows.

(a) Shared phrases. Textually similar duplicate bug reports have more shared phrases (e.g., Bigram, Trigram) than unique words (e.g., unigram). For example, rather than the word ”scroll”, phrases such as “horizontal scroll”, and “horizontal scroll installation” are more prevalent in these reports.

(b) Components’ prevalence. Observed behaviors (OB) and expected behaviors (EB) are more prevalent in textually similar duplicate bug reports, which could lead to their increased textual similarity. We found up to 90% overall similarity between two duplicate reports from this category.

(c) Missing components. We observed a higher percentage of missing components in textually dissimilar duplicate bug reports than in textually similar ones. Bug reports with missing components are likely to have lower similarity scores. Although we notice minimal keyword overlaps, the root cause was mostly similar for both bugs from the same duplicate pair.

(d) Unique phrases. Textually dissimilar duplicate bug reports often use completely unique phrases, which could lead to their dissimilarity. We found that although the EB was somewhat similar, the OB and S2R components were written differently for the two bug reports of the same duplicate pair.

From RQ₁ and RQ₂, we see that textually similar and dissimilar duplicate bug reports could be different in their lexicon, underlying semantics, and structures. Our analysis above also shows that the performance gap between these two sets is the smallest when the Machine Learning approach is used for duplicate bug report detection (RQ₁). Machine Learning approaches might be able to capture more contextual information than the other approaches, which could be useful for the detection task [47, 5, 10].

In RQ₁, we used pre-trained word embeddings from GloVe to train our Siamese CNN model [20] for duplicate bug report detection. Pre-trained word embedding has proven to be invaluable for improving the performance of various natural language understanding tasks (e.g., text classification [6], sentiment analysis [49]). However, GloVe has been pre-trained on natural language texts (e.g., Wikipedia) [48], which might not be relevant to the texts from bug reports. Thus, we use domain-specific embedding to retrain our Siamese CNN model. We set the max token size to 20,000 and the embedding dimension to 100, which are similar to the parameters used in RQ₁. We generate the embedding matrix with Skip-gram algorithm [46], use the whole dataset of 92,854 bug reports, and apply the same deep learning architecture to Siamese CNN, as used in RQ₁ [20].

Datasets constructed from bug-tracking systems are often heavily imbalanced. The number of duplicate bug reports is considerably smaller than that of non-duplicate bug reports [1]. Hence, we use oversampling [68] to handle the data imbalance problem during model training [68]. To the best of our knowledge, the original work [20] did not use sampling in their Siamese CNN model. However, for an in-depth investigation, we replicate another variant of the original DL-based model [20] applying oversampling. Table VIII summarizes the experimental results of our DL-based model for three different scenarios: pre-trained embedding only (original work [20]), pre-trained embedding + oversampling, and domain-specific embedding + oversampling. When compared between these model scenarios– pre-trained embedding + oversampling, and domain-specific embedding + oversampling, we see the noticeable impact of domain-specific embeddings on duplicate bug report detection.

From Table VIII, we see that, in terms of F1-measure, the model’s performance with textually dissimilar duplicate bug reports has increased by 1.18% for Eclipse, 6.56% for Firefox, and 3.21% for Mobile. Furthermore, the AUC improved by 6.56% for Eclipse and remained comparable for the other two systems. Thus, domain-specific embeddings have a positive impact on detecting textually dissimilar duplicate bug reports. However, they have mostly negative impacts, except in a few cases, on detecting textually similar bug reports. From Table VIII, we see that the model’s F1-measure decreased by 7.24% for Eclipse and 4.32% for the Mobile system. Furthermore, the AUC decreased by 10.48% for the Mozilla system. Thus, while domain-specific embeddings have the potential to tackle the challenge of textual dissimilarity, they have a mixed impact on detecting duplicate bug reports.