Automating Quantum Software Maintenance: Flakiness Detection and Root Cause Analysis

To extend this research and detect more flaky reports systematically and automatically, we employed embedding transformers to represent GitHub IRs and PRs and measured the cosine similarity with previously identified flaky tests.

Based on the Hugging Face leaderboard [4], we selected three top-performing (at the time of experiment design) embeddings on generic tasks. We utilized the pre-trained ‘mixedbread-ai/mxbai-embed-large-v1’ transformer [5, 6] from Hugging Face library [7] to extract contextual embeddings¹¹1The embeddings were derived from the penultimate layer of the model, ensuring that they captured the nuanced features of the text. of the GitHub IRs and PRs. We also evaluated other transformers, such as ‘SFR-Mistral’ [8] and ‘e5-mistral-7b-instruct transformers’ [9, 10], using $k$-means clustering [11] to assess their effectiveness in distinguishing between flaky and non-flaky test cases. Our analysis showed that the “mixedbread-ai/mxbai-embed-large-v1” model provided the most distinct and separable representation for this task.

Using this model, we generated embeddings from the tokenized text of all scraped GitHub IRs and PRs from the 12 repositories, as well as from the previously identified flaky cases. We calculated cosine similarity scores between the embeddings and ranked the issues based on their similarity to known flaky cases. Excluding identical matches from our initial flaky set, at least two authors cross-examined the top-ranked issues, associated PRs, and code commits to determine if they were related to flaky tests and to establish their cause categories.

In the first iteration, we identified 15 new flaky tests from the top-ranked cosine similarity scores. We then augmented the original dataset with these new cases and repeated the process, resulting in the detection of an additional 10 flaky tests in the second iteration. In total, we identified 25 new flaky tests across the 12 repositories, increasing the original dataset size by 54%.

To create a balanced dataset, we also collected non-flaky tests from GitHub reports during the cosine similarity analysis. Test cases with a cosine similarity score of less than 0.5 were labeled as non-flaky. Additionally, we included instances that our method incorrectly labeled as flaky to further challenge the classification task. In total, we compiled 71 non-flaky cases to match the size of our expanded flaky dataset.

We extracted descriptions and comments from IRs and PRs for each issue observation in the dataset, recorded code differences, and noted affected files before and after fixes. Method-level code changes were also extracted for a more concentrated analysis.

Manual verification ensured that the extracted artifacts matched the identified IRs and PRs. The dataset was organized into ‘full’ and ‘method’ directories, each further divided into ‘flaky’ and ‘non-flaky’ sections.

Leveraging the extracted codebase data (discussed in Section II-A2), we crafted input prompts to address our research questions. The prompts are provided in the supplementary material (https://doi.org/10.5281/zenodo.13913775).

To explore our research questions and assess how context size affects the answers, we designed the following experiments.

For RQ1, which aims to classify whether a particular IR (or PR, if no issue is associated with it) is flaky or non-flaky, we tested two levels of context: $R_{p}$ (partial), which includes only the initial IR (or PR) description, and $R_{f}$ (full), which includes the description along with all associated comments.

For RQ2, we expanded the context for the language model by adding the code involved in the PR before the fix was applied, also at two levels: $C_{p}$ (partial), which includes the method-level code, and $C_{f}$ (full), which provides the complete code listing.

By combining the context levels from RQ1 and RQ2, we generated four experimental conditions: $\{R_{p},R_{f}\}\times\{C_{p},C_{f}\}$. These conditions range from $(R_{p},C_{p})$, which uses only the description and method-level code, to $(R_{f},C_{f})$, which includes the description with comments and the full code listing.

For RQ3, the amount of information provided did not change; we simply followed up by asking which specific root cause a particular flaky test relates to, using the nine classes of root causes defined by [2]: “Randomness (PRNG),” “Floating Point Operations,” “Software Environment,” “Multi-threading,” “Visualization,” “Unhandled Exceptions,” “Network,” “Unordered Collection,” or “Others”. We find that we do not need to alter these classes for our extended dataset.

We utilized LangChain [12], an integration framework that abstracts prompt templating, retrieval strategies, and chaining, allowing us to manage conversational memory effectively. This setup enabled us to simulate a developer-to-AI interaction, appending additional context to study its impact on the LLM’s reasoning.

To assess the performance of various LLMs, we study two open-source LLMs, namely Meta LLaMA-70B-Instruct v.3.1, and LLaMA-405B-Instruct v.3.1 [13], and two closed-source LLMs, namely OpenAI GPT-4o-mini v.2024-10-03 and GPT-4o v.2024-10-07 [14, 15]. All captured OpenAI and Meta Instruct models were accessed remotely through serverless APIs of OpenAI [16] and Google VertexAI [17], respectively.²²2We also fine-tuned CodeBERT [18] for flaky test detection using a few-shot learning approach. While training was successful, the model failed to generalize effectively for the test set when applied to GitHub IRs and PRs.

In cases where the issue and fix resided in different repositories, we manually aligned the IRs with the fix repositories to maintain consistency in our analysis.

In some cases of the dataset, a flaky bug will have two PRs associated with it. For the four observations in the flaky data and two in the non-flaky data, we split the observation into two two instances. A single issue that is observed in both PRs will be present in both of the codebases respectively.

The differences between methods were automatically extracted using PyDriller [19], which operates only on Python code. Given that Python dominates our dataset (12 out of 14 repositories are written primarily in Python), this limitation is not a significant concern.

However, even in Python-focused datasets, not all repositories contain method-level data. For example, some fixes might target configuration files or global variables within Python scripts. Therefore, we capture and report the total number of observations for each experimental setup to account for such cases.

Using the full context, the best-performing model is GPT-4o-mini, with an F1-score of 0.8871 and an MCC of 0.7774. This outcome is somewhat unexpected, as GPT-4o is generally considered more powerful. However, when only partial context is provided, GPT-4o outperforms GPT-4o-mini with an F1-score of 0.8443 vs. 0.8229 and MCC of 0.6971 vs. 0.6558. Thus, GPT-4o effectively identifies flakiness with respectable performance when provided just the initial report. This suggests that classification with limited context might be more challenging, and a more sophisticated model excels in such cases. LLaMA-405B-Instruct and LLaMA-70B-Instruct ranked third or fourth (depending on the context).

We assess whether adding code at the method-level ($C_{p}$) or including a full code listing ($C_{f}$) improves classification accuracy. We compare recall values from RQ1 and RQ2 to evaluate this. For GPT-4o, adding method-level code ($C_{p}$) increases recall, with the highest score observed in the $\{R_{f},C_{p}\}$ setup, as expected. Interestingly, providing a complete code listing ($C_{f}$) reduces recall, which aligns with the observation that a model, much like a human programmer, might struggle to identify specific methods to focus on when faced with the entire codebase.

For GPT-4o-mini, recall decreases for RQ2, but less so for $C_{p}$ compared to $C_{f}$. This suggests that analyzing both natural language and code is a more complex task, one that requires a more advanced model.

The behavior of LLaMA-405B-Instruct is similar to GPT-4o, showing an increase in recall for $C_{p}$ and a decrease in recall for $C_{f}$. In contrast, LLaMA-70B-Instruct exhibits behavior similar to GPT-4o-mini, with decreased performance in both contexts.

Note that we should be cautious about drawing strong conclusions when comparing $C_{p}$ and $C_{f}$, as the number of observations for the $C_{p}$ cases is smaller than for the $C_{f}$ cases (44 vs. 62, respectively, due to the reasons discussed in Section II-C3).

The most challenging task, requiring both multi-label classification and code analysis, is presented by RQ3. As anticipated, the performance of all models declines compared to RQ1 and RQ2. The highest results are achieved by the most powerful model, GPT-4o, when using method-level data ($C_{p}$) and the initial issue or pull request description ($R_{p}$), though the performance is still low, with an F1-score of 0.5839 and an MCC of 0.5928. With complete code ($C_{f}$), and thus a higher number of observations, the results further drop to an F1-score of 0.4919 and MCC of 0.4805.

GPT-4o-mini, LLaMA-405B-Instruct, and LLaMA-70B-Instruct have similar performances, but their rankings differ depending on whether the F1-score or MCC is used. For F1-score, from 2nd to 4th place, the order is LLaMA-405B-Instruct, GPT-4o-mini, and LLaMA-70B-Instruct. For MCC, the ranking is GPT-4o-mini, LLaMA-405B-Instruct, and LLaMA-70B-Instruct. These results suggest that there is considerable room for improvement in this area.