Identifying Flakiness in Quantum Programs

As mentioned above, the use of PRNGs is the most common (21%) cause of reports of flakiness in quantum programs. The PRNGs produce different outputs from run to run, which may result in a flaky test. Given that most of the test cases are executed on simulators of QCs and that the simulators rely heavily on PRNGs to simulate the non-deterministic nature of QCs, it is not surprising that this is the primary cause of flaky test reports.

PRNGs are used in quantum programs and associated test suites. Our first example (Listing 1) demonstrates a problem with the quantum program found in issue report #3533 of qiskit-terra. The constructor of a two-qubit decomposer (details in [15]) has a non-deterministic component in Lines 6 and 7. This decomposer can generate random results when generating controlled versions of the same unitary gate. For example, Listing 2 creates a UnitaryGate instance in Line 1, and the equality operation in Line 5 returns “True” or “False” at random.

In Listing 3, the second example (based on issue report #5217 in qiskit-terra) illustrates a test case issue. The logic of the test test_append_circuit, which checks if appending quantum circuits work properly, is correct. However, Lines 3 and 6 cause flakiness in the test because the function random_circuit generates a random quantum circuit using a randomly selected seed (by default). Thus, the test fails occasionally due to randomness. We will discuss how to fix the flakiness in Listings 1 and 3 in Section III-C1.

This category includes flaky tests caused by specific software or library dependency issues. For example, pull request #1369 in netket discusses a flaky test observed only in the GitHub Actions Python 3.10 environment. The pull request starts with the following comment.

“No idea why, but on the GitHub runner python 3.10 this test keeps failing. I never reproduced locally so I think it’s not a real failure. Simplyfying [sic] the test to avoid.”

A common way to fix this issue is to alter the software environment as will be shown in Section III-C2.

This category of flaky tests is caused by multi-threading issues, e.g., concurrency and overload. As an example, issue report #5904 in qiskit-terra describes a flaky test caused by address collisions due to parallel builds. The code in this example can be seen in Listing 4, showing that sphinx-build generates documentation in parallel over $N$ processes, where $N$ is the number of CPUs (i.e., by the argument -j auto). While the parallelization improves the throughput, it causes an error when multiple jobs of sphinx-build compete against each other. A fix pattern for this cause is given in Section III-C3.

We define a flaky test as floating-point-related if it occurs due to challenges such as round-off errors. For example, in netket pull request #1147 (Listing 5), the hard-coded tolerance value of $10^{-5}$ (shown in Line 5) causes a test case to fail intermittently.

To tackle this problem, developers modify the assertion to alter tolerance, round the actual value, or remove a flaky test case altogether. We will provide details of how developers address this challenge in Section III-C4.

This group of flaky tests is related to image generation. For example, qiskit-terra has a test manager that schedules visual tests sequentially and allows these tests to communicate with one another. Issue #3283 in qiskit-terra reports a flaky test in the visualizer of the test manager due to out-of-date reference indexes.

In this category, we are unable to identify a recurring fix pattern. There is a unique solution for each of the three reports (all associated with qiskit-terra).

This group of flaky tests is caused by the code that does not appropriately handle exceptions. For example, Listing 6 shows the stack trace in the issue report of #398 in Microsoft/QuantumLibraries. In this case, the testing can occasionally fail whenever the “number” does not fall into the expected range. We will explore a fix pattern for this cause in Section III-C5.

A flaky test in this category occurs as a result of network-related issues, such as an unstable network or server. For example, issue #584 of qiskit-ibm-runtime reports a flaky test due to timeouts and socket connection problems. As shown in Listing 7, test_websocket_proxy fails because jobs have been completed before websocket connection can be established. A method to fix this cause is given in Section III-C6.

This category of flaky tests is Python-specific (although, hypothetically, it may appear in other languages). Dictionaries (hash maps or hash tables) are implemented as an unordered collection from Python 3.3 to 3.7 [6]. When tests have order dependencies, test outcomes can become non-deterministic, which leads to flakiness. For example, pull request #8627 in qiskit-terra reveals a flaky test that uses insertion order to compare two dictionaries, hence non-deterministic results. We will explore a fix for this cause in Section III-C7.

Flaky test reports are included in this special category if there is only one observation of the cause and the cause has not been classified in previous studies (e.g., [1, 5, 6]).

For example, in Listing 8 (issue report #62 in Microsoft/Quantum), there is a space between the left double quotation mark and Microsoft, which accidentally injects a random value into the test and causes intermittent failures.

Another example is keeping local outputs in Jupyter Notebook and preventing flaky tests from being executed in the continuous integration pipeline (pull request #453 in TensorFlow/quantum).

A final example is given in pull request #795 of qiskit-aer. It relates to the changing configuration of a QC (as engineers keep tweaking the computer’s configuration). Engineers update the configuration files of the simulator of this QC whenever the configuration changes. However, the test case expected values were hard-coded, which caused intermittent test case failures. To resolve this issue, testers started dynamically reading configuration parameters from configuration files instead of hard-coding them.

A flaky test cause is classified in this special case as unknown if there is insufficient information about its cause or fix, i.e., it is impossible to identify the cause of a flaky test without detailed conversations or corresponding commits. For example, a flaky test stack trace is provided in issue report #185 of qiskit-machine-learning. The report is later closed because of insufficient information.

It is easier to compare a variable with a constant when a seed used to initialize a PRNG is fixed. Thus, 9 out of 10 randomness-related flaky test reports (i.e., 19% of all flaky test reports) are fixed by setting a random seed value to a constant (or in combination with a reduced convergence threshold as shown in pull request #8820 of qiskit-terra). For example, to avoid flakiness in Listing 3, Listing 9 (pull request #5599 in qiskit-terra) shows a solution that replaces the default non-constant random seed value with a fixed seed value.

While flakiness can be controlled by fixing seeds for the PRNGs, this approach can potentially make testing less effective because it limits possible execution paths that can expose real bugs [8]. Further, a change in the PRNG algorithm may make the test case flaky again.

We have not seen robust fixes to this issue that involve running the test case multiple times to compute the distribution¹¹1Although we saw a strategy of running the test three times and declaring success if the test passes at least once (pull request #724 in netket). of successful and failed executions, then performing distribution analysis to assess whether new changes to the code alter the distribution (see [8] for details). This could be due to higher computation and execution time requirements. Additionally, programmers may not want to build such a test framework from scratch and do not realize that test frameworks like this, e.g., FLEX [8], already exist.

Finally, we find one flaky test that is fixed by replacing the PRNG with a deterministic formula (pull request #3585 of qiskit-terra, as shown in Listing 10).

Flaky tests can be removed by updating or changing the development environment. In the seven observed flaky test reports related to the software environment (see Table II), four of them (9% of all the reports) are fixed by upgrading or changing the dependencies. Listing 11 demonstrates a fix by removing the dependency on a specific numpy version (more details can be found in pull request #4656 of qiskit-terra).

The remaining three reports are fixed by changing the configuration of the continuous integration pipeline (issue #319 in Microsoft/qdk-python), simplifying the test case (pull request #1369 in netket), or simply closing the report because of dependency changes (issue #1466 in qiskit-aer).

Multi-threading-related flaky tests can be resolved by limiting the number of threads to one. Setting a single thread resolved two of the six flaky tests (4% of all reports) related to multi-threading. For example, in pull request #780 of qiskit-experiments, developers observe occasional timeout issues when running multiple tests because multi-threading is disabled in certain environments. Though the flakiness may be resolved by disabling multi-threading, performance overheads may arise. Another example can be seen in Listing 12 (pull request #6539 in qiskit-terra), which shows the fix for the flaky test in Listing 4 by disabling the parallelization.

For flaky tests related to floating point operations, one can adjust tolerance. Three out of five reports, 6% of all the reports, are fixed this way. For example, one can increase the tolerance manually, e.g., by doubling the tolerance value as shown in Listing 13. However, hard-coded loose tolerances may result in increased false negative test results, affecting the correctness of the program.

Setting dynamic tolerance may be a more robust solution. For example, to fix the issue shown in Listing 5, the developers compute the tolerance level dynamically, as shown in Lines 2 and 3 of Listing 14.

Developers also round the actual value to combat numeric errors (e.g., pull request #4835 of qiskit-terra), or remove a flaky test case altogether if it is deemed non-essential (e.g., pull request #8582 of qiskit-terra).

Flaky tests caused by unhandled exceptions can be mitigated by adding an exception handler or removing the exception (all three reports, or 6% of all the reports, are fixed this way). For example, developers add an if condition to remove any unexpected negative coefficients in pull request #399 of Microsoft/QuantumLibraries (see Listing 15).

We observe one (i.e., 2% of all the reports) network-related flaky test report, i.e., issue #584 and pull request #588 in qiskit-ibm-runtime. Here, tests for callback functions are dependent on socket tests. However, sometimes those callback tests finish before the socket tests, which causes flakiness. Developers increase callback test iterations so that socket tests have sufficient time to finish first (see Listing 16). This may be a suboptimal solution as the number of iterations is hard-coded. The ideal solution would be to synchronize callback tests with the completion of socket tests.

For flaky tests caused by unordered Python dictionaries, developers can use key values for ordering instead of using the insertion order. In Python 3.8 and later, dictionaries are order-preserved, so upgrading the Python environment can solve the problem. However, Python upgrades can cause dependency issues. One flaky test report (i.e., 2% of all reports) had this cause and fix.