Quantum Vulnerability Analysis to Guide Robust Quantum Computing System Design

The flow for generating a quantum executable from a given algorithm and evaluating it on a quantum chip is illustrated in Fig. 1. The quantum compiler will be given information about the target quantum chip and compiler strategy, such as optimization levels, initial layout method, mapping method, etc. Based on that input, the compiler will follow all intermediate compiling steps to generate a compiled circuit for execution on a quantum computer.

When operating a superconducting quantum computer, operations may fail due to poor environmental conditions, inaccurate control, state decoherence, and more. The error rate associated with an operation, Operational Error Rates, is closely estimated by randomized benchmarking, described in detail below, which approximates the extent of failures without revealing the exact source. The lifetime of a qubit, or its ability to retain a quantum state, is determined by its relaxation time (T1) and decoherence time (T2), so-called Retention Errors. The decoherence and relaxation time represent the qubit’s average time to retain its energized and superimposed states, respectively.

Randomized Benchmarking: Operator performance must be accurately characterized to use a quantum computer effectively. Unfortunately, quantum computer noise models are complex, and it is unscalable to completely characterize system noise via process tomography [39]. In addition to scaling considerations, characterization procedures must separate noise associated with quantum gates from errors stemming from state preparation and measurement to ensure that computation quality can be adequately estimated. Randomized benchmarking[27, 26] is a method of assessing quantum computer hardware that achieves an average error rate for operations through a process known as twirling. At the high level, twirling implements long sequences of random gate operations and fits the resulting data to a curve to determine the average error. Because randomized benchmarking considers only the exponential decay of sequences of random gates, sensitivity to measurement noise in the resulting average error is minimized. Meanwhile, the T1 and T2 errors on gate operation are included as part of the gate’s error rate reported by the randomized benchmarking [36]. Randomized benchmarking, while applicable to systems of any dimension, is predominantly employed for single-qubit or two-qubit gates [11]. This preference arises from the exponential growth in the number of required gates as the system’s dimension increases, making the method less practical for larger systems. Despite this, the technique can be adapted to pinpoint errors due to unintended crosstalk [12]. Notably, IBM utilizes randomized benchmarking in each calibration cycle to ascertain error rates for their quantum computer’s single and two-qubit gates, as reflected in the system’s properties.

Success Rate: The success rate (SR) is used to gauge quantum program performance on a quantum computer. We compute SR by dividing the number of correct outputs by total executions, shown in Equation 1. For more details on quantum computing, we refer to [35].

Early quantum computing research was focused on designing quantum hardware [28], instruction set architecture [8], and quantum computer microarchitecture [10, 9, 32, 33]. Afterward, the temporal and spatial noise variation challenges of SC quantum computers were studied to discover mapping and allocation-enhanced compilations to make algorithm execution more robust to diverse errors[46, 31, 20, 21, 22]. Additionally, works such as [18] contribute to the understanding of how noise, fidelity, and computational cost interplay in quantum processing, enriching the broader discourse on quantum system performance. Currently, the focus of quantum computing is on optimizing the success rate by applying different compiler strategies, such as mitigating the effect of errors by enhancing the quantum instructions [43, 14], decreasing measurement errors [48, 13], mitigating crosstalk errors [34, 7], combining pre- and post-execution software approaches to improve performance [47, 19], and compiling with specific constraints [23, 7].

While many studies have improved quantum program performance, two areas have been underexplored: 1) accurate SR prediction for specific compiled circuits and 2) better modeling of error/algorithm relationships in current quantum systems. Regarding SR prediction methods, popular alternatives to noisy quantum computer simulations include using machine learning for SR prediction, which treats the entire computation as a black box [25], developing detailed noise models, and methods like statistical fault injection [17, 37, 41] and estimated success probability (ESP) [47, 46, 36].

Machine Learning-based: The Machine Learning-based Success Rate (SR) prediction method, referenced in [25], simplifies quantum computations into a black box model. This approach can blur distinctions between circuits with varied gate parameters and requires retraining when adapting to different quantum machine sizes. Data collection for larger machines is resource-intensive, especially given the method’s requirement to gather data within a single calibration period. Furthermore, its validation, limited to specific compiled circuits, may not account for the complexities of larger machines or diverse error-mitigation strategies.

Fault Injection-based: The statistical fault injection method employs classical computers to simulate full-state quantum computations. During this process, errors are systematically injected into each basis gate based on specific triggering probabilities [15]. A very recent work [37] employs fault injection methods to evaluate quantum vulnerabilities. This study proposes the use of quantum fault injection to scrutinize circuit vulnerabilities, with a specific emphasis on radiation-induced errors. We note, however, that the fault injection approach may face challenges when applied to large machines. This complexity is accentuated when sampling a broad spectrum of circuits that exhibit variations in qubit size, circuit depth, and concurrent fault counts, especially in the context of larger machines and intricate algorithms. Moreover, the quantum vulnerability factor (QVF) metric proposed in [37], can face challenges when assessing circuits reaching the quantum supremacy, typically seen in machines with 50+ qubits [4]. The QVF calculation hinges on the contrast function, which necessitates prior knowledge of P(A), the expected correct state. In the absence of this knowledge, the correct state must be determined through a noise-free simulation. Relying on classical computing for full-state quantum circuit simulation poses a significant challenge, as such methods approach the limits of classical computational capabilities. Alternatively, our research introduces a different approach, formulating a noise model addressing 1- and 2-qubit gate errors, measured errors, and crosstalk errors. Our methodology is crafted with a focus on scalability.

Estimated Success Probability-based: The estimated success probability (ESP), shown in Equation 3, predicts the correct output trial probability by multiplying the success rate, or fidelity, of each gate ($g_{i}^{s}$) and measurement ($m_{i}^{s}$) operations, generated by one minus the gate ($g_{i}^{e}$) and measurement ($m_{i}^{e}$) error rate in equation 2. While ESP considers all circuit operations, the product treats all gate errors that contribute to the final SR estimation equally when some gate errors influence the final circuit outcome more or less than others. As a basic demonstration of the inaccuracy of ESP modeling, the gate success products in Equation 3 commute, whereas most operations in quantum circuits are fixed in ordering [29]. Based on such position differences of the gates on the compiled circuits, gate errors will contribute differently based on their propagation path to the measurement, which influence the estimated success rate differently than their original gate error rates. The detailed analysis is presented in Sec.III-B. On the other hand, the simplicity of the ESP metric has made it frequently applied in quantum compiler design and circuit optimization efforts as a method to predict quantum program success on quantum computers [47, 2, 38, 3, 32, 36, 24]. However, if a better SR estimator was available, the effectiveness of the aforementioned quantum computer optimizations could potentially experience significant improvements.

We used statistical fault injection, ESP_CP [46], and ESP, as illustrated in Fig. 2, to estimate the Success Rate (SR) of the Bernstein-Vazirani (BV) and Quantum Fourier Transform (QFT) algorithms across varying scales on three distinct quantum computers. The BV algorithm was selected due to its relatively shallow depth, allowing us to demonstrate the effects of significant algorithm size increments. Conversely, the QFT, characterized by deeper circuits, exhibits only modest size increases. For a comprehensive evaluation, both algorithms were tested with three different input sizes, yielding actual success rates ranging from 70% to 5%. ESP_CP, a variant of ESP, focuses solely on multiplying the success rates of gates situated on the critical path. Unfortunately, the predicted outcomes from both methods show significant deviation from the actual SR, with discrepancies ranging from 25% to 60% and relative error rates ranging between 70% to 470%. Further compounding the issue, both methods produce SR estimations that diverge sharply from the real machine results as the circuits increase in size, indicating that their tendency toward scaling is not well performed.

The ESP model, upon closer examination, presents potential sources of inaccuracies in SR prediction. Illustrated in Fig. 3, the ESP model accurately predicts the SR only for the middle circuit. When considering a compiled circuit, its final SR results from the product of individual qubit success rates, which are pre-measured. Therefore, only errors impacting a measurement gate influence the final output.

For scenarios akin to the first circuit, the ESP model tends to overestimate error rates. Here, the error from the red-boxed $Z$ gate doesn’t influence the measurement gate, meaning the $Z$ gate’s error only impacts the success rate of $Q2$. Yet, this isn’t captured by the subsequent $Q1$ measurement gate.

Contrastingly, for situations resembling the third circuit, the ESP model tends to underestimate error rates. Errors originating from the two red-boxed $H$ gates not only affect the measurements of their associated qubits but also influence other qubits via the CNOT gate. Despite this, the gate error to success rate transformation, as outlined in equation 2, only accounts for this error once. Any subsequent error impacts on other qubit measurements arising from error propagation are disregarded in the ESP model.

Such observations underscore that certain errors exert more influence on the final output than others, highlighting the need for a refined approach. This analysis emphasizes the importance of taking into account circuit structure and architectural vulnerabilities when estimating SR on actual quantum computers.

In Section III-B, we discover that the SR of quantum computation is the outcome of the success rate of each qubit being measured. Each measured qubit’s correctness is influenced by gate errors propagated to it. Our proposed quantum vulnerability analysis (QVA) is a systematic methodology that follows error propagation. The QVA will estimate the vulnerability of the compiled circuit by performing error modeling based on the error rates from randomized benchmarking calibration.

The QVA generates the cumulative quantum vulnerability (CQV) metric. Definition of CQV: CQV presents the final circuit’s vulnerability by predicting the failure rate (FR) for the compiled circuit. We emphasize that the CQV will not predict the correct result but the possibility of an incorrect result during runtime. The calculation of $1-CQV$ represents the estimated success rate of a compiled circuit on the target machine calculated with the CQV. In Fig. 4.a, we present the complete workflow of QVA.

Circuit Array Generator: To understand the error propagation path within a quantum circuit during runtime, a connection between when an error occurs and how much it affects the compiled circuit must be established. For more granularity, we quantify the compiled circuit to a finer degree by representing the algorithm at the cycle level. Cycle-level representation for a quantum circuit is analogous to the classical electrical circuit diagram to replace the previous analysis at the level of the complete compiled circuit. The Circuit Array Generator block transfers the compiled circuit further to a $2D$ array where each element represents the attributes of the physical qubit at that cycle, as shown in Fig. 4.b. The circuit array records each physical qubit’s attribution in every cycle, including the gate type, gate error, associated virtual qubit, its cumulative success rate, etc. Based on the cycle level compiled circuit, a snapshot of the operating quantum chip at a given cycle can be linked with the corresponding cycle in the compiled circuit.

Calculating ($1-CQV$): The CQV methodology aims to predict the failure rate for a given compiled circuit on a designated quantum chip by effectively modeling the errors based on its propagation. The Calculating $1-CQV$ block of Figure 4.a will first receive a circuit array with attributes filled. Next, we perform a crosstalk error calibration based on [34] and update it to the $gate\ error$ by multiplying their success rate based on Equation 2. To determine the success rate estimation, which is $1-CQV$, we introduce an algorithm (referenced as Algorithm 1). This algorithm progressively updates the cumulative success rate (CSR) of each physical qubit based on the circuit array, considering the propagation of errors.

The algorithm initiates by setting the CSR for all entries in the Circuit Array to a perfect score, i.e., CSR = 1 (100% success), as depicted in line 3. Subsequent steps, from lines 3 to 16, loop through all the gates, updating CSR values. For single-qubit gates, lines 6 and 7 modify the CSR for that gate by multiplying its total success rate with the preceding CSR value of the same qubit from the last cycle.

Complex operations like the CNOT gate necessitate a deeper understanding. Here, errors from one qubit can cascade to itself and affect the paired qubit. In lines 8 to 11, for every occurrence of such two-qubit interactions in a given compiled circuit, we introduce a weight $w$. This weight, which lies between 0 and 1, signifies the fraction of cumulative error originating from the paired qubit that might propagate via the CNOT gate.

This error propagation model stems from the constraints imposed by the error rates disclosed through randomized benchmarking. The intricacy lies in the fact that these reported error rates cannot be disaggregated into individual types, such as phase, bit, or decoherence errors. Each type behaves differently when channeled through the CNOT gate.

For a target qubit involved in a CNOT gate, its CSR is computed as a product of its own success rate ($g^{s}$), its preceding CSR, and the success rate inherited from its paired qubit. This inherited success rate factors in only the weighted portion of the cumulative error. It’s crucial to remember that the success rate (or the associated error rate) for any quantum element (qubit, gate, or circuit) can be deduced using equation 2, which is based on the complementary relationship of success and error rates.

For idle cycles in the quantum circuit, we attribute an error value of zero. In the case of repeated gates, either compiler-introduced or manually inserted by the programmer, we appoint the error value based on the known error rates of such gates in the target machine.

In conclusion, the $1-CQV$ value, which represents the overall success rate prediction, is derived by multiplying the CSRs of all measurement gates. Upon examining Algorithm 1, it becomes evident that its complexity is $O(G)$, where $G$ represents total gate counts for all types. This complexity arises because the algorithm iteratively checks each physical qubit’s gate in every cycle to update the corresponding cumulative success rate. Consequently, the algorithm scales linearly with the gate count, encompassing all gate types, including ideal gates. This linear scalability of our noise model offers a significant advancement, facilitating both current NISQ-era and future quantum computing endeavors without being constrained by the limitations of classical computational power.

CNOT Weight Selection: In Sec. V, we have provided a study to observe the weight selection impact for various compiled circuits with different error profiles. Then we used a machine learning-based method to learn the weight value and used it to infer the proper weight in the CQV calculation. For more details. please refer to section V.

Algorithm size within Quantum Volume: Here, we present the CQV prediction performance for all six benchmarks on the Quantum machines IBMQ_Montreal, IBMQ_Toronto, and IBMQ_Mumbai. As shown in Fig. 9, we presented the 1-CQV prediction accuracy compared with ESP with varying compiled circuits for a five-qubit QPE algorithm on IBMQ_Montreal on Apr. 5th, 2022. The different configurations are guidelines for the compiler to generate the final compiled circuit based on the given logical circuit and target device.

We observe that $1-CQV$ is much closer to the real SR than the ESP. After being shown to the right of Fig. 9, the average absolute error rate for ESP over all the configurations compared to ground truth SR is 29.8%, while $1-CQV$ achieves an average error of 4.8%. Such an error rate difference means that $1-CQV$ achieves an 84% error reduction compared to the ESP, which also means the CQV calculation is adaptable to the variation of errors across different calibration periods and provides excellent predictions.

As shown in Fig. 10, we present the 1-CQV prediction for all the benchmarks on the single machine IBMQ_Montreal. We include experiments with an SR higher than 0.1%. The relative prediction error is produced by dividing the absolute prediction error by the SR. The trend emerges that 1-CQV prediction outperforms the ESP by achieving an average of 6 times less relative prediction error rate and the best improvement of 30 times. Meanwhile, the relative prediction error jumps clearly when the algorithm size increases, which follows the nature of the benchmark by employing significantly more two-qubit gates. When increasing algorithm size, not only does the number of two-qubit gates increase, but the number of swapping operations also increases. As shown in Fig. 11, 1-CQV presents a stable and accurate average relative prediction error across all machines and benchmarks. Based on the results, we conclude that the CQV achieved the goal of designing a more precise SR estimator consistently across different dates, machines, and algorithms than the state-of-the-art SR estimator.

Algorithm size Beyond Quantum Volume: From the results shown in Fig. 9, 10, 11, $1-CQV$ proves to predict SR with a closer distance to the real SR even when it falls in the 10% to 0.1% range. The primary reason for the low success rate is that the size of the compiled circuits equals or exceeds the desired quantum volume of the target quantum machine. Quantum volume can be defined as the product of the number of virtual qubits and maximum circuit depth supported by the machine.

After making a full-spectrum comparison among all the backends and benchmarks, from the observations of the results, we can say that the CQV has better error modeling than the ESP noise model. Fig. 10 demonstrates that the CQV prediction is stable across different algorithms with a much lower relative error rate. Additionally, the results show that CQV performs much better when the algorithm reaches or exceeds the quantum volume, which is a valuable property when the limited quantum volume is the bottleneck of the current NISQ era. Therefore, we conclude that the $1-CQV$ prediction is accurate across the full spectrum of algorithm sizes and success rates.

In the rapidly evolving landscape of quantum computing, the intricacy of quantum circuits is escalating, bringing us closer to addressing real-world challenges. As we navigate this frontier, the need for prediction models that are both accurate and scalable becomes paramount. Our study, therefore, focuses on the scalability of the Quantum Vulnerability Analysis (QVA), examining its efficiency across a spectrum of quantum circuit sizes.

The Quantum Fourier Transform (QFT) was chosen as the benchmark for our evaluation. With input sizes spanning from 5 to 120 qubits, it’s noteworthy that machines operating with around $50$ qubits are on the threshold of quantum supremacy [4]. Our scalability experiments for QVA were conducted based on the topology and error profile of the IBMQ_Washington machine, a state-of-the-art 127-qubit quantum computer.

To ensure our execution time assessment was both transparent and unbiased, we considered only the inference time of the Graph Neural Network (GNN), which provides the CNOT weight value, and the execution time of the CQV noise model. This choice was made because the GNN training is an offline process, executed just once. As shown in Table II, our CQV approach demonstrated linear scalability, with execution time increasing proportionally with the CNOT gate counts. It is worth mentioning that the execution of the CQV takes place on the CPU, including the pre-trained GNN to infer the weight value $w$. This linear trajectory underscores the potential of our model to efficiently manage complex quantum circuits in the future.

However, the full-state quantum circuit simulator-based approaches face challenges in scalability, particularly with circuits that exceed 32 qubits. This limitation highlights the simulator’s constraints when tasked with emulating large-scale, real-world quantum systems. In contrast, our approach requires only a fraction of the computational power for estimating success rates. To both validate our predictions and refine our noise model, particularly for circuits approaching quantum supremacy, we’ve devised a novel method: by merging the original circuit with its inverse and using the input states as a benchmark, we can effectively measure the circuit’s performance and fine-tune our noise model. It’s pertinent to note potential overfitting for specific circuits due to the reliance on reverse circuits. Nevertheless, we believe our approach adds a valuable technique to the toolbox for exploring circuit vulnerabilities.

Furthermore, while our preliminary results on QVA’s scalability are promising, they represent just the tip of the iceberg. Our model’s design is inherently versatile, unencumbered by specific error rates, gate types, or circuit structures. This adaptability not only allows for the integration of partial error correction techniques in larger devices but also hints at the vast potential for future optimization strategies, further refining the success rate estimation process.