QuaLITi: Quantum Machine Learning Hardware Selection for Inferencing with Top-Tier Performance

Qubits are the fundamental units of a quantum computer, equivalent to bits for classical computers. A qubit stores information in a quantum state, which is represented using a 2x1 vector. It is mathematically denoted as $\ket{\psi}=$ $\big{[}\begin{smallmatrix}\alpha\\ \beta\end{smallmatrix}\big{]}$, where where $|\alpha|^{2}$ represents the probability of qubit being measured to 0 and $|\beta|^{2}$ represents the probability of qubit being measured to 1. There are two special states with $\alpha=1$, $\beta=0$ ( $\ket{0}=$ $\big{[}\begin{smallmatrix}1\\ 0\end{smallmatrix}\big{]}$) and $\alpha=0$, $\beta=1$ ( $\ket{1}=$ $\big{[}\begin{smallmatrix}0\\ 1\end{smallmatrix}\big{]}$). These are known as basis states and are quantum analogous to classical 0 and 1 bit values respectively. The quantum state of a qubit is changed with the help of quantum gates, which are unitary matrix operations. These gates work either on a single qubit (e.g., Hadamard gate, Pauli X/Y/Z gates, etc.) or on multiple qubits (CNOT gate, Pauli CY/CZ gate, SWAP gate, Toffoli gate, etc.). Combining qubits and quantum gates, we obtain quantum circuits that are ordered sequences of quantum gates placed on qubits. All the gate operations of qubits are eventually collapsed to classical bit values (either 0 or 1) [22]. A special kind of quantum circuit, known as Parametric Quantum Circuit (PQC) consists of parametric rotation gates (such as U, Pauli RX/RY/RZ, etc.) that can be tuned classically using traditional optimization algorithms. These PQCs can be thought of as trainable ML models that form an integral part of QNNs in QML.

Modern NISQ quantum computers today are typically accessed via a cloud service from vendors such as IBM [23], Google [24], and Amazon [25]. The general way to access a quantum computer is (1) the user would write the program containing the specific quantum circuit for their task, (2) the user would then send the program to the cloud service along with the target hardware to run the quantum circuit on and some extra metadata (such as number of shots, optimization level for transpilation, number of ancilla qubits, etc.), (3) the cloud service would allocate the user’s program to the desired hardware, after which the hardware would run the program, generate the results and send them back to the cloud service, (4) finally, the cloud service sends the results from the quantum hardware back to the user.

However, there is a major problem that arises from such an access model. Due to the limited availability of quantum computers, a single quantum computer has a long access queue that can have up to hours of wait time until the user’s program reaches the top of the queue. This problem is further aggravated for QML algorithms, that require iterative runs for optimizing the rotation gate parameters for which selecting a quantum hardware that has less wait time is an important aspect for consideration in the QML training pipeline.

The performance of QNNs, like all quantum computing systems, is significantly influenced by various forms of noise, each impacting the accuracy and efficiency of the system in distinct ways. A common source of error within QNNs is decoherence, a phenomenon where qubits lose their quantum state due to unintended interactions with the external environment. This loss is essentially an energy dissipation from the qubits, leading to a degradation of the quantum coherence of the system and, consequently, its computational capabilities. Another error is crosstalk which occurs when there are unwanted interactions specifically between qubits that are quantum mechanically coupled. These interactions can alter the state of neighboring qubits in an uncontrolled manner, introducing errors into the computation process.

The implementation of quantum gates also introduces potential sources of error. Quantum gates in QNNs are typically realized through the application of microwave pulses in systems utilizing superconducting qubits, or laser pulses in the case of trapped-ion qubits. Any inaccuracies or imperfections in these pulses can result in gate errors, where the intended quantum operation is applied incorrectly, leading to deviations from the expected computational outcome. Lastly, the process of measuring quantum states introduces another avenue for error, known as readout errors. Quantum measurement operations can vary widely depending on the physical implementation of the qubits. For instance, photonic qubits are often measured using photon detectors, trapped-ion qubits may be measured through the intensity of fluorescence, and superconducting qubits might be measured via resonator coupling. Each measurement technique has its own set of potential inaccuracies, whether due to imprecise measurements, equipment limitations, or inherent inaccuracies in the measurement apparatus itself. These readout errors can significantly affect the accuracy of the quantum computation, as they directly influence the interpretation of the quantum system’s final state.

Many efforts have been attempted to reduce the effect of quantum hardware limitations. For example, [16] was proposed to run concurrent executions of different training data within the same batch for the same QNN circuit on different available qubits. [17, 18] propose converting Variational Quantum Algorithms (VQA) into pulse schedules such that the pulses are native to the quantum hardware and have tunable parameters. By training these parameters, the authors train the original model. Works like [19, 20] take the effect of noise into account and incorporate them into the VQA, such as creating noise-resilient state preparation circuit for better training, and injecting quantum noise during training to make it noise-aware and performing post-measurement processing, such as quantization and normalization of measurement outputs. Finally, [21] propose training unitary operator representation of a QML ansatz ($2^{N}*2^{N}$ in size for $N$ qubits) as compared to training the ansatz itself. The authors use a gradient descent algorithm for optimization and also use partitioning of the unitary operator for further time complexity reduction. Out of all these works, only [16] takes hardware queue wait time into consideration in its study and achieves up to 20x speedup. The proposed approach is complementary in nature and could be augmented in conjunction with [16] for additional benefit.

We first choose a set of quantum hardware and their corresponding configurations for training and inferencing QML models. Considering the latest release of Qiskit 1.0, we select Fake20QV1() (20 qubits), Fake27QPulseV1() (27 qubits) and Fake127QPulseV1() (127 qubits) noisy simulators (containing real hardware calibrated data) from qiskit.providers.fake_provider library for conducting our experiments. The coupling map of each of these hardware is shown in Fig. 3. We select five topologically different 8 qubit coupling maps (labeled I-V) for each hardware as shown visually in Fig. 3(d) and with individual qubit information in Table I. Considering the exponential simulation runtime increase with growing qubits [26, 27], we pick small-scale datasets such as the Iris and UCI Digits dataset for the training process. Furthermore, we select all 3 classes in the Iris dataset for classification and 2 sets of 2 class datasets in the Digits dataset: 0,1 and 8,9 for performing classification²²2Henceforth, we refer to them as Digits01 and Digits89 respectively..

We use 8-qubit QNNs containing classical data loading circuits, Parametric Quantum Circuit (PQC), and measurement operations. We select the input data loading scheme based on the dimensions of the input. Considering we have 8 qubit-QNN, the Iris dataset ($4$ size vector) can fit with an $n$ features to $n$ qubits embedding, while the Digits dataset ($64$ size vector) can fit with a $2^{n}$ features to $n$ qubits embedding. So, we use angle embedding ($4$ features on $4$ qubits) for the Iris dataset and amplitude embedding ($64$ features on $6$ qubits) for the Digits dataset [28]. Strongly Entangling Layers (SEL) [29] are selected for the PQC part since they create strong entanglement due to the presence of many entangling gates such as CNOT gates, and also have many trainable parameters which overall improves trainability of the PQC [30]. Furthermore, we take the number of classes into account and accordingly set the number of layers for each case. Specifically for the Iris dataset (with 3 classes) we choose a model with 6 SEL and for the reduced Digits datasets (with 2 classes) we choose a model with 3 SEL. Finally, the measurement operations consist of expectation value measurement in the Pauli-Z basis. For all the runs, we perform training on 10 epochs, use Adam optimizer (learning rate $=10^{-3}$), and use a batch size of 16.

Another aspect to consider in the model selection process is the ansatz used in SEL. The SELs have a range parameter $r$ that can dictate the target qubit for a given control qubit of the CNOT gate. Suppose the control qubit is on $i^{th}$ qubit (ranging $0$ to $n-1$), the total number of qubits in the QNN is $n$ and the range value is $r$, then the target qubit will be on qubit number $(i+r)$(mod $n$). As an example, we show for an 8 qubit SEL (entangling part in particular) how having different range values changes the arrangement of CNOT gates in Fig. 4. We select four range values $r=\{1,2,3,4\}$, define a model for each range value and train these models under noise using the chosen datasets to determine the model with most appropriate entanglement. For each case, the training is done on 27 qubit hardware with configuration I (linear coupling map). We note the inferencing results for this experiment in Table II. Note that the tabulated values present mean inferencing performance for 10 inferencing runs to account for fluctuations due to noise³³3For all the runs, we use mean accuracy of 10 inferencing runs.. We observe that for the Iris dataset, $r=1$ performs the best, and for Digits01 and Digits89 $r=3$ and $r=1$ perform the best, respectively. Therefore, we select these models for further analysis.

We identify the potentially best-performing configurations by analyzing their various properties such as coherence times (both T1, and T2 times), single and two-qubit error rates, single-layer post-transpilation depth, and readout error. In general, the presence of CNOT gates due to the SWAP gate insertion procedure during transpilation makes the resulting QNN circuit sensitive to the two-qubit error rate. Next, the native gate set of a particular hardware plays an important factor in determining the overall QNN depth. The readout errors (preparing $\ket{0}$, measuring $1$ and preparing $\ket{1}$, measuring 0) can further degrade the performance. Finally, individual single qubit error rates can lead to the erroneous computation of quantum state, however, their effect can be considered relatively small compared to other factors. For the best results, (i) the two qubit error rate, post-transpilation depth, readout errors and single qubit error rate should be low (inversely proportional $\downarrow$), and (ii) the coherence times should be high (directly proportional $\uparrow$). Furthermore, both the types of aforementioned readout errors should be low and both T1 and T2 times should be high. Taking this into account, we can combine the T1 and T2 times by taking their harmonic means and the readout errors using arithmetic mean. The harmonic mean is sensitive to lower values, so a pair of T1, and T2 values having even a single low value will have a lower harmonic mean, implying a lower overall coherence time for the configuration. Similarly, high readout error values will yield a higher arithmetic mean. Next, we create a score metric for every property in the range of [0,100]. We do this as follows: (i) For inversely proportional properties, we take the inverse/reciprocal of the property and normalize them in the range of [0,100]. The normalization is done by taking the minimum and maximum of the property value into account. Suppose, if $p$ is the list of all the scores of the property after taking inverse, then for a property score $p_{i}$ ($1\leq i\leq 15$ since there are 5 configurations for 3 hardware), the normalized score will be $p_{i}^{\prime}=\frac{p_{i}-min(p)}{max(p)-min(p)}*100$. (ii) For directly proportional values, we directly normalize the property score value in the range [0,100] using the aforementioned formula without taking the inverse. Finally, once the individual score for each property is computed, we combine these individual scores in a weighted fashion to obtain a final overall score for each configuration. Based on the criticality of the properties discussed earlier, we assign a weight of 0.3 to two qubit error rate, 0.2 to post-transpilation layer depth, coherence times, and readout error rates, and 0.1 to single-qubit error rate. These results are tabulated in Table III in decreasing order of final score from top to bottom. From the table, we observe that all the 27 qubit hardware configurations are top performing, owing to high coherence times and lower error rates.

From the analysis performed in Table III, it would be motivating to select the top-scoring configurations with the highest scores. However, as we can see the top 5 scoring configurations all belong to the 27 qubit hardware. From a training standpoint, while the individual coupling map configurations might be different we still dedicate all 10 epochs of training only to 27 qubit hardware. If the 27 qubit hardware has a large queue of pending jobs, then the overall training time overhead will be equivalently compounded based on the number of epochs allotted to it for training. To address this challenge, we propose an alternative configuration selection strategy to just selecting the top-scoring configurations, where (i) We select one of the top-scoring configurations from each hardware at least once during training, and (ii) the next hardware selected for training should be different from the current hardware chosen. We specifically use these two criteria since they allow the usage of top-performing hardware and avoid the case of choosing the busiest hardware back to back, potentially saving training wait time in the hardware queue. We satisfy these two conditions by selecting five configurations (2 epochs per configuration) such that the first three configurations are among the best-performing configurations from each hardware and the next two are chosen in a similar fashion but we also ensure that these configurations have the least queue wait times. For example, the top three scored configurations (in order) for each hardware are (i) 20Q: II, IV, III (ii) 27Q: II, I, V (iii) 127Q: V, II, IV. From these, we can randomly select one configuration from each hardware, switch to a different hardware, and repeat this process 5 times. We employ randomness in the selection here as the scores of specified configurations for each hardware are relatively close so selecting one configuration over the other will not make much difference in the final inferencing performance. A set of configurations chosen in this fashion for the Iris dataset could be 20Q (IV), 127Q (V), 27Q (I), 20Q (III), and 127Q (IV). The final training configurations for all the datasets selected using this procedure are shown in Table IV in order of training from left to right. Note that we assume all the chosen configurations have the least wait time.

The results for this training procedure are shown in Fig. 5. We observe that 27 qubit hardware is the best-performing hardware for inferencing, followed by 20 qubit hardware and finally 127 qubit hardware. For 20, 27 and 127 qubit hardware respectively, we note mean inferencing accuracy of (i) $92.76\%$, $93.91\%$ and $91.24\%$ for Iris dataset, (ii) $84.85\%$, $85.83\%$ and $78.47\%$ for Digits01 dataset and (iii) $77.65\%$, $83.45\%$ and $72.57\%$ for Digits89 dataset.

Variation with hardware: From Fig. 5, we observe inferencing performance variation as we switch to different hardware. In particular, we note that 27 qubit hardware performs the best, followed by 20 qubit hardware and finally the worst-performing 127 qubit hardware. This trend can be explained by examining internal hardware characteristics.

We show the mean hardware characteristics for all coupling configurations combined such as mean two qubit error rates in Table V and the native basis gate set for every hardware. We observe that 27 qubit hardware has the best two-qubit error rate (0.0085) (which matches with the high two-qubit error scores of all 27 qubit hardware in Table III). The 20-qubit and 127-qubit hardware have two-qubit error rates that are a magnitude of order higher (0.0172 and 0.0147 respectively) than the 27-qubit hardware (which explains why it shows the best performance). Furthermore, the 20 qubit hardware has a different basis gate set compared to the other two, which leads to lower post-transpilation depth. For example, an amplitude embedding circuit along with a single SEL with r=1 post-transpilation has an average depth of 608 on 127 qubit hardware while having 321 on 20 qubit hardware. Therefore, even if the two-qubit error rate is higher, the lower post-transpilation depth compensates for the error rate and gives higher performance for 20-qubit hardware, as compared to the 127-qubit hardware. However, if 27 qubit hardware has a large queue, then we may lose performance by selecting other hardware in the process of optimizing wait time. This is a trade-off between the wait time and inferencing performance that should be made while choosing the desired configurations.

Variation with dataset: As mentioned earlier, we utilize angle embedding to embed a vector of size 4 onto 4 qubits using parametric rotation angles for the Iris dataset, and amplitude embedding to embed 64 size feature vectors onto 6 qubits. Under ideal conditions, both the state preparation circuits perform well. However, when coupling constraints of hardware are taken into account, an amplitude embedding circuit requires depth exponential in the number of qubits used [31]. This contributes significantly to the overall post-transpilation circuit depth, leading to degradation in performance. This can be seen when we compare the performances of the Iris and Digits dataset in Fig. 5. We note that Iris dataset always shows greater than $90\%$ inferencing performance for any configuration, and Digits dataset shows inferencing performance below $90\%$, sometimes even below $80\%$.

We also observe significant performance differences within the two Digits datasets. Structurally, both datasets have images of the same dimensions (8x8 images, or 64x1 size vector for the case of training). This means that both datasets will have the same amplitude embedding depth post-transpilation. Numerically, we observe a post-transpilation depth for the amplitude embedding circuit of roughly 490 for both datasets. The remaining difference then would be the digits present in the images themselves, specifically the structure of the digits. For the case of Digits01, 0 has a round ‘o’ shape while 1 has line ‘$|$’ shape, while for Digits89 digit 8 has two ‘o’ shapes while 9 has one ‘o’ shape attached to a ‘$|$’ shape. Under noiseless conditions, the selected model can classify both datasets with greater than $90\%$ accuracy. However, under noise, the same model would probably find it easier to distinguish between images containing two distinct shapes (‘o’ and ‘$|$’) like 0 and 1, as compared to images having common shapes (‘o’ shape) like 8 and 9. This can potentially explain the lower performance in the Digits89 dataset as compared to the Digits01 dataset.

We visualize previous results from the viewpoint of coupling maps. We restructure multi-hardware training results from Fig. 5 in Fig. 6 in the form of boxplots for each coupling map and every dataset. An indicator of a good coupling map configuration is the lower fluctuation it shows across different hardware. Across all the three datasets combined for coupling maps I to V respectively, we observe mean fluctuations of $3.74\%$, $3.46\%$, $2.59\%$, $3.33\%$, and $2.96\%$. From these values, we can conclude that coupling maps III, and V are the most resilient for inferencing multi-hardware trained models.

We observe queue depth variation of real hardware (IBM Brisbane, IBM Kyoto and IBM Osaka) with time. We show boxplot for queue depth (recorded from 04/10/2024 16:30 to 04/12/2024 19:30) in Fig. 7. We note that IBM Brisbane is the busiest out of all three, followed by IBM Kyoto and IBM Osaka. From the boxplots, we note the overall fluctuation in queue depth (i.e. standard deviation) as 88 jobs for IBM Brisbane, 74 jobs for IBM Kyoto, and 43 jobs for IBM Osaka. To get an idea of how much wait time these queue depths translate to, we choose a dummy 8-qubit circuit (consisting of only a single RZ gate on every qubit) and send it for execution on each circuit. The queue wait times for this circuit is tabulated in Table VI. We observe that for the aforementioned time period, Kyoto has the largest wait time per circuit (44.3s), followed by Brisbane (14.1s) and Osaka (1s). Using this data, we estimate the single inference run time. For our datasets, we select a 70:30 split for training and testing, which would mean (for inferencing) 45 data points for the Iris dataset and 108 data points for both the Digits01 and Digits89 datasets. Assuming each datapoint run would wait for the computed wait time, we obtain an overall inferencing wait time of 634, 1993, and 45 seconds on IBM Brisbane, IBM Kyoto, and IBM Osaka respectively for the Iris dataset and 1522, 4784, and 108 seconds respectively for both the Digits datasets for a single inferencing run. We can also extrapolate the average wait time to estimate training wait time. For example, consider the Iris dataset for training on the IBM Osaka machine. The training set will have 105 images, so for an average wait time of 1s and 10 epochs of training, the total training wait time will be roughly 17 minutes. We tabulate all the training wait times for all datasets on all hardware in Table VI. From these values, we note up to nearly as high as 45X reduction (per epoch) in training wait time when the training hardware is switched from IBM Kyoto to IBM Osaka. This (potentially) is more than twice the speedup that is achieved in [16].

We also show that our methodology is scalable to larger datasets. In particular, we train a hybrid quantum-classical neural network (with few convolution layers followed by QNN having 8 qubits) using a multi-hardware training setup for the Cifar-10 dataset [32]. We use a reduced version of the dataset consisting of airplane (class 0) and frog (class 6) classes (300 images per class, 70:30 train-test split). Based on the best-scoring configurations from Table III, we select the following configurations: 27Q(II), 20Q(II), 127Q(V), 27Q(I) and 20Q(IV). For all configurations combined, we note mean inferencing accuracy of $80.96\%$, $82.84\%$, and $80.59\%$ for 20, 27, and 127-qubit hardware respectively. Once again, we observe that 27-qubit hardware performs the best, followed by 20-qubit hardware, and finally 127-qubit hardware. We show the boxplots of inferencing performance for various hardware in Fig. 8. We also compare these results with another model that is trained only on the best-scoring configuration 27Q(II). From this model, we note mean inferencing accuracy of $82.78\%$, $82.33\%$, and $81.82\%$ respectively for 20, 27, and 127-qubit hardware respectively. From this, we note a mean accuracy reduction of $0.84\%$ across all hardware when we switch from single hardware to multi-hardware training for the Cifar dataset.