Classical-to-Quantum Transfer Learning for Spoken Command Recognition Based on Quantum Neural Networks

The state-of-the-art baseline system of spoken command recognition (SCR) [1, 2, 3] is built upon the advancement of deep learning (DL) technology [4]. The DL technologies highly rely on two important aspects: (1) powerful computational resources arisen from the graphical processing unit (GPU); (2) the availability of access to a large amount of labelled or unlabelled training data [5, 6]. Despite the rapid empirical progress of the SCR systems, deep learning models are becoming computational expensive and, now that Moore’s law is faltering [7], it is necessary to contemplate a future technology to further deal with a huge amount of speech data. However, new exciting possibilities are opening up due to the imminent advent of quantum computing devices that directly exploit the laws of quantum mechanics to evade the technological limits of classical computation [8].

The exploitation of quantum computing machines to carry out quantum machine learning (QML) [9] is still in its initial exploratory stage. In particular, a quantum neural network (QNN) [10], which is capable of carrying out a universal quantum computation, attracts much attention in the domain of QML because it can be seen as a quantum analog of the classical deep neural network (DNN). Hence, the QNN model allows the algorithm of back-propagation to train the model parameters of the QNN. However, in the age of noisy intermediate-scale quantum (NISQ) devices [11], it is necessary to simulate quantum experiments with noisy quantum circuits that may degrade the baseline performance. A compromised QNN, namely variational quantum circuit (VQC), has been proposed to overcome the influence of quantum noise in the QNN. The advantages of the VQC in the QNN arise from many aspects: (1) a VQC is a quantum circuit with adjustable parameters that are optimized according to a predefined metric; (2) VQC is flexibly placed on a NISQ machine and can be resilient to the quantum noisy effects on quantum circuits.

Prominent examples in the hybrid QNN for machine learning tasks include quantum reinforcement learning [12], quantum image processing [13], and quantum circuit learning (QCL) [14], where the VQC model plays an important role as a quantum component. Particularly as for the application of QNNs for SCR, our pioneering work [15] investigates the use of quantum convolutional neural networks to extract quantum speech features as the input to state-of-the-art classical deep neural networks. However, the work [15] does not employ the QNN as an end-to-end SCR model, and it is still unknown about the performance of QNNs for SCR. Hence, in this work, we attempt to build a hybrid classical-quantum QNN model for SCR. In particular, we make use of a hybrid transfer learning approach to speed up the end-to-end training framework based on the QNN model.

Classical transfer learning [16, 17] is a typical example of machine learning that has been originally inspired by biological intelligence, and it originates from the knowledge acquired in a specific context which can be transferred to a different area. For example, when we learn a foreign language, we always make use of our previous linguistic knowledge to speed up the learning rate [18]. This general idea has been successfully applied to artificial intelligence and machine learning domains. It has been shown that in many situations, instead of training a full network from scratch, it should be more efficient to start from a pre-trained network so that only some model components need further fine-tuning for a particular task of interest. The transfer learning methodology is very related to the training of a hybrid classical-quantum QNN, where the classical component can be pre-trained in another generic model and directly transferred to the hybrid QNN.

This work aims to investigate a hybrid classical-to-quantum transfer learning paradigm in the context of a QNN-based SCR system. Our baseline system, namely CNN-DNN, consists of two core components: (1) convolutional neural network (CNN) for speech feature extraction; (2) DNN for recognizing commands. In this work, we first build a hybrid classical-quantum model, namely CNN-QNN, where the CNN is still kept for feature extraction and the DNN is replaced by the QNN model. In doing so, a classical CNN-DNN becomes a hybrid classical-quantum CNN-QNN. Although the hybrid model follows an end-to-end learning pipeline, the frequent data conversion between classical and quantum states greatly slows down the computing efficiency in the training stage. Moreover, the current NISQ device admits only a small number of qubits for the QNN so that the representation power of the CNN-QNN may become quite limited. Thus, a new training strategy needs to be proposed. Here, we put forth a novel hybrid transfer learning strategy. In more detail, given a well-trained classical CNN-DNN, the CNN model is taken to initialize the CNN component of the CNN-QNN and several additional training steps are conducted to fine-tune the parameters of the QNN.

The rest of the paper is organized as follows: Section 2 introduces some notations. Section 3 introduces the architecture of the VQC-based QNN. Section 4 presents the framework of our SCR system. Section 5 shows the hybrid transfer learning algorithm. Our simulation results will be reported in Section 6, and the paper is concluded in Section 7.

We denote $\mathbb{R}^{d}$ as the d-dimensional real coordinate space. Given a vector $\textbf{v}=[v_{1},v_{2},...,v_{d}]^{T}\in\mathbb{R}^{d}$, a $d$-qubit quantum state $|\textbf{v}\rangle=\otimes_{i=1}^{d}|v_{i}\rangle=|v_{1}\rangle\otimes|v_{2}\rangle\otimes\cdot\cdot\cdot\otimes|v_{d}\rangle$ is a quantum state associated with a $2^{d}$-dimensional vector in a Hilbert space, where for a scalar $v_{i}$, the quantum state $|v_{i}\rangle$ can be written as Eq. (1).

Figure 1 exhibits the architecture of the QNN including three components: (1) quantum encoding; (2) VQC; (3) measurement. The introduction of each component is shown as follows:

1. 1.

   The framework of quantum encoding bridges the relationship between the classical data input x and its quantum state $|\textbf{x}\rangle$. In other words, quantum encoding is associated with the generation of quantum embedding from the classical input vector $\textbf{x}=[x_{1},x_{2},x_{3},x_{4}]^{T}$. The quantum state $|\textbf{x}\rangle$ can be written as Eq. (2).

   $$\begin{split}&|\textbf{x}\rangle=|x_{1}\rangle\otimes|x_{2}\rangle\otimes|x_{3}\rangle\otimes|x_{4}\rangle\\ &=\left[\begin{matrix}\cos(x_{1})\\ \sin(x_{1})\end{matrix}\right]\otimes\left[\begin{matrix}\cos(x_{2})\\ \sin(x_{2})\end{matrix}\right]\otimes\left[\begin{matrix}\cos(x_{3})\\ \sin(x_{3})\end{matrix}\right]\otimes\left[\begin{matrix}\cos(x_{4})\\ \sin(x_{4})\end{matrix}\right]\\ &=\left(\otimes_{i=1}^{4}R_{Y}(2x_{i})\right)|0\rangle^{\otimes 4}.\end{split}$$

   (2)

   Then, the circuits of quantum encoding in Figure 1 produce the following quantum state as Eq. (3).

   $$\begin{split}&\hskip 5.69054pt\left(\otimes_{i=1}^{4}R_{Y}(\pi x_{i})\right)|0\rangle^{\otimes 4}\\ &=\left[\begin{matrix}\cos(\pi x_{1})\\ \sin(\pi x_{1})\end{matrix}\right]\otimes\left[\begin{matrix}\cos(\pi x_{2})\\ \sin(\pi x_{2})\end{matrix}\right]\otimes\left[\begin{matrix}\cos(\pi x_{3})\\ \sin(\pi x_{3})\end{matrix}\right]\otimes\left[\begin{matrix}\cos(\pi x_{4})\\ \sin(\pi x_{4})\end{matrix}\right].\end{split}$$

   (3)

2. 2.

   The model of the VQC in the dashed square consists of CNOT gates and adjustable rotation gates $R_{X},R_{Y},R_{Z}$. The CNOT gates mutually impose quantum entanglement between any two quantum wires, so that the qubits from all the wires can be entangled. The rotation angles $\alpha_{i}$, $\beta_{i}$, and $\gamma_{i}$ are adjustable and can be taken as the trainable parameters for $R_{X}$, $R_{Y}$, and $R_{Z}$, respectively.

3. 3.

   The outputs of the quantum states should be projected by measuring the expectation values of $4$ observables $\hat{\textbf{z}}=[\hat{z}_{1},\hat{z}_{2},\hat{z}_{3},\hat{z}_{4}]$ for a classical vector z as Eq. (4).

   $$\mathcal{M}:|x\rangle\rightarrow\textbf{z}=\langle x|\hat{\textbf{z}}|x\rangle.$$

   (4)

Moreover, the rotation gates $R_{X}$, $R_{Y}$, and $R_{Z}$, which are associated with the unitary matrices in Figure 2, stand for a linear mapping between quantum inputs and quantum outputs. One key advantage of the QNN is that fewer model parameters are involved in the QNN model. For example, Figure 1 shows 4 quantum wires and 6 VQC layers, which result in 72 trainable model parameters in the QNN.

Besides, the QNN model can be trained in an end-to-end pipeline based on the back-propagation algorithm with different stochastic gradient descent (SGD) optimizers such as Adam [19], RMSprop [20], and Adadelta [21]. This is because the update of the QNN parameters follows a first-order optimization technique to minimize a loss function over the dataset, which is represented as Eq. (5).

where $\Theta=[\Theta_{1},\Theta_{2},...,\Theta_{d}]^{T}$ are the parameters to be learnt, $\mathcal{L}$ is the loss over the data, and $\eta$ is the learning rate. Given a small $\epsilon$, the partial derivative term can be approximated by using the finite difference method as Eq. (6).

Next, we illustrate the architecture of the hybrid classical-quantum QNN model for SCR in Figure 3. The CNN framework consists of four 1D convolutional layers (Conv1D) followed by batch normalization (BN) and the ReLU activation. A max-pooling layer with a kernel of $4$ is also used after each Conv1D. The output of the CNN framework is a set of high-dimensional CNN features, which should be compressed to low-dimensional features (Dense) connected to the quantum encoding framework in a QNN. The encoded quantum states go through the QNN model and the measured outputs correspond to the classification labels for a certain task. The output of the QNN is connected to a classification layer by a non-trainable matrix. This hybrid classical-quantum QNN is denoted as CNN-QNN.

Accordingly, a CNN-DNN model is shown in Figure 4, where the QNN is replaced by a classical DNN. The feature reduction component is removed, but more model parameters are included in the DNN model.

As discussed in the introduction, hybrid classical-to-quantum transfer learning is significantly appealing in the current technological era of NISQ devices. Although there are so many technical limitations in the usage of quantum computing nowadays, e.g., a small number of available qubits and noisy quantum circuits, the NISQ computers are approaching the quantum supremacy milestone [22, 23]. At the same time, we could make use of the very successful and well-tested tools of classical deep learning, especially for speech and language processing tasks in which some specific datasets are not large enough to attain well-trained neural networks. However, prior knowledge of some networks that are pre-trained on generic datasets can be shared with a new model for a specific task.

In classical machine learning, transfer learning has been widely used in certain applications. For example, the generative pre-trained transformer (GPT) [24] and bidirectional encoder representations from transformers (BERT) [25] are always adapted to particular language processing tasks by simply appending a few neural networks on top of the pre-trained language models. Similarly, a hybrid classical-to-quantum transfer learning algorithm comprises of applying exactly those classical pre-trained models as feature extractors and then post-processing these features on a quantum computer. As for our transfer learning algorithm applied to SCR, a pre-trained generic CNN-DNN model is obtained beforehand based on a classical generic dataset as shown in Figure 5. The CNN model of the pre-trained CNN-DNN, namely CNN’, is transferred to the CNN-QNN, which generates CNN’-QNN. During the training stage of the CNN’-QNN, the CNN’ model is fixed, and only the parameters of the QNN need further fine-tuning on the classical specific dataset. Since the transferred CNN is not involved in the training process, a small number of specific training data is enough to optimize the model parameters of the QNN. Besides, the hybrid classical-to-quantum transfer learning significantly speeds up the training efficiency because the overhead of the frequent communication between classical and quantum devices can be avoided.

In this section, we assess our hybrid transfer learning algorithm for CNN-QNN in the following aspects:

1. 1.

   Performance gain can be obtained by applying the hybrid transfer learning technique.

2. 2.

   The hybrid transfer learning can work with both noiseless and noisy quantum circuits.

This work focuses on a hybrid classical-to-quantum transfer learning algorithm for QNNs applied to SCR. We first set up the VQC-based QNN, and then design a CNN-QNN-based SCR system. We employ hybrid transfer learning to transfer a pre-trained CNN framework to our CNN-QNN system so that better performance could be obtained. Our experiments on the Google speech command dataset show that the hybrid classical-to-quantum transfer learning is of significance in enhancing classification accuracy and lowering cross-entropy loss value for the CNN-QNN model.