Introduction to Quantum Reinforcement Learning: Theory and PennyLane-based Implementation

Reinforcement learning is mathematically modeled with Markov Decision Process (MDP) as a tuple $(\mathcal{S},\mathcal{A},P,R,T)$, where $\mathcal{S}$ is a finite set of state information, and $\mathcal{A}$ is a finite set of action information. The function $P:\mathcal{S}\times\mathcal{A}\to P(\mathcal{S})$ is a transition probability function, with $P(s^{\prime}\mid s,a)$ being the probability of transitioning into state $s^{\prime}$ if an agent starts executing action $a$ in state $s$. The function $R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to\mathbb{R}$ denotes the reward function, with $R_{t}=R(s_{t},a_{t},s_{t+1})$. The MDP has a finite time horizon $T$, and solving an MDP means finding a optimal policy $\pi_{\theta}^{*}\in\Pi:\mathcal{S}\times\mathcal{A}\to\left[0,1\right]$, where $\pi_{\theta}$ is neural network-based policy with parameter $\theta$; Observing $s$, $\pi_{\theta}$ determines agent’s action $a\in\mathcal{A}$ to maximize the cumulative rewards received during the finite time $T$.

When the environment transitions and the policy are stochastic, the probability of a $T$-step trajectory is defined as $P(\tau\mid\pi_{\theta})=\rho(s_{0})\prod_{t=0}^{T-1}P(s_{t+1}\mid s_{t},a_{t})\pi_{\theta}(a_{t}\mid s_{t})$ where $\rho$ is the initial state distribution. Then, the expected return $\mathcal{J}(\pi_{\theta})$ is defined as $\mathcal{J}(\pi_{\theta})=\int_{\tau}P(\tau\mid\pi_{\theta})R(\tau)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[R(\tau)\right]$ where the trajectory $\tau$ is a sequence of states and actions in the environment. The objective of reinforcement learning is to learn a policy that maximizes the expected return $\mathcal{J}(\pi_{\theta})$ when the agent acts according to the policy $\pi_{\theta}$. Therefore, the optimization objective is expressed by

with $\pi_{\theta}^{*}$ being the optimal policy.

Quantum computers use a qubit as the basic unit of computation, which represent a quantum superposition state between two basis state $|0\rangle$ and $|1\rangle$. It is controlled by unitary gates in a quantum circuit to perform various quantum operations. It can be represented as a normalized two-dimensional complex vector as:

and there also is a geometrical representation of a qubit space, using polar coordinates $\theta$ and $\phi$:

where $0\leq\theta\leq\pi$ and $0\leq\phi\leq\pi$. Qubit state is mapped into the surface of 3-dimensional unit sphere, which is called Bloch sphere. Quantum gate is a unitary operator transforming a qubit state into another qubit state, which can be represented as a $2\times 2$ matrix with complex entries. There are some important quantum gates, Pauli-$X$, Pauli-$Y$, and Pauli-$Z$, rotating by $\pi$ around their corresponding axes in Bloch sphere. The rotation operator gates $R_{x}(\theta),R_{y}(\theta)$, and $R_{z}(\theta)$ rotate by $\theta$ instead of $\pi$ in Pauli-$X$, Pauli-$Y$, and Pauli-$Z$ gates, and it is known that any single-qubit unitary gate in $SU(2)$ can be written as a product of three rotation operators of each axis. In addition, there are quantum gates which operate on multiple qubits, called controlled rotation gates. They act on a qubit according to the signal of several control qubits, which generates quantum entanglement between multiple qubits. Among them, Controlled X(or CNOT) gate is one of the most used control gates, changing the sign of the second qubit if the first qubit is $|1\rangle$. These gates allow quantum algorithms to work using their features on a quantum circuit that will be introduced later.

The variational quantum circuit (or parameterized quantum circuit) is a quantum circuit using learnable parameters to perform various numerical tasks, such as optimization, approximation, and classification. Operation of general VQC model can be divided into 4 steps. First one is state preperation step, the input information is encoded into corresponding qubit states, which can be treated in the quantum circuit. Next step is variational step, entangling qubit states by controlled gates and rotating qubits by parameterized rotation gates. This process can be repeated in a multi-layer manner with more parameters, which possibly enhance the performance of the circuit. In the third step, processed qubit states are measured and decoded to the form of appropriate output information. Last step is conducted outside the circuit. The quantum circuit parameters are updated in the direction of optimizing the objective function of the algorithm by a classical CPU algorithm, like Adam optimizer. Then the circuit updated with the new parameters performs the calculation again from the beginning. This circuit is known to be able to approximate any continuous function like classical neural network[13], so VQC is often called Quantum Neural Network (QNN)[14]. It has been widely applied in quantum machine learning researches.

In recent studies of quantum reinforcement learning [15, 16], VQC substitutes the policy training DNN of existing DRL. At each episode, agent with given state information determines its action from policy-VQC and parameters are updated with a classical CPU algorithm like Adam Optimizer. This paper approaches similarly, by using a VQC depicted in Fig. 1.
The quantum circuit in Fig. 1 is a prototype of policy-VQC, which consists of the basic structure of VQC. State encoding part of the circuit includes $R_{y}$ gates parameterized by normalized state input $s$, having their values between $-\pi$ and $\pi$. Variational part in the center consists of entangling $CX$ Gates and $R_{x}$, $R_{y}$, $R_{z}$ gates parameterized with free parameter $\theta$. This part is called a layer, and several can be repeatedly stacked in a circuit. After that, measured output of the circuit is decoded into the action space, yielding the action probabilities. Then the obtained $\pi_{\theta}$ is evaluated and updated in a classical computer.

The quantum reinforcement learning system in this paper is as described in Fig. 2. In the beginning of an episode, quantum-classical hybrid agent receives a state information from the environment, and determine its action by $\theta_{\pi}$ made from the VQC. Then the policy of the agent is evaluated and updated by PPO algorithm, which is introduced before and described in Algorithm 1. The PPO algorithm are effectively the same as in the previous study[12]. The replay buffer functions in the same way as in traditional approaches, keeping track of the $\langle s,a,R,s^{\prime}\rangle$ tuples. One does not have to fundamentally or drastically change an algorithm in order to apply the power of VQCs to it. The algorithm presented in Algorithm 1.

As quantum computing and quantum machine learning research is actively progressing, many development libraries for researchers have emerged, such as TensorFlow-quantum, Qiskit, and PennyLane. Among them, PennyLane was created to support quantum machine learning research, allowing anyone to easily test the performance of quantum circuits through quantum simulators. The quantum simulator supported by PennyLane allows the CPU to imitate the operation of QPU, and especially supports the use of parameters in the form of PyTorch tensor and gradient operation [17]. Thanks to these features, PennyLane makes it easy for anyone who has previously done machine learning research using PyTorch to start researching quantum machine learning. Based on this background, in this paper, a quantum reinforcement learning model was implemented using PennyLane and PyTorch as shown in Fig. 3.

Cartpole, the implementation environment in this paper, is a test environment for reinforcement learning provided by OpenAI [18]. This is a game where the agent moves the cart back and forth to avoid dropping the stick on the cart and the longer one holds the stick, the greater the reward. At every moment, the player observes the cart’s position, velocity, and the angle and angular velocity of the rod to determine which direction to accelerate accordingly. The VQC in Fig. 1 using 4 qubits is suitable for policy making in this environment. Each of the four pieces of information provided by the environment is normalized and fed into the circuit as values between $-\pi$ and $\pi$, and the two measures are decoded into the probability values of taking two actions via the softmax function. This process continues until the agent can take the optimal action on the given state information by optimizing the given parameters in the reinforcement learning algorithm. The experimental result is showed later in this paper.

Our experiment is conducted with the software packages, PyTorch for speed and convenience of tensor operation and and PennyLane for quantum circuit simulation. The quantum simulator provided by Pennylane is very convenient to use, but it is difficult to use many qubits because of its slow computational speed. Therefore, the CartPole environment was used as a simple environment that can be operated with a circuit of small qubits. We used classical parameter optimizer as Adam optimizer with learning rate 0.001 for quantum policy and 0.00001 for classical critic. Other hyperparameter settings are $\gamma$ = 0.98, $\lambda$ = 0.95, $\epsilon$ = 0.01. The baseline model is a random version of this model, using random parameters in every time step without optimization.

Fig. 4 shows the performance of the proposed quantum reinforcement learning model. Comparing with random actions, one can see that the model is learning to find the optimal action. Also, it can be seen that the deviation of rewards during the learning process is extremely high. This is due to uncertainty within quantum systems, although the impact of reinforcement learning algorithms facilitating exploration cannot be ignored either. This uncertainty simultaneously implies the possibilities and limitations of quantum reinforcement learning. This allows effective policy exploration with relatively few tens of parameters, but makes it difficult to maintain the good results once reached. Leveraging these characteristics is an important challenge for quantum reinforcement learning.