Optimizing measurement-based cooling by reinforcement learning

Cooling mesoscopic and microscopic resonators down to their minimum-energy state is fundamental to observe classical-quantum transition and to exploit quantum advantage in nanoscience Milburn and Woolley (2008); Aspelmeyer et al. (2014). The ground-state preparation is a crucial and implicit step in quantum information processes, including but not limited to the continuous-variable quantum computations Lloyd and Braunstein (1999); You and Nori (2011); Toyoda et al. (2015); Um et al. (2016), the ultrahigh precision measurements Bocko and Onofrio (1996); Caves et al. (1980), and the quantum interface constructions Sharma et al. (2018). Various strategies are designed to reach an effective temperature as low as possible in the trapped atom and ion systems Wilson-Rae et al. (2007); Gigan et al. (2006); Wang et al. (2011). In atomic laser cooling, popular strategies are consisted of the laser Doppler cooling Sharma et al. (2018); Zhang et al. (2013); Epstein et al. (1995), the resolved-sideband cooling, and the electromagnetically induced transparency (EIT) cooling Morigi et al. (2000); Roos et al. (2000).

Beyond the paradigms extracting the system energy through dissipative channels based on the blue-shifted (anti-Stokes) sidebands, a versatile approach to cooling the mechanical states of motion is provided by the interaction with electromagnetic radiation or quantum measurement. Back-action-evading measurement techniques that can surpass the standard quantum limit have attracted enormous interests. Through pulsed measurement process in optomechanics Vanner et al. (2011, 2013); Bennett et al. (2016); Rossi et al. (2018); Brunelli et al. (2020), they can dramatically change the mechanical thermal occupation with no initial cooling. A genuine quantum mechanical cooling engine is proposed Buffoni et al. (2019), whereby the fuel is the energy exchanged with an apparatus performing invasive quantum measurements.

Among these measurement-based techniques, quantum state engineering based on measurements on ancillary systems have been proposed recently in theory Nakazato et al. (2003); Li et al. (2011) and demonstrated in experiment Xu et al. (2014). Rather than directly detecting the target system, a net nonunitary propagator is realized by inserting projective measurements on the ground state of the detector system in between the joint unitary-evolution segments of target and detector. The induced postselection of the ground state of the target system (typically modelled as a resonator) reduces its high-energy distribution in the ensemble. In another word, the resonator is gradually steered by the outcomes of the conditional measurement (CM) to its ground state via dynamically filtering out its vibrational modes. Ranging from cooling the nonlinear mechanical resonators Puebla et al. (2020), cooling by one shot measurement Pyshkin et al. (2016), expanding cooling range by an external driving Yan and Jing (2021), to accelerating cooling rate by optimized measurement intervals Yan and Jing (2022), an unexplored weakness of the CM strategies is their limited success probability inherited from the projective operation. An amount of experimental overhead rises unavoidably with more samples in ensemble. In sharp contrast to CM, the unconditional measurement (UM) strategy performs a nonselective and impulsive measurement in all the bases of the bare Hamiltonian of detector in the end of each round of the joint evolution Zhang et al. (2019a); Harel and Kurizki (1996). It is more likely to realize a unit-success-probability cooling but suffers from a much slower cooling rate than CM, indicating much more number of measurements toward the ground-state cooling. To compromise the cooling rate and the success probability, the interpolating-configuration of conditional and unconditional measurements constitutes an optimization problem.

The integration of a small-scale quantum circuit with a classical optimizer, e.g., the neural network, provides a paradigm by designing a sequence of parametrized quantum operations that are well suited to implement robust and high-fidelity algorithms. Many reinforcement learning (RL) algorithms constructed by the neural network, that demonstrated remarkable capabilities in the board and video games Silver et al. (2016, 2017, 2018); Mnih et al. (2015), have substantiated a widely and timely interest in studying quantum physics Carleo et al. (2019), such as quantum error correction Convy et al. (2022); Fösel et al. (2018), quantum simulation Bolens and Heyl (2021); Yuan et al. (2021), and quantum state preparation Guo et al. (2021); Bukov et al. (2018); Zhang et al. (2019b), to name a few. The proximal policy optimization (PPO) algorithm, as a typical RL algorithm with a significant sample complexity, scalability, and robustness for hyperparameters, has proven to be a fruitful tool in quantum optimization control Sivak et al. (2022); Kim and Jeong (2021); Yao et al. (2021).

In this work, we propose a measurement-based cooling architecture as a hybrid sequence of UM and CM strategies. It involves a double optimization: for each step along the sequence, either UM or CM can be considerably improved by using a local optimized measurement interval; and for the global efficiency of the sequence, its arrangement can be separably optimized through reinforcement learning. Particularly, in a typical measurement-based cooling model, i.e., the Jaynes-Cummings (JC) model, where a mechanical resonator (the target system) is coupled to a qubit (the detector system), conditional and unconditional measurements are alternatively performed to cool down the resonator to its ground state. A feedback scheme is triggered upon calling a CM to determine whether or not to launch the next round of evolution-and-measurement according to the measurement outcome. Analogous to the optimized measurement-interval obtained for CM Yan and Jing (2022), we analytically derive an optimized interval for UM. Then the free-evolution intervals between any neighboring measurements, either UM or CM, can be optimized for cooling. The global sequence of measurements or the implementing order of UM and CM can be further optimized with reinforcement learning. The optimizer is fed with the cooperative cooling performance, a function of the average population of the resonator, the success probability of the detector in the measured subspace, and the fidelity of the resonator in the ground state. Eventually we find an optimal sequence holding an overwhelming advantage over all the others.

The rest of this work is structured as follows. We briefly revisit the general framework for the cooling protocols based on conditional and unconditional measurements in Secs. II.1 and II.2, respectively. In Sec. II.2, an analytical expression of the optimized measurement-interval is obtained for UM. In Sec. III, we introduce the interpolation diagram for the cooling architecture based on these two measurements, define the cooperative cooling performance to comprehensively quantify various strategies, and present the optimized result through reinforcement learning. The PPO algorithm and the optimal-control procedure are provided in Appendixes A and B, respectively. The whole work is discussed and summarized in Sec. IV.

Thermal resonator could be steadily yet slowly cooled down by unconditional measurement strategy equipped with an optimized measurement-interval in Eq. (14). And this strategy is performed with a unit probability in the absence of postselection over the measurement outcome. In sharp contrast, conditional measurement strategy is a more efficient cooling protocol but with a poor success probability. It is therefore desired to find an optimized sequence of measurements as a hybrid of UM and CM to hold a great performance taking both cooling efficiency and experimental overhead into account. In this section, we present an algorithm that employs the reinforcement learning to generate the optimized control sequence indicating when and which measurement is performed.

The performance of any cooling-by-measurement strategy can be characterized or evaluated by the cooling ratio $\bar{n}(t)/\bar{n}_{\rm th}$, the success probability $P_{g}$ of the detector in the measured subspace, and the fidelity of the resonator in its ground state $F=\langle n=0|\rho_{a}(t)|n=0\rangle$ Li et al. (2011). To compare various interpolation sequences of UM and CM in cooling performance and to evaluate the figure of merit for the reinforcement learning, we can define a cooperative cooling quantifier as

Notably, the logarithm function is used to obtain a positive value with almost the same order as $F$ and $P_{g}$ in magnitude. Then $\bar{n}(t)$, $P_{g}$, and $F$ could be considered in a balanced manner. In fact, the average population could be reduced by several (normally less than $10$) orders in magnitude under an efficient cooling protocol. In the EIT cooling Feng et al. (2020), $\log_{10}[\bar{n}_{\rm th}/\bar{n}(t)]\sim(2,3)$; and in the resolved sideband cooling  Triana et al. (2016), $\log_{10}[\bar{n}_{\rm th}/\bar{n}(t)]\sim(4,5)$. Although Eq. (15) is not a unique choice, it is instructive to find that a lower average population, a larger success probability, and a higher ground-state fidelity to yield a better cooling performance.

The RL-optimization is shown in Fig. 2(a). It is constituted by the “agent” part based on a series of neural network and the “environment” part performing the cooling-by-measurement actions on quantum system. In the reinforcement learning, the agent has a cluster of parameters, which would be learned and trained using the data collected through its interaction with the environment. In our architecture, the agent would choose an action, i.e., conditional or unconditional measurement, on the resonator, given its current state. Then the environment takes this action and returns the updated resonator-state $\rho_{a}$ and a “reward” $R$ after the measurement. The reward is generated by the indicator in Eq. (15) to estimate whether the action is good or bad, that would be used to update the agent’s parameters. During one “episode”, the agent would interact with the environment for $N$ times, i.e., the number of measurements during the whole sequence, which has been fixed from the beginning. A total reward is eventually counted. And the agent is trained to maximize the total reward through artificial episodes until it converges. Then the agent could provide a realistic control sequence of the measurement strategies with their own (optimized) measurement intervals. The cooling-by-measurement sequence can be realized in a circuit model in Fig. 2(b). Rounds of free-evolutions and measurements are successively arranged. The evolution time between two neighboring measurements depends on the measurement strategy and the resonator state at the end of the last round. We follow the PPO algorithm in the agent structure, the data-collecting methods, and the updating parameters, whose details can be found in Appendix A. The interpolation algorithm of UM and CM and the implementation of the measurement sequence are illustrated by a pseudocode in Appendix B.

We consider to cool down a mechanical microresonator in gigahertz Ding et al. (2011); Chan et al. (2011) with various interpolation sequences of UM and CM. Using the resonator-frequency $\omega_{a}=1.4$ GHz, the coupling strength between resonator and detector $g=0.04\omega_{a}$ and the initial temperature of resonator $T=0.1$ K, it is found that the average population starts from $\bar{n}_{\rm th}=8.85$. The cooling performances under the sequences entirely consisting of UM and CM are shown by the blue-solid lines with circle markers and the orange-dotted lines in Figs. 3(a)-(d), labeled by $S_{u}$ and $S_{c}$, respectively. It is found that under the conditional measurement strategy with $N=16$, the average population $\bar{n}$ is reduced by five orders in magnitude [see Fig. 3(a)] and the ground-state fidelity is over $F>0.9999$ [see Fig. 3(b)] with less than $10\%$ of the success probability [see Fig. 3(c)]. In sharp contrast, under the same number of unconditional measurements, $\bar{n}$ is merely reduced to $\bar{n}\approx 3.36$ and the ground-state fidelity $F\approx 0.78$, despite with a unit success probability. In terms of all the individual quantifiers, i.e., $\bar{n}$, $F$, and $P_{g}$, the results under the hybrid sequences of UM and CM labelled by $S_{k}$, $k=1,2,4$, are among the former two limits $S_{u}$ and $S_{c}$. As illustrated by Figs. 3(e), (f), and (g), the three sequences start from a CM (indicated by $1$), switch to the UM (indicated by $0$) after $k$ rounds of free-evolution and measurement, switch back to CM after a single round, and then repeat the preceding arrangement. In comparison to the entire UM sequence, the interpolation with CM promotes the cooling efficiency in $\bar{n}$. A larger $k$ gives rise to a smaller proportion of the unconditional measurements and a less probability $P_{g}$ that the detector remains in its measured subspace.

With respect to the cooperative cooling performance given by Eq. (15), it is found [see Fig. 3(d)] that $\mathcal{C}(S_{1})>\mathcal{C}(S_{2})>\mathcal{C}(S_{4})>\mathcal{C}(S_{u})$ and yet $\mathcal{C}(S_{2})\approx\mathcal{C}(S_{c})$. Such that a regular interpolation sequence could therefore have a better cooperative cooling performance than the entire CM sequence. While the dependence of $\mathcal{C}$ for arbitrary hybrid sequence on its proportion of CM strategies might not be monotonic. We are then motivated to find an optimized sequence by virtue of the PPO algorithm. A typical RL-optimized sequence of cooling strategies labeled by $S_{\rm opt}$ is described in Fig. 3(h). With four orders reduction in the average population (close to the cooling efficiency provided by $S_{c}$), an almost unit ground-state fidelity $F>0.9999$, and a moderate success probability $P_{g}\approx 30\%$ (much larger than that by $S_{c}$), the optimized sequence achieves an overwhelming cooperative cooling performance $\mathcal{C}(S_{\rm opt})=2.73$ according to Eq. (15) over all the other measurement sequences. Therefore, we have achieved a compromise of cooling rate and success probability through the reinforcement leaning method with a much less overhead than the brute-force searching. The RL-optimized sequence is not unique, yet the current results of $\bar{n}$, $F$, $P_{g}$, and $\mathcal{C}$ in Fig. 3 are almost invariant as long as there is one CM in the first several rounds.

The RL-optimized algorithm applies to a wide range of initial temperature for the resonator. Starting from various $\bar{n}_{\rm th}$ determined by the temperature, the average populations could be reduced by three to five orders in magnitude under the optimized measurement sequences, as demonstrated in Fig. 4(a). It is found that under a higher temperature, it is harder to suppress the transitions between the ground state and the excited states of the detector. Then both the relative magnitude in the population reduction [see Fig. 4(a)] and the cooperative cooling performance [see Fig. 4(b)] manifest a monotonically decreasing behavior as temperature increases.

Similar to Fig. 3(h), here we present in Figs. 4(c), (d), (e), and (f) the optimized sequences fully determined by the PPO algorithm, which still outperform any regular interpolated sequence in the cooling quantifier $\mathcal{C}$. Comparing these four sub-figures corresponding to various temperatures, it is interesting to find that a larger portion of the unconditional measurements is required along the optimized sequence for a higher temperature. It is consistent with the fact that under CM the success probability $P_{g}$ to find a detector in its ground state decreases exponentially with increasing temperature of the target resonator. Then more UMs are used to save a rapidly declining $P_{g}$ for obtaining a larger $\mathcal{C}$. In addition, for $T>0.05$ K, RL-optimized sequence always starts from a conditional measurement, which is important to have a significant cooling rate for $\bar{n}$ during the first several rounds of the whole sequence.

The profiles shown in Fig. 3(h) and Figs. 4(d), (e), and (f) manifest a common pattern for all the RL-optimized sequences. It is found in the previous several rounds that a conditional or projective measurement should be performed on the detector, when the resonator is normally in a comparatively high-temperature state, and several unconditional measurements ensued before further cooling. This pattern is consistent with the variations of both energy and entropy in nonunitary controls Gherardini et al. (2020). The energy variation induced by a projective measurement is $k_{B}TH(\rho)$ on average, where $H(\rho)$ is the Shannon entropy of the whole system after a free evolution. Then in the end of the first round, a projective measurement is desired to cut down as much energy as it could, which is followed by several rounds of unconditional measurements to save the success probability. Thus in general we anticipate to see more UMs than CMs in the first several rounds and more CMs than UMs in the remaining rounds.

Preceding analysis over the cooling performance neglects the environment-induced dissipation. We now consider the cooling process in an open-quantum-system scenario, in which the free evolution between neighboring measurements is influenced by a finite-temperature environment. The dynamics is then described by the master equation

where $\mathcal{D}[A]$ represents the Lindblad superoperator

In Fig. 5(a) and (b), we present the average population $\bar{n}$ and the cooperative cooling performance $\mathcal{C}$ respectively with various dissipation rates. To compare the cooling performances in the presence of thermal decoherence to the dissipation-free situation, we apply the RL-optimized sequence provided in Fig. 3(h). It is found that a larger dissipation rate gives rise to a weaker cooling performance in terms of both $\bar{n}$ and $\mathcal{C}$, exhibiting the struggle between cooling effects by measurement and the accumulated heating effects by environment. Nevertheless, for typical mechanical resonators in gigahertz with $\gamma/\omega_{a}\sim 10^{-5}$ Ding et al. (2011); Chan et al. (2011), our optimized cooling protocol is still capable to reduce $\bar{n}$ by three orders in magnitude with about $N=10$ measurements [see the green dashed line in Fig. 5(a)]. In the mean time, the asymptotic value of $\mathcal{C}$ still overwhelms the CM strategy labeled by $S_{c}$ in Fig. 3(d).

Even in the absence of thermal decoherence, $\bar{n}$ does not keep decreasing. Fundamentally, it is under the constraint of the third law of thermodynamics, that the absolute zero cannot be attained within a finite number of operations. Actually, either $\tau_{\rm opt}^{c}$ or $\tau_{\rm opt}^{u}$ approaches infinity as $\bar{n}\rightarrow 0$, which indicates that the whole cooling process has to be truncated by a maximum timescale.

We emphasize again that the preceding hybrid cooling sequences based on the conditional and unconditional measurements are optimized in both global and local perspectives. Globally, we use the reinforcement learning to find the optimized order for UM and CM. The local optimization depends on the selected measurement interval to obtain a minimum average-population $\bar{n}$ under one measurement. For UM in Eq. (14), $\tau_{\rm opt}^{u}(t)$ is not necessarily obtained by an instant feedback mechanism during a realistic practice. The measurement sequence $\{\tau_{1}(t_{1}),\tau_{2}(t_{2}),\cdots,\tau_{N}(t_{N})\}$ can be actually obtained prior to the cooling measurements. $\tau_{1}(t_{1})$ depends on the initial population-distribution $p_{n}$, and $\tau_{k}(t_{k})$, $k\geq 2$, can be calculated on the effective temperature that is uniquely determined by the dynamics of $p_{n}(t)$ through Eq. (12). In other words, we can avoid the feedback error and imprecision induced by detecting the resonator states during the experiment.

In summary, we present an optimized cooling architecture on a sequential arrangement of both conditional and unconditional measurements. We analyse and compare the advantages and disadvantages of both CM and UM on cooling rate and success probability. We obtain analytically for the first time an analytical expression for the optimized unconditional measurement-interval $\tau_{\rm opt}^{u}=\pi/(\Omega_{d}+\Omega_{d+1})$ in parallel to that for conditional measurement Yan and Jing (2022). Here the dominant Rabi frequency $\Omega_{d}$ depends on the dominant distribution of resonator in its Fock state with $n_{d}=k_{B}T/(\hbar\omega_{a})$ and the coupling strength between target and detector. The combination of the advantages of both measurement strategies gives rise to an optimized hybrid cooling algorithm assisted by the reinforcement learning. It is justified by the cooperative cooling performance as we defined to quantify the comprehensive cooling efficiency for arbitrary cooling-by-measurement strategy. Our work therefore pushes the cooling-by-measurement to an unattained degree in regard of efficiency and feasibility. It offers an appealing interdisciplinary application of quantum control and artificial intelligence.

We acknowledge financial support from the National Science Foundation of China (Grants No. 11974311 and No. U1801661).