An application of data driven reward of deep reinforcement learning by dynamic mode decomposition in active flow control

Kármán vortex street is a famous phenomenon when flow separating from a bluff body such as a circular cylinder Roshko (1955); Berger and Wille (1972); Williamson and Govardhan (2004). The periodic flow separation and instability cause vibrations and noise, which can be weakened through passive Bearman (1965); Bearman and OWen (1998); Bagheri, Mazzino, and Bottaro (2012); Deng, Mao, and Xie (2019) and active Poncet (2004); Amitay, Smith, and Glezer (1998); Mittal (2001) methods. Active flow control (AFC) has been studied by a lot of researchers and has witnessed a fast growth Choi, Jeon, and Kim (2008). Various forcing devices have been applied on the active control of flow over a bluff body, such as rotary Poncet (2004), streamwise Cetiner and Rockwell (2001); Leontini, Jacono, and Thompson (2013), and transverseBlackburn and Henderson (1999); Kumar Chauhan et al. (2016) oscillations of the bluff body, distributed forcing Kim and Choi (2005), synthetic jets Amitay, Smith, and Glezer (1998); Smith and Glezer (1998); Glezer and Amitay (2002), rotating control cylinders Mittal (2001); Zhu et al. (2015); Schulmeister et al. (2017), etc.

Besides the various control approaches, the control policies of AFC are worth further investigation. The high non-linearity of the dynamic system of AFC makes it challenge to find efficient control policies. For the studies of open-loop AFC (no feedback mechanism), a periodic signal is often used as the control policy in the AFC of flow over a bluff body Nakano and Rockwell (1994); Konstantinidis, Balabani, and Yianneskis (2005); Jeon et al. (2004); Fujisawa, Takeda, and Ike (2004). In these cases, the control system cannot interact with the feedback of flow field. For the active closed-loop control cases, many control algorithms based on mathematical analysis are utilized in the determination of the control policies of AFC, such as optimal control theory Min and Choi (1999); He et al. (2000); Protas and Styczek (2002); Li et al. (2003) and reduced-order models Cortelezzi, Chen, and Chang (1997); Gillies (1998); Li and Aubry (2003); Protas (2004); Bergmann, Cordier, and Brancher (2005). However, these model-based control policies are usually based on either harmonic or constant forcing Brunton and Noack (2015); Schoppa and Hussain (1998). The complex high-dimensional nonlinear systems of actual AFC are challenge to these conventional control algorithms.

In recent years, many artificial intelligence techniques caught the attention of researchers Brunton, Noack, and Koumoutsakos (2020). Reinforcement learning (RL) is one such approach that interacts with the environment according to policies, learns from experience, and improves policies. In the case of RL, an agent (controlled by the artificial neural networks (ANNs) described in Section IV) interacts with an environment through three channels of exchange of information in a closed-loop fashion Sutton and Barto (2018). First, the agent is given access at each time step to a state $s_{t}$ of the environment. Second, the agent performs an action $a_{t}$, that influences the time evolution of the environment. Finally, the agent receives a reward $r_{t}$ depending on the state of the environment following the action. The selection of reward function is important and related to the learning result. The RL model keep on finding policies learnt from the experience of interacting with the environment, in order to discover control sequences that yield the highest possible reward. The environment can be any system that can provides the interface $(s_{t},a_{t},r_{t})$. As a model-free approach, RL is quite suitable for complex, high-dimensional, nonlinear systems. With deep ANNs used in the framework, deep reinforcement learning (DRL) can extract features from data in high-dimensional state and action spaces. DRL has been successfully applied to resolve several problems such as: playing Atari games Mnih et al. (2013), controlling complex robots Kober, Bagnell, and Peters (2013); Gu et al. (2017) and generating realistic dialogs Li et al. (2016).

Since these challenging systems successfully controlled by DRL have the properties of high-dimension and non-linearity similar to the features of AFC, the DRL arouses researchers’ interests as one new approach of AFC. In recent years, DRL has been applied on AFC in several cases. Rabault et al. Rabault et al. (2019) introduced DRL technique to the AFC problem of flow over a circular cylinder. They used synthetic jets set on the surface of the cylinder, injecting and suctioning mass flow into the field to control the flow. With the control policies learnt through DRL, the mass flow rate of the jets is controlled to stabilize the vortex alley in a short period of time. Based on this framework, Xu et al. Xu et al. (2020) placed two small rotational cylinders behind the main cylinder to control the flow. The rotation law of the small cylinders is directed by DRL model. Xu et al. also achieved a good result on stabilizing the vortex alley, which shows that the DRL-based AFC method performs well on the flow over a cylinder through various control approaches like synthetic jets and rotational cylinders. Tang et al. Tang et al. (2020) analyzed the AFC based on DRL at different Reynolds number conditions. The results demonstrated that this AFC method through DRL technique is robust and has good controlling performance over a range of Reynolds number.

In Rabault et al.’s work, the reward of the DRL model is based on the drag coefficient and lift coefficient of the cylinder Rabault et al. (2019). The drag coefficient and lift coefficient are extracted from flow field, where a lot of other detailed information are ignored or not considered in computing the coefficients. Compared with the total performance, the complete flow field snapshots contains the whole information for analysis and control of the flow, which can lead to a finer result. However, the extremely high-dimensional data of the flow field snapshots is highly memory-cost and unfeasible for computation. Seeking a low-dimensional description from the snapshots while keeping dynamically important flow field information is necessary and important. One way to achieve this goal is using a reduced order model (ROM). The basic idea of ROM is to obtain a low-dimensional description of the flow features which provides the spatiotemporal information of a flow field needed for feedback control through some matrix analysis means such as proper orthogonal decomposition (POD) and dynamic mode decomposition (DMD) Kutz et al. (2016).

With the application of ROM, the dynamically important information of the periodic vortex shedding can be extracted from the complete flow field information, and be used to make AFC policy. As a mainstream ROM, POD Berkooz, Holmes, and Lumley (1993) has been widely used on the AFC of the flow over a circular cylinder based on various forcing devices, such as rotary Graham, Peraire, and Tang (1999); Bergmann, Cordier, and Brancher (2005), transverse displacement Siegel, Cohen, and McLaughlin (2006) and synthetic jets Feng, Wang, and Pan (2011); Ma et al. (2015). Another popular ROM which has been applied in flow field analysis and AFC in recent years is DMD. Dynamic mode decomposition, proposed by Schmid Schmid (2010), extracts the spatiotemporal information of the flow and constructs a low-dimensional, linearized ROM for the high-dimensional flow field snapshots and flow evolution. Different from POD which computes orthogonal modes based on the energy information, DMD can capture dominant flow modes with purely frequency content Kou and Zhang (2019). Although having extensive applications on flow field analysis, the standard DMD mainly focus on the basic periodic dynamics of the flow, which has limitations for problems with external actuation or external control. As a developed method from DMD, dynamic mode decomposition with control (DMDc) Proctor, Brunton, and Kutz (2016), can consider the effect of control and extract more accurate reduced-order models and dynamics modes for the complex systems with external control. As a good approach to catch modes of the flow, DMD has been utilized to analyse the computational and experimental data of the flow over a bluff body in many studies Tu et al. (2013); Hemati, Williams, and Rowley (2014); Bagheri (2013); He, Wang, and Pan (2013); Zhang, Liu, and Wang (2014).

For its application on AFC, a study on bistable flow of the flow around a circular cylinder under the synthetic-jet control was carried out by Wang et al. Wang and Feng (2017) through total DMD (TDMD). DMD was employed to extract dynamic modes, spectrum, and mode coefficients to describe the flow dynamics to reveal the underlying flow physics of the bistable flow. The result showed that the bistable flow contains dual dominant frequencies corresponding to the antisymmetric and symmetric vortex-shedding modes and the DMD method provides an effective and efficient approach to analyze the bistable flow. Jardin and Bury Jardin and Bury (2012) investigated the influence of the control of pulsed tangential jets on the flow past a circular cylinder. The time-series data were analysed using a spectral–Lagrangian dual approach based on DMD and vortex tracking, which revealed the interaction mechanisms (between the tangential jets and the separated shear layers) governing the forced flow. To control the flow over a circular cylinder, Shaafi and Vengadesan Shaafi and Vengadesan (2014) set a control rod of the same diameter upstream the cylinder. The control rod was subjected to different rotation rates. Modal structures obtained from DMD revealed that the flow structures behind the main cylinder are suppressed due to wall and the flow is dominated by the wake of control rod.

In this paper, the AFC through DRL applied on the flow over a circular cylinder is studied. To achieve a reward function that contains more global flow field information, DMDc is performed on the snapshots to extract the temporal and spatial characteristics of the flow field. The modes and the corresponding frequencies of the flow with AFC can be computed via DMDc. To stabilize the vortex street, the amplitudes of each dynamic modes are expected to decrease. With the decomposition of the flow field information, a data-driven reward function of the DRL network is computed from the combination of these mode amplitudes. With the improved reward function, the modified AFC method based on DRL is performed on the flow over a circular cylinder and achieve good control results. With the active control guided by the policy learnt from the flow field information, the drag of the flow over a circular cylinder is reduced, as well as the fluctuations of drag and lift. In addition, the recirculation area (the downstream region of the cylinder where the horizontal component of the velocity is negative) of the controlled flow is increased and the low-velocity region in the wake is extended. Furthermore, a DMD analysis is conducted on the complete field of the baseline flow and the flow with control for detailed information and the comparison of the dynamic modes of the baseline flow and the controlled flow is discussed.

This paper is organized as follows: In Section II, the simulation base of the flow over a circular cylinder controlled by synthetic jets is introduced. In Section III, an introduction to the DMD and DMDc algorithms is presented. Section IV introduces DRL framework and its application, and proposes a new data-driven reward for the learning in this paper. Section V shows the controlling results and the DMD analysis of it, and the paper is concluded in Section VI.

The synthetic-jet AFC is applied on the flow past a circular cylinder immersed in a two-dimensional channel, with the configuration of the simulation adapted from the classical benchmark computations carried out by Schäfer Schäfer et al. (1996). The simulation acts as the environment that the DRL agent interacts with. With all quantities considered non-dimensionalized, the geometry of the simulation is shown in Fig. 1. The benchmark consists of a circular cylinder of non-dimensional diameter $D=1$ placed in a two-dimensional domain of total non-dimensional length $L=22$ and height $H=4.1$. The center of the cylinder is located at a shift of 0.05 from the horizontal centreline of the flow domain. Two jets normal to the cylinder wall are symmetrically implemented on the upper and lower sides of the cylinder, at angles $\theta_{1}=90^{\circ}$ and $\theta_{2}=270^{\circ}$ relative to the flow direction. The jets are controlled through their non-dimensional mass flow rates, $Q_{i},~{}(i=1,2)$. The total mass flow rate injected by the jets is set to zero, i.e. $Q_{1}+Q_{2}=0$. Therefore, the AFC in this case is in fact according to one scalar value $Q_{1}$. Negative values of $Q_{i}$ correspond to suction.

In this viscous, incompressive flow, the governing equations are the two-dimensional, time-dependent Navier–Stokes equations and the continuity equation, expressed in a non-dimensional form:

The Reynolds number is defined as $Re=\bar{U}D/\nu$, where $\nu$ is the kinematic viscosity and the characteristic velocity $\bar{U}$ is the mean inflow velocity magnitude.

The inflow velocity profile, according to Schäfer Schäfer et al. (1996), can be expressed as follows:

where $U(y)$ is the horizontal non-dimensionalized velocity component at the inlet. The mean velocity magnitude is $\bar{U}=2U(0)/3=1$. No-slip boundary conditions are imposed on the top and bottom walls and on the solid walls of the cylinder. An outflow boundary condition is imposed based on the assumption that the derivative of the velocity along the X-axis is zero at the outlet. The Reynolds number in this case is set to $Re=100$. For the boundary conditions of synthetic jets, the radial velocity profiles of the two jets are set as:

where $\theta$ is the angular coordinate and $\theta_{i},~{}(i=1,2)$ is the angular position of the center of each jet. $\omega=10^{\circ}$ is the width of each jet.

The computational mesh is generated by Gmsh Geuzaine and Remacle (2009), an open-source finite element mesh generator. The computational domain is discretized into triangular cells, and refined around the cylinder surface. Based on a finite-element framework FEniCS Logg et al. (2012); Alnæs et al. (2015), the governing Navier-Stokes equations are solved in a segregated manner Valen-Sendstad et al. (2012). More precisely, the incremental pressure correction scheme (IPCS) method Goda (1979) with an explicit treatment of the nonlinear term is used.

A non-dimensional, constant numerical time step $dt=5\times 10^{-3}$ is used. The drag and lift forces on the cylinder at each time step are calculated respectively as

and

where $\boldsymbol{\sigma}$ is the Cauchy stress tensor, $\boldsymbol{n}$ is the unit vector normal to the cylinder surface, and $\boldsymbol{e}_{x}=(1,0)$ and $\boldsymbol{e}_{y}=(0,1)$. They are normalized into drag coefficient and lift coefficient

and

The Strouhal number (St) is defined as $St=fD/\bar{U}$, where $f$ is the vortex shedding frequency computed by performing fast Fourier transform (FFT) on the lift coefficients $C_{L}$ depending on time. The injected mass flow rates are normalized as $Q_{i}^{*}=Q_{i}/Q_{ref}$, where $Q_{ref}=\int_{-D/2}^{D/2}\rho{}U(y)dy$. The normalized mass flow rates $Q_{i}^{*}$ is constrained in this study as $\left|Q_{i}^{*}\right|<0.068$.

The mesh-independence study for this benchmark is conducted, with meshes of three different resolutions, which is shown in Table 1. The maximum value of $C_{D}$ in this case is within $0.2\%$ of what is reported in the benchmark of Schäfer et al. Schäfer et al. (1996), and similar agreement is found for $St$, which validates the simulation. As can be seen in TABLE 1,the resolution of the main mesh, which is used in this paper, is fine enough for the simulation with the discrepancies less than $0.04\%$ in all listed quantities when compared with the fine mesh. The mesh used for the simulation is plotted in Fig. 2.

Artificial neural network (ANN), as a basement of RL, is briefly introduced as follows. As a mathematical model for biological neural network, ANN is a system with many artificial neurons connecting by some topology. An artificial neuron can compute the output with the inputs passed from connected neurons with adaptive weights and bias, which are updated by an algorithm like stochastic gradient descent to minimize a cost function on a training set Goodfellow et al. (2016). ANN has excellent nonlinear mapping ability and performs well on making a nonlinear link between the inputs and outputs LeCun, Bengio, and Hinton (2015). A multilayer feed-forward network with enough neurons in hidden layers can approximate any continuous function with arbitrary accuracy Hornik (1991).

The DRL model, as an intelligent system with learning ability, can be applied in active flow control. As one of the most important part of DRL, the environment in this study is the direct numerical simulation (DNS) of the flow field, as described in Section II. With three basic elements related to the study, the DRL can be applied on the active control of the flow pass a circular cylinder. They are: the state $s_{t}$ (here, an array of point measurements of velocity obtained from the simulation), the action $a_{t}$ (here, the active control of the jets, imposed on the simulation by the learning agent) and the reward $r_{t}$ (here, objective function computed from the flow field information related to the stability of the vortex alley, which will be described in detail later). After these three elements determined, the learning agent can interact with the environment by: choosing the action $a_{t}$ according the current state $s_{t}$, performing the action at the state in the environment and computing the reward $r_{t}$ and the next state $s_{t+1}$. With numerous interactions with the environment, the learning agent is trained to choose action from the current state that yields the highest possible reward. In this study the DRL agent can learn the control experience of flow past a circular cylinder from interaction with the DNS-based environment, which will guide the adjustment of mass flow of jets.

The reinforcement learning algorithm used in this paper is proximal policy optimization (PPO) Schulman et al. (2017). Different from the value-based learning methods, e.g., deep Q-learning (DQN) Mnih et al. (2015), which is restricted to discrete and low-dimensional state and action space, PPO, as a policy-based learning method, can handle the high-dimensional action problems with the continuity of state and action space. Another advantage of PPO is that it is less complex mathematically and requires little to no metaparameter tuning. Like many other policy-based methods, PPO uses an actor-critic structure Konda and Tsitsiklis (2000). Two ANNs are used in the actor-critic structure, with the network parameters $\Theta^{\mu}$ and $\Theta^{Q}$, respectively. The actor network approximates to the policy function $a=\mu(s|\Theta^{\mu})$, which learns a policy that yields the highest possible reward $r$ so that a definite action from the current state can be given. The critic network approximates the action-state value function $Q(s,a|\Theta^{Q})$ to criticize the given action according to the action $a_{t}$ in state $s_{t}$ and following policy $\mu$.

During the training process, the action is obtained from the state $s_{t}$ for each step with the actor network as $a_{t}=\mu(s_{t})$. Then the agent interacts with the environment according to the action $a_{t}$ and the state $s_{t}$, feed back with the next state $s_{t+1}$ and the reward $r_{t}$. The parameters of the actor network, $\Theta^{\mu}$, and the critic network, $\Theta^{Q}$, are updated according to the reward $r_{t}$ in each step. The PPO method is episode-based, which means that the interactions between the agent and the environment are broken into a number of training interaction sequences Sutton and Barto (2018). As the PPO algorithm has already been used in a variety of fluid mechanics works, the reader interested in more details on the PPO algorithm itself is invited to refer to its previous application on AFC Rabault et al. (2019).

The framework of DRL network used in this paper is built based on Tensorforce Kuhnle, Schaarschmidt, and Fricke (2017), an open-source DRL library builds upon Tensorflow. The learning is performed in parallel with the multi-agent technique developed by Rabault Rabault and Kuhnle (2019). The structure and the hyperparameters of the PPO model are determined by referring the previous work on this topic of Rabault and Kuhnle Rabault and Kuhnle (2019). The ANNs are composed of two dense layers of 512 fully connected neurons, plus the input layer and the output layer. For each step PPO agent interacting with the environment, 50 numerical time steps are computed by the direct numerical simulation, corresponding to approximately $7.5\%$ of the vortex shedding period. Therefore, the action time step $T=50dt,~{}(dt=5\times 10^{-3}$), which means the PPO agent is allowed to interact with the simulation and update its control only each 50 time steps. The total steps of an episode is chosen as 4000, with the episode duration $T_{total}=20.0$, which spans approximately 6.5 vortex shedding periods. For each episode, 80 actions are determined by the network and performed by the agent. The hyperparameters are chosen as: learning rate is $1\times 10^{-3}$ and likelihood-ratio-clipping is 0.2.

The reward $r_{t}$, as the feedback from the simulation, varies by the determination of researchers and affects the final training result and learnt policy of the DRL model. In this paper, to utilize the complete flow field information, DMDc is performed on the snapshots obtained from simulation in the training process to achieve a reward, to instruct the agent in learning a more global control policy. For reducing the computation of DMDc, the flow field snapshot matrices composed with 30 columns of pressure measurements are used in this study. The control snapshot matrices composed with 30 columns of actions of the agent. Therefore, the snapshot matrices $\boldsymbol{X}_{0}$, $\boldsymbol{X}_{1}$ and $\boldsymbol{\Gamma}$ in 22 can be written as:

where $p_{t}$ and $p_{t+1}$ refer to the pressure measurements of the state $s_{t}$ and the next state $s_{t+1}$. $p_{t-i}$ and $a_{t-i}$ refer to the pressure measurement and the action at $i$ time intervals before the current state, respectively. The time interval of each adjacent measurement are set to 50 numerical time steps, equal to the action time step of the agent, so that the matrices $\boldsymbol{X}_{0}$, $\boldsymbol{X}_{1}$ and $\Gamma$ can be updated each time the agent determines the action and interacts with the simulation. The 30 snapshots span approximately 4 vortex shedding periods.

To determine the expression of the reward, the modes of the baseline flow (with no actuation, i.e. $Q_{1}^{*}=Q_{2}^{*}=0$) is first studied by performing DMD on the snapshots of the complete flow field. The dynamic modes of the flow are $\phi_{i},~{}i=0,2,...,28$, with corresponding eigenvalues $\mu_{i},~{}i=0,2,...,28$ and modal amplitudes $\alpha_{i},~{}i=0,2,...,28$ computed according to Eq. 30. The eigenvalues with non-negative imaginary part and the spectrum of the dynamic modes are illustrated in Fig. 7. The zeroth mode is the time-averaged flow and has a reflection symmetry about the centerline $y=0$. The first mode, as the main mode, corresponds to the part of the flow field that oscillates with the fundamental vortex shedding frequency. This mode has the same spatiotemporal symmetry and the same streamwise spacing between the vortices as the full nonlinear cylinder flow. The second mode and the third mode are due to the interaction of the first mode with themselves. The second harmonic oscillates with twice the fundamental frequency, while three times the fundamental frequency for the third mode.

The zeroth mode an energetic mode corresponding to the time-averaged flow. The other modes are corresponding to the unsteady flow. In addition, the amplitudes of the other modes can also present the irregularity of the transient flow, e.g., for the baseline flow without actuation, the amplitudes of the other modes are smaller than those of the flow with stochastic mass flow rate of the jets. Therefore, to obtain a relatively stable flow over a circular cylinder, the global kinetic energy of the flow is expected to decrease. We hope that controlled flow can evolved into a regular flow like periodic flow, so the amplitudes of the other modes is expected to decrease either. With some test on the flow with actuation, we find that the amplitudes of the other modes vary greatly according to the irregularity mass flow rate $Q_{i}^{*}$, which make it difficult for the agent to learnt the policy. Thus, the amplitude of the zeroth mode is considered more in the reward to make sure the learning of the agent is stable and robust. The expression of the reward $r_{t}$ are determined as follows:

The weights in Eq. 34 are chosen to make terms of the first mode and the other modes are close in magnitude, and the bias are chosen to make the reward close to zero.

The training of the PPO model in this paper is performed on Intel Xeon CPU E7-8890 v4 (48 cores @2.20GHz). With the multi-agent technique, 80 agents perform and interact with the environment at the same time in parallel. 400 epochs of training process can be finished in 5 periods with 80 epochs done in each period. The time cost for one period of 80 epochs is about 40 minutes.

For the actual application of this AFC method in experiments, 63 probes are set in the wake of the cylinder to measure the flow physical quantities e.g. pressure and velocity, shown in Fig. 4. With the reward in Eq. 34 computed from the DMDc of the snapshots of the pressure measurements, the DRL-based AFC method is applied with a training process of 400 epochs.

After the agent is trained, it is tested in the simulation-based environment of the AFC. The mass flow injected by the jets is guided according to the learnt policy. The drag coefficient $C_{D}$ curves of the baseline and controlled flow depending on the numerical time steps are plotted in Fig. 5. The corresponding normalized mass flow rate of the jets $Q_{1}^{*}$ are shown in Fig. 5. With the robust policy learnt through DRL, the agent can stabilize the flow in about 2500 time steps. As can be seen, the model with DMD-based reward is able to reduce the drag of the flow over a cylinder by applying AFC following training through DRL. For the relatively stable stage in the control process (after 10000 steps), the mean value of $C_{d}$ is reduced from $3.21$ (baseline) to $2.958$ (with control), which represents a drag reduction of approximately $8\%$. In addition to this reduction in drag, the fluctuations of the drag coefficient are reduced to approximately $0.0315$ by the active control, i.e. a decrease of roughly $55\%$ compared with the value of $0.0701$ for the baseline. Similarly, fluctuations in lift are reduced, from $2.12$ (baseline) to $0.807$ (controlled), with a factor of approximately $0.38$. With FFT, the Strouhal number of the mass flow of the jets of the relatively stable stage, $St_{jet}$, are computed. Similar to the $St$ of the controlled flow, $St_{jet}=0.2686$, is approximately $0.89$ times the $St$ of the flow without actuation. With the actuation evolving into an approximately periodic signal, the characteristic frequency of the system is modified. The frequency of the vortex shedding of the controlled case is approximately $11\%$ lower than the baseline case.

The velocity magnitude contours of the baseline flow and the flow with control are illustrated in Fig. 6. With the control policy, the jets modify the flow configuration and stabilize the vortex alley. In the case with active control, the low-velocity region of the wake behind the cylinder is extended and larger than in the baseline case. The velocity fluctuations induced by the vortices are globally less strong, and less active close to the upper and lower walls. The wake of the vortices is increased in width and reduced in velocity magnitude. For numerical comparison, the recirculation area, defined as the region in the downstream neighbourhood of the cylinder where the horizontal component of the velocity is negative, are measured in both cases. The average recirculation area of the controlled flow is $0.0223$, with a relative improvement of $109\%$ compared with the average recirculation area of $0.01067$ for the baseline flow. With the active flow control, dramatically increase is achieved in the recirculation area, which shows the efficiency of the learnt control policy at reducing the effect of vortex shedding.

The DMD are performed on the complete flow field snapshots of the baseline and controlled cases for more detailed analysis. Snapshot matrices composed of 50 columns of measurement on the total grid points, with a non-dimensional time interval of $0.25$ are used for each case. With a truncation of 29, the number of the dynamic modes is equal to the modes used to computed the reward. The imaginary verses real part of the eigenvalues of the baseline flow and the controlled flow, $\mu_{j,b}$ and $\mu_{j,c}$, respectively, is plotted in Fig. 7. With the computed amplitudes $\alpha_{j,b}$ (baseline) and $\alpha_{j,c}$ (with control), the spectrum is shown in Fig. 7. As can be seen from the spectrum, in the controlled case, the frequency of the first mode (fundamental frequency) is decreased when compared with the baseline case, as well as the frequencies of the second and third mode. With the AFC, the amplitude of the zeroth mode is sightly reduced. The amplitude of the first mode (main mode) is greatly reduced, with an approximately $45\%$ reduction The amplitudes of the second and the third mode both decrease, while the other modes not marked in the Fig. 7 also have large reductions on amplitudes. The results show that though the amplitude of the zeroth mode is considered more while the amplitudes of the first to fourteenth are considered less in the reward, based on the modified reward, the agent still can learn a policy of AFC to reduce the amplitudes of first to fourteenth modes greatly. The large reductions on the amplitudes of the modes also show the AFC learnt by the method of this paper is effective in stabilizing the flow over a circular cylinder.

The velocity magnitude contours of the zeroth to the third dynamic modes of the flow without actuation and with AFC reconstructed from the DMD on velocity measurement are plotted in Fig. 8. The zeroth modes (Fig. 8(a), 8(b) ) show the similar modification as in the flow field velocity magnitude contours (Fig. 6), like extended low-velocity region behind the cylinder, larger wake width and reduced velocity magnitude of the wake in the controlled case. For the first to the third mode, with the AFC, the space of the fluctuation of the mode increases, consistent with the change of the frequencies in FIG.7. The starting position of the modes (i.e. the nearest horizontal position with nonzero measurement in the downstream of the cylinder) moves aft along the positive x-axis, as well as the region of larger velocity magnitude. The width of the region of the modes with non-zero velocity is reduced by the AFC, which means the area affected by the flow pass the cylinder is reduced. For the first mode, the velocity amplitude at the approximate position of $x=5$ (marked with red boxes in Fig. 8(c), 8(d)) is increased, which represents the main mode actuated by the control jets.

Furthermore, training with different epochs are tested in this paper. For the case of reward based on the probe measurement described above, the drag coefficient $C_{d}$ curves of training processes of epochs of 400 and 800 are shown in Fig. 9. The agent achieves a better control when the training epochs is doubled. The reduction of the mean value of $C_{d}$ at the relatively stable stage is approximately $8.6\%$. The mean value of the recirculation area is $0.0239$, with an improvement of $124\%$, compared with $109\%$ of the 400-epoch case. In addition, a study with the reward based on the DMDc of the full pressure information of the complete flow field is also done. In the training process of the DRL agent, DMDc is performed on the spanshots composed of 12837 pressure values of total grid points. For the more high-dimensional information compressed by DMDc, the agent needs more steps to learn from the interaction with the environment. It takes 800 epochs for the reward of the DRL model based on information of the complete flow field decreases to a low value. The drag coefficient $C_{D}$ curves of the reward based on information of probes and the complete flow field are plotted in Fig. 9, compared with the case of probe measurement (training of 800 epochs). The reward based on the complete flow field brings a great improvement on the reduction of the drag. The reduction of the mean value of $C_{d}$ at the relatively stable stage is approximately $10.8\%$, which is a large improvement by comparison with $8.6\%$ of the probe-based case. The results shows the good robustness of this method on different training periods and measurement resolution. Moreover, there are approaches to improve the AFC results of this method further, by extending the learning process of the agent and using more complete information of the flow field.

In this study, the framework initially presented in the work of Rabault et al. Rabault et al. (2019) is extended by using a reward of more flow information based on DMDc in the DRL/PPO model. The DRL agent can learning the AFC policy for synthetic jets on a cylinder, and control the configuration of the 2D Kármán vortex street, through the optimization of a reward function modified in this paper. The improved reward function are composed with the amplitudes of the dynamic modes of the flow, which represents the main dynamic characteristics extracted from the detailed flow field information. With the AFC, a drag reduction of approximately $8\%$ is achieved in the case of reward computed from the DMDc on probe measurements, while a drag reduction of up to approximately $10.8\%$ is achieved in the case of reward based on the complete flow field. An improvement of $109\%$ in the recirculation area is observed with the learning process of 400 epochs and an improvement of up to $124\%$ is obtained in the recirculation area with the learning process of 800 epochs. The good results in drag reduction and improvement of recirculation area show the effectiveness of the control policy, as well as the framework based of the data-driven reward. The AFC results of this method can be improved further by extending the learning process of the agent and using more complete information of the flow field.

From the further analysis through the DMD on the complete flow field information, the modifications of the modes for the flow with AFC can be observed. The fundamental frequency of the controlled flow is decreased of approximately $11\%$. With the active control the amplitudes of the modes of the flow are reduced with the active control, especially the main mode. Larger space between the fluctuation of the mode and mode positions in further downstream of the cylinder are observed in the velocity magnitude contours reconstructed from the DMD-modes. These results show that the modifications of the modes for the flow through control is beneficial to stabilize the vortex street, which also shows that the AFC policy learnt through with the improved reward is effective in stabilize the flow over a cylinder.

Since measuring probes are used in this study, the framework of the DRL model with DMD-based reward presented in this paper is able to be applied to experiments in the further study. The configuration of the DRL model can be modified into the case with more steps of episode (larger episode duration) and less epochs of learning for the application in experiments. As concluded from this study, with more probes positioned in the flow domain, more complete information of the field can be measured, which will results in an improvement in the control result. Besides the experimental application, this method has many open prospects of research. It can be applied to other AFC problems with more complex geometry like airfoils in the study of stall. It also can be considered to be utilized in more complex simulations such as three-dimensional large eddy simulations. In this case, the amplitudes of the dynamic modes are used in the computation of the reward function, while more information like the characteristics in the mode vectors can be considered in the reward function in the further study.