Hierarchical Deep Reinforcement Learning for Age-of-Information Minimization in IRS-aided and Wireless-powered Wireless Networks

DRL has been introduced recently as an effective solution for AoI minimization by adapting the scheduling and beamforming strategies according to time-varying traffic demands and channel conditions. The authors in [7] designed the freshness-aware scheduling solution by using the DQN method. The DQN agent continuously updates its scheduling strategy to maximize the freshness of information in the long term. The authors in [8] focused on a multi-user status update system, where a single sensor node monitors a physical process and schedules its information updates to multiple users with time-varying channel conditions. Based on the user’s instantaneous ACK/NACK feedback, the DRL agent at the information source can decide on when and to which user to transmit the next information update, without any priori information on the random channel conditions. The authors in [9] focused on a multi-access system where the base-station (BS) aims to maximize its information collected from all wireless users. Given a strict time deadline to each wireless user, the PPO algorithm was used to adapt the scheduling policy by learning the users’ traffic patterns from the past experiences. The authors in [12] considered a different multi-user scheduling scheme that allows a group of sensor nodes to transmit their information simultaneously. The scheduling decision is adapted to minimize the AoI by using the double DQN (DDQN) method [13], an extension of the DQN method by using two sets of deep neural networks (DNNs) to approximate the Q-value. The AoI-energy tradeoff study in [14] revealed that the sensor nodes’ energy consumption can be reduced without a significant increase in the AoI, by using DQN to adapt the content update in the caching node.

The wireless power transfer and energy harvesting are promising techniques to sustain the massive number of low-cost sensor nodes. However, energy harvesting is usually unreliable depending on the channel conditions. The dynamics and scarcity in energy harvesting make it more challenging for the sensor nodes’ energy management and AoI minimization. The authors in [15] focused on AoI minimization in a cognitive radio network with dynamic supplies of the energy and spectrum resources. The optimal scheduling policy is derived by a dynamic programming approach, revealing a threshold structure depending on the sensor nodes’ AoI states. The authors in [16] revealed that the optimal policy allows each sensor to send a status update only if the AoI value is higher than some threshold that depends on its energy level. Considering stochastic energy harvesting at sensor devices, the authors in [17] studied the AoI minimization problem with the long-term energy constraints and proposed Lyapunov-based dynamic optimization to derive an approximate solution. Generally the dynamic optimization approaches are not only computational demanding, but also relying on the availability of system information. Without knowing the dynamic energy arrivals, the authors in [18] reformulated the AoI minimization problem into MDP. The online Q-learning method was proposed to adaptively schedule the wireless devices’ information update. The authors in [19] focused on the RF-powered wireless network, where the wireless devices can harvest RF power from a dedicated BS and then transmit their update packets to the BS. To minimize the long-term AoI, the DQN algorithm was used to adapt the scheduling between the downlink energy transfer and the uplink information transmission. Both the DQN and dueling DDQN methods were used in [20] to adapt the sensing and information update policy for AoI minimization in a spectrum sharing cognitive radio system. Considering energy harvesting ad hoc networks, the authors in [21] solved the AoI minimization problem by using the advantage actor-critic (A2C) algorithm to adapt the scheduling and power allocation policy, which shows faster runtime and comparable AoI performance to the optimum. The above-mentioned works typically solve the AoI minimization by using the conventional model-free DRL methods. These methods become inflexible and unreliable due to slow convergence with the increasing state and action spaces.

The IRS’s reconfigurability can be used to enhance the channel quality/capacity or reduce the transmission delay by tuning the phase shifts of the reflecting elements [4]. Only a few existing works have discussed the IRS’s application for AoI minimization in wireless networks. The authors in [22] focused on a mobile edge computing (MEC) system and proposed using the IRS to minimize the workload processing delay by optimizing the IRS’s passive beamforming strategy. The authors in [23] set delay constraints to the wireless users’ uplink information transmissions and revealed that the IRS’s passive beamforming can help reduce the wireless users’ transmit power. The authors in [24] employed the IRS to enhance the AoI performance by jointly optimizing the uses’ scheduling and the IRS’s passive beamforming strategies. The combinatorial scheduling decision is adapted by the model-free DRL algorithm, while the passive beamforming optimization relies on the solution to the conventional semi-definite relaxation (SDR) problem. The authors in [25] employed the UAV-carrying IRS to relay information from the ground users to the BS. The AoI minimization requires the optimization of the UAV’s altitude, the ground users’ transmission scheduling, and the IRS’s passive beamforming strategies. Comparing to [24] and [25], our work in this paper exploits the performance gains in both the uplink and downlink of the IRS-assisted system. The IRS’s passive beamforming not only assists uplink information transmission but also enhances or balances the AP’s downlink energy transfer to the users.

We consider a time-slotted frame structure to avoid contention between different nodes. Each data frame is equally divided into $T$ time slots allocated to different sensor nodes. Let $\mathcal{T}\triangleq\{1,2,\ldots,T\}$ denote the set of all time slots. In each time slot, we need to firstly decide the AP’s operation mode, i.e., the time slot is used for either the downlink energy transfer or the uplink data transmission. We use $\psi_{0}(t)\in\{0,1\}$ to denote the AP’s mode selection in each time slot, i.e., $\psi_{0}(t)=1$ indicates the downlink energy beamforming and $\psi_{0}(t)=0$ represents the uplink information transmission. We further use $\psi_{k}(t)\in\{0,1\}$ to denote the uplink scheduling policy, i.e., $\psi_{k}(t)=1$ represents that the $k$-th sensor node is allowed to access the channel for uploading its sensing information to the AP. We require that at most one sensor node can access the channel in each time slot, which implies the following scheduling constraint:

We denote $\boldsymbol{\Psi}(t)=[\psi_{0}(t),\psi_{1}(t),\ldots,\psi_{K}(t)]$ as the AP’s scheduling policy, which depends on the sensor nodes’ traffic demands, channel conditions, and energy status.

Let $\mathcal{N}\triangleq\{1,2,\ldots,N\}$ denote the set of the IRS’s reflecting elements and $\theta_{n}(t)\in(0,2\pi]$ denote the phase shift of the $n$-th reflecting element in the $t$-th time slot. We define the IRS’s phase shifting vector in the $t$-th time slot as $\boldsymbol{\theta}(t)=[e^{j\theta_{n}(t)}]_{n\in\mathcal{N}}$. Note that the IRS can set different beamforming vectors, denoted as $\boldsymbol{\theta}_{d}(t)\triangleq[e^{j\theta_{d,n}(t)}]_{n\in\mathcal{N}}$ and $\boldsymbol{\theta}_{u}(t)\triangleq[e^{j\theta_{u,1}(t)}]_{n\in\mathcal{N}}$, for the downlink and uplink phases, respectively. The channel matrix from the AP to the IRS in $t$-th time slot is given by ${\bf G}(t)\in\mathbb{C}^{M\times N}$. The channel vectors from the IRS and the AP to the $k$-th sensor node are denoted by ${\bf h}^{{r}}_{k}(t)\in\mathbb{C}^{N\times 1}$ and ${\bf h}^{{d}}_{k}(t)\in\mathbb{C}^{M\times 1}$, respectively. The AP can estimate the channel information by a training period at the beginning of each time slot.

When $\psi_{0}(t)=1$, the IRS-assisted downlink energy transfer ensures the sustainable operation of the system. Given the IRS’s passive beamforming strategy ${\boldsymbol{\theta}}_{d}(t)$, the equivalent downlink channel vector ${\bf f}_{d,k}(t)$ from the AP to the $k$-th sensor node can be expressed as follows:

where we define $\boldsymbol{\Theta}_{d}(t)\triangleq\text{diag}(\boldsymbol{\theta}_{d}(t))$ as a diagonal matrix with the diagonal element $\boldsymbol{\theta}_{d}(t)$. The phase shifting matrix $\boldsymbol{\Theta}_{d}(t)$ represents the IRS’s passive beamforming strategy in the downlink energy transfer. Let ${\bf w}_{d}(t)\in\mathbb{C}^{M\times 1}$ denote the AP’s transmit beamforming vector in the downlink energy transfer phase. Given the AP’s transmit power $p_{s}$, the AP’s beamforming signal is given by ${\bf x}(t)=\sqrt{p_{s}}{\bf w}_{d}(t)s_{0}(t)$, where $s_{0}(t)\in\mathbb{C}$ denotes a random complex symbol with the unit power. Then, the received signal at the $k$-th sensor node is given as ${y}_{k}(t)={\bf f}^{{H}}_{d,k}(t){\bf x}(t)+n_{k}(t)$, where $(\cdot)^{H}$ denotes conjugate transpose and $n_{k}(t)$ is the normalized Gaussian noise with zero mean and unit power. Considering a linear energy harvesting (EH) model [27], the received signal ${y}_{k}(t)$ can be converted to energy as follows:

where $\eta$ denotes the energy conversion efficiency. The energy harvested from the noise signal is assumed to be negligible. It is clear that the AP can control the energy transfer to different sensor nodes by optimizing the downlink beamforming vector ${\bf w}_{d}(t)$.

In particular, the AP can steer the beam direction toward the sensor nodes with insufficient energy supply. Besides, the IRS’s passive beamforming strategy $\boldsymbol{\Theta}_{d}(t)$ controls the downlink channel conditions ${\bf f}_{d,k}(t)$ to individual receivers. Due to the broadcast nature of wireless signals, the AP’s energy transfer to different sensor nodes depends on the joint beamforming strategies $({\bf w}_{d}(t),\boldsymbol{\Theta}_{d}(t))$ in different time slots. A more practical non-linear EH model can be also applied to our system. In this case, the harvested power firstly increases with the received signal power and then becomes saturated as the received signal power continues to increase, e.g., [28]. This can be approximated by a piecewise linear EH model: $E^{h}_{k}(t)=\min\Big{\{}\eta p_{s}|{\bf f}^{{H}}_{d,k}(t){\bf w}_{d}(t)|^{2},\,\,p_{\text{sat}}\Big{\}}$, where $p_{\text{sat}}$ denotes the saturation power. Our algorithm in the following part adopts the linear EH model in (3) and can be easily applied to the piecewise linear model with minor modifications.

We assume that the uplink channels are the same as the downlink channels in each time slot due to channel reciprocity, similar to that in [11] and [19]. Let $p_{k}(t)$ denote the transmit power of the $k$-th sensor node when it is scheduled in the $t$-th time slot, i.e., $\psi_{k}(t)=1$. The signal received at the AP is given by $\textbf{y}_{k}=\sqrt{p_{k}(t)}{\bf f}_{u,k}(t)s_{k}+{\bf n}_{k}(t)$, where ${\bf f}_{u,k}(t)$ denotes the uplink channel from the $k$-th sensor node to the AP and $s_{k}(t)$ denotes its information symbol. Similar to (2), the IRS-assisted uplink channel is given by ${\bf f}_{u,k}(t)={\bf h}^{{d}}_{k}(t)+{\bf G}_{k}(t){\boldsymbol{\Theta}}_{u}(t){\bf h}^{{r}}_{k}(t)$, where ${\boldsymbol{\Theta}}_{u}(t)\triangleq\text{diag}(\boldsymbol{\theta}_{u}(t))$ denotes the IRS’s uplink passive beamforming strategy. Without loss of generality, the noise signal ${\bf n}_{k}(t)$ received by the AP can be normalized to the unit power. Thus, the received SNR can be characterized as $\gamma_{k}(t)=p_{k}(t)|{\bf f}_{u,k}^{H}(t){\bf w}_{u}(t)|^{2}$, where ${\bf w}_{u}(t)$ represents the AP’s receive beamforming vector. By using the time division protocol, the sensor nodes’ uplink transmissions can avoid mutual interference. The AP can simply align its receive beamforming vector ${\bf w}_{u}(t)$ to the uplink channel ${\bf f}_{u,k}(t)$. As such, we can denote the received SNR as $\gamma_{k}(t)=p_{k}(t)||{\bf f}_{u,k}(t)||^{2}$ and characterize the uplink throughput as $r_{k}(t)=\tau_{k}\log\bigl{(}1+p_{k}(t)||{\bf f}_{u,k}(t)||^{2}\bigr{)}$, where $\tau_{k}\in[0,1]$ denotes the uplink transmission time. Given the data size $d_{k}$, we require $r_{k}(t)\geq d_{k}$ to ensure the successful transmission of the sensing information.

In each time slot, the sensor node’s energy consumption is given by $E_{k}^{c}(t)=\tau_{k}(t)(p_{k}(t)+p_{c})$, where $p_{c}$ denotes a constant circuit power to maintain the node’s activity. The power consumption $\tau_{k}(t)p_{k}(t)$ in uplink data transmission is linearly proportional to the transmit power $p_{k}(t)$ and the transmission time $\tau_{k}$. The transmit power $p_{k}(t)$ can vary with the channel conditions to ensure the rate constraint $r_{k}(t)\geq d_{k}$. Let $E_{\max}$ denote the sensor nodes’ maximum battery capacity and ${E}_{k}(t)$ be the energy state in the $t$-th time slot. Then, we have the following energy dynamics:

Here we denote $(x)^{+}\triangleq\max\{x,0\}$ for simplicity.

In the $t$-th time slot, when the $k$-th sensor node is allowed to update its sensing information with $\psi_{k}(t)=1$, all other sensor nodes have to wait for scheduling in the other time slots. The sensor nodes’ AoI values will be either increased by 1 or reset to 1 if the transmission is unsuccessful or successful, respectively, as shown in (5). In this case, we can minimize the energy consumption $E^{c}_{k}(t)=\tau_{k}(t)(p_{k}(t)+p_{c})$ of the scheduled sensor node conditioned on the successful transmission of its sensing data, i.e., $r_{k}(t)\geq d_{k}$. This will preserve more energy for its future use. Thus, we have the following energy minimization problem:

The uplink transmission only considers the node-$k$’s rate constraint in (8b). Hence, the AP’s receive beamforming vector ${\bf w}_{u}$ can be aligned with the uplink channel ${\bf f}_{u,k}={\bf h}^{{d}}_{k}+{\bf G}_{k}{\boldsymbol{\Theta}}_{u}{\bf h}^{{r}}_{k}$. Given ${\bf w}_{u}={\bf f}_{u,k}/||{\bf f}_{u,k}||$, the optimal passive beamforming strategy $\boldsymbol{\Theta}_{u}$ needs to maximize the IRS-assisted channel gain $|{\bf f}_{k}^{H}{\bf w}_{u}|^{2}$ as follows:

which can be easily solved by the SDR method similar to that in [24] and [29]. The transmission control parameters $(\tau_{k},p_{k})$ can be also optimized to minimize the energy consumption in (8a). Let $e_{k}\triangleq\tau_{k}p_{k}$ denote the node’s energy consumption in RF communications. Given the optimal $\boldsymbol{\Theta}_{u}$ to (9), we can find the optimal $(\tau_{k},p_{k})$ by the following problem:

Problem (10) is convex in $(\tau_{k},e_{k})$ and satisfies the Slater’s condition, which allows us to find a closed-form solution by using the Lagrangian dual method [30]. After this, we can easily find the optimal transmit power as $p_{k}=e_{k}/\tau_{k}$. If the energy budget holds, i.e., $e_{k}+p_{c}\tau_{k}\leq E_{k}$, the node-$k$’s data transmission will be successful and thus we can update its AoI as $A_{k}(t+1)=1$.

In downlink energy transfer with $\psi_{0}(t)=1$, we aim to supply sufficient energy to all sensor nodes that can sustain their uplink information transmission to minimize the time-averaged AoI performance. However, it is difficult to explicitly quantify how downlink energy transfer affects the AoI performance. Instead, our intuitive design is to transfer more energy to those sensor nodes with the relatively worse AoI conditions. If the node-$k$ has a higher AoI value, we expect to transfer more energy to the node-$k$. This allows the node-$k$ to increase its sampling frequency and report its sensing information with a shorter transmission delay, and thus reducing its AoI value in the following sensing and reporting cycles. By this intuition, in the downlink phase we aim to maximize the AoI-weighted energy transfer to all sensor nodes, formulated as follows:

where $E^{c}_{k}=\tau_{k}(p_{k}+p_{c})$ denotes the node-$k$’s energy consumption in the uplink transmission. For each node-$k$, we define the weight parameter as $v_{k}=A_{k}+\alpha_{k}E_{k}^{-1}$, which is increasing in the AoI value $A_{k}$ while inversely proportional to the energy capacity $E_{k}$. Thus, we prefer to transfer more energy to the sensor node with a higher AoI value and a lower energy supply, which is prioritized by a larger weight parameter $v_{k}$ in (11a). Such a heuristic is expected to reduce the average AoI of all sensor nodes in a long term. The constant $\alpha_{k}$ characterizes the tradeoff between the sensor node’s energy supply and AoI performance. A larger value of $\alpha_{k}$ indicates that the sensor node is more sensitive to the energy insufficiency. Besides, we expect that any sensor node may need to upload its data in future time slots, but we do not know when it will be scheduled to transmit. For a conservative consideration, we impose the constraint (11b) to ensure that all sensor nodes will have sufficient energy to upload their data in the next time slot. If we remove (11b), it becomes possible that some node-$k$ may not have sufficient energy to upload its data after beamforming optimization. If this node-$k$ happens to be scheduled by the DRL agent in the next time slot, its data transmission will be unsuccessful and thus its AoI will continue increasing at the AP. Therefore, we include the constraint in (11b) as a one-step lookahead safety mechanism to ensure that every sensor has sufficient energy for data transmission when it is scheduled in the next time slot.

The DQN algorithm relies on two DNNs to stabilize the learning, denoted by the online Q-network and target Q-network. Given the DNN parameters ${\boldsymbol{\mu}}_{t}$ and ${\boldsymbol{\mu}}_{t}^{\prime}$ for the two Q-networks, their outputs are given by $Q_{{\boldsymbol{\mu}}_{t}}({\bf s}_{t},{\bf a}_{t})$ and $Q_{{\boldsymbol{\mu}}_{t}^{\prime}}({\bf s}_{t},{\bf a}_{t})$, respectively. At each learning episode, the DQN agent observes the system state ${\bf s}_{t}=({\bf A}(t),{\bf E}(t))$ and chooses the best scheduling action ${\bf a}_{t}$ with the maximum value of $Q_{{\boldsymbol{\mu}}_{t}}({\bf s}_{t},{\bf a}_{t})$. To enable random exploration, the DQN agent can also take a random action with a small probability. Once the action ${\bf a}_{t}$ is fixed and executed in the network, the DQN agent observes the instant reward $v_{t}({\bf s}_{t},{\bf a}_{t})$ and records the transition to the next state ${\bf s}_{t+1}$. Each transition sample $({\bf s}_{t},{\bf a}_{t},v_{t},{\bf s}_{t+1})$ will be stored in the experience replay buffer. Meanwhile, the DQN agent estimates the target Q-value as $y_{t}=v_{t}({\bf s}_{t},{\bf a}_{t})+\varepsilon Q_{{\boldsymbol{\mu}}_{t}^{\prime}}({\bf s}_{t+1},{\bf a}_{t+1})$ by using the target Q-network with a different weight parameter ${\boldsymbol{\mu}}_{t}^{\prime}$.

DQN’s success lies in the design of the experience replay mechanism that improves the learning efficiency by reusing the historical transition samples to train the DNN in each iteration [33]. The DNN training aims to adjust the parameter ${\boldsymbol{\mu}}_{t}$ to minimize a loss function $\ell({\boldsymbol{\omega}}_{t})$, which is defined as the gap or more formally the temporal-difference (TD) error between the online Q-network $Q_{{\boldsymbol{\mu}}_{t}}({\bf s}_{t},{\bf a}_{t})$ and the target value $y_{t}$, specified as follows:

The expectation in (15) is taken over a random subset (i.e., mini-batch) of transition samples from the experience replay buffer. For each mini-batch sample, we can evaluate the target value $y_{t}$ and generate the Q-value $Q_{{\boldsymbol{\mu}}_{t}}({\bf s}_{t},{\bf a}_{t})$. The weight parameters ${\boldsymbol{\mu}}_{t}$ can be updated by the backpropagation of the gradient information. The DQN method stabilizes the learning by periodically copying the DNN parameter ${\boldsymbol{\mu}}_{t}$ of the online Q-network to the target Q-network.

Different from the value-based DQN method, the policy-based approach improves the value function by updating the parametric policy $\pi_{{\boldsymbol{\omega}}}$ in gradient-based methods [33]. Given the system state ${\bf s}_{t}$, the policy $\pi_{{\boldsymbol{\omega}}}({\bf a}_{t}|{\bf s}_{t})$ specifies a probability distribution over different actions ${\bf a}_{t}\in\mathcal{A}$. The DNN training aims at updating the policy network to improve the expected value function, rewritten as $J({\boldsymbol{\omega}})=\sum_{{\bf s}\in\mathcal{S}}d({\bf s})V_{\pi}({\bf s})=\sum_{{\bf s}\in\mathcal{S}}d({\bf s})\sum_{{\bf a}\in\mathcal{A}}\pi_{\boldsymbol{\omega}}({\bf a}|{\bf s})Q^{\pi}({\bf s},{\bf a})$, where $d({\bf s})$ is the stationary state distribution corresponding to the policy $\pi_{\boldsymbol{\omega}}$ and $Q^{\pi}({\bf s},{\bf a})$ denotes the Q-value of the state-action pair $({\bf s},{\bf a})$ following the policy $\pi_{\boldsymbol{\omega}}$. Now the expected value function $J({\boldsymbol{\omega}})$ becomes a function of the policy parameter ${\boldsymbol{\omega}}$. The policy gradient theorem in [35] simplifies the evaluation of the policy gradient $\nabla_{\boldsymbol{\omega}}J({\boldsymbol{\omega}})$ as follows:

where the expectation is taken over all possible state-action pairs following the same policy $\pi_{\boldsymbol{\omega}}$.

For practical implementation, the policy gradient $\nabla_{\boldsymbol{\omega}}J({\boldsymbol{\omega}})$ can be evaluated by sampling the historical decision-making trajectories. At each learning epoch $t$, the DRL agent interacts with the environment by the state-action pair $({\bf s}_{t},{\bf a}_{t})$, collects an immediate reward $v_{t}$, and observes the transition to the next state ${\bf s}_{t+1}$. Let ${\boldsymbol{\ell}}\triangleq\{{\bf s}_{1},{\bf a}_{1},v_{1},{\bf s}_{2},{\bf a}_{2},v_{2},{\bf s}_{3},\ldots,v_{T}\}$ denote the state-action trajectory as the DRL agent interacts with the environment. We can estimate the Q-value $Q^{\pi}({\bf s}_{t},{\bf a}_{t})$ in (16) by $g_{t}=\sum_{\tau=t}^{T}\varepsilon^{\tau-t}v_{t}$. As such, we can approximate the policy gradient $\nabla_{\boldsymbol{\omega}}J({\boldsymbol{\omega}})$ in each time step by the random sample $g_{t}\nabla_{\boldsymbol{\omega}}\ln\pi_{\boldsymbol{\omega}}({\bf a}_{t}|{\bf s}_{t})$ and update the policy network as ${\boldsymbol{\omega}}\leftarrow{\boldsymbol{\omega}}+\alpha_{\boldsymbol{\omega}}g_{t}\nabla_{\boldsymbol{\omega}}\ln\pi_{\boldsymbol{\omega}}({\bf a_{t}}|{\bf s}_{t})$, where $\alpha_{\boldsymbol{\omega}}$ denotes the step-size for gradient update. Besides the stochastic approximation, we can also employ another DNN to approximate the Q-value $Q^{\pi}({\bf s}_{t},{\bf a}_{t})$ in (16) and replace the Monte Carlo estimation $g_{t}$ by the DNN approximation $Q_{\boldsymbol{\mu}}({\bf s}_{t},{\bf a_{t}})$ with the weight parameter ${\boldsymbol{\mu}}$, similar to that in the DQN algorithm. This motivates the actor-critic framework to update both the policy network and the Q-value network [35]. The actor updates the policy network while the critic updates the Q-network by minimizing a loss function similar to (15). We can take the derivative of the loss function and update the weight parameter as ${\boldsymbol{\mu}}\leftarrow{\boldsymbol{\mu}}+\alpha_{\boldsymbol{\mu}}\delta_{t}\nabla_{\boldsymbol{\mu}}Q_{\boldsymbol{\mu}}({\bf s}_{t},{\bf a_{t}})$, where $\delta_{t}=y_{t}-Q_{{\boldsymbol{\mu}}_{t}}({\bf s}_{t},{\bf a}_{t})$ denotes the TD error. The Q-value estimation can be also replaced by the advantage function $A_{\pi}({\bf s}_{t},{\bf a}_{t})\triangleq Q^{{\pi}}({\bf s}_{t},{\bf a}_{t})-V_{\pi}({\bf s}_{t})$ to reduce the variability in predictions and improve the learning efficiency.

The gradient estimation in (16) requires a complete trajectory by using the same target policy $\pi_{\boldsymbol{\omega}}$. It is actually the on-policy method that refrains us from using past experiences and limits the sample efficiency. This drawback can be avoided by a minor revision to the policy gradient:

where the behavior policy $\pi_{\boldsymbol{\omega}}^{o}$ is used to collect the training samples. This becomes the off-policy gradient that allows us to estimate it by using the past experience collected from a different and even obsolete behavior policy $\pi_{\boldsymbol{\omega}}^{o}$. Hence, we can improve the sample efficiency by maintaining the experience replay buffer, similar to the DQN method. To further improve training stability, the off-policy trust region policy optimization (TRPO) method imposes an additional constraint on the gradient update [36], i.e., the new policy $\pi_{\boldsymbol{\omega}}$ should not change too much from the old policy $\pi_{\boldsymbol{\omega}}^{o}$. Therefore, the policy optimization is to solve the following constrained problem:

where $D_{KL}(P_{1},P_{2})\triangleq\int_{-\infty}^{\infty}P_{1}(x)\log\left(P_{1}(x)/P_{2}(x)\right)\,dx$ represents a distance measure in terms of the Kullback-Leibler (KL) divergence between two probability distributions [36]. The inequality constraint in (18) enforces that the KL divergence between two policies $\pi_{\boldsymbol{\omega}}$ and $\pi_{\boldsymbol{\omega}}^{o}$ are bounded by $\delta_{KL}$. The advantage function $A_{\pi_{\boldsymbol{\omega}}^{o}}$ in the objective of (18) is the approximation of the true advantage $A_{\pi_{\boldsymbol{\omega}}}$ corresponding to the target policy $\pi_{\boldsymbol{\omega}}$. However, the exact solution to the optimization problem (18) is not easy. Normally we require the first- and second-order approximations for both the objective and the constraint in (18).

The proximal policy optimization (PPO) algorithm proposed in [10] further improves the objective in (18) by limiting the probability ratio or the importance weight $\rho_{\boldsymbol{\omega}}({\bf s},{\bf a})\triangleq\frac{\pi_{\boldsymbol{\omega}}({\bf s},{\bf a})}{\pi^{o}_{\boldsymbol{\omega}}({\bf s},{\bf a})}$ within a safer region $[1-\epsilon,1+\epsilon]$.

where the function $\text{clip}(\rho_{\boldsymbol{\omega}},1-\epsilon,1+\epsilon)$ returns $\rho_{\boldsymbol{\omega}}$ if $\rho_{\boldsymbol{\omega}}\in[1-\epsilon,1+\epsilon]$ and returns $1-\epsilon$ (or $1+\epsilon$) if $\rho_{\boldsymbol{\omega}}<1-\epsilon$ (or $\rho_{\boldsymbol{\omega}}>1+\epsilon$). The parameter $\epsilon$ is used to control the clipping range. The approximate value function $\tilde{J}(\pi_{\boldsymbol{\omega}})$ ensures that the target policy ${\pi_{\boldsymbol{\omega}}}$ will not deviate too far from the behavior policy ${\pi_{\boldsymbol{\omega}}^{o}}$ for either positive or negative advantage $A_{\pi_{\boldsymbol{\omega}}^{o}}$. With the clipped value function $\tilde{J}(\pi_{\boldsymbol{\omega}})$, we further introduce a Lagrangian dual variable $\beta_{KL}$ and reformulate the constrained problem (18) into the unconstrained maximization as follows:

The policy parameter ${\boldsymbol{\omega}}$ for the new value function in (20) can be easily updated by using the stochastic gradient ascent method. The parameter $\beta_{KL}$ can be also adapted according to the difference between the measured KL divergence $\mathbb{E}_{\pi_{\boldsymbol{\omega}}^{o}}\Big{[}D_{KL}(\pi_{\boldsymbol{\omega}},\pi_{\boldsymbol{\omega}}^{o})\Big{]}$ and its target $\delta_{KL}$.