Model-free Nearly Optimal Control of Constrained-Input Nonlinear Systems Based on Synchronous Reinforcement Learning

The control method of nonlinear constraint systems has been widely researched and is a valuable practical problem. In the framework of optimal control, an optimal policy that minimize a user-defined performance index and the corresponding value function can be obtained by solving Hamilton-Jacobi-Bellman (HJB) equation, which is a partial differential equation in continuous-time nonlinear systems and the analytical solution is usually hard to find. Therefore, function approximation using neural networks (NNs) is a useful tool to solve HJB equation. These algorithms belong to the family of adaptive dynamic programming (ADP) or reinforcement learning (RL) methods.

ADP/RL methods have solved several discrete-time optimal control problems such as nonlinear zero-sum games [1] and multi-agent systems [2], [3]. A considerable number of literatures have also focused on solving the continuous-time constrained-input optimal control problem via ADP/RL. In [4], an off-line policy iteration (PI) algorithm that can iteratively solve HJB equation is proposed. The full knowledge of system dynamics and an initial stabilized controller are needed in this algorithm. In these years, many works are intended to relax the restriction of the a priori knowledge of system dynamics. The concept of integral reinforcement learning (IRL, [5]) and generalized PI [6] is brought in [7], [8] to estimate the optimal value function in constrained-input problems. The initial stabilized controller is not needed in these methods, but the input gain matrix of the system is still used in policy improvement step.

Another way to solve this problem is to use an identifier NN. Xue et al. [9] proposed an algorithm based on actor-critic-identifier structure to estimate the system dynamics and solve HJB equation online. Comparing with other model-free methods, an extra identifier might bring additional burden on real-time computing.

In terms of IRL, Vamvoudakis [10] proposed a completely model-free Q-learning algorithm to solve continuous-time optimal control problem. Q-function combines the value function and the Hamiltonian in Pontryagin’s minimum principle and provides the information of policy improvement without requiring any a priori. Similarly, the restriction of the system input dynamics can be relaxed by adding exploration signal [11]. Our previous work [12] also present a novel model-free algorithm that combines the exploration signal and synchronous reinforcement learning to solve the adaptive optimal control problem in continuous-time nonlinear systems.

In this paper we present an algorithm to solve the continuous-time constrained-input optimal control problem without knowing any information of system dynamics or an initial stabilized controller. The identifier NN in [9] is also not needed in this method. The remainder of this paper is organized as follows: Section 2 provides the mathematic formulation of the optimal control for continuous-time systems with input constraints. In section 3 we present the weights tuning law of actor-critic NNs based on exploration-HJB and synchronous reinforcement learning. In the process of learning, policy evaluation and improvement can be done simultaneously and no knowledge of system dynamics is required. The convergence of weights and stability of the closed-loop system is proved through Lyapunov analysis. Finally, we design two simulations to verify the effectiveness of our method.

Consider a continuous-time input-affine system

where $x\in\mathbb{R}^{n}$ and $u\in\mathbb{R}^{m}$ is the measurable state and input vector respectively. $\xi$ denotes the initial state of the system. $f(x)\in\mathbb{R}^{n}$ is called as the system drift dynamics and $g(x)\in\mathbb{R}^{n\times m}$ is the system input dynamics. The system dynamics $f(x)+g(x)u$ is assumed to be Lipschitz in a compact set $\Omega$ and satisfies $f(0)=0$.

The objective of optimal control is to design a controller that minimize a user-defined performance index. The performance index in this paper is defined as

both $Q(x)$ and $U(u)$ are positive definite functions.

In this paper, the constrained-input control problem is considered. The input vector needs to satisfy the constraint $|u_{i}|\leq\lambda(i=1,...,m).$ To deal with this constraint, a non-quadratic integral cost function of input is usually employed [13]:

where $\bm{\vartheta}(s)=\left[\vartheta(s_{1}),...,\vartheta(s_{m})\right]^{\top}$ and $\bm{\vartheta}^{-1}(s)=\left[\vartheta^{-1}(s_{1}),...,\vartheta^{-1}(s_{m})\right]^{\top}$. $\vartheta$ is a bounded one-to-one monotonic odd function. The hyperbolic tangent function $\vartheta(\cdot)=\tanh(\cdot)$ is often used. $R$ is a positive definite symmetric matrix. For the convenience of computing and analysis, $R$ is assumed to be diagonal, i.e. $R={\rm diag}(r_{1},...,r_{m}).$

The goal is to find the optimal control law $u^{*}$ that can stabilize system (1) and minimize performance index (2). The following value function is used:

where $\mu(x)$ is a feedback policy and $\mu(0)=0.$ The value function is well-defined only if the policy is admissible [4].

Assume that the set of admissible policy $\mathcal{A}(\Omega)$ of system (1) is not empty and $V^{\mu}\in\mathcal{C}^{1}(\Omega)$. It is obvious that there exists an optimal policy $\mu^{*}(x)\in\mathcal{A}(\Omega)$ so that

The optimal value function satisfies $V^{*}(x)=V^{\mu^{*}}(x)$. In the remainder of the paper both of these two symbols may be used to describe the same variable.

According to the definition of the value function (4), the infinitesimal version of (2) is obtained as

where $\nabla V(x)=\partial V(x)/\partial x\in\mathbb{R}^{n}$. Define the Hamiltonian as

For optimal value function $V^{*}$ and optimal control law $u^{*}$, the following Hamilton-Jacobi-Bellman (HJB) equation is satisfied:

The optimal policy can be derived by minimizing the Hamiltonian, for input-affine system (1) and the performance index (2) and (3), the optimal policy can be directly denoted as

Let

the optimal policy can be obtained as

It is hard to solve the optimal value function and policy in (7) directly and the a priori knowledge of system dynamics is required. Therefore, the online data-driven algorithm is a useful tool to relax the restriction.

In this section a data-driven method that learning the optimal policy is introduced. The framework of integral reinforcement learning (IRL) is proposed in [5]. A completely model-free algorithm based on IRL is developed by Lee et al. [11] to solve the optimal control problem of input-affine systems. However, the initial stabilized controller is still required to start the iteration.

In our previous work [12], an algorithm called synchronous integral Q-learning is proposed, which can continuously update the weights of both two NNs to estimate the optimal value function and policy. In this paper we extend the algorithm to solve the constrained-input problem. By adding a bounded piece-wise continuous non-zero exploration signal $e$ into input, the system (1) is derived as

Substituting (11) into HJB equation (7). For any time $t>T$ and time interval $T>0$, integrating (7) from $[t-T,t]$ along the trajectory of system (11), the following integral temporal difference equation is obtained as

The following HJB equation is derived by substituting (9):

Both $V^{*}(x)$ and $v^{*}(x)$ can be solved in integral exploration-HJB equation (13) and the optimal policy is computed by (10). Note that the knowledge of system dynamics is not needed in these two equations.

Since the analytical solution of $V^{*}(x)$ and $v^{*}(x)$ cannot be determined. We can approximate them by choosing a proper structure of neural networks (NNs). Assume $V^{*}(x)$ is a smooth function. Based on the assumption of system dynamics, $v^{*}(x)$ is also smooth. Therefore, there exists two single-layer NNs, namely actor and critic NN, so that:

1) $V^{*}(x)$ and its gradient can be universal approximated:

2) $v^{*}(x)$ can be universal approximated:

where $\phi_{c}(x):\mathbb{R}^{n}\to\mathbb{R}^{N_{c}},w_{c},\varepsilon_{c}$ is the activation function, weights and the reconstruction error of the critic NN, respectively. $N_{c}$ is the number of neurons in the hidden layer of the critic NN. Similarly, in actor NN, we denote $\phi_{a}(x):\mathbb{R}^{n}\to\mathbb{R}^{N_{a}},w_{a},\varepsilon_{a}$ is the activation function, weights and the reconstruction error, respectively. $N_{a}$ is the number of neurons in the hidden layer of the actor NN. $\varepsilon_{c}(x)$ and $\varepsilon_{a}(x)$ are bounded on a compact set $\Omega$. According to Weierstrass approximation theorem, when $N_{c},N_{a}\to\infty$, we have $\varepsilon_{c},\nabla\varepsilon_{c},\varepsilon_{a}\to 0.$

Define the Bellman error $\varepsilon_{B}$ based on (13) as

where $\Delta\phi_{c}=\phi_{c}(x(t))-\phi_{c}(x(t-T))$. The integral reinforcement is defined as

Using the characteristic of Kronecker product, the following equation can be derived from (17):

where $W=[w_{c}^{\top},{\rm col}\{w_{a}\}^{\top}]^{\top}$. ${\rm col}\{w_{a}\}$ is the reshaped column vector of $w_{a}$, and

Before introducing the weights tuning law, it is necessary to make some assumptions of NNs.

These assumptions are trivial and can be easily satisfied except the assumption of $g(x).$ With the assumption that system dynamics is Lipschitz, Bellman error $\varepsilon_{B}$ is bounded on a compact set. When $N_{c},N_{a}\to\infty$, $\varepsilon_{B}\to 0$ universally [4].

In this paper we use $\vartheta(\cdot)=\tanh(\cdot)$. The non-quadratic cost $U(u)$ is derived as

where $\overline{R}=[r_{1},...,r_{m}],\bm{1}_{m}$ is the $m$-dimensional vector with all elements are one. According to (17) and (19), the approximate exploration-HJB equation is obtained as

in which $\varepsilon_{HJB}(x)$ is the error raised from the reconstruction errors of two NNs. Since the ideal weights $w_{c}$ and $w_{a}$ are unknown, they are approximated in real time as

The optimal policy is obtained as

We use an actor NN to approximate the optimal policy as

where $\hat{v}_{2}(x)=\hat{w}_{a2}^{\top}\phi_{a}(x)$. Note that $\hat{v}_{1}$ and $\hat{v}_{2}$ have the same approximate structure. However, the estimated weights in (22) will not be directly implemented on the controller for Lyapunov stability. Define $\hat{W}=[\hat{w}_{c}^{\top},{\rm col}\{\hat{w}_{a1}\}]^{\top}$, the approximate Bellman error of the critic NN is obtained as

In order to minimize the error, we define the objective function of critic NN as $K=\frac{1}{2}E^{\top}E.$ The modified gradient-descent law is obtained as

where $\alpha_{1}$ is the learning rate that determines the speed of convergence. In order to guarantee the stability of the closed-loop system and the convergence. Define the approximate error of the actor NN as

The gradient-descent tuning law of $\hat{w}_{a2}$ is set as

where $Y$ is a designed parameter to guarantee stability. Define $m_{s}=1+\delta^{\top}\delta$ and $\overline{\delta}=\delta/m_{s}$, the following theorem is derived.

Define the error of weights as $\tilde{w}_{c}=w_{c}-\hat{w}_{c},\tilde{w}_{a1}=w_{a}-\hat{w}_{a1}$ and $\tilde{w}_{a2}=w_{a}-\hat{w}_{a2}$. Consider the following Lyapunov candidate

The derivative of (28) to time is

The first term in (29) is obtained as

From (16), we have

Using $w_{a}=\hat{w}_{a2}+\tilde{w}_{a2}$ and the fact that $x^{\top}\tanh(x)\geq 0,\forall x$ for the first term on the right side of (31), it can be obtained as

Therefore, (30) turns into

where $\varepsilon_{1}=\nabla\varepsilon_{c}^{\top}(f+ge)-2\varepsilon_{a}^{\top}R\lambda\tanh(\hat{w}_{a2}^{\top}\phi_{a}/\lambda)$. According to assumption 1, it satisfies

where $\sigma_{\min}(R)$ denotes the minimum singular value of matrix $R$. Substituting the derivative of the approximate HJB equation (20) into (33), we have

The last term in (35) satisfies

Since $Q$ and $U$ are positive definite, there exists a $q>0$ such that $x^{\top}qx<Q+U$. Therefore, (35) can be derived as

where

$\varepsilon_{h}$ is the upper bound of $\varepsilon_{HJB}$.

Now analyzing the second term in (29). According to the tuning law (25), we have the following error dynamics

Substituting HJB equation (20) into (38), one has

According to the definition of $U$, we have

According to [7], the third term in (40) can be written as

where $\Arrowvert\varepsilon_{\hat{D}}\Arrowvert\leq\ln 4$. Similarly, we have

where $\Arrowvert\varepsilon_{D}\Arrowvert\leq\ln 4$. The following inequality is derived from (40):

In [7], a continuous approximation of $x\ {\rm sgn}(x)$ is provided:

Combining (43) and (44), the approximate HJB error is

in which the approximate error satisfies $0\leq\varepsilon_{\kappa}\leq 7.18/\kappa$.

Substituting (31) into (45), we can get

where

and

Substituting (46) into (38), the error dynamics can be rewritten as

The second term in (29) is obtained as

For a sufficiently small reinforcement interval, the integral term in (48) can be approximated as