Accelerated Continuous-Time Approximate Dynamic Programming via Data-Assisted Hybrid Control¹¹1Research supported in part by NSF grant number CNS-1947613.²²footnotemark: 2

Recent technological advances in computation and sensing have incentivized the development and implementation of data-assisted feedback control techniques previously deemed intractable due to their computational complexity. Among these techniques, reinforcement learning (RL) has emerged as a practically viable tool with remarkable degrees of success in robotics [1], autonomous driving [2], water-distribution systems [3], among other cyber-physical applications, see [4]. These types of algorithms, are part of a large landscape of adaptive systems that aim to control a plant while simultaneously optimizing a performance index in a model-free way, with closed-loop stability guarantees.

In this paper, we focus on a particular class of infinite horizon RL problems from the perspective of approximate optimal control and approximate adaptive dynamic programming (AADP). Specifically, we study the optimal control problem for nonlinear continuous-time and control-affine deterministic plants, interconnected with approximate adaptive optimal controllers [5] in an actor-critic configuration. These types of adaptive controllers aim to find, in real time, the solution to the Hamilton-Jacobi-Bellman (HJB) equation by measuring the output of the nonlinear dynamical system while making use of two approximation structures:

• 1.

  a critic, used to estimate the optimal value function of the optimal control problem, and

• 2.

  an actor, used to estimate the optimal feedback controller.

Our goal is to design online adaptive dynamics for the real-time tuning of the aforementioned structures, while simultaneously achieving closed-loop stability and high transient performance. To achieve this, and motivated by the widespread usage of momentum-based gradient dynamics in practical RL settings [6], we study continuous-time actor-critic dynamics inspired by a class of ordinary differential equations (ODEs) that can be seen as continuous-time counterparts of Nesterov’s accelerated optimization algorithm [7]. Such types of algorithms have gained popularity in optimization and related fields due to the fact that they can minimize smooth convex functions at a rate of order $\mathcal{O}(1/t^{2})$ [8]. The main source for the acceleration property in these ODEs comes from the addition of momentum to gradient-based dynamics, in conjunction with a vanishing dynamic damping coefficient. However, as recently shown in [9] and [10], the non-uniform convergence properties that emerge in these types of dynamics complicates their use in feedback systems with plant dynamics in the loop. In this paper, we overcome these challenges by incorporating resets into the proposed momentum-based algorithms, similar to restarting heuristics studied in the machine learning literature, see [11] and [7]. Our resulting actor-critic controller is naturally modeled by a hybrid dynamical system that incorporates continuous-time and discrete-time dynamics, which we analyze using tools from [12].

A traditional assumption in the literature of continuous-time actor-critic RL is that the regressors used in the parameterizations satisfy a persistence of excitation condition along the trajectories of the plant. However, in practice, this condition can be difficult to verify a priori. To circumvent this issue, in this paper we consider a data-assisted approach, where a finite amount of past “sufficiently rich” recorded data is used to guarantee asymptotic learning in the closed-loop system. As a consequence, the resulting data-assisted hybrid control algorithm concurrently uses real-time and recorded data, similar in spirit to concurrent-learning (CL) techniques [13]. By using Lyapunov-based tools for hybrid dynamical systems, we analyze the interconnection of an actor-critic neural-network (NN) controller and the nonlinear plant, establishing that the trajectories of the closed-loop system remain ultimately bounded around the origin of the plant and the optimal actor and critic NN parameters. Since the resulting closed-loop system has suitable regularity properties in terms of continuity of the dynamics, our stability results are in fact robust with respect to arbitrarily small additive disturbances that can be adversarial in nature, or that can arise due to numerical implementations. To the best knowledge of the authors, these are the first theoretical stability guarantees of continuous-time accelerated actor-critic algorithms for neural network-based adaptive dynamic programming controllers in nonlinear deterministic settings.

The rest of this paper is organized as follows: Section 2 presents the notation and some concepts on hybrid dynamical systems, Section 3 presents the problem statement and some preliminaries on optimal control. Section 4 introduces the hybrid momentum-based dynamics for the update of the critic NN, Section 5 presents the update dynamics for the actor NN, and Section 6 studies the properties of closed-loop system. In Section 7 we study a numerical example illustrating our theoretical results.

Notation: We denote the real numbers by $\mathbb{R}$, and we use $\mathbb{R}_{\geq 0}\subset\mathbb{R}$ to denote the non-negative real line. We use $\mathbb{R}^{n}$ to represent the $n$-dimensional Euclidean space and $\left\lvert\cdot\right\rvert$ to denote its usual vector norm. Given $A\in\mathbb{R}^{n\times n}$, we use $\left\lvert A\right\rvert$ to denote the induced 2-norm for matrices, and we infer its distinction with the vector norm depending on the context. We use $\text{Tr}\left(A\right)$ to denote the trace operator on matrices. Given a compact set $\mathcal{A}\subset\mathbb{R}^{n}$ and a vector $z\in\mathbb{R}^{n}$, we use $|z|_{\mathcal{A}}\coloneqq\min_{s\in\mathcal{A}}\left\lvert z-s\right\rvert$ to represent the minimum distance of $z$ to $\mathcal{A}$. We also use $r\mathbb{B}$ to denote a closed ball in the Euclidean space, of radius $r>0$, and centered at the origin. We use $I_{n}\in\mathbb{R}^{n\times n}$ to denote the identity matrix, and $(x,y)$ for the concatenation of the vectors $x$ and $y$, i.e., $(x,y)\coloneqq[x^{\top},y^{\top}]^{\top}$. A function $\gamma:\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$ is said to be of class-$\mathcal{K}$ ($\gamma\in\mathcal{K}$), if it is continuous, zero at zero, and nondecreasing. A function $\beta:\mathbb{R}_{\geq 0}\times\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$ is said to be of class-$\mathcal{K}\mathcal{L}$ ($\beta\in\mathcal{KL}$) if $\beta(\cdot,s)\in\mathcal{K}$ for each $s\in\mathbb{R}_{\geq 0}$, it is non-increasing in its second argument, and $\lim_{s\to\infty}\beta(r,s)=0$ for each $r\in\mathbb{R}_{\geq 0}$. The gradient of a real valued function $f:\mathbb{R}^{n}\to\mathbb{R}$ is defined as a column vector and denoted by $\nabla f$. For a vector valued function $g:\mathbb{R}^{n}\to\mathbb{R}^{m}$, we use $\diffp{g(x)}{x}\in\mathbb{R}^{m\times n}$ to denote its Jacobian matrix.

Consider a control-affine nonlinear dynamical plant

$$\dot{x}=f(x)+g(x)u,$$

(3)

where $x\in\mathbb{R}^{n}$ is the state of the system, $u\in U\subset\mathbb{R}^{m}$ is the input, and $f:\mathbb{R}^{n}\to\mathbb{R}^{n}$ and $g:\mathbb{R}^{n}\to\mathbb{R}^{n\times m}$ are locally Lipschitz functions. Our goal is to design a stable algorithm able to find –in real time– a control law $u^{*}$ that minimizes the cost functional $V:\mathbb{R}^{n}\times\mathcal{U}_{V}\to\mathbb{R}$ given by:

$$V(x_{0},u)\coloneqq\int_{0}^{\infty}r\Big{(}x\big{(}\tau\big{)},u\left(x(\tau)\right)\Big{)}d\tau,$$

(4)

where $x\big{(}t\big{)}$ represents a solution to (3) from the initial condition $x(0)=x_{0}$, that results from implementing a feedback law $u$, belonging to a class of admissible control laws $\mathcal{U}_{V}$ characterized as follows:

To leverage the form of (7), we consider the following parameterization of the optimal value function $V^{*}(x)$:

$$V^{*}(x)=\theta_{c}^{*^{\top}}\phi_{c}(x)+\epsilon_{c}(x)\quad\forall x\in K,$$

(8)

where $K\subset\mathbb{R}^{n}$ is a compact set, $\theta_{c}^{*}\in\mathbb{R}^{l_{c}}$, $\phi_{c}:\mathbb{R}^{n}\to\mathbb{R}^{l_{c}}$ is a vector of continuously differentiable basis functions, and $\epsilon_{c}:\mathbb{R}^{n}\to\mathbb{R}$ is the approximation error. The parameterization (8) is always possible on compact sets due to the continuity properties of $V$ and the universal approximation theorem [16]. This parametrization results in an optimal Hamiltonian of the form $H_{p}^{*}\coloneqq H(x,u^{*},\diffp{\phi_{c}}{x}^{\top}\theta_{c}^{*}+\nabla\epsilon_{c})$ given by:

$\displaystyle H_{p}^{*}(x)$

$\displaystyle=\theta_{c}^{*^{\top}}\psi(x,u^{*}(x))+Q(x)+R\left(u^{*}(x)\right)+\nabla\epsilon_{c}(x)^{\top}\left(f(x)+g(x)u^{*}(x)\right),$

(9)

where we defined $\psi:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}^{l_{c}}$ as:

$$\psi(x,u)\coloneqq\diffp{\phi_{c}(x)}{x}\left(f(x)+g(x)u\right).$$

(10)

We note that the explicit dependence of $\psi:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}^{l_{c}}$ on the control action $u$, defined in (10), is a fundamental departure from the previous approaches studied in the context of concurrent learning (CL) NN actor-critic controllers, such as those considered in [17] and [18]. In particular, we note that in the context of CL the data used to estimate the optimal value function $V^{*}$ is generated from measurements of the optimal Hamiltonian which, by definition, incorporates the optimal control law $u^{*}$. Hence, the need to include $u$ as part of the regressor vectors $\psi$ becomes crucial; this dependence characterizes how far our recorded measurements of a Hamiltonian are from the optimal Hamiltonian $H_{p}^{*}$. Indeed, this distance will explicitly emerge in our convergence and stability analysis. Naturally, the dependence of (10) on $u$ will impose stronger conditions on the recorded data needed to estimate $V^{*}$.

Using the optimal value parametrization described in Section 4 the optimal control law can written as:

$$u^{*}(x)=-\frac{1}{2}\Pi_{u}^{-1}g(x)^{\top}\left[\diffp{\phi_{c}(x)}{x}^{\top}\theta_{c}^{*}+\nabla\epsilon_{c}(x)\right],\quad\forall x\in K.$$

(22)

Therefore, using $\diffp{\phi_{c}(x)}{x}$ and $g(x)$ we can implement an actor neural-network given by:

$$\hat{u}(x)=\omega(x)^{\top}\theta_{u},$$

(23)

where $\omega:\mathbb{R}^{n}\to\mathbb{R}^{l_{c}\times m}$ is defined as:

$$\omega(x)\coloneqq-\frac{1}{2}\diffp{\phi_{c}(x)}{x}g(x)\Pi_{u}^{-1}.$$

(24)

To guarantee convergence of $\hat{u}$ to $u^{*}$, we design update dynamics for $\theta_{u}\in\mathbb{R}^{l_{c}}$ based on the minimization of the error:

	$\displaystyle\varepsilon(x,\theta_{c},\theta_{u})$	$\displaystyle\coloneqq\frac{1}{2}\Bigg{[}\alpha_{1}\frac{\varepsilon_{a}(x,\theta_{c},\theta_{u})^{\top}\varepsilon_{a}(x,\theta_{c},\theta_{u})}{1+\text{Tr}\left(\omega(x)^{\top}\omega(x)\right)}+\alpha_{2}\varepsilon_{b}(\theta_{c},\theta_{u})^{\top}\varepsilon_{b}(\theta_{c},\theta_{u})\Bigg{]},$
	$\displaystyle\varepsilon_{a}(x,\theta_{c},\theta_{u})$	$\displaystyle\coloneqq\hat{u}(x)-\omega(x)^{\top}\theta_{c}=\omega(x)^{\top}\left(\theta_{u}-\theta_{c}\right),$
	$\displaystyle\varepsilon_{b}(\theta_{c},\theta_{u})$	$\displaystyle\coloneqq\theta_{u}-\theta_{c},{}$		(25)

which satisfies:

$\displaystyle\nabla_{\theta_{u}}\varepsilon(x,\theta_{c},\theta_{u})$

$\displaystyle=\Omega(x)(\theta_{u}-\theta_{c}),$

where

$\displaystyle\Omega(x)$

$\displaystyle\coloneqq\alpha_{1}\frac{\omega(x)\omega(x)^{\top}}{1+\text{Tr}\left(\omega(x)^{\top}\omega(x)\right)}+\alpha_{2}I\in\mathbb{R}^{l_{c}\times l_{c}}\quad\forall x\in\mathbb{R}^{n}.{}$

(26)

Based on these definitions, we consider the following gradient-descent dynamics for the actor neural-network:

$\displaystyle\dot{\theta}_{u}=F_{u}(\theta_{u},x,\theta_{c})\coloneqq-k_{u}\nabla_{\theta_{u}}\varepsilon(x,\theta_{c},\theta_{u}),{}$

(27)

where $k_{u}\in\mathbb{R}_{>0}$ is a tunable gain. A scheme representing these update dynamics is shown in Figure 2.

Consider the closed-loop resulting from the interconnection between the plant (3), the critic update dynamics (17), the actor update dynamics (27) and the feedback law in (23) shown in Figure 3(a), and given by:

	$\displaystyle\dot{x}$	$\displaystyle=f(x)+g(x)\hat{u}(x),\qquad x^{+}=x,$		(28a)
	$\displaystyle\dot{y}$	$\displaystyle=F_{c}(y,x,\hat{u}(x)),\qquad\quad\leavevmode\nobreak\ y^{+}=G_{c}(y),$		(28b)
	$\displaystyle\dot{\theta}_{u}$	$\displaystyle=F_{u}(\theta_{u},x,\theta_{c}),\qquad\qquad\theta_{u}^{+}=\theta_{u},$		(28c)

and with flow set and jump set given by $C=\mathbb{R}^{n}\times C_{c}\times\mathbb{R}^{l_{c}}$ and $D=\mathbb{R}^{n}\times D_{c}\times\mathbb{R}^{l_{c}}$ respectively, where $C_{c}$ and $D_{c}$ are as defined in (17). Let $z\coloneqq(x,y,\theta_{u})$ be the overall state of the closed-loop system, and define:

$\displaystyle\mathcal{A}$

$\displaystyle\coloneqq\left\{0\right\}\times\mathcal{A}_{c}\times\left\{\theta_{c}^{*}\right\}.$

The following is the main result of this paper.

Theorem 2 establishes asymptotic convergence to a neighborhood of the compact set $\mathcal{A}$ as $\left(\overline{\epsilon_{\text{HJB}}},\overline{d\epsilon_{c}}\right)\to 0$ from any compact set $K_{z}$ modulo some error $\nu$, under a suitable choice of tunable parameters. To the best knowledge of the authors this is the first result providing stability certificates for continuous-time actor-critic reinforcement learning using recorded data and accelerated value-function estimation dynamics with momentum. In addition, since the resulting closed-loop system in (28) is given by a well-posed hybrid system, the stability results are robust with respect to arbitrarily small additive disturbances on the states and dynamics [12, Ch. 7].

In this section, we present a numerical experiment that illustrates our theoretical results. In particular, we study the following nonlinear control-affine plant:

	$\displaystyle\dot{x}=f(x)+g(x)u,$		(29a)
	$\displaystyle f(x)=\begin{bmatrix}-x_{1}+x_{2}\\ -\frac{1}{2}\Bigg{(}x_{1}-x_{2}\Big{(}1-\cos(2x_{1}+2)^{2}\Big{)}\Bigg{)}\end{bmatrix},$		(29b)
	$\displaystyle g(x)\coloneqq\begin{bmatrix}0\\ \cos(2x_{1})+2\end{bmatrix},$		(29c)

with local state and control costs given by $Q(x)=x^{\top}x$ and $R(u)=u^{2}$ [18]. The optimal value function for this setting is given by $V^{*}(x)=\frac{1}{2}x_{1}^{2}+x_{2}^{2}$ with optimal control law given by $u^{*}(x)=-(\cos(2x_{1})+2)x_{2}$. Using this information, we choose $\phi_{c}(x)=(x_{1}^{2},x_{1}x_{2},x_{2}^{2})$, and we implement the prescribed hybrid momentum-based dynamics in (17) for the update of the critic neural network, and the update dynamics for the actor described in (27). We obtain the results shown in Figure 3(b) with $x(0,0)=(-10,10)$, $\theta_{c}(0,0)=(1,1,1)$ and $\theta_{u}\in[0,1]^{3}$. We compare the results with the case in which the critic neural-network is updated with the gradient-descent dynamics of [17], and where the sufficiently rich data is a set of $16$ data points obtained by sampling the dynamics (29) in a grid around the origin of size $4\times 4$. In our simulations we use $T_{0}=0.1,T=5.5$ for the momentum-based dynamics in (17). These particular values are obtained by using the level of richness $\underline{\lambda}$ of the data-set, and the inequalities in (18) in order to ensure compliance with Assumption 3. For both reinforcement learning dynamics we use $k_{c}=1,k_{u}=1,\rho_{d}=1$ and $\rho_{i}=1$. As shown in the figure both update dynamics are able to converge to $\left\{\theta_{c}^{*}\right\}$, with $\theta_{c}^{*}=(1/2,0,1)$ describing the optimal value function $V^{*}$. However, the hybrid-based dynamics are able to significantly improve the transient performance of the learning mechanism.³³3The code used to implement this simulation can be found in the following repository: https://github.com/deot95/Accelerated-Continuous-Time-Approximate-Dynamic-Programming-through-Data-Assisted-Hybrid-Control

In this paper, we introduced the first stability guarantees for deterministic continuous-time actor-critic reinforcement learning with accelerated training of neural network structures. To do so, we studied a novel hybrid momentum-based estimation dynamical system for the critic NN, which estimates, in real time, the optimal value function. Our stability analysis leveraged the existence of rich recorded data taken from a finite number of samples along optimal trajectories and inputs of the system. We showed that this finite sequence of samples can be used to train the controller to achieve online optimal performance with fast transient performance. Closed-loop stability was established using tools from hybrid dynamical systems theory. Potential extensions include the study of similar accelerated training dynamics for the actor subsystem, as well as considering reinforcement learning problems in hybrid plants.