On the Problem of Reformulating Systems with Uncertain Dynamics as a Stochastic Differential Equation

Consider a nonlinear system whose state at time $t\geq 0$ is $x(t)\in\mathbb{R}^{n}$, control inputs are $u(t)\in\mathbb{R}^{m}$, such that

where $T>0$, $x(0)=x_{0}\in\mathbb{R}^{n}$ almost surely, i.e., $x(0)$ is known exactly, and $f:\mathbb{R}^{n}\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{n}$ is twice continuously differentiable.

In many applications, $f$ is not known exactly, and prior knowledge is necessary to safely control (1). One such approach consists of assuming that $f$ lies in a known space of functions $\mathcal{H}$, and to impose a prior distribution in this space $\mathbb{P}(\mathcal{H})$. For instance, by assuming that $f$ lies in a bounded reproducing kernel Hilbert space (RKHS), a common approach consists of imposing a Gaussian process prior on the uncertain dynamics $f\sim\mathcal{GP}(m,k)$, where $m:\mathbb{R}^{n+m}\,{\rightarrow}\,\mathbb{R}^{n}$ is the mean function, and $k:\mathbb{R}^{n+m}\times\mathbb{R}^{n+m}\,{\rightarrow}\,\mathbb{R}^{n\times n}$ is a symmetric positive definite covariance kernel function which uniquely defines $\mathcal{H}$ [1, 2]. An alternative consists of assuming that $f(x,u)=\phi(x,u)\theta$, where $\phi:\mathbb{R}^{n+m}\rightarrow\mathbb{R}^{n\times p}$ are known basis functions, and $\theta\in\mathbb{R}^{p}$ are unknown parameters. With this approach, one typically sets a prior distribution on $\theta$, e.g., a Gaussian $\theta\sim\mathcal{N}(\bar{\theta},\Sigma_{\theta})$, and updates this belief as additional data about the system is gathered.

Given these model assumptions and prior knowledge about $f$, safe learning-based control algorithms often consist of designing a control law $u$ satisfying different specifications, e.g., minimizing fuel consumption $\|u\|$, or satisfying constraints $x(t)\in\mathcal{X}\ \forall t\in[0,T]$, with $\mathcal{X}$ a set encoding safety and physical constraints.

Next, we describe an issue with the mathematical formulation of the safe learning-based control problem that has appeared in recent research [3, 4, 5, 6], slightly changing notations and assuming a finite-dimensional combination of features for clarity of exposition but without loss of generality. As in [5], consider the problem of safely controlling the uncertain system

where $t\in[0,T]$, $x(0)=x_{0}$, $\bar{\theta}\in\mathbb{R}^{p}$, and $\Sigma_{\theta}\in\mathbb{R}^{p\times p}$ is positive definite, with $\Sigma_{\theta}=B_{\theta}B_{\theta}^{\top}$ its Cholesky decomposition. Note that this formulation can be equivalently expressed in function space, where $f$ is drawn from a Gaussian process with mean function $m(x,u)=\phi(x,u)\bar{\theta}$ and kernel $k(x,u,x^{\prime},u^{\prime})=\phi(x,u)\Sigma^{-1}_{\theta}\phi(x^{\prime},u^{\prime})^{\top}$. Our representation can be seen as a weight-space treatment of the GP approaches used in [3] and [4].¹¹1For a squared exponential kernel, one needs $p\rightarrow\infty$ for this equivalence, see [1] for more details. Nevertheless, the issue discussed in this paper remains valid for such kernels.

These works then proceed by introducing the Brownian motion $W(t)$, making the change of variable $\theta\textrm{d}t=\bar{\theta}\textrm{d}t+B_{\theta}\textrm{d}W(t)$, and reformulating (2) as a stochastic differential equation (SDE)

with $t\in[0,T]$ and $x(0)=x_{0}$. Unfortunately, (3) is not equivalent to (2). Indeed, the solution to (3) is a Markov process, whereas the solution to (2) is not. Intuitively, the increments of the Brownian motion $W(t)$ in (3) are independent, whereas in (2), $\theta$ is randomized only once, and the uncertainty in its realization is propagated along the entire trajectory. By making this change of variables for $\theta\textrm{d}t$, the temporal correlation between the trajectory $x(t)$ and the uncertain parameters $\theta$ is neglected. In the next section, we provide a few examples to illustrate the distinction between these two cases. The first demonstrates the heart of the issue on a simple autonomous system, whereas the second shows that analyzing the SDE reformulation (3) is insufficient to deduce the closed-loop stability of the system in (2).

As (2) and (3) are generally not equivalent, the stability and constraint satisfaction guarantees derived for the SDE (3) in recent research [3, 4, 5, 6] do not necessarily hold for the system (2).²²2For instance, the generator of the Markov process solving the SDE (3) (see [9]) is used to prove stability in [3, 4, 5, 6]. Unfortunately, (2) does not yield a Markov process. Thus, it would be necessary to adapt the concept of generator to solutions of (2) before concluding the stability of the original system. This could yield undesired behaviors when applying such algorithms, developed on an SDE formulation of dynamics (3), to safety-critical systems where uncertainty is better modeled by (2), i.e., dynamical systems with uncertain parameters that are not changing over time.

Although (3) is not equivalent to (2), it is interesting to ask whether (3) is a conservative reformulation of (2) for the purpose of safe control. For instance, given a safe set $\mathcal{X}\subset\mathbb{R}^{n}$, if one opts to encode safety constraints through joint chance constraints of the form

where $\delta\in(0,1)$ is a tolerable probability of failure, there may be settings where a controller $u$ satisfying (10) for the SDE (3) may provably satisfy (10) for the uncertain model (2). Indeed, as solutions of (3) may have unbounded total variation (as in the example presented above), which is not the case for solutions of (2), we make the conjecture that for long horizons $T$, a standard proportional-derivative-integral (PID) controller may better stabilize (2) than (3), and that similar properties hold for adaptive controllers.

Alternatively, approaches which bound the model error through the Bayesian posterior predictive variance [10] or confidence sets holding jointly over time [11, 12] exist. Given these probabilistic bounds, a policy can be synthesized yielding constraints satisfaction guarantees.