Towards Global Optimal Control via Koopman Lifts

This paper considers infinite horizon optimal control problems of the form

Here, $x:\mathbb{R}\to\mathbb{R}^{n_{x}}$ denotes the state trajectory and $u:\mathbb{R}\to\mathbb{R}^{n_{u}}$ denotes the control input. The initial value $x_{0}\in\mathbb{R}^{n_{x}}$ is assumed to be given. Throughout this paper the following assumptions are imposed.

The constant $l^{\star}\in\mathbb{R}$ in (1) denotes the optimal average cost,

assuming that this limit exists. Notice that $l^{\star}$ does not need to be known explicitly in order to solve (1), as adding constant offsets to the objective does not affect the solution of an optimization problem. In this paper, the constant $l^{\star}$ is merely introduced for mathematical reasons, such that $V(x_{0})$ remains finite and well-defined on infinite horizons under mild regularity assumptions that shall be introduced further below.

In practical instances, one is often interested in whether optimal solutions for the state trajectory of (1) converge to an optimal steady-state or, in more generality, to an optimal periodic limit orbit. These optimal steady states or more general periodic orbits are defined as follows.

In order to prepare the following analysis, it is helpful to introduce the shorthands

assuming that $(x_{\mathrm{p}},u_{\mathrm{p}})$ is an optimal periodic orbit. Here, $f_{x}$ and $f_{u}$ denote the partial derivatives of $f$ with respect to $x$ and $u$ and an analogous notation is then also used for the mixed second order derivatives of $l$. The above matrix-valued functions can be used to construct sufficient conditions under which an optimal periodic orbit is locally stabilizable. Here, one relies on the theory of periodic Riccati differential equations [5] of the form

It is well-known [5] that if Assumptions 1 and 2 are satisfied and if $R$ has full rank, then the existence of a periodic and positive definite function $P$ satisfying (3) is sufficient to ensure that the solution $x^{\star}$ of (1) converges to the optimal periodic orbit $x_{\mathrm{p}}$ as long as $\,\mathrm{dist}\left(x_{0},x_{\mathrm{p}}\right)\,$ is sufficiently small—that is, if $x_{0}$ is in a small neighborhood of the optimal orbit $x_{\mathrm{p}}$. However, this statement is of a rather local nature. This means that, if we wish to understand the global behavior of system (1), an analysis of the periodic Riccati equation is not sufficient. The following sections focus on analyzing global solutions of (1), under the assumption that $\,\mathrm{dist}\left(x_{0},x_{\mathrm{p}}\right)\,$ is not necessarily small.

Pontryagin’s maximum principle [28] can be used to derive necessary conditions for the minimizers of (1). The first order variational optimality condition is summarized as follows. Let $H:\mathbb{R}^{n_{x}}\times\mathbb{R}^{n_{u}}\times\mathbb{R}^{n_{x}}\to\mathbb{R}$ be the Hamiltonian function of (1),

If Assumptions 1 and 2 hold, $H$ is, by construction, twice continuously differentiable in all variables. Next, let

denote the associated parametric minimizer of $H$. At this point, we introduce the following regularity assumption.

Notice that for the practically relevant special case that $f$ is affine in $u$ and $l$ strongly convex in $u$, Assumption (4) always holds whenever Assumptions 1 and 2 are satisfied.

Next, if Assumptions 1, 2, and 4 are satisfied, it is well-known [28] that any minimizer $(x,u)$ of (1) necessarily satisfies Pontryagin’s differential equation

for a co-state function $\lambda\in\mathbb{W}_{1}^{n_{x}}$. In the following, we summarize this differential equation in the form

where $y=[x^{\intercal},\lambda^{\intercal}]^{\intercal}$ denotes the stacked state and $F$ a stacked version of the right-hand side functions of (6) and (7).

Besides Pontryagin’s differential equation (8), optimal solutions of (1) satisfy necessary boundary conditions. First, of course, the initial condition $x(0)=x_{0}$ must hold. Moreover, the co-state $\lambda$ satisfies a necessary limit condition, which can be summarized as follows.

Notice that Proposition 5 is—at least in very similar versions—well-known in the literature [28, 21]. However, as this result is important for understanding the developments in this paper, the following proof briefly recalls the main argument, why this proposition holds.

Proof. Since (3) admits a positive definite solution, the value function $V$ in (1) is well-defined and differentiable in a neighborhood of the optimal orbit $x^{\star}$. Thus, (1) can be replaced by an equivalent finite-horizon optimal control problem as long as $V$ is used as a terminal cost. Now, Pontryagin’s principle for optimal control problems with Mayer terms [28] yields the boundary condition

for any horizon length $t>0$. Clearly, since $x^{\star}$ converges to the optimal periodic orbit $x_{\mathrm{p}}$, $\lambda^{\star}$ must converge to the associated dual limit orbit $\lambda_{\mathrm{p}}(t)=\nabla V(x_{\mathrm{p}}(t))$. ∎

In the next sections we will see that Proposition 5 implies that the states of Pontryagin’s differential equation for stabilizable systems must evolve along certain stable manifolds. As it shall demonstrated, these manifolds can be characterized using a Koopman mode analysis [26].

Let $\Gamma_{t}:\mathbb{R}^{2n_{x}}\to\mathbb{R}^{2n_{x}}$ denote the flow of Pontryagin’s differential equations such that

for all $z\in\mathbb{R}^{2n_{x}}$. If Assumptions 1, 2, and 4 hold, then $\Gamma_{t}$ is a well-defined, continuously differentiable function. In the mathematical literature [2] it is well-known that $\Gamma_{t}$ is a so-called symplectic flow. In order to reveal this symplectic structure it is helpful to introduce the block matrix

Now, the structural properties of $\Gamma_{t}$ can be summarized as follows.

Proof. Let us first recall that the right-hand side $F$ of Pontryagin’s differential equation is given by (6) and (7), which can also be summarized as

This implies that the derivative of $F$ with respect $y$ can be written in the form

Here, we have used Assumptions 1 and 2 to ensure that the second derivatives of $H$ with respect to $x$ and $u$ exist. Moreover, we have used Assumption 4, which implies that $H_{uu}$ is invertible and, as a consequence, that the implicit function theorem holds. In particular, we have

In this form, it becomes clear that the matrix $\Omega F_{y}$ is symmetric and we arrive at the intermediate result

In order to proceed, we write the first order variational differential equation for $\Gamma_{t}$ in the form

Now, the main idea of this proof is to show that the function

vanishes, $\Delta(t)=0$. Here, we first have $\Delta(0)=0$ by construction, since $\left[\frac{\partial}{\partial z}\Gamma_{0}\right]=I$. Moreover, the derivative of $\Delta$ with respect to time is given by

Thus, in summary, we must have $\Delta(t)=0$ for all $t\in\mathbb{R}$, which yields the statement of this lemma. ∎

The following corollary summarizes two immediate consequences of Lemma 6 which are both equivalent to stating that $\Gamma_{t}$ is an incompressible flow.

Proof. By taking the determinant on both sides of (9) and using that $\mathrm{det}(\Omega)=1$, we find that

Since $\frac{\partial}{\partial z}\Gamma_{0}=I$ and since $\frac{\partial}{\partial z}\Gamma_{t}$ is continuous, this is only possible if (21) holds. Next, by taking the logarithm on both sides of (21) and differentiating with respect to time, we find¹¹1Alternatively, the equation $\mathrm{Tr}(F_{y})=0$ can also directly be found by substituting the explicit expression (17) from the proof of Lemma 6.

This completes the proof of the corollary. ∎

The developments from the previous section can be used to analyze the structural properties of the Pontryagin-Koopman operator, which are defined to be the Koopman operator that is associated with the flow $\Gamma_{t}$ of Pontryagin’s differential equation, or, more formally:

It is well known [19, 26] that $U_{t}$ is a linear operator satisfying $U_{t_{1}+t_{2}}=U_{t_{1}}U_{t_{2}}$ for all $t_{1},t_{2}\in\mathbb{R}$. Moreover, $U_{t}$ satisfies $U_{t}^{-1}=U_{-t}$, which follows by substituting $t=t_{1}=-t_{2}$ in the previous equation and using that $\Gamma_{0}=\mathrm{id}$.

In order to introduce a notion of symplecticity in the space of observables of the Pontryagin-Koopman operator, we introduce the bilinear form

which is defined as

for all $\varphi,\Phi\in\mathbb{W}_{1,2}^{2n_{x}}$. Because we have $\Omega^{\intercal}=-\Omega$, the bilinear form $\omega$ is skew-symmetric,

and $\left(\mathbb{W}_{1,2}^{2n_{x}},\omega\right)$ is a symplectic space [2].

Proof. Assumptions 1, 2, and 4 imply that the function $\Gamma_{t}$ and its inverse $\Gamma_{t}^{-1}=\Gamma_{-t}$ are both continuously differentiable. This implies in particular that the equivalence

holds. Next, the definition of the operator $U_{t}$ and the chain rule for differentiation imply that

for all $\Phi\in\mathbb{W}_{1,2}^{2n_{x}}$. Furthermore, because Assumptions 1, 2, and 4 hold, it follows from Lemma 6 that

Let us multiply this equation with $\left[\frac{\partial}{\partial z}\Gamma_{t}\right]\Omega$ from the left and substitute $\Omega^{2}=-I$. This yields

Next, we multiply the latter equation with $\left[\frac{\partial}{\partial z}\Gamma_{t}\right]^{-1}\Omega$ from the right and use the relation $\Omega^{2}=-I$ once more, in order to arrive at the equation

Thus, by substituting the previous equations, we get

In order to further transform this integral, we need to introduce the change of variables $z^{\prime}=\Gamma_{t}(z)$. Because Corollary 7 ensures that

this change of variables is volume preserving. Thus, this implies that

for all $\varphi,\Phi\in\mathbb{W}_{1,2}^{2n_{x}}$. Thus, $U_{t}$ is a symplectic operator. ∎

In order to prepare the analysis of the properties of the spectrum of symplectic Koopman operators, it is helpful to introduce a notion of duality. However, instead of defining duality in a Hilbert space, we propose to introduce the following notion of symplectic duality.

Proof. For any function $\Phi\in\mathbb{W}_{2,2}^{2n_{x}}$, we have

which follows by differentiating (24) on both sides. Next, we know from Theorem 9 that $U_{t}$ is a symplectic operator. Thus, we can differentiate the equation

on both sides with respect to $t$, which yields

Now, if $\varphi,\Phi\in\mathbb{W}_{2,2}^{2n_{x}}$, we can expand the derivative on the right as

and, after changing variables, $z\leftarrow\Gamma_{t}(z)$, recalling that $\mathrm{det}\left(\frac{\partial}{\partial z}\Gamma_{t}\right)=1$, and resorting terms,

Moreover, we recall that the equation

also holds (see Equation (18)). Thus, in summary, we have

Because the latter equation holds for all $\varphi,\Phi\in\mathbb{W}_{2,2}^{2n_{x}}$, we find that $L^{\star}=-L$ is the symplectic adjoint of $L$, as claimed by the statement of this lemma. ∎

As already mentioned in the introduction, the spectrum of general Koopman operators has been analyzed by many authors; for example in [1, 6, 26]. In the context of this paper, we call a weakly differentiable function $\Psi$ an eigenfunction of the differential Pontryagin-Koopman operator $L$, if the Lebesgue measure of the set $\,\mathrm{supp}(\Psi)\,$ is either unbounded, or, if it exists, is not equal to $0$, and

for a potentially complex eigenvalue $\kappa\in\mathbb{C}$. The fact that the symplectic adjoint of the operator $L$ is given by $L^{\star}=-L$ has important consequences on its spectrum, which can be summarized as follows.

Proof. Because the functions $\Psi_{1},\Psi_{2}\in\mathbb{W}_{2,2}^{2n_{x}}$ are eigenfunctions of $L$, they satisfy

Thus, since $\omega$ is a bilinear form, we have

and, after resorting terms,

Thus, if $\omega(\Psi_{1},\Psi_{2})\neq 0$, we must have $\kappa_{1}+\kappa_{2}=0$. This is equivalent to the statement of the theorem. ∎

Let $\sigma(L)\subseteq\mathbb{C}$ denote the spectrum of $L$; that is, the set of eigenvalues of the linear operator $L$. In the following, we use the notation $\Psi_{\kappa}\in\mathbb{W}_{2,2}^{2n_{x}}$ to denote an eigenfunction that is associated with an eigenvalue $\kappa\in\sigma(L)$. Moreover, for a function $q:\mathbb{R}^{2n_{x}}\to\mathbb{R}^{2n_{x}}$, we use the shorthand notation

to denote the matrix that is obtained by evaluating the skew symmetric bilinear form $\omega$ for all possible combinations of the components of $q$. We call $\,q\,$ a skew-orthogonal function in the symplectic space $(\mathbb{W}_{1,2}^{2n_{x}},\omega)$, if it satisfies $\omega(q,q)=I$. Notice that the construction of such skew orthogonal functions is straightforward by using the standard skew-symmetric variant of the Gram-Schmidt algorithm [2].

Notice that the above definition uses the “control engineering notation” for generalized functions, which means that we use $a$ as if it was a standard function, although this notation suppresses the distributional nature of $a$. Thus, in mathematical terms, this notation has to be translated as “$a$ represents a distribution of order $0$; that is, a linear operator on $\sigma(L)$ that is Lipschitz continuous with respect to the $L_{\infty}$-norm”.²²2Notice that, for example, $a$ could be a Dirac distribution, which is not a function in the traditional sense but, by construction, a Lipschitz continuous linear operator. The following statement is a consequence of Theorem 13.

Proof. By substituting (31) in the equation $\omega(q,q)=I$, we find that the equation

holds. Let us have a closer look at the terms in (32). Clearly, the unit matrix on the left has full rank. But, on the other side, we have an integral over the rank $1$ matrices $a(\kappa^{\prime})a(\kappa)^{\intercal}$. This integral term can only have full rank, if there are at least $2n_{x}$ pairs of eigenvalues $(\kappa^{\prime},\kappa)\in\sigma(L)\times\sigma(L)$ for which

But now Theorem 13 implies that all these pairs must be such that $\kappa^{\prime}=-\kappa$ and, after sorting all eigenvalues with respect to their real-part, we find that there must be at least $n_{x}$ eigenvalues with non-negative real part and $n_{x}$ associated mirrored eigenvalues with non-positive real-part, as claimed by the statement of this corollary. ∎