Model Predictive Control of Nonlinear Dynamics
Using Online Adaptive Koopman Operators

Model Predictive Control (MPC) has emerged as a powerful framework for solving complex control challenges across diverse engineering applications, from process industries to robotics and autonomous systems. At the heart of MPC lies its ability to optimize control decisions based on predicted future system behavior. This predictive capability critically depends on the accuracy of the underlying control model. Yet for complex systems, the mathematical models used for control often deviate from the true system dynamics. To address this limitation, researchers have developed adaptive and data-driven MPC approaches that explicitly account for modeling uncertainties, leading to more robust control performance ([Adetola et al.(2009)Adetola, DeHaan, and Guay, Klenske et al.(2016)Klenske, Zeilinger, Schölkopf, and Hennig, Ostafew et al.(2016)Ostafew, Schoellig, and Barfoot, Ostafew et al.(2014)Ostafew, Schoellig, and Barfoot]). For instance, the online adaptation procedure can be formulated such that the residual dynamics, which is the difference between the control model and the true dynamics, is learned online by fitting non-parametric or parametric function approximators such as Gaussian processes ([Hewing et al.(2020)Hewing, Kabzan, and Zeilinger]) and random Fourier features ([Zhou and Tzoumas(2024)]).

Koopman operator theory has gained popularity in recent years for its utility as an alternative approach to describing nonlinear dynamics ([Brunton and Kutz(2019), Mauroy et al.(2020)Mauroy, Mezić, and Susuki, Pan and Duraisamy(2024)]). Specifically, nonlinear dynamics can be represented as linear ones in the embedded space of feature maps, and several computational methods have been developed to obtain finite-dimensional approximations of Koopman operators, which are then utilized for prediction and control. The Koopman operator framework offers a potentially powerful advantage: it enables the transformation of nonlinear dynamical systems into linear representations through data-driven methods. This linearization affords access to the extensive theoretical machinery and computational tools developed for linear control systems, while preserving the ability to handle underlying nonlinear dynamics ([Korda and Mezić(2018), Arbabi et al.(2018)Arbabi, Korda, and Mezić]). While Koopman operator-based models can be learned by linear regression types of methods such as Extended Dynamic Mode Decomposition (EDMD) ([Williams et al.(2015)Williams, Kevrekidis, and Rowley]), utilization of neural networks to learn (in contrast to prescribing from a dictionary) feature spaces has been shown to be promising for complex nonlinear dynamics and incorporated into data-driven Koopman operator-based control ([Han et al.(2022)Han, Euler-Rolle, and Katzschmann, Xiao et al.(2023)Xiao, Zhang, Xu, Liu, and Liu, Uchida and Duraisamy(2023), Pan and Duraisamy(2020)]).

While the use of Koopman operators enables expressive and flexible modeling for data-driven control, modeling errors may arise due to several possible factors, e.g., lack of data quantity/quality, inadequate model structures, etc. Whereas there are several methods to tackle this issue from the control theoretic viewpoints ([Zhang et al.(2022)Zhang, Pan, Scattolini, Yu, and Xu, Son et al.(2020)Son, Narasingam, and Sang-Il Kwon, Uchida et al.(2021)Uchida, Yamashita, and Asama]), most of them are based on EDMD-type models. Also, while [Han et al.(2022)Han, Euler-Rolle, and Katzschmann] develops a model uncertainty-aware Koopman MPC with the use of probabilistic neural networks and [Uchida and Duraisamy(2023)] proposes a model refinement technique to handle the modeling error of neural network-based Koopman models in the context of control, there is a relative scarcity of exploration of online update methods of such control models.

In this paper, we propose an online adaptation method for Koopman operator-based MPC to avoid performance deterioration due to the modeling error. Whereas [Zhang et al.(2019)Zhang, Rowley, Deem, and Cattafesta, Hemati et al.(2014)Hemati, Williams, and Rowley, Sinha et al.(2020)Sinha, Nandanoori, and Yeung, Alfatlawi and Srivastava(2020)] explore the idea of online adaptation in the context of Koopman operator-based computational modeling, they do not assume controller design problems. [Deem et al.(2020)Deem, Cattafesta, Hemati, Zhang, Rowley, and Mittal] presents an adaptive control of flow separation based on the online dynamic mode decomposition, which is an EDMD-type model without nonlinear feature maps. [Singh et al.(2024)Singh, Sah, and Keshavan] also develops an adaptive Koopman MPC, in which the linear operator $[A\ B]$ in (6) is updated to $[A+\Delta A\ B+\Delta B]$ s.t. $\Delta A$ and $\Delta B$ are parameterized by additional neural networks, which are trained online w.r.t. an adaptation loss function. On the other hand, the proposed method provides a more tractable yet effective online adaptation procedure. We only use a single loss function to train neural networks throughout offline and online model learning, which results in fewer hyperparameters and less complexity of the model learning. At the same time, the proposed method enables flexible online model learning since it allows an adaptation of the feature maps in addition to the linear operator $[A\ B]$. Considering that model learning involving deep neural networks results in high-dimensional, non-convex optimizations and typically becomes unstable, we adopt the soft update of target networks ([Lillicrap et al.(2015)Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver, and Wierstra]), a common technique to stabilize learning used in Reinforcement Learning (RL). Also, we provide a discussion on which parameters should be prioritized in the online update procedure based on a specific formulation of linear embedding models in [Iacob et al.(2024)Iacob, Tóth, and Schoukens] to further improve the stability and robustness of online learning. The overview of the proposed method is shown in Fig. 1.

In Section 2, MPC with Koopman operator-based linear embedding models is presented. This is followed by the formulation of offline model learning based on the Koopman operator framework in Section 3. The proposed method is formalized in Section 4 and numerical evaluations are provided in Section 5.

The model parameters of the linear embedding model (6),(7) are the feature maps $g:\mathcal{X}\rightarrow\mathbb{R}^{N}$ and matrices $A,B,$ and $C$. In this section, we formulate a problem of learning these parameters in an offline manner, where it is assumed that only data samples generated by a nominal model $f_{\text{known}}$ are available but we do not have access to data from the true system (3).

While the Koopman MPC with a linear embedding model learned by Problem 3 is expected to perform well if the nominal model $f_{\text{known}}$ is sufficiently close to the true dynamics, updating the control model online will further improve the performance or foster robustness in case there exists a large discrepancy between the nominal model and the true dynamics.

Let $\Theta:=\{A,B,C,\theta\}$ be the model parameters of a linear embedding model (6), (7). Also, we use a notation $\text{LinearMPCSolver}(g(x_{k});\Theta)$ to denote a solution of the Koopman MPC (Problem 1) given a current state $x_{k}$. Assuming that a new data sample $x_{k+1}$ is available at time $k+1$ s.t. $x_{k+1}=f(x_{k},u_{k})$ where $u_{k}=\text{LinearMPCSolver}(g(x_{k});\Theta)$, we add $x_{k+1}$ to a relay buffer, from which a data batch is sampled at each time step to update $\Theta$ in an online manner. Similar to Problem 3, the model parameters are updated online by minimizing (19).

It is, however, known that using neural networks as function approximators often makes learning processes unstable due to its non-convexity and high-dimensionality. To this end, the proposed method adopts the soft update of target networks ([Lillicrap et al.(2015)Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver, and Wierstra]). A target network is paired with a main network and its parameters are slowly updated towards that of the main network, by which abrupt changes in the outputs will be avoided to enhance the stability of the learning process.

In the proposed method, we initialize both the main and the target networks (two linear embedding models with the model parameters labeled $\Theta$ and $\Theta_{\text{target}}$, respectively) by the offline model from Problem 3. The target network is then updated at each time step by an interpolation $\Theta_{\text{target}}\leftarrow\tau\Theta+(1-\tau)\Theta_{\text{target}}$, where $\tau$ is a hyperparameter to adjust the smoothness of the update. The actual control input is computed by the Koopman MPC solver with the slowly changing model parameters $\Theta_{\text{target}}$ so that we can suppress undesirable behavior of the closed-loop dynamics due to large fluctuations of the control model. The proposed Adaptive Koopman MPC with Soft Update is summarized in Algorithm 4.

[t] Adaptive Koopman MPC with Soft Update Require: Prior Model or simulator $f_{\text{known}}$ of known dynamics  Step 1: Train an offline model  Simulate $f_{\text{known}}$ and collect data:
$\mathcal{D}_{\text{known}}=\{(x,u,y)\mid y=f_{\text{known}}(x,u),u=\text{MPCSolver}(x;f_{\text{known}})\}$  Solve Problem 3 to train a linear embedding model:
$x^{\text{pred}}_{k+1}=C_{\text{prior}}(A_{\text{prior}}g_{\text{prior}}(x_{k};\theta_{\text{prior}})+B_{\text{prior}}u_{k})$ on the data set $\mathcal{D}_{\text{known}}$  Step 2: Adaptive Data-driven MPC  $(A,B,C,\theta)\leftarrow(A_{\text{prior}},B_{\text{prior}},C_{\text{prior}},\theta_{\text{prior}})$    \tcpMain network parameters $(A_{\text{target}},B_{\text{target}},C_{\text{target}},\theta_{\text{target}})\leftarrow(A_{\text{prior}},B_{\text{prior}},C_{\text{prior}},\theta_{\text{prior}})$ \tcpTarget network params $(\Theta,\Theta_{\text{target}})\leftarrow(\{A,B,C,\theta\},\{A_{\text{target}},B_{\text{target}},C_{\text{target}},\theta_{\text{target}}\})$  $\mathcal{B}\leftarrow\emptyset$    \tcpReplay buffer $x_{0}\leftarrow$Initial condition  \For$k=0,1,2,\cdots$ $u_{k}\leftarrow\text{LinearMPCSolver}(g(x_{k});\Theta_{\text{target}})$    \tcpKoopman MPC $x_{k+1}\leftarrow f(x_{k},u_{k})$    \tcpNext state from the true environment $\mathcal{B}\leftarrow\mathcal{B}\cup\{(x_{k},u_{k},x_{k+1})\}$    \tcpAdd new data to replay buffer $\mathcal{D}_{\text{online}}\leftarrow\text{BatchSample}(\mathcal{B})$  $\Theta\leftarrow\text{GradientDescent}(\mathcal{D}_{\text{online}})$  $\Theta_{\text{target}}\leftarrow\tau\Theta+(1-\tau)\Theta_{\text{target}}$    \tcpSoft update

As a numerical example, we consider a cartpole system with a cart mass $m_{c}$, a pole mass $m_{p}$, and a pole length $2l$, which is described by the following ordinary differential equations (ODE) ([Yuan et al.(2022)Yuan, Hall, Zhou, Brunke, Greeff, Panerati, and Schoellig]):

where $x(t)$, $\theta(t)$, $F(t)$, and $g$ are the cart position, the pole angle, the force to the cart, and the acceleration due to gravity, respectively. The state of the system is supposed to be $x_{k}:=[x(k\Delta t)\ \dot{x}(k\Delta t)\ \theta(k\Delta t)\ \dot{\theta}(k\Delta t)]^{\mathsf{T}}$ s.t. $x(t)$, $\dot{x}(t)$, $\theta(t)$, and $\dot{\theta}(t)$ are sampled with a sampling period $\Delta t=1/15$ [s]. Also, $F(t)$ is given by the control input $u_{k}$ determined by MPC s.t. $F(t):=u_{k}$ for $k\Delta t\leq t\leq(k+1)\Delta t$. The reference signal is set to $x^{\text{ref}}_{k}:=0.$

It is assumed that we are given the ODE (22), (23) with $(m_{c},m_{p},l)=(0.75,0.075,0.375)$ as a nominal model, whereas the true dynamics is governed by the same equations with different parameter values: $(m_{c},m_{p},l)=(1,\ 0.1,\ 0.5)$. In Step 1 of Algorithm 4, we collect data from the nominal model that consists of 500 trajectories, each of which has a length of 60 time steps, and train a linear embedding model with the feature maps given by $g(x_{k};\theta):=[x_{k}^{\mathsf{T}}\ g_{5}(x_{k};\theta)\ g_{6}(x_{k};\theta)]^{\mathsf{T}}$, where $[g_{5}(\cdot;\theta)\ g_{6}(\cdot;\theta)]$ is a feed-forward neural network with three hidden layers, each of which has 64 neurons. Note that including the state $x_{k}$ itself in the feature maps eliminates the decoding error and ensures the condition (9) by an analytical decoder $C:=[I_{n}\ 0]$. Therefore, we set $\lambda_{2}=0$ in (19). Following the discussion in Section 4.1, we fix $A$ to the learning result of Problem 3 and update $g$ and $B$ online with $\tau:=0.05$.

For comparison, we also test three controllers in addition to the proposed method. As a baseline of non-adaptive MPC, we consider the nominal MPC, which is described in Section 2. As data-driven adaptive MPC, the Gaussian Process MPC (GP-MPC) ([Hewing et al.(2020)Hewing, Kabzan, and Zeilinger]) and the MPC with random Fourier Features (RFF-MPC) ([Zhou and Tzoumas(2024)]) are considered. GP-MPC and RFF-MPC learn the residual dynamics $r(x_{k},u_{k})$ in (3) with sparse Gaussian processes and random Fourier features in online manners, respectively. For all the MPC methods, we set $H:=20$, $Q_{\text{state}}:=\text{diag}(5,0.1,5,0.1)$, and $R:=0.1$. For each controller, a simulation with the same setting is performed 10 times with randomly chosen initial conditions. The simulations are implemented with safe control gym ([Yuan et al.(2022)Yuan, Hall, Zhou, Brunke, Greeff, Panerati, and Schoellig]) on a system with an AMD Ryzen 7 7700X 8-core processor and 32 GB of memory.

The results are shown in Fig. 2, where a sample trajectory and the average error defined by $\frac{1}{10}\sum_{i=1}^{10}\|x_{k,i}-x^{\text{ref}}_{k}\|_{2}$ ($x_{k,i}$: state of the $i$-th trajectory) for each method are on the left and right panels, respectively. Since nominal MPC does not take the effect of the residual dynamics into account, it does not track the reference within the simulation window of six seconds. On the other hand, all the adaptive MPC methods successfully stabilize the states by the end of the simulations. Compared to GP- and RFF-MPC, the proposed adaptive Koopman MPC outperforms in terms of the average errors. Table 1 shows the average execution times of the simulation for individual controllers. Since GP- and RFF-MPC are adaptive methods based on nonlinear MPC, they take longer than the nominal MPC. On the other hand, the proposed adaptive Koopman MPC achieves an even shorter execution time than the nominal MPC thanks to the convexity of its formulation.

Finally, a sensitivity analysis w.r.t. the residual dynamics is performed, where we consider various extents of the discrepancy between the nominal model and the true dynamics. Figure 3 shows the results, where the true dynamics parameters are varied by 10 to 30 % w.r.t. the nominal model and the same experiment is performed for each case. Whereas the proposed method results in higher average errors after $t=1.5$ [s] for relatively small extents of the residual dynamics (Figs. 3(a,b)), it outperforms the other controllers at the beginning of the simulation in all cases. Also, the proposed method shows the most robust performance across the given range of residual dynamics.

This work introduced an adaptive Model Predictive Control (MPC) framework that leverages Koopman operators for nonlinear dynamics. While MPC is widely adopted across diverse control applications, its performance can deteriorate when the control model fails to accurately capture the true system dynamics. To address this challenge, we developed an approach that combines the Koopman operator framework with a linear embedding model, enabling online parameter updates to compensate for model discrepancies. The offline training learns the features and operator matrices jointly, allowing for greater expressivity. By maintaining linearity in the control model while accommodating nonlinear dynamics, our method achieves both computational efficiency and adaptability. Online learning may become unstable if the model is parameterized by complex model forms such as neural networks, which result in a high-dimensional non-convex optimization. The proposed method uses the soft update of target networks so that abrupt changes in the model will be avoided and we can stabilize online updates. Also, we provide a discussion on which model parameters to prioritize for the online update based on a specific system description of linear embedding models in [Iacob et al.(2024)Iacob, Tóth, and Schoukens]. Experimental validation on a cartpole system demonstrates that our method achieves favorable control performance while requiring significantly lower computational resources compared to existing adaptive MPC approaches. Furthermore, the results reveal enhanced robustness to residual dynamics. While our initial results demonstrate promise and basic viability, expanding the validation to more complex industrial and robotic systems represents an important next step in establishing this framework’s broad applicability.

This work has benefited from several discussions and code-sharing with Hongyu Zhou and Vasileios Tzoumas, for which we are grateful. This work was funded by AFOSR grant FA9550-17-1-0195.