A Data-driven Hierarchical Control Structure for Systems with Uncertainty

As robots increasingly venture outside of the lab, the effect of uncertainty within the system and/or at the robot-environment interactions becomes more pronounced. Motivating examples include the influence of surface effects for Unmanned Aerial Vehicles (UAVs), unknown flow dynamics for marine robots, and inherently uncertain leg-ground contacts for legged robots.

Central to exploiting system and robot-environment uncertainty is the ability to quickly identify deviations from nominal behaviors based on data collected during robot deployment. Pre-deployment model identification and calibration tools (e.g., [1, 2, 3]) can help improve model-based control. However, even though such models can be obtained with high precision, the presence of uncertain, time-varying disturbances during deployment may turn the utilized model invalid to the extent that an otherwise well-tuned controller will fail [4]. A way to tackle this is to incorporate Uncertainty Quantification (UQ) into control design.

Methods using UQ, like a-priori estimation, rely heavily on the employed model structure. Generated models are usually complex and hard to incorporate into controller design in practice [5, 6]. When prior information about the environment is limited, methods based on principles of adaptive control [7] can apply. An appealing feature of these methods is that they can provide some form of performance and safety guarantees (e.g., regarding system stability). Despite their overall effectiveness, such methods can still be challenged in two key ways. First, when the intensity of the uncertainty affecting system behavior violates the underlying controller assumptions. Second, when modeling errors [8], such as unmodeled dynamics, render the model invalid [4]. To address the problem of unmodeled dynamics, data-driven control techniques have been proposed.

Machine learning methods, such as Gaussian Process [9, 10], or Deep Neural Networks [11], can be used to either model the system [12, 13] or determine control inputs [14, 15] following a black-box input-output training procedure. However, as state dimensionality and system structure complexity increase (e.g., by using ‘deeper’ neural networks), the aforementioned methods may be challenged when it comes to be implemented in real-time for robotics research (e.g., [16, 17, 18]). One way to address this issue is by adopting a hierarchical structure to involve deep neural networks in control design. Instead of designing new controllers directly, reference trajectories are generated to deal with uncertainty through learning [19, 20]. Unfortunately, certain limitations still exist even when adopting a hierarchical structure. Neural-network-based approaches require a large body of data to train well [21]. At the same time, they remain limited in terms of offering some form of performance and safety guarantees, although this is currently an active research topic (e.g., [22, 20, 23]).

Besides neural-network-based approaches, dimensionality reduction and dynamics decomposition approaches can play an important role in data-driven algorithms. Methods like DMD (Dynamic Mode Decomposition [24]), EDMD (Extended DMD [25]), POD (Proper Orthogonal Decomposition [26, 27]) and their various kernel [28] and tensor [29] extensions have been successfully applied to across areas. A benefit of decomposition algorithms is that they can significantly reduce the amount of data required for approximating a system’s model through data (e.g., [30, 31, 32]).

Fueled by the potential of hierarchical methods and dimensionality-reduction approaches, this paper presents a new Data-driven Hierarchical Control (DHC) structure to handle uncertainty. The approach hinges on DMD with Control (DMDc [33]) to learn a higher-level controller and then generate a refined reference to improve an existing lower-level controller’s performance (Fig. 1). Our approach can be particularly appropriate in practice when a low-level, high-rate, pre-tuned controller is already in place, and a second higher-level controller is wrapped around the lower-level one to allow for the system to operate under uncertainty. The quadrotor UAV is one illustrative case, where it is now common practice to employ a hierarchical control structure [34, 35]. The low-level controller is responsible for controlling the attitude of the robot, while the high-level controller determines its position.

Succinctly, the paper’s contribution is twofold.

• •

  We propose a hierarchical control structure that refines a reference signal sent to a lower-level controller to deal with uncertainty. The structure builds on top of a model-identification block based on DMDc, and is shown that it retains stability of the underlying controller.

• •

  We analyze the sensitivity of DMDc to noisy data, and provide loose and tight bounds for the model identification component of our method.

Furthermore, we evaluate and validate the methodology using aerial robots both in simulation and experimentally. In turn, this effort can help design controllers able to learn how to harness uncertain aerodynamics, such as ground effect. A supplementary video can be found at https://youtu.be/OznDCskVnJU.

Dynamic Mode Decomposition (DMD) can characterize nonlinear dynamics through analysis of an approximated linear system [36]. Most relevant to our work here, DMD with control (DMDc)–an extension of the original DMD [37]—aims to incorporate the effect of external inputs [33].

The goal of DMDc is to analyze the relation between a future system measurement $\xi_{k+1}\in\mathbb{R}^{n}$, a current measurement $\xi_{k}\in\mathbb{R}^{n}$, and the current input $u_{k}\in\mathbb{R}^{m}$. For each triplet of measurement data $(\xi_{k+1},\xi_{k},u_{k})$, a pair of linear operators $A\in\mathbb{R}^{n\times n}$, and $B\in\mathbb{R}^{n\times m}$ is determined to approximate

Operators $A$ and $B$ are selected as the best-fit solution for all triplets of available data. Given observations and inputs up to time instant $M$ arranged in vectors $\Xi=[\xi_{1},\xi_{2}\dots,\xi_{M-1}]$, $\Xi_{P}=[\xi_{2},\xi_{3}\dots,\xi_{M}]$, and $U=[u_{1},u_{2}\dots,u_{M-1}]$, approximation (1) can be rewritten compactly as

Then we could seek the best-fit solution as:

where $\dagger$ denotes the pseudo-inverse. The problem can be solved immediately by Singular Value Decomposition (SVD), or QR decomposition, among others [38, 39].

The model identification block extracts dynamics when measurement data are noisy. Below we analyze the sensitivity to noisy data and derive two estimation error bounds.

Lemma 2.1. Consider the system

The sensitivity of system (3) to perturbations in s and $H$ is

where $\kappa(H)$ is the condition number of linear operator H, and $\theta=\arccos{\frac{\|H\textbf{z}\|}{\|\textbf{s}\|}}.$

Proof of Lemma 2.1: Start with the normal equations

The first-order perturbation relation is

which we re-arrange to get

Define $r=\textbf{s}-H\textbf{z}$, and let $\sigma_{1},\dots,\sigma_{p}$ be singular values of $H$. Then,

Setting $\|H\|=\sigma_{1}$, $\|(H^{T}H)^{-1}\|=1/\sigma_{q}^{2}$, we can derive [40]

Dividing both sides by $\|\textbf{z}\|$ and after some algebraic manipulation so that $\|\delta H\|$ and $\|\delta\textbf{s}\|$ only appear in ratios of $\frac{\|\delta H\|}{\|H\|}$ and $\frac{\|\delta\textbf{s}\|}{\|\textbf{s}\|}$, as $\kappa(H)=\sigma_{1}/\sigma_{q}$, we get

Then with

we arrive at (4).$\hfill\blacksquare$

Corollary 2.2. The estimation error of problem (2) is bounded by the measurements’ noise intensity, i.e.

where $\kappa(\Omega)$ is the condition number of observation matrix $\Omega$, and $\theta=\arccos{\frac{\|A_{P}\Omega\|}{\|\Xi_{P}\|}}.$

Proof of Corollary 2.2: The proof follows directly from Lemma 2.1 by setting $H=\Omega$, $\textbf{z}=A_{P}^{T}$, and $\textbf{s}=\Xi_{P}^{T}$. $\hfill\blacksquare$

Lemma 2.3 [A tighter bound]. Let $\epsilon=\frac{\|\delta\textbf{s}\|}{\|\textbf{s}\|}$. In the case that $\|\delta\textbf{s}\|=\|\delta H\|$, the sensitivity of (3) is reduced to

Proof of Lemma 2.3: From system (3), we have

and it follows that

Then,

Setting $\|\delta\textbf{s}\|=\|\delta H\|$ yields

which concludes the proof. $\hfill\blacksquare$

Corollary 2.4. When the disturbance occurs in observed states, i.e. $\|\delta\Xi_{P}\|=\|\delta\Xi\|=\|\delta\Omega\|$ a tighter error bound is

Proof of Corollary 2.4: The proof follows directly from Lemma 2.3 by setting $H=\Omega$, $\textbf{z}=A_{P}^{T}$, and $\textbf{s}=\Xi_{P}^{T}$. $\hfill\blacksquare$

Toy Example: Consider the simple linear system

We add zero-mean white noise to corrupt state observations, i.e. $N(0,\sigma^{2})$, with $\sigma\in[0.05,1]$ in $0.05$ increments. The initial condition is set at $[0.01,0.01]$. We perform a Monte Carlo simulation for $10,000$ trials to produce the average error. With reference to Fig. 2, the effect of the disturbance increases as variance increases (as expected). The general bound given in (5) expands at higher rates than the tighter one we obtain via (7) as the noise variance increases.

We demonstrate the utility and evaluate the performance of DHC in simulating a planar quadrotor with different noise intensity, and then test experimentally with a quadrotor flying under the influence of ground effect.

The paper introduced a Data-driven Hierarchical Control (DHC) structure with two key functions. 1) DHC utilizes an online data-driven approach to learn a model of input reference and true outputs. 2) DHC utilizes the derived model to keep refining a reference signal given to an underlying low-level controller to ensure system performance in the presence of system and/or system-environment uncertainty. We showed that DHC retains the stability properties of the underlying lower-level controller, and further determined lower bounds for our method’s sensitivity to noisy data.

The utility and performance of the proposed approach are tested in simulation and experimentally with a quadrotor. The test cases were designed to excite specific types of uncertainty that are common in practice: 1) constant model parameter errors in simulation; and 2) variable uncertain robot-environment interaction experimentally through operation under ground effect. Results suggest that DHC can successfully be wrapped around existing controllers in practice, to improve their performance by discovering and harnessing unmodeled dynamics during deployment.