Robust Fuzzy Q-Learning-Based Strictly Negative Imaginary Tracking Controllers for the Uncertain Quadrotor Systems

In this section, we briefly present the negative imaginary systems theory and its formal definition for linear time invariant (LTI) systems.

Negative imaginary systems theory was introduced by Lanzon and Petersen in [11, 6]. A generalized definition was developed in [7] to accommodate systems with free body motion. We can understand the negative imaginary property by considering the case of single-input single-output (SISO) system. For instance, the negative imaginary property is defined by considering the properties of the imaginary part of the frequency response $G(j\omega)$ and requiring the condition $j\left(P(j\omega)-P(j\omega)^{\ast}\right)\geq 0$ to be satisfied for all $\omega\in(0,\infty)$. Roughly speaking, the NI systems are stable systems having a phase lag between 0 and $-\pi$ for all $\omega>0$. That is, their Nyquist plot lies below the real axis when the frequency varies in the open interval $(0,\infty)$ . This is similar to positive real systems, where the Nyquist plot is constrained to lie in the right half of the complex plane.

Consider an LTI system that has the following state space representation,

Here, the matrices $A\in\mathbb{R}^{n\times n},B\in\mathbb{R}^{n\times m},C\in\mathbb{R}^{m\times n},$ $D\in\mathbb{R}^{m\times m}.$ The transfer function matrix corresponding to this system is given by:

For the above LTI system, an NI system is defined as follows;

Similarly, the strictly negative imaginary system (SNI) is also defined as follows;

The fundamental result of interconnected negative imaginary systems theory can be summarised as follows;

where $\lambda_{max}$ is the maximum eigenvalue.

Feedback linearization is a widely used approach in linearizing nonlinear dynamical systems [40]. The basic idea of feedback linearization is to find a transformation that transforms given nonlinear dynamics into an equivalent linear dynamical system (see Fig. 2).

Consider the following nonlinear system

where $x\in\mathbb{R}^{n}$ represents the state vector, $u\in\mathbb{R}^{p}$ represents the input vector, and $y\in\mathbb{R}^{m}$ represents output vector. The main idea is to design the control law as follows;

such that the dynamics from the new input $v$ to the output $y$ are linear.

The negative imaginary property arises in many practical systems. For example, such systems arise when considering the transfer function from a force actuator to a corresponding colocated position sensor (for instance, a piezoelectric sensor) in a lightly damped structure [41, 6, 42]. Another area where the underlying system dynamics are NI, are nano-positioning systems; see e.g., [42, 43, 44]. Similar issues also arise in application to the area of robotics, where position measurements are also widely used.

A fuzzy controller or model uses fuzzy rules, which are linguistic if-then statements, consisting of fuzzy sets, fuzzy logic, and fuzzy inference, to express complex processes without the use of complicated models [18, 19]. These rules embed expert’s control/modeling knowledge and represent experience in connecting fuzzy variables using linguistic terms [45]. Fig. 3 describes the overall architecture of Fuzzy Logic control. The common structure of a fuzzy rule is:

where $\epsilon_{i},\forall i=1,...,n$ is a fuzzy set of the $i^{th}$ input. $n$ is the total number of fuzzy rules. While $\zeta=(\zeta_{1},\zeta_{2},...,\zeta_{n})$ represents the script input vector, $\phi$ denotes the output variable obtained from the defuzzification process, and $\mu$ indicates a fuzzy consequent offered by the expert.

The fundamental operators ‘and/or/not’ combine the conditions of the variable inputs that are satisfied. Based on the degree of match between fuzzy input and the rules, which rules need to be implemented and the firing strength $w_{r_{i}}$ of the rule $i$ is defined according to the given input area.

Sugeno (TS) fuzzy inference, which is the most commonly used, works with singleton output membership functions that are either constant or a linear function of the input values. By using a weighted average over all rule outputs to compute the script output, the defuzzification process for the Sugeno system is more computationally efficient than that of a Mamdani system. The final output can be obtained using the Wang-Mendel technique [46]:

where $\phi_{i}$ is the consequent of rule $i$.

Q-learning is a subclass of reinforcement learning, which is one of the three basic machine learning platforms; along with supervised learning and unsupervised learning. Reinforcement learning represents the learning process in the nature around us. Reinforcement learning relies on a goal-directed problem solving mechanism. Q-learning is a model-free control method, where the control policy generated depends on the interaction or the feedback between the environment and the agent without knowing any information about the system model. Therefore, a Q-reinforcement learning policy is defined as a learning control law for the system to decide on the actions to be performed.

The main components in reinforcement learning in general are the agent states $S$, the agent allowable actions $A$, and the rewards $R$. Here, the agent or the controller observes states $S_{t}\in S$ and therefore performs an action (control signal) $A_{t}\in A$. Hence, the environment (the system under control) updates the new states $S_{t+1}\in S$ and produces a reward $R_{t+1}\in R$. The reward is then received by the agent and chooses a new action $A_{t+1}\in A$. This feedback interaction proceeds until a terminal state $S_{T}\in S$ is reached.

The main aim of the agent (the controller) in reinforcement learning is to find or learn the optimal control policy. This control policy is to maximize a discounted accumulated reward function $J_{t}$, which may take the following form:

Here, $\sigma$ is a discounting factor that defines the significance of the future rewards, $0<\sigma<1$.

The action-value function (Q-Function), $Q^{\pi}(s,a)$, indicates the expected return with initial state $s$ and initial action $a$ and all forward actions controlled by policy $\pi$:

In the Q-learning algorithm, the update rule is governed by the following equation:

where the learning rate $\eta$ represents the degree the new information overrides the old one.

Fuzzy Q-Learning is a fuzzy extension of the Q-learning algorithm. In creating an FQL model, one first needs to specify the input states and their corresponding fuzzy sets and then build a Fuzzy Inference System (FIS) to integrate with the Q-Learning algorithm. In Section III-B, a fuzzy Q-Learning algorithm for our Quadrotor system is developed.

The general dynamic model of a quadrotor using the Euler-Lagrangian method has been derived in the article [47, 48]. To derive the quadrotor’s motions in 3D space, an inertial frame $E$ whose origin is at the defined home location and a body frame $B$ whose origin is at the quadrotor’s center of gravity are formed as in Fig. 4.

Let $\delta=(x,y,z,\alpha,\theta,\psi)\in\Re^{6}$ denote the generalized coordinates and let $\rho=(x,y,z)\in\Re^{3}$ denote the position vector of the air vehicle with respect to an inertial frame. Moreover, the orientation of the quadrotor is represented by the three Euler angles, namely the roll angle $\alpha$ around the $x$ axis, the pitch angle $\theta$ around the $y$ axis, and the yaw angle $\psi$ around the $z$ axis, that generate a rotational vector $\Omega=(\alpha,\theta,\psi)\in\Re^{3}$. Based on the Euler angles, coordinates are transferred between inertial and body frames as required.

The overall dynamic model describing the nonlinear quadrotor behavior is summarized as follows [49]:

where

in which $I_{i}$ stands for the moments of inertia about the respective axes and $m$ represents the mass of the quadrotor. ($C_{dx},C_{dy}$, $C_{dz}$) and ($C_{ax},C_{ay}$, $C_{az}$) are the translational and rotational drag coefficients. Further, ($c_{\alpha},c_{\theta},c_{\psi}$) and ($s_{\alpha},s_{\theta},s_{\psi}$) represent ($cos(\alpha),cos(\theta),cos(\psi)$) and ($sin(\alpha),sin(\theta),sin(\psi)$), respectively. While $K_{m}$ is the rotor torque coefficient, $K_{f}$ is the rotor thrust coefficient. Additionally, $J_{r}$ stands for the inertia of the propeller and $\Omega_{r}$ = $\sum_{i=1}^{4}$ (-1)ⁱ⁺¹ $\omega_{pi}$ is the total angular speed of propellers, wherein $\omega_{pi}$ is the speed of the $i^{th}$ rotor. Moreover, the control signal vector $U=[U_{1},U_{2},U_{3},U_{4}]$ includes the total thrust and the pitch, roll, and yaw moments. Finally, $L$ represents the distance from the center of the quadrotor to the actuator axes.

The proposed state-space model of the quadrotor in (13) cannot be converted into an equivalent linear and controllable system by static state feedback due to under-actuation. It is, however, possible to resort to dynamic state feedback to achieve full-state linearisation. After that, starting from the new feedback linearized design, a linear controller can be designed. In this section, the feedback linearization approach for the quadrotor UAV attitude and altitude will be presented.

According to (7), the control input vector $U$ can be chosen to eliminate all nonlinear terms in (13), leading to a linear system. Using dynamic inversion, $U$ can be obtained by:

Substituting (15) into (II-B) results in the following linear system:

In our approach, for convenience, drag coefficients are assumed to be zero since drags are negligible at low speed [50]. The attitude and altitude quadrotor dynamic formulas in (13), therefore, can be re-written as:

Based on (17), $U_{1},U_{2},U_{3}$, and $U_{4}$ can be selected as bellows:

where $k_{1}=\frac{I_{z}-I_{y}}{I_{x}}$, $k_{2}=\frac{J_{r}}{I_{x}}$, $k_{3}=\frac{I_{x}-I_{z}}{I_{y}}$, $k_{4}=\frac{J_{r}}{I_{y}}$, $k_{5}=\frac{I_{y}-I_{x}}{I_{z}}$, $h_{1}=\frac{1}{I_{x}}$, $h_{2}=\frac{1}{I_{y}}$, $h_{3}=\frac{1}{I_{z}}$.

By substituting (18) into (17), the following linear system is obtained:

The Laplace transforms of the above equation gives the following transfer function matrix:

The transfer function matrix $P$ in (20) has two poles on the imaginary axis (double integrator). Based on the NI definition given in [51], it is clear that the linearised attitude and altitude dynamic model satisfies the NI property.

In the Q Learning method, the values of $Q$ for the state-action (SA) pairs in the table are continuously computed and updated at each iteration using a Q function. However, if the number of SA pairs is large, the computational load will dramatically increase, which might fail in real-time implementation. For scaling up to large or continuous SA spaces, TS fuzzy function approximation can be used in combination with Q-Learning to generate the Fuzzy Q-Learning control theory [52, 37]. In this new method, the state of the agent $S$ is the crisp set of the fuzzy inputs $\zeta$. Hence, the firing strength of each rule determines the degree to which the agent matches a state.

Additionally, the rules do not have fixed consequences, meaning that there are no predefined state-action pairs. They can be increased through the exploration/exploitation algorithm. Thus, the learning agent has to find the best conclusion for each rule via competing actions (e.g., the action with the best Q-value):

where $a[i,j]$ is the $j^{th}$ possible action in rule $i$ and $Q[i,j]$ is a corresponding Q value. Furthermore, $S_{i}$ is defined by $\zeta_{1}$ is $S_{i,1}$ and $\zeta_{2}$ is $S_{i,2}$ … and $\zeta_{n}$ is $S_{i,n}$, in which $S_{i,j}$, j = 1,…,n are TS fuzzy rules.

Therefore, the global action $a(\zeta)$ for input vector $\zeta$ and the selected rule consequences $a_{i}$ can be expressed as follows:

On the other hand, the corresponding Q-value is calculated by the following equation:

where $q[i,i^{+}]$ is the actual q-value of the chosen action $i^{+}$ in the fired rule $i$, using an exploration/exploitation strategy and an action $i^{*}$ that has the maximum Q value for the fired rule $i$, such that $q[i,i^{*}]$ = $maxq[i,j]$, $j=1,..,J$.

Based on gradient descent, the rate of change in q values are calculated as follows:

where

Thus, the q values are updated as:

where $\eta$ indicates the learning rate that determines how much the model updates the action value depending on the reward prediction error.

The pseudo code of this algorithm is presented as in in Table II:

The control strategy for the quadrotor plant is shown in Fig. 5. The tracking errors arise from the subtraction of the desired attitude and altitude set points and their actual values. The error signals are considered as the input vector of both SNI tracking controllers and the FQL agents. The SNI tracking control system whose gains are tuned offline by minimizing the tracking error function and then adjusted online by the fuzzy Q-Learning algorithm has the following general representation as follows:

where $\gamma$ is the DC gain, $\tau$ is the time constant, and $\beta$ is a feed forward gain.

Note that the SNI controller in (27) is in a different form to the one used in [3, 19, 4], since it contains a negative feed forward term $\beta$. The primary reason for using a feed forward term in the SNI controller is due to stability conditions. This will be explained in the next section.

The proposed SNI gains are tuned online by the fuzzy Q-Learning algorithm. The input signal $\zeta$ of the fuzzy Q-Learning agent is bounded within a range of [-2 2]. There are six Gaussian-shaped membership functions (e.g., NB, NS, Z, PS, PB) for the state variables, as depicted in Fig. 6.

The output of each agent is a vector with two output variables $\phi=[\gamma,\tau]$, each of them for each gain of the SNI controller. Each output variable involves a group of three singleton fuzzy actions. For the output variable $\gamma$, its singleton fuzzy sets are {-20, 0, 20} and for the output variable $\tau$, the group of the fuzzy singletons is {-0.002, 0, 0.002}. Each linguistic value, produced at the output variable, denotes the percentage variation of each gain according to its initial value.

As mentioned in Section II. D, the agent communicates with the environment and performs updates to the state-action pairs in the Q-Table. Moreover, there are two primary operations: exploring and exploiting to acquire information through its experience. While the exploiting action uses the available data to make a decision via the max value of those actions, the exploring action enables the agent to explore new states that may not be selected during the exploitation process. To accelerate the learning process at the beginning of the experiment, where the fuzzy Q-Learning agent mainly explores, the agent will search for new knowledge during a certain number of iterations; see Fig 9(c). After that, the agent conducts exploitation for 99% of the time and the remaining 1% of the time is exploration for a given state.

The agent reward $R$ is selected the same as in [37]:

where $e_{t+1}$ and $e_{t}$ are the tracking errors in the next and current states. The input error signal is limited in range of [-2 2]m. If the tracking error is 0, $R$ becomes 1. The obtained reward result can take either positive or negative values. If the reward values are positive, the error in the following state is decreased (reward). Otherwise, if the reward values are negative, the next state’s error is raised (punishment).

The future discounted accumulated reward therefore, is rewritten as:

.

The performance of the proposed adaptive controller is evaluated under a nominal condition. According to the simulation results observed in Fig. 8, it is obvious that most of the proposed controllers are capable of tracking the desired reference values with satisfactory performance, reasonable rise time and low $RMSE$ values, as summarized in Table IV. In contrast, although the PID controller tracks well the constant z setpoints and the sin-wave trajectory, it cannot handle the desired extreme square and step trajectories by experiencing a high overshoot of approximately 25%. Thus, it can be seen that the proposed SNI control methods are more robust and have a better tracking capability than the traditional PID controller.

Additionally, the above experiment has been repeated under the different samples of $\eta$, $\sigma$, and the initial exploration period. Firstly, as shown in Fig. 9(b), increasing the constant discount factor results in the slight variations of the RSME metrics. However, the best improvement is observed as $\sigma$ = 0.7. Similarly, learning rate $\eta$ = 0.1 produces the smallest RMSE values among the potential candidates; see Fig. 9(a). Lastly, the obtained results in Fig. 9(c) demonstrate that having an exploration phase at the start time noticeably reduces the tracking errors, especially the vertical direction, due to the faster learning time. Furthermore, the smallest root means square error (RMSE) results are obtained if the beginning exploration phase is implemented in the first 0.7s of the simulation. Consequently, a 0.7s exploration phase is set at the beginning of all simulations.

For robustness analysis, the performance of the closed-loop control system will be examined against exogenous disturbances (e.g., wind gusts and air turbulence) and parameter changes. The Dryden wind gust model in MATLAB is applied to replicate a realistic wind gust scenario under realistic flight circumstances. The model employs continuous gusts along three axes based on a spatially varying stochastic process with a maximum of 5m/s [53]. In addition to the Dryden gust turbulence, the effects of air stream are also considered in our numerical experiment by the use of the 1-cos air turbulence model in [54]. The combined effects of extreme wind gusts and turbulence applied to the system throughout the simulation time is shown in Fig. 10.

Furthermore, the model uncertainties, consisting of the moment of inertia about each axis and mass, are set in consideration of 15% bias. However, the quadrotor simulation model parameters in the linear-system generator are kept as in Table III. Therefore, the residual nonlinearities and uncertainties acting on the quadrotor must be accommodated by the tracking controllers.

Evidently, the effects of the multiple disturbances alters the performance of all four controllers. Nonetheless, the fuzzy-QL-SNI controller outperforms the fuzzy-SNI, SNI, and PID by stabilizing the system with less drift from the desired pose references, as shown in the plots in Fig. 11. The PID method is unable to achieve the desired tracking performance when tracking the extreme pitch trajectory. Although the tracking performances are better for the yaw and z step trajectories, the flight paths experience the high-overshoot for yaw control and chattering phenomena in the steady-state phase, and chattering phenomena in the steady-state phase, demonstrating a slow recovery from disturbances. The similar results can be seen in Table V, the PID control approach produces large $SO$ figures of 0.48rad for roll, 0.25rad for pitch, 0.51rad for yaw, and 0.11m for altitude. Moreover, the PID controller has the maximum $RMSEs$ of 0.31rad for roll, 0.72rad for pitch, 0.34rad for yaw, and 0.09m for altitude while its $t_{s}$ values for the roll and pitch are high, namely 2.5s and 1.4s. However, the SNI controllers are able to handle the wind turbulence and the system’s uncertain parameters, much faster and with less error than the PID. This is reflected in both columns of Table V and Fig. 11. Furthermore, the Fuzzy-QL-SNI controller’s performance yields much more desirable values than the other ones. Particularly, the proposed adaptive strategy obtains the lowest $RMSE$ values of [0.026rad, 0.32rad, 0.25rad, 0.067m] for roll, pitch, yaw, and height, which are approximately 78%, 61%, and 60% lower than those with the PID, SNI and Fuzzy-SNI. The response of the Fuzzy-QL-SNI is noticeably faster with the $t_{s}$ numbers of [0.35, 0.17, 5.8, 1.5]s, almost twice as fast as that of the other control schemes. Furthermore, there are only slight differences in the evaluation metrics of the two given cases considered for the Fuzzy-QL-SNI. These observations validate the fast adaption capability and the robustness of our closed-loop adaptive control system.

Another strength of the Fuzzy-QL-SNI controller is that it delivers low-frequency and unsaturated output control signals. This is demonstrated in Fig. 12, where the Fuzzy-QL-SNI controller produces smooth moment and thrust control signals, even under external and internal disturbances without high-frequency oscillations, which is a drawback of the other control methods [55, 56].