Optimal PID and Antiwindup Control Design as a Reinforcement Learning Problem

A controller can become saturated when it has maximum and minimum constraints on its control signal and is given a set-point or a disturbance that carries the control signal outside these limits. If the actuator constraints are given by two scalars $u_{\text{min}}<u_{\text{max}}$, then we define the saturation function to be

If saturation persists, the controller is then operating in open-loop and the integrator continues to accumulate error at a non-diminishing rate. That is, the integrator experiences windup. This creates a nonlinearity in the controller and can destabilize the closed-loop system. Methods for mitigating the effects of windup are referred to as anti-windup techniques. For a more detailed overview of the windup phenomenon and simple anti-windup techniques, the reader is referred to Åström and Rundqwist (1989).

In this paper, we focus on one of the earliest and most basic anti-windup methods called back-calculation (Fertik and Ross, 1967). Back-calculation works in discrete-time by feeding into the control signal a scaled sum of past deviations of the actuator signal from the unsaturated signal. The nonnegative scaling constant, $\rho$, governs how quickly the controller unsaturates (that is, returns to the interval $(u_{\text{min}},u_{\text{max}})$). Precisely, we define $e_{u}(t)=\text{sat}(u(t))-u(t)$ and $I_{u}(t_{n})=\sum_{i=1}^{n-1}e_{u}(t_{i})\Delta t$, then we redefine the PID controller in (2) to be the following

From (3) it is clear that if the controller is operating within its constraints, then (4) emits the same control signal as (2). Otherwise, the difference $\text{sat}(u)-u$ adds negative feedback to the controller if $u>u_{\text{max}}$, or positive feedback if $u<u_{\text{min}}$. Further, (4) agrees with (2) when $\rho=0$; therefore, the recovery time of the controller to the operating region $[u_{\text{min}},u_{\text{max}}]$ is slower the closer $\rho$ is to zero and more aggressive when $\rho$ is large. A scheme of this approach is shown in figure 2 and the effect of the parameter $\rho$ is shown in figure 7 in Section 4.

The fundamental components of RL are the policy, the objective, and the environment. We assume the environment is modeled by a Markov decision process with action space $\mathcal{U}$ and state space $\mathcal{S}$. Therefore, the environment is modeled with an initial distribution $p(s_{0})$ with a transition distribution $p(s_{n+1}|s_{n},u_{n})$, where $s_{0},s_{n},s_{n+1}\in\mathcal{S}$ and $u_{n}\in\mathcal{U}$. Here, we define $s_{n}=[e_{y}(t_{n})\ I_{y}(t_{n})\ D(t_{n})\ I_{u}(t_{n})]^{T}$ and $u_{n}\in\mathcal{U}$ refers to the saturated input signal given by (7) at time $t_{n}$. The vector of parameters in (6) is referred to as $K$. Formally, the PID controller with anti-windup compensation in (7) is given by the mapping $\mu(\cdot,K)\colon\mathcal{S}\rightarrow\mathcal{U}$ such that

Each interaction the controller (8) has with the environment is scored with a scalar value called the reward. Reward is given by a function $r:\mathcal{S}\times\mathcal{U}\to\mathbb{R}$; we use $r_{n}$ to refer to $r(s_{n},u_{n})$ when the corresponding state-action pair is clear. We use the notation $h\sim p^{\mu}(\cdot)$ to denote a trajectory $h=(s_{1},u_{1},r_{1},\ldots,s_{N},u_{N},r_{N})$ generated by the policy $\mu$, where $N$ is a random variable called the terminal time.

The goal of RL is to find a controller, namely, the weights $K$, that maximizes the expectation of future rewards over trajectories $h$:

where $s_{0}\in\mathcal{S}$ is a starting state and $0\leq\gamma\leq 1$ is a discount factor. Our strategy is to iteratively maximize $J$ via stochastic gradient ascent, as maximizing $J$ corresponds to finding the optimal PID gains. Optimizing this objective requires additional concepts, which we outline in the next section.

Equation (9) is referred to as the value function for policy $\mu$. Closely related to the cost function is the Q-function, or state-action value function, which considers state-action pairs in the conditional expectation:

Returning to our objective of maximizing (9), we employ the policy gradient theorem for deterministic policies (Silver et al., 2014):

We note that Eq. (9) is maximized only when the policy parameters $K$ are optimal, which then leads to the update scheme

where $\alpha>0$ is the learning rate.

The optimization of $J$ in line (12) relies on knowledge of the $Q$-function (10). We approximate $Q$ iteratively using a deep neural network with training data from replay memory (RM). RM is a fixed-size collection of tuples of the form $(s_{n},u_{n},s_{n+1},r_{n})$. Concretely, we write a parametrized $Q$-function, $Q(\cdot,\cdot,W_{c}):\mathcal{S}\times\mathcal{U}\to\mathbb{R}$, where $W_{c}$ is a collection of weights. This framework gives rise to a class of RL methods known as actor-critic methods. Precisely, the actor-critic methods utilize ideas from policy gradient methods and $Q$-learning with function approximation (Konda and Tsitsiklis, 2000; Sutton et al., 2000). Here, the actor is the PID controller given by (8) and the critic is $Q(\cdot,\cdot,W_{c})$.

An advantage of our approach is that the weights for the actor can be initialized with user-specified PID gains. For example, if a plant is operating with known gains $k_{p},k_{i}$, and $k_{d}$, then these can be used to initialize the actor. The idea is that these gains will be updated by stochastic gradient ascent in the approximate direction leading to the greatest expected reward. The quality of the gain updates then relies on the quality of the $Q$-function used in (12). The $Q$-function is parameterized by a deep neural network and is therefore initialized randomly. Both the actor and critic parameters are updated after each roll-out with the environment. However, depending on the number of time-steps in each roll-out, this can lead to slow learning. Therefore, we continually update the critic during the roll-out using batch data from RM.

As previously described, the actor (PID controller) is updated after each episode. We are, however, free to change the PID gains at each time-step. In fact, previous approaches to RL-based PID tuning such as Brujeni et al. (2010) and Sedighizadeh and Rezazadeh (2008) dynamically change the PID gains at each time-step. There are two main reasons for avoiding this whenever possible. First, the PID controller is designed for set-point tracking and is an inherently intelligent controller that simply needs to be improved subject to the user-defined objective (reward function); that is, it does not need to ‘learn’ how to track a set-point. Second, when the PID gains are free to change at each time-step, the policy essentially functions as a gain scheduler. This switching of the control law creates nonlinearity in the closed-loop, making the stability of the overall system more difficult to analyze. This is true even if all the gains or controllers involved are stabilizing (Stewart, 2012). See, for instance, example 1 of Malmborg et al. (1996).

Of course, gain scheduling is an important strategy for industrial control. The main point here is that RL-based controllers can inherit the same stability complications as gain scheduling. In the next subsection, we demonstrate the effect of updating the actor at different rates on a simple linear system.

Consider the following continuous-time transfer function:

We discretize (14) with time-steps of 0.1 seconds. In this example, we initialize a PI controller with gains $k_{p}=0.2,k_{i}=0.05$. The following results are representative of other initial PI gains. Note, however, we cannot set $k_{p}=k_{i}=0$ because this forces $\mu(\cdot,K)\equiv 0$ and the parameters will not change between updates.

In our experiments we implement algorithms V1 and V2. We also consider “V1.5”, in which the PID parameters are updated every tenth time-step. Note that the fundamental difference between V1 and V2 is that the former corresponds to an online implementation of the algorithm, while the latter can be seen as an offline version. V1.5 represents a dwell time in the learning algorithm.

Figure 3 only shows the value of $k_{p},k_{i}$ at the end of each episode for each implementation. Nonetheless all three implementations reach approximately the same values; the final closed-loop step responses for each implementation are shown in figure 5.

Another way of visualizing the PID parameters is in the $k_{p}-k_{i}$ plane. We plot the boundary separating the stable and unstable regions using the parametric curve formulas due to Saeki (2007). Figure 4 is a scatter plot with the $k_{p},k_{i}$ value at the end of each episode along with the aforementioned boundary curve. We note that the stability regions refer to the closed-loop with (14) and a fixed $k_{p}-k_{i}$ point, rather than the nonlinear system induced by updating $k_{p}-k_{i}$ values online.

We see in figure 6 that all three implementations achieve similar levels of performance as measured by the reward function (13) ($\lambda=0.50$). Although, V1 and V1.5 plateau sooner than V2, the initial dip in reward from V1 can be explained by the sequence of unstable $k_{p}-k_{i}$ values around the boundary curve in figure 4.

Finally, we note that V1 and V1.5 reach their peak performances after approximately 25 minutes (real-time equivalent) of operation. In our experiments, the actor learning rate $\alpha$ had the most drastic effect on the convergence speed. Here, we show the results for a relatively small $\alpha$ (see Appendix A) to clearly capture the initial upheaval of the parameter updates as well as the long-term settling behavior. In principle, we could omit the latter aspect and simply stop the algorithm, for example, once the reward reaches a certain average threshold.

In this example, we incorporate an anti-windup tuning parameter and employ algorithm V2. Consider the following transfer function:

In order to tune $\rho$, it is necessary to saturate the input: If the actor always operates within the actuator limits, then

because $I_{u}\equiv 0$, meaning $\rho$ will never be updated in (12). This can be understood from figure 7 at the bottom, as there is a non-zero difference between the dashed and solid lines only after the first step change (this corresponds to the difference shown in figure 2). Further, although (16) also holds for states $s_{n}$ corresponding to input saturation, which therefore do not contribute to the update in (12), we still store them in RM for future policy updates.

In our experiment, the set-point is initialized to $1$, then switches to $1.5$ (plus a small amount of zero-mean Gaussian noise), then switches back to 1. The switches occur at varying time-steps. At the beginning of an episode, with $10\%$ probability, the switches set-point is set to 3 instead of 1.5. Figure 7 shows a slower recovery time for smaller $\rho$ and a more aggressive recovery for larger $\rho$ values.

We emphasize that the actor can be initialized with hand-picked parameters. To illustrate this, we initialize $k_{p}$ and $k_{i}$ using the SIMC tuning rules due to Skogestad (2001). Figure 8 shows little change in the $k_{p}$ parameter, while $k_{i}$ and $\rho$ adjust significantly, leading to a faster integral reset and smoother tracking than the initial parameters.