Exploring the use of deep learning in task-flexible ILC*

Consider the conventional frequency domain ILC configuration illustrated by Figure 1, where $P\in\mathcal{R}^{n_{\mathrm{y}}\times n_{\mathrm{u}}}$ corresponds to the proper transfer matrix representation of a discrete time (DT) linear-time-invariant (LTI) multiple-input multiple-output (MIMO) plant with $\mathcal{R}$ denoting the set of real rational functions in the complex variable $z\in\mathbb{C}$. Furthermore, the proper $K\in\mathcal{R}^{n_{\mathrm{u}}\times n_{\mathrm{y}}}$ represents a LTI stabilizing DT feedback controller, which is typically constructed using rigid-body decoupling strategies, see [15]. The aim of the conventional frequency domain ILC framework is to construct an optimal feedforward policy $f$, which minimizes the position tracking error $e$ in the presence of the motion trajectory $r$. Under the assumption that the reference trajectory is trial invariant, the error propagation per trial $k\in\mathbb{N}_{\geq 0}$ is given by:

From (4), several observations can be made. First, it can be observed that the contribution of $r$ to the position tracking error is dependent on the robustness filter $Q$, which is optimally chosen as identity to negate the contribution of the reference trajectory towards the tracking error. Secondly, learning filter $L$ aims to minimize the criterion $\|Q(I-JL)\|_{\infty}<1$, where $\|\centerdot\|_{\infty}$ stands for the $\mathcal{H}_{\infty}$ norm, such that the tracking error is steered to zero, which is optimally achieved when $L=J^{-1}$. Note that these assumptions on $Q$ and $L$ yield the optimal feedforward update $f_{k+1}=P^{-1}r$, which results in perfect position tracking. Moreover, when the convergence criterion is satisfied, the limit policies, i.e. $e_{\infty}=\lim_{k\rightarrow\infty}e_{k}$, $f_{\infty}=\lim_{k\rightarrow\infty}f_{k}$, correspond to:

In spite of its simplicity and efficacy, the conventional ILC is hindered by significant limitations, the most notable of which is its confinement to a single task. Consequently, its practical utility is restricted to particular types of machinery.

The aim of this paper is to address the challenge of augmenting the task-flexibility of the conventional ILC by utilizing an imitation learning based controller. This approach facilitates the generalization of the optimal feedforward policy, created by the conventional ILC, for a wider range of motion profiles. The primary objective of this paper is to devise a feedforward controller that employs a learning-based mechanism, which satisfies the following requirements:

• (R1)

  The learning-based feedforward approach enables the generalization of the performance of the conventional ILC across multiple trajectories.

• (R2)

  The scalability of the learning-based feedforward approach is imperative for its implementation in systems with a high sampling rate.

For a given dynamic system with a proper discrete transfer function $G\in\mathcal{R}$ under a sampling time $T_{\mathrm{s}}\in\mathbb{R}_{+}$, a reference trajectory $r$ of duration $T=n_{\mathrm{d}}T_{\mathrm{s}}$ seconds can be defined as

Given a specific combination of the $L$ and $Q$ filters, consider that an ILC policy $\pi^{*}$ exists which maps a given reference trajectory $r$ to the optimal feedforward compensation signal $f^{*}$, see (5). This can be formally expressed as:

The three-part student policy $\pi_{\mathrm{s}}:r\rightarrow f_{\pi_{\mathrm{s}}}$ can be decomposed into three distinct components:

The proposed approach involves propagating the parameter $\eta$ through the three sub-policies, with the aim of iteratively optimizing both $n_{l}$ and $\mu$ via (17). However, given the significant computational burden associated with this approach, there is a need for a more straightforward alternative or a reformulation of the problem. With this goal in mind, we introduce the concepts of the Expert space and Student space to provide alternative perspectives for addressing the optimization problem at hand.

Example:

1. 1.

   Expert policy in student space:

   $$\pi_{\mathrm{e}}^{\mathrm{s}}:r_{l}\rightarrow f_{l}$$

   (20)

   where $r_{l},f_{l}\in\mathbb{R}^{n_{l}\times 1}$

2. 2.

   Student policy in student space:

   $$\pi_{\mathrm{s}}^{\mathrm{s}}:r_{l}\rightarrow\widehat{f_{l}}$$

   (21)

   where $r_{l},\widehat{f_{l}}\in\mathbb{R}^{n_{l}\times 1}$

Stated differently, the expert space is comprised of all the decoding policies, $\pi_{\mathrm{D}}$, which project signals into $n_{\mathrm{d}}$ dimensions, while the student space is composed of all the encoding policies, $\pi_{\mathrm{E}}$, which project signals into $n_{l}$ dimensions.

Based on the preceding definitions, it is worth noting that our primary objective is to determine the student policy in the expert space, $\pi_{\mathrm{s}}^{\mathrm{e}}$. In light of these definitions, the distance metric specified in (16) can be reformulated as:

The encoding and decoding sub-policies in this work employ DPCA, a well-established linear dimensionality reduction technique, due to its computational simplicity. Other commonly-used linear and non-linear dimensionality reduction methods are also available and have been reviewed in [17]. DPCA involves the identification of a linear subspace with $n_{l}$ dimensions in an $n_{\mathrm{d}}$ dimensional space, where $n_{l}$ is significantly smaller than $n_{\mathrm{d}}$. This subspace is defined by a set of orthonormal bases that maximize the variance of the original data when projected onto this subspace. The orthonormal bases computed through this process are commonly referred to as principal components.

The selection of the principal components for an $n_{l}$ dimensional latent space for the data points in $C$ involves choosing the right eigenvectors that correspond to the first $n_{l}$ singular values of $H$. It should be emphasized that the projection of a data point onto the latent space can be computed through the following method:

In the light of Remark 1, for a given dataset $H\in\mathbb{R}^{n_{\mathrm{d}}\times n_{\mathrm{t}}}$ the encoding ($\pi_{\mathrm{E,U}}$) and decoding ($\pi_{\mathrm{D,Y}}$) sub-policies for using in the student policy $\pi_{\mathrm{s}}$ can be defined as follows:

A silicon wafer scanning application is considered where the scanning takes place during the constant velocity interval of the motion profile, see [1]. In this context, Figure 5 illustrates the position tracking error in $x$-direction during the constant velocity interval of the reference trajectories $r_{1}$ and $r_{2}$ respectively. In addition to the performance of mass feedforward, TAIL-ILC and NN-ILC, the figure also indicates the performance of the expert ILC policy. This is to facilitate the comparison of the performance of the two deep learning based ILC variants with the baseline. As demonstrated in the left Figure, i.e. the performance of the feedforward controllers on $r_{1}$, the expert ILC policy exhibits the highest overall performance. Nonetheless, it is noteworthy that the TAIL-ILC policy outperforms in terms of the peak tracking error achieved compared to the alternative feedforward approaches, whereas the NN-ILC policy demonstrates a superior performance in terms of the convergence time of the error. Nonetheless, when analyzing the right Figure, i.e. the performance of the feedforward approaches for a previously unseen trajectory $r_{2}$, the expert ILC policy needs to re-learn the relevant feedforward signal. Conversely, the TAIL-ILC and NN-ILC policies are capable of achieving similar performance to the re-learned expert ILC policy without any further training. Additionally, when combined with a classical mass feedforward controller, both the TAIL-ILC and NN-ILC policies are observed to yield superior performance in terms of peak error and settling time compared to the classical mass feedforward controller alone.

Table III provides a comparison of the training and prediction properties of the TAIL-ILC and NN-ILC student policies. Here, we compare the following parameters:

1. 1.

   $T_{\text{train}}$: Time to train the neural network

2. 2.

   $T_{\text{predict}}$: Time to make predictions for 10 randomly selected test set trajectories.

3. 3.

   $e_{\text{train}}$: Control signal prediction error averaged over 10 randomly selected train set trajectories.

4. 4.

   $e_{\text{test}}$: Control signal prediction error averaged over 10 randomly selected test set trajectories.

5. 5.

   $e_{\text{peak \ tracking}}$: Peak tracking error achieved with the predicted control signals averaged over 10 randomly selected train set trajectories.

Moreover, as observed in Figure 5, the average signal prediction error has decreased for TAIL-ILC in case of previously seen trajectories whereas the NN-ILC has improved performance for previously unseen trajectories.

The NN-ILC and TAIL-ILC are two approaches of ILC that differ in their treatment of reference trajectories and feedforward signals. NN-ILC is capable of handling trajectories of different lengths, as it deals with them sample-wise. In contrast, TAIL-ILC processes trajectories and signals in their entirety, making it challenging to manage trajectories of varying durations due to the fixed input-output dimensionality of neural network learning models. Additionally, NN-ILC is better equipped to handle instantaneous changes in reference trajectories compared to TAIL-ILC. A possible solution to reconcile these perspectives is to use a different class of learning models, such as a recurrent neural network.

In NN-ILC, the training dataset used for $\pi_{\mathrm{NN}}$ encompasses all the samples from all the trajectories in the training set, along with their associated feedforward signals. Conversely, TAIL-ILC employs a training dataset for $\pi_{\mathrm{C}}$ that solely includes the parameters of the trajectories and feedforward signals within the latent space, resulting in a significantly smaller dataset in comparison to the total number of samples. This characteristic leads to TAIL-ILC presenting shorter training and prediction times when compared to NN-ILC, as demonstrated by the results presented in Table III.

Figure 5 demonstrates that NN-ILC outperforms TAIL-ILC in terms of generalizing performance to previously unobserved trajectories. The improved performance can be attributed to NN-ILC’s treatment of reference trajectories as points in an $n$-dimensional space corresponding to $n^{\mathrm{th}}$ order motion profiles, which allows it to learn a mapping to the corresponding feedforward signal time samples. As a result, the trained network can more accurately extrapolate performance to previously unobserved points in the space of possible reference trajectories. In contrast, TAIL-ILC relies primarily on analogies between individual tasks on a higher level, which may result in suboptimal performance when confronted with previously unobserved trajectories at the sample level.