Quantifying Knowledge Distillation Using
Partial Information Decomposition

Modern-day machine learning requires large amounts of compute for both training and inference. Knowledge distillation [1, 2] can be used to compress a complex machine learning model (the teacher) by distilling it into a relatively simpler model (the student). The term “distillation” in this context means obtaining some assistance from the teacher while training the student so that the student performs much better than when trained alone (see Figure 1). In its earliest forms, knowledge distillation involved the student trying to match the output logits of the teacher [1]. More advanced methods focus on distilling multiple intermediate representations of the teacher to the corresponding layers of the student [2, 3, 4, 5]. We also refer the reader to [6, 7] for surveys.

Information theory has been instrumental in both designing [3, 4] and explaining [8, 9] knowledge distillation techniques. However, less attention has been given to characterizing the fundamental limits of the process from an information-theoretical perspective. Our goal is to bridge this gap by first introducing new measures to quantify the “transferred knowledge” and “knowledge to distill” for a teacher and a student model given a target downstream task. We bring in an emerging body of work called Partial Information Decomposition (PID) [10, 11, 12] to explain knowledge distillation. We define the knowledge to distill using the PID measure of “unique” information about the task that is available only with the teacher but not the student. As it follows, the transferred knowledge is succinctly quantified by the measure of “redundant” information that is common between the teacher and student.

We propose a multi-level optimization that maximizes redundant information (transferred knowledge) as a regularizer for more effective distillation. While PID has been explored in a few avenues of machine learning,

it has remained a challenge to maximize these measures as a regularizer since computing them itself requires solving an optimization. Our optimization leads to a novel knowledge distillation framework – Redundant Information Distillation (RID) – which precisely captures task-relevant knowledge and filters out the task-irrelevant information from the teacher. In summary, our main contributions are as follows:

•  Quantifying transferred knowledge and knowledge to distill: Given a downstream task, and a teacher and a student model, we formally define the knowledge to distill as the unique information in the teacher (Definition 3.1) and the transferred knowledge as the redundant information (Definition 3.2). Through examples and theoretical results (Theorem 3.1), we first show that redundant information succinctly captures task-related knowledge transferred to the student as opposed to the existing frameworks which directly align teacher ($T$) and student ($S$) representations, e.g., Variational Information Distillation (VID) [3] maximizes the mutual information $I(T;S)$. Theorem 3.1 points out a fundamental limitation of the existing knowledge distillation frameworks for capacity-limited students: they blindly align the student and the teacher without precisely capturing the task-related knowledge.

•  Maximizing redundant information as a regularizer: To alleviate this limitation, we propose a strategy to incorporate redundant information as a regularizer during model distillation to precisely maximize the task-relevant transferred knowledge. We first circumvent the challenge of computing the redundant information measure proposed in [12] by utilizing the quantity termed intersection information defined in [13], which we prove (in Theorem 4.1) to be a lower-bound for redundant information. The significance of Theorem 4.1 is that it enables us to obtain an optimization formulation to maximize a lower-bound of redundant information without making distributional assumptions, a contribution that is also of independent interest outside the domain of knowledge distillation.

•  A novel knowledge distillation framework: We propose a new framework called Redundant Information Distillation (RID) whose distillation loss is tailored to maximize the redundant information (i.e., the transferred knowledge as per Definition 3.2). We carry out a number of experiments to demonstrate the advantage of this new framework over VID [3], an existing knowledge distillation framework that maximizes the mutual information between the teacher and the student representations (Section 5). Experiments are carried out for the CIFAR10 and the CIFAR100 datasets, as well as for a transfer learning setting where the teacher is trained on the ImageNet dataset and transferred to the students over the CUB-200-2011 dataset. Our framework explains knowledge distillation and shows more resilience under less-informative nuisance teachers.

Related Works: Multi-layer knowledge distillation was introduced in FitNets [2]. Henceforth, a large number of techniques, based on different statistics derived for matching a teacher-student pair, have been proposed. In particular, [3, 4, 14, 15] leverage an information-theoretic perspective to arrive at a solution (also see surveys [6, 7]). In this paper, we focus on VID [3] as a representative framework of the larger class of distillation frameworks which maximize $I(T;S)$ as the distillation strategy. We also discuss Task-aware Layer-wise Distillation (TED) [5] as a framework that filters out task-related information. Specifically, [5] highlight the importance of distilling only the task-related information when there is a significant complexity gap between the teacher and the student. Towards this end, [16] point out the existence of non-distillable classes due to the unmatched capacity of the student model. We discuss some more related works on knowledge distillation [17, 18, 19, 20, 21] in Appendix A.

Information theory has been instrumental towards explaining the success of knowledge distillation. [9] utilize information bottleneck principles [22, 23] to explain how a teacher model may assist the student in learning relevant features quickly. [8] observe the training process as systematically discarding knowledge from the input. Accordingly, the distillation helps the student to quickly learn what information to discard. Despite these attempts, we observe that there exists a gap in characterizing the fundamental limits of knowledge distillation which we seek to address using the mathematical tool of PID.

PID is also beginning to generate interest in other areas of machine learning [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 35, 39, 40, 41]. However, it has not been leveraged in the context of knowledge distillation before. Additionally, while most related works predominantly focus on efficiently computing PID, e.g., [40, 28, 36, 42] that itself requires solving an optimization over the joint distribution, there are limited works that further incorporate it as a regularizer during model training. [25] leverage Gaussian assumptions to obtain closed-form expressions for the PID terms, enabling them to use unique information as a regularizer during training for fairness (also see [38, 35] for more details on Gaussian PID). Our work makes novel connections between two notions of redundant information and demonstrates how PID can be integrated as a regularizer in a multi-level optimization without Gaussian assumptions, which could also be of independent interest outside the context of knowledge distillation.

Background on PID: Partial Information Decomposition (PID), first introduced by [10], offers a way to decompose the joint information in two sources, say $T$ and $S$, about another random variable $Y$

(i.e., $I(Y;T,S)$ where $I(A;B)$ denotes the mutual information between $A$ and $B$ [43]) into four components as follows:

1. 1.

   Unique information $Uni(Y:T\backslash S)$ and $Uni(Y:S\backslash T)$: information about $Y$ that each source uniquely contains

2. 2.

   Redundant information $Red(Y:T,S)$: the information about $Y$ that both $T$ and $S$ share

3. 3.

   Synergistic information $Syn(Y:T,S)$: the information about $Y$ that can be recovered only by using both $T$ and $S$.

See Figure 2 for a graphical representation. These PID components satisfy the relationships given below:

While this system of equations cannot be solved to arrive at a deterministic definition for each PID term, defining only one of the terms is sufficient to define the rest. Consequently, a wide array of definitions exists, each based on different desired properties [10, 12, 11, 13]. Among these, the definition proposed in [12] is motivated with an operational interpretation of unique information from decision theory. Moving on to the context of knowledge distillation, we map $T$ to be the teacher representation, $S$ to be the student representation, and $Y$ to be the downstream task that the student is being trained for. That makes $I(Y;T)$ and $I(Y;S)$ be the total knowledge about $Y$ that is in the teacher and in the student, respectively.

Notation and Problem Setting: We consider a layer-wise distillation scheme where the teacher representation $T(X)$ is distilled into the student representation $S_{\eta_{s}}(X)$, where $X$ is the input. The target of the student is to predict the task $Y$ from $X$. Both $T(\cdot)$ and $S_{\eta_{s}}(\cdot)$ are deterministic functions of $X$ and the randomness is due to the input being random. Note that the student representation depends on the parameters of the student network denoted by $\eta_{s}$ and hence written as $S_{\eta_{s}}$. However, when this parameterization and dependence on $X$ is irrelevant or obvious, we may omit both and simply write $T$ and $S$. We denote the supports of $Y,T$ and $S$ by $\mathcal{Y},\mathcal{T}$ and $\mathcal{S}$, respectively. In general, upper-case letters denote random variables, except $P$ and $Q$ which represent probability distributions, $C,H,W$ which stand for the representation dimensions, and $K$ which indicates the number of layers distilled. Lowercase letters are used for vectors unless specified otherwise. Lowercase Greek letters denote the parameters of neural networks.

Knowledge distillation is achieved by modifying the student loss function to include a distillation loss term in addition to the ordinary task-related loss as follows:

Here, $\lambda_{1},\lambda_{2}>0$. When the task at hand is a classification, $Y$ denotes the true class label and we use the cross entropy loss defined as $\mathcal{L}_{CE}(Y,\hat{Y})=-\mathbb{E}_{P_{X}}\left[\log P_{\hat{Y}(X)}(Y)\right]$ as the ordinary task-related loss for the student. Here, $\hat{Y}(X)$ is the student’s final prediction of $Y$. The teacher network is assumed to remain unmodified during the distillation process.

In this section, we propose information theoretic metrics to quantify both the task-relevant information that is available in the teacher for distillation, and the amount of information that has already been transferred to the student. We mathematically demonstrate favorable properties of our proposed measures in comparison to other candidate measures. Our mathematical results highlight the limitations of existing knowledge distillation frameworks that naively align the student with the teacher with no regard for task-relevance.

With the knowledge to distill is defined as the unique information $Uni(Y:T\backslash S)$, we see that the more the distillation happens, the more the $Uni(Y:T\backslash S)$ shrinks. Note that under the knowledge distillation setting, the total knowledge of the teacher $I(Y;T)$ is constant since the teacher is not modified during the process. Since $I(Y;T)=Uni(Y:T\backslash S)+Red(Y:T,S)$ we therefore propose $Red(Y:T,S)$ as a measure for knowledge that has already been transferred.

We leverage the unique and redundant information definitions given by [12] for an exact quantification of these quantities.

Comparison to Existing Approaches for Knowledge Distillation: A multitude of knowledge distillation frameworks exist which are based on maximizing the mutual information between the teacher and the student (i.e., $I(T;S)$) [3, 4, 14, 15]. While a distillation loss that maximizes $I(T;S)$ can be helpful to the student when the teacher possesses task-related information, we show that it creates a tension with the ordinary loss when the teacher has little or no task-relevant information. Moreover, even though the teacher contains task-related information, the limited capacity of the student may hinder a proper distillation when this kind of framework is used. The following examples provide critical insights, exposing the limitation of $I(T;S)$. Our proposed measure $Red(Y:T,S)$ resolves these cases in an explainable manner by succinctly capturing task-relevant knowledge.

Example 1: (Uninformative teacher) An uninformative teacher representation (i.e., $T$ with $I(Y;T)=0$) gives $Uni(Y:T\backslash S)=Red(Y:T,S)=0$ for any $S$, agreeing with the intuition. Hence, an algorithm that maximizes exactly the transferred knowledge $Red(Y:T,S)$ will have a zero gradient over this term. In contrast, algorithms that maximize the similarity between $S$ and $T$ quantified by $I(T;S)$ will force $S$ to mimic the uninformative teacher, causing a performance worse than ordinary training without distillation. As a simplified example, let $U_{1},U_{2}\sim Ber(0.5)$ and $Y=U_{1},T=U_{2}$. Then, the teacher cannot predict the intended task $Y$. Note that, in this case, $I(T;S)$ is not maximized when the student representation is $S=Y$. Instead, it is maximized when $S=U_{2}$.

Example 2: (Extra complex teacher) Let $U_{1}\sim Ber(0.2),U_{2}\sim Ber(0.5)$ and $Y=U_{1},T=(U_{1},U_{2})$. Then, the teacher can completely predict the intended task $Y$. Assume the student is simpler than the teacher and has only one binary output. In this situation, $I(T;S)$ is not maximized when $S=U_{1}$ because $I((U_{1},U_{2});U_{1})\approx 0.72<1=I((U_{1},U_{2});U_{2})$ where the right-hand side is achieved when $S=U_{2}$. However, $S=U_{1}$ is a maximizer for $Red(Y:T,S)$ (i.e., $Red(Y:T,S)=Red(U_{1}:T,U_{1})=I(Y;T)$). Theorem 3.1 presents a more general case.

Theorem 3.1 formally exposes the limitations of maximizing $I(T;S)$ for capacity-limited students as $I(T;S)$ does not emphasize task-related knowledge.

The uninformative random variable $G$ here can be seen as a stronger version of the nuisance defined in [44, Section 2.2]. In the above scenario, the task-related part of the student loss will have a tension with the distillation loss when $H(Z)<H(G)$, in which case, the distillation actually adversely affects the student. In contrast, a loss that maximizes $Red(Y:T,S)$ will always be aligned with the task-related loss.

These examples show that the frameworks based on maximizing $I(T;S)$ are not capable of selectively distilling the task-related information to the student. In an extreme case, they are not robust to being distilled from a corrupted teacher network. This is demonstrated in the experiments under Section 5.

It may appear that using information gain (conditional mutual information) $I(Y;T|S)=I(Y;T,S)-I(Y;S)$ as the measure for the amount of knowledge available to distill resolves the cases similar to Example 1. However, Example 3 below provides a counterexample.

Example 3: (Effect of synergy) Consider a scenario similar to Example 1, where the teacher is uninformative regarding the interested task. For example, let $U_{1},U_{2}\sim Ber(0.5)$ and $Y=U_{1},T=U_{1}\oplus U_{2}$ where $\oplus$ denotes the binary XOR operation. Suppose we were to consider conditional mutual information $I(Y;T|S)$ as the measure of the amount of knowledge available to distill information available in the teacher. Then, $I(Y;T|S)=H(Y)$ when $S=U_{2}$, indicating non-zero knowledge available to distill in the teacher. This is un-intuitive since in this case both $I(Y;T)=I(Y;S)=0$ and neither the teacher nor the student can be used alone to predict $Y$. In contrast, the proposed measures give $Uni(Y:T\backslash S)=Red(Y:T,S)=0$ indicating no available or transferred knowledge.

Next, we present Theorem 3.2 which highlights some important properties of the proposed metrics. These properties indicate that the proposed measures agree well with our intuition for explaining distillation.

In this section, we propose a distillation framework – Redundant Information Distillation (RID) – which maximizes the transferred knowledge $Red(Y:T,S)$ with a focus on classification problems. We first show that our measure of transferred knowledge is lower-bounded by an alternative definition of redundant information (also called the intersection information – denoted by $Red_{\cap}(Y:T,S)$). Next, we leverage this lower-bound to develop a multi-level optimization framework to selectively distill task-relevant knowledge.

Next we show, from Theorem 4.1, that $Red_{\cap}(Y:T,S)$ is a lower-bound for $Red(Y:T,S)$. The significance of this result is that maximizing the lower-bound $Red_{\cap}(Y:T,S)$ would also increase $Red(Y:T,S)$.

Next, we discuss our proposed approach for maximizing the lower-bound $Red_{\cap}(Y:T,S)$ (also see Figure 3 and Algorithm 1 for our complete strategy). Our proposed framework is based on selecting $Q$ in Definition 4.1 to be $Q=f_{t}(T)$, and parameterizing $f_{t}(\cdot)$ and $f_{s}(\cdot)$ using small neural networks. To denote the parameterization, we will occasionally use the elaborated notation $f_{t}(\cdot;\theta_{t})$ and $f_{s}(\cdot;\theta_{s})$, where $\theta_{t}$ and $\theta_{s}$ denote the parameters of $f_{t}$ and $f_{s}$, respectively. With the substitution of $Q=f_{t}(T)$, Definition 4.1 results in the optimization problem given below:

We divide the problem (P1) into two phases and employ gradient descent on two carefully designed loss functions to perform the optimization. In the first phase, we maximize the objective w.r.t. $\theta_{t}$ while $\theta_{s}$ and $S$ are kept constant (recall that $T$ is fixed in all cases because the teacher is not being trained during the process). For this, we append an additional classification head $g_{t}(\cdot;\phi_{t})$ parameterized by $\phi_{t}$ to the teacher’s task aware filter $f_{t}$. We then minimize the following loss function with respect to $\theta_{t}$ and $\phi_{t}$:

where $V_{c,h,w}$ denotes the corresponding element of $V=f_{t}(T(X);\theta_{t})-f_{s}(S(X);\theta_{s})\in\mathbb{R}^{C\times H\times W}$. Here, $C,H,$ and $W$ are the number of channels, height, and width of the outputs of $f_{s}$ and $f_{t}$. $\sigma=[\sigma_{1},\dots,\sigma_{C}]^{T}$ is a stand-alone vector of weights that are optimized in the second phase. Minimizing the cross-entropy term $\mathcal{L}_{CE}(Y,g_{t}(f_{t}(T;\theta_{t});\phi_{t}))$ of $\mathcal{L}_{t}(\theta_{t},\phi_{t})$ above amounts to maximizing $I(Y;f_{t}(T;\theta_{t}))$. The second term prohibits $f_{t}(T)$ from diverting too far from $f_{s}(S)$ during the process, so that the constraint $I(Y;f_{t}(T;\theta_{t})|f_{s}(S;\theta_{s}))=0$ can be ensured.

During the second phase, we freeze $\theta_{t}$ and maximize the objective over $\theta_{s},S_{\eta_{s}}$ and $\sigma$. The loss function employed in this phase is as follows:

where

and $\lambda_{1}$ and $\lambda_{2}$ are scalar hyperparameters which determine the prominence of ordinary learning (through $\mathcal{L}_{CE}(Y,\hat{Y}_{\eta_{s}})$) and distillation (through $\mathcal{L}_{s}(\theta_{s},\sigma,\eta_{s})$). $V$ and $\sigma$ are as defined earlier. $\hat{Y}_{\eta_{s}}$ denotes the final prediction of the student network.

The first term of the loss function is the ordinary task-related loss. The next two terms correspond to the distillation loss, which is our focus in the following explanation. Consider phase 2 as an estimation problem that minimizes the $\sigma$-weighted mean squared error, where $Q=f_{t}(T)$ is the estimand and $f_{s}(\cdot)$ is the estimator. The magnitudes of the positive weights $\sigma$ are controlled using the term $||\sigma||^{2}$. We observe that this optimization ensures $I(Y;Q|f_{s}(S))=0$ given that the following assumption holds.

Assumption: Let the estimation error be $\epsilon=f_{t}(T)-f_{s}(S)$. Assume $I(\epsilon;Y|f_{s}(S))=0$, i.e., given the estimate, the estimation error is independent of $Y$.

With the above assumption, we see that

Therefore, the constraint in problem P1 is satisfied by this selection of random variables. Therefore, along with the maximization of $I(Y;Q)$ during phase 1, the proposed framework can be seen as performing the optimization in Definition 4.1 in two steps.

This, along with Theorem 4.1, completes our claim that the proposed framework maximizes a lower bound for the transferred knowledge. RID loss terms can be extended to multiple layers as follows: Replace the single term $\mathcal{L}_{t}(\theta_{t},\phi_{t})$ in (10) with a sum of terms $\sum_{k=1}^{K}\mathcal{L}_{t}^{(k)}(\theta_{t}^{(k)},\phi_{t}^{(k)})$ corresponding to each pair of layers $T^{(k)}$ and $S_{\eta_{s}}^{(k)}$ where$k=1,\dots,K$. Similarly, replace $\mathcal{L}_{s}(\theta_{s},\sigma,\eta_{s})$ in (12) with $\sum_{k=1}^{K}\mathcal{L}_{s}^{(k)}(\theta_{s}^{(k)},\sigma^{(k)},\eta_{s})$. The framework is summarized in Algorithm 1. The advantage of RID over the VID framework [3] which maximizes $I(T;S)$ can be observed in the experiments that are detailed in Section 5.

Setup: We compare the performance of the proposed RID framework with (i) VID; and (ii) our adaptation of TED in this domain under two different conditions. In the first setting, the teacher network is fully trained with the complete training set, whereas in the second setting, the teacher is just randomly initialized without any training at all. Experiments are carried out on CIFAR-10 [45], CIFAR-100 [46], ImageNet [47] and CUB-200-2011 [48] datasets (with the last two being used in a transfer learning setup). Additionally, we train a student without any knowledge distillation, which we label as “BAS”. Table 1 provides details of the model architectures used in each setting.

We distill three layers from the teacher to the corresponding student layers. In the case of RID, each teacher layer $T^{(k)}$ has its own filter $f_{t}^{(k)}$ parameterized with $\theta_{t}^{(k)}$. Student filters are parameterized in a similar manner. Moreover, each teacher filter $f_{t}^{(k)}(\cdot)$ has its own classification head $g^{(k)}_{t}(\cdot)$ parameterized with $\phi^{(k)}$. All the student representations are parameterized by the complete weight vector $\eta_{s}$. In the beginning, the teacher filters are trained for $n_{w}$ number of warm-up epochs with just the cross-entropy loss $\sum_{k=1}^{K}\mathcal{L}_{CE}(Y,g_{t}^{(k)}(f_{t}^{(k)}(T^{(k)};\theta_{t}^{(k)});\phi_{t}^{(k)}))$. Then, the optimization alternates between the first and the second phases, with each cycle taking $q$ epochs in total. Within a cycle, phase 1 is carried out for $r\times q$ epochs followed by phase 2 for rest of the epochs (see Algorithm 1). Values of all the hyperparameters are given in Appendix D. TED is implemented similarly, except the fact that now, the student filters also have a classification head $g_{s}^{(k)}(\cdot)$ (see Appendix C). Since TED acts as a fine-tuning method, we use the BAS student as the base model to apply TED on.

In order to evaluate the validity of the Definitions 3.1 and 3.2, we compute the PID components [12] of the joint mutual information $I(Y;T,S)$ of the innermost distilled layer using the estimation method given in [28]. See Appendix D for additional details.

Results: Figure 4 shows the classification accuracy on CIFAR-10 dataset for each student model RID, VID, and TED when distilled with either a trained teacher or an untrained one. It also shows the classification accuracies of the baseline model BAS (without any distillation) and the trained teacher. Figures 5(a) and 5(b) present the PID values and the marginal mutual information estimates for the teacher and the student models for CIFAR-10 and CIFAR-100, respectively. Tables D.4 and D.4 in Appendix D present the classification accuracies for CIFAR-100 and the transfer learning setup ImageNet $\rightarrow$ CUB-200-2011, respectively.

Discussion: From Figure 4, we see that both RID and VID perform equally well when the teacher is trained. However, the performance degradation of VID under the untrained teacher is significantly larger w.r.t. that of the RID. This can be attributed to the fact that the VID student trying to naively mimic the teacher. RID leads to more resilient and effective distillation under nuisance teachers as it succinctly quantifies task-relevant knowledge rather than simply aligning student and teacher representations. TED, despite being rather unstable, performs very close to the baseline model in both cases. As evident from Figure 5, in the case of the trained teacher (whose $I(Y;T)>0$), we observe that the knowledge available for distillation $Uni(Y:T\backslash S)$ in the teacher decreases with the increasing number of epochs. Consequently, the amount of knowledge transferred $Red(Y:T){S}$ increases. When the teacher is not trained (i.e., $I(Y;T)=0$), $Uni(Y:T\backslash S),Red(Y:T,S)\approx 0$ as expected. Both BAS and RID models show an increase in $I(Y;S)$ even when distilled from the untrained teacher. However, in this case, VID shows a very low $I(Y;S)$ as expected, caused by the distillation loss forcing to mimic the teacher. A Python implementation of the experiments is available at https://github.com/pasandissanayake/kd-rid.

With the growing interest in knowledge distillation, our work provides critical insights into the explainability of knowledge distillation. We propose using $Uni(Y:T\backslash S)$ to quantify the knowledge available in a teacher model for distillation w.r.t. a student and a downstream task. This, in turn, leads to the definition of the amount of knowledge that has been distilled to a student as $Red(Y:T,S)$. We show that knowledge distillation frameworks which use mutual information between the teacher and the student representations to achieve knowledge distillation have a fundamental problem: These frameworks force the student to mimic the teacher regardless of the usefulness of the teacher’s information to perform the task at hand. In contrast, through many examples we demonstrate that the proposed metrics can correctly characterize the amounts of knowledge available for distillation and the already transferred knowledge. Moreover, we show the advantage of the proposed metric by implementing a new distillation framework – Redundant Information Distillation (RID) – and comparing its performance with the existing technique VID [3]. While VID and RID perform similarly when the teacher is well-trained for the downstream task, VID performance degrades largely when the teacher is not trained but RID performs close to a student model trained without distillation.

Limitations and future work: While the RID framework uses an alternative definition for redundant information, computation of exact $Red(Y:T,S)$ during training can be computationally prohibitive due to the optimization over $\Delta_{P}$. Moreover, characterizing the extent to which the assumption in Section 4 holds is not explored in this work. Extending the mathematical formulation in Section 4 to analyze other knowledge distillation frameworks is an interesting path for future research. Other potential research directions include: (i) distilling from an ensemble of teachers [49] in a way that the adverse effects of corrupted teachers are mitigated; (ii) dataset distillation [50]; or (iii) distillation for model reconstruction from counterfactual explanations [51]. Incorporating fundamentally different definitions for PID components, such as [41] which provides explicit formulae, as regularizers can also be interesting.

Acknowledgments: This work was supported in part by NSF CAREER Award 2340006 and Northrop Grumman Seed Grant.