AMLNet: Adversarial Mutual Learning Neural Network for Non-AutoRegressive Multi-Horizon Time Series Forecasting

We conducted experiments on four publicly available real-world datasets: Sanyo [20], Hanergy [21], Solar [22], and Electricity [23]. The Sanyo and Hanergy datasets consist of solar power generation data from two PV plants in Australia. The data for Hanergy spans from 01/01/2011 to 31/12/2016 (6 years), while the data for Sanyo covers the period from 01/01/2011 to 31/12/2017 (7 years). The Solar dataset comprises solar power data from 137 PV plants in Alabama, USA, gathered between 01/01/2006 and 31/08/2006. The Electricity dataset contains electricity consumption data from 370 households, recorded from 01/01/2011 to 07/09/2014.

A summary of the data statistics is provided in Table I. For the Sanyo and Hanergy datasets, we considered data between 7 am and 5 pm and aggregated it at half-hourly intervals. Additionally, weather and weather forecast data were collected and used as covariates in the experiments (refer to [24] for further details). For the Solar and Electricity datasets, the data was aggregated into 1-hour intervals. Following the approach in [7, 24], calendar features were incorporated based on the granularity of each dataset. Specifically, Sanyo and Hanergy datasets used features such as month, hour-of-the-day, and minute-of-the-hour, Solar dataset used month, hour-of-the-day, and age, and Electricity dataset used month, day-of-the-week, hour-of-the-day, and age. For consistent preprocessing, all data was normalized to have zero mean and unit variance [24].

Given is: 1) a set of $N$ univariate time series (solar or electricity series) $\{\mathbf{Y}_{i,1:T_{l}}\}^{N}_{i=1}$, where $\mathbf{Y}_{i,1:T_{l}}\coloneqq[{y}_{i,1},{y}_{i,2},...,{y}_{i,T_{l}}]$, $T_{l}$ is the input sequence length, and ${y}_{i,t}\in\Re$ is the value of the $i$th time series (generated PV solar power or consumed electricity) at time $t$; 2) a set of associated time-based multi-dimensional covariate vectors $\{\mathbf{X}_{i,1:T_{l}+T_{h}}\}_{i=1}^{N}$, where $T_{h}$ denotes the length of the forecasting horizon. Our goal is to predict the future values of the time series $\{\mathbf{Y}_{i,T_{l}+1:T_{l}+T_{h}}\}^{N}_{i=1}$, i.e. the PV power or electricity usage for the next $T_{h}$ time steps after $T_{l}$.

The covariates for the Sanyo and Hanergy datasets include: weather data $\{\mathbf{W1}_{i,1:T_{l}}\}_{i=1}^{N}$, weather forecasts $\{\mathbf{WF}_{i,T_{l}+1:T_{l}+T_{h}}\}_{i=1}^{N}$ and calendar features $\{\mathbf{Z}_{i,1:T_{l}+T_{h}}\}_{i=1}^{N}$, while the covariates for Solar and Electricity datasets include only calendar features.

Specifically, AMLNet produces the probability distribution of the future values, given the past history:

where the input of model at step $t$ is the concatenation of ${y}_{i,t-1}$ and ${x}_{i,t}$.

NAR forecasting models [15, 6, 5, 19] directly eliminate AR connection from the decoder side, instead modeling separate conditional distributions for each prediction independently. Unlike AR models, NAR models enable parallelized training and inference processes. However, NAR models can produce disjointed forecasts and introduce discontinuities [16] due to the erroneous assumption of independence, limiting their ability to capture interdependencies among predictions. AST [5] stands as the sole approach addressing this within the NAR framework, utilizing adversarial training to enhance global perspective. Recent NAR models focused on reducing output space interdependence have primarily emerged in the realm of Natural Language Processing (NLP) tasks [12, 25]. Various strategies have emerged, with KD [26, 27] garnering substantial attention. KD effectively transfers knowledge from a larger teacher to a smaller student network by offering softer, more informative target distributions. KD methods for NAR either distill the prediction distribution of a pre-trained AR teacher model [12] or incorporate hidden state patterns of the AR model [28].

Classic KD methods are offline and can incur computational and memory overhead due to the reliance on powerful pre-trained teachers. In response, online KD techniques [29, 30, 31, 32] have emerged, showing superior results. These methods treat all networks as peers, enabling mutual exchange of output information and requiring less training time and memory. DML [29] introduced collaborative training, where each model can be both a student and a teacher. Further advancements, such as Wu and Gong’s work [32], assemble teachers into online KD, enhancing generalization. Notably, online KD techniques can capture intermediate feature distributions through adversarial training, as seen in AFD [30] and AMLN [31].

Generative Adversarial Networks (GANs) [33], comprising a generator $G$ and a discriminator $D$ engaged in adversarial training, were initially proposed for sample generation. However, the adversarial training paradigm has found applications in diverse domains, including computer vision, NLP [30, 31], and time series forecasting [34, 5].

In contrast to prior work, our AMLNet introduces several advancements: 1) We pioneer the application of online KD for forecasting, introducing AMLNet as the first model to employ online KD methods for training a NAR forecasting model to capture target sequence interdependencies. Specifically, AMLNet trains a deep AR and a deep NAR model mutually as ensemble teachers before transferring their knowledge to a shallow NAR student. 2) While Wu and Gong [32] construct ensemble teachers and adjust KD loss weights based on training epoch number, they overlook teacher model instability during training. AMLNet utilizes outcome-driven KD, assigning dynamic weights to KD losses based on teacher model performance, specifically tailored for probabilistic forecasting. 3) We address the issue of discontinuous predictions stemming from NAR model hidden states, proposing hint-driven KD to capture hidden state distribution information. Unlike previous approaches [30, 31], designed for networks with differing layer counts, our method is tailored to AMLNet’s architecture.

The central components of AMLNet, as depicted in Figure 1, encompass a shared encoder $f_{\theta_{e}}$ consisting of $n_{e}$ layers, a deep AR Peer1 (P1) decoder $f_{\theta_{P1}}$ with $n_{P}$ hidden layers, a deep NAR Peer2 (P2) decoder $f_{\theta_{P2}}$ also possessing $n_{P}$ hidden layers, and a shallow NAR Student (S) decoder $f_{\theta_{S}}$ equipped with $n_{S}$ hidden layers. It’s noteworthy that Informer [19] serves as the foundational framework for AMLNet, although other deep learning forecasting models can replace Informer.

To harness temporal patterns from historical data, the encoder extracts insights from past time steps by processing the concatenation of covariate $x_{t}$ and ground truth $y_{t}$ as input at time step $t$:

The shared encoder temporal patterns $h_{e;1:T_{l}}$ are uniformly leveraged across all decoders, exploiting the consistent input pasting sequence. This shared approach significantly reduces network parameters and computational overhead.

The P1, P2 and S decoders are formulated:

where $T_{d}e$ is the length of start token used by Informer [19], and the prediction $y$ consists of a mean and variance. NAR models without input sequences for the decoder have poor performance and copying input pasting series to the decoder as its input could enhance model performance [12].

Notably, the AR model excels at capturing output space interdependence, whereas the NAR model is adept at mitigating error propagation. Through a mutually beneficial relationship, the deep AR P1 and NAR P2 decoders coalesce as peers, leveraging each other’s strengths to augment their individual abilities. This collective knowledge is then harnessed to train the shallow NAR S decoder, effectively establishing a dynamic ensemble, as illustrated in Figure 1.

Conventional optimization objective of probabilistic time series forecasting is the Negative Log Likelihood (NLL) loss, denoted as $\mathcal{L_{NLL}}$. This loss function, prevalent in training probabilistic forecasting models [8, 7], is formally defined in Eq. (6):

where $y_{T_{l}+1:T}$ represents the ground truth, while $\hat{y}_{T_{l}+1:T}$ pertains to the predicted distribution, encompassing the mean ${\mu}t$ and standard deviation $\sigma{t}$ across the forecasting horizon.

Addressing the inherent limitations of both autoregressive (AR) and non-autoregressive (NAR) models when subjected to the NLL loss, we propose a novel approach by enlisting the counterpart model as a peer network. This peer relationship serves as a guiding mechanism to overcome the respective limitations, approximating both the ground truth and the predicted distribution of their teacher network.

In the realm of online Knowledge Distillation (KD), traditional methodologies involve aggregating prediction and KD losses with fixed predefined proportions [29] or gradually increasing the proportion of KD loss over training epochs [32]. However, these approaches overlook the diverse abilities of the peer networks and the varying quality of their predictions during training. Consequently, the contribution of each peer network to the student network should be weighted according to its performance.

By allocating a constant weight to the KD loss irrespective of the peers’ performance, inaccurately predicted distributions could propagate errors within the network, limiting overall performance. Conventional remedies, such as removing misclassified data for offline KD, are unsuitable for online distillation or forecasting tasks due to reduced training data.

To address these challenges, we introduce an attention-based KD loss that assigns higher weights to well-forecasted samples. The KD loss functions for the P1, P2, and S decoders are formulated in Eq. (8), (9), and (10). P1 and P2 aim to mimic each other’s predicted distributions, while S simultaneously distills knowledge from the outputs of both P1 and P2. The Kullback Leibler (KL) divergence serves as a measure of discrepancy between predicted distributions. Given distributions $P$ and $Q$, their KL divergence is defined in Eq. (7).

In these equations, $\alpha_{o}$ modulates the weight of the outcome-driven KD loss, while the weight $\omega_{e}(\cdot)$ captures the importance of the teacher model’s predictions. Considering a Gaussian distribution of data, we define $\omega_{e}(\cdot)$ as follows:

where $\mu_{T_{l}+1:T}$ and $\sigma_{T_{l}+1:T}$ are mean and standard deviation of teacher predicted distribution. The weight $\omega_{e}(\cdot)$ ranges from 0 to 1 and changes during training. A higher weight signifies a more accurate teacher prediction, optimizing the student network’s approximation to the teacher’s outputs.

Time series data exhibit a natural continuity, yet NAR forecasting models often yield discontinuous and unrelated predictions [16]. This disconnect arises due to these models ignoring output interdependence and position information, and we trace this phenomenon to the behavior of hidden states. To illustrate this, we conducted a case study where an AR model outperformed an NAR model. In Figure 2 (d), we display a smooth and continuous PV power measurement from the Sanyo dataset alongside forecasts from an AR model (DeepAR), an NAR model (Informer), and our proposed AMLNet. Notably, the trajectory of DeepAR is noticeably more continuous and smooth than that of the NAR models. To quantify this observation, we employed the Dynamic Time Warping (DTW) algorithm [35] to measure the similarity between two series. Specifically, we computed the DTW distance and Mean Absolute Percentage Errors (MAPEs) between the DeepAR prediction and the ground truth, yielding 3.05 and 0.102 kW, respectively. In contrast, the Informer yields MAPEs and DTW values of 4.91 and 0.143 kW, indicating that the DeepAR forecasts align more closely with the ground truth.

We visualize the cosine distances between hidden states at the last hidden layer in Figure 2, where each pixel at row $i$ and column $j$ represents the distance between hidden states $h_{i}$ and $h_{j}$ at steps $i$ and $j$. Lighter colors correspond to lower distances. Given two hidden states $h_{i}$ and $h_{j}$, their cosine distance is calculated as $1-\frac{h_{i}\cdot h_{j}}{||h_{i}||\cdot||h_{j}||}$. Notably, the average cosine distance between DeepAR hidden states $h_{i}$ and their six closest neighbors is 0.016, while the corresponding value for Informer is 0.084. Clearly, the cosine distances of DeepAR hidden states are substantially lower, indicating greater similarity to their neighbors. Such similarity suggests that the hidden states of DeepAR generate predictions with consistent and gradual variations, yielding predictions that are more continuous and smooth. From our analysis, several key observations emerge:

• •

  AR models exhibit similar hidden states patterns, while NAR models lack this property.

• •

  Dissimilar hidden states could lead to discontinuous predicted trajectory.

• •

  Hidden states hold meaningful information, and distilling knowledge from them could be beneficial [31, 30].

Thus, we capitalize on the hidden state patterns through online KD to generate continuous predictions while utilizing their inherent information. Unlike offline KD methods which regularize the distance between pairs of hidden states [28], the hidden states of online KD models exhibit greater variability compared to model outputs during training [30]. Direct regularization in this context impairs the learning process and fails to guarantee convergence. To address this, we adopt an adversarial training approach to extract distributional information from hidden states.

In our approach, Peer 1 (P1) learns from Peer 2 (P2) to counter error accumulation, while P2 learns output interdependence from P1. The Student (S) inherits abilities from both P1 and P2. Given a shared encoder, we focus solely on the decoders’ hidden states. The adversarial training involves two components: a generator producing feature mappings and a discriminator classifying these mappings. Each decoder layer serves as a generator for feature mappings, and each P1 and P2 layer is paired with a discriminator acting as a classifier (refer to Figure 1).

The discriminators receive hidden states $h\in\Re^{T_{h}\times d_{hid}}$ and output probabilities ranging between 0 (fake) and 1 (real). A discriminator’s architecture consists of a sequence of ConvLayer-BatchNorm-LeakyReLU-ConvLayer-LinearLayer-Sigmoid operations. The initial ConvLayer has an output dimension of 16, stride of 2, and kernel size of 3, while the second ConvLayer has an output dimension of 1, stride of 1, and kernel size of 3.

In our training process, the generators aim to fool discriminators by approximating the distribution of hidden states from their teacher networks. Discriminators, on the other hand, strive to distinguish the origin of hidden states. Specifically, we denote the $i$th layer of P1, P2, and S decoders as generators $G_{i,P1}$, $G_{i,P2}$, and $G_{i,S}$, which produce hidden states $h_{i,P1}$, $h_{i,P2}$, and $h_{i,S}$, respectively. The $i$th P1 discriminator $D_{i,P1}$ is trained to classify P1-generated mappings as real (output: 1) and P2 or S-generated mappings as fake (output: 0). Analogously, the $i$th P2 discriminator $D_{i,P2}$ distinguishes P2-generated mappings as real and P1 or S-generated mappings as fake. Its parameters $G_{i,P1}$ is optimised by minimising the hidden states-driven KD loss $\mathcal{L}_{i,P1}(h_{i,P1})$:

where hyperparameter $\alpha_{h}$ controls the weight of hint-driven KD loss. Similar, $i$th P2 layer $G_{i,P2}$ minimises the KD loss:

For hint-driven KD, the student network’s shallow layers imitate low-level features from teacher networks, while deep layers learn higher-level features. In the context of the S decoder with fewer layers, we enable each shallow S layer to acquire knowledge from multiple deep network layers. More specifically, the $i$th shallow S network layer learns features from the $j=[1+(i-1)\times\lfloor\frac{n_{d}-1}{n_{S}-1}\rfloor,min(\lceil\frac{n_{d}-1}{n_{S}-1}\rceil+(i-1)\times\lfloor\frac{n_{d}-1}{n_{S}-1}\rfloor,n_{d})]$th layers of the teachers. Notably, the term $min(\lceil\frac{n_{d}-1}{n_{S}-1}\rceil+(i-1)\times\lfloor\frac{n_{d}-1}{n_{S}-1}\rfloor,n_{d})$ ensures that $j$ does not exceed the depth of the teacher network.

Consequently, each $i$th shallow S network layer ($G_{i,S}$) distills knowledge from both P1 and P2 decoders, attempting to deceive corresponding discriminators ($D_{j,P1}$ and $D_{j,P2}$) for $j$ within the specified range. The KD loss for the $i$th S layer is formulated in Eq. (14). Here, $\alpha_{h}$ governs the weight of hint-driven KD loss.

In parallel, the discriminators are trained to classify the origin of hidden states. The $i$th P1 discriminator ($D_{i,P1}$) distinguishes features generated by the $i$th P1 layer as real (output: 1), and P2 or S-generated features as fake (output: 0). Similarly, the $i$th P2 discriminator ($D_{i,P2}$) discriminates P2-generated features as real and P1 or S-generated features as fake. The loss functions for the P1 and P2 discriminators are defined in Eq. (15) and (16), respectively. Here, $k$ represents the indexes of shallow NAR layers endeavoring to deceive the $i$th discriminator. For example, the first and second S layers aim to deceive the first P1 discriminator in Figure 1, resulting in $D_{1,P1}$ having $k={1,2}$.

The optimization process for AMLNet involves minimizing the forecasting loss, as well as the outcome-driven and hint-driven KD losses. During each training iteration, the encoder and P1 and P2 decoders are optimized first by minimizing Equations (17) and (18). Subsequently, the S decoder is optimized by minimizing Equation (19). Lastly, each discriminator associated with the $i$th P1 or P2 layer is optimized by minimizing Equation (15) or (16). During testing, only the encoder and S decoder are utilized to generate results. The entire training and testing process is outlined in Algorithm 1.

We compare the performance of AMLNet with six methods: four state-of-the-art deep learning (DeepAR, LogSparse Transformer, N-BEATS and Informer), a statistical (SARIMAX) and a persistence model. 1) Persistence is a typical baseline in forecasting which considers the time series of the previous day as the prediction for the next day; 2) SARIMAX [4] is an extension of ARIMA which cann handle seasonality with exogenous variables; 3) DeepAR [8] is a widely used RNN-based forecasting model; 4) LogSparse Transformer [7] is a Transformer-based forecasting model, it is denoted as ”LogTrans” in Table III; 5) N-BEATS [6] consists of blocks of fully-connected neural networks, organised into stacks using residual links. We introduced covariates at the input of each block to facilitate multivariate series forecasting; 6) Informer [19] is a Transformer-based forecasting model. We modified it for probabilistic forecasts to generate the mean value and variance. Note that Persistence, N-BEATS and Informer are NAR models while the others are AR models.

All models were implemented using PyTorch 1.6 and evaluated on Tesla V100 16GB GPU. The deep learning models were optimised by mini-batch gradient descent with the Adam optimiser and a maximum number of epochs 200. Following the experimental setup in [24, 9], we used the following training, validation and test split: for Sanyo and Hanergy - the data from the last year as test set, the second last year as validation set for early stopping and the remaining data (5 years for Sanyo and 4 years for Hanergy) as training set; for Solar and Electricity - the last week data as test set (from 25/08/2006 for Solar and 01/09/2014 for Electricity), the week before as validation set. For all data sets, the data preceding the validation set is split in the same way into three subsets and the corresponding validation set is used to select the best hyperparameters. We selected the hyperparameters with a minimum loss on the validation set. We used Bayesian optimisation for hyperparameter search with a maximum number of iterations 20. The models used for comparison were tuned based on the authors’ recommendations. For the Transformer-based models, we used learnable position and ID (for Solar, Electricity and Exchange sets) embedding. For AMLNet, the constant sampling factor for Informer backbone was set to 2, and the length of start token $T_{d}e$ is fixed as half of the forecasting horizon. The learning rate of generator $\lambda_{G}$ and discriminator $\lambda_{D}$ was fixed, the loss function regularisation parameters $\alpha_{o}$ and $\alpha_{h}$ were chosen from {0.001, 0.05, 0.1, 0.5}, the dropout rate $\delta$ was chosen from {0, 0.1, 0.2}, the hidden layer dimension size $d_{hid}$ was chosen from {8, 12, 16, 24, 48}, the Informer backbone Pos-wise FFN dimension dimension size $d_{f}$ and number of heads $n_{h}$ were chosen from {8, 12, 16, 24, 48, 96} and {4, 8, 16, 24, 32}, number of encoder $n_{e}$, P1 and P2 decoder $n_{S}$ and shallow NAR decoder layers $n_{S}$ were chosen from and {2, 3, 4}. Note that number of encoder layer is not less than number of decoder layers, P1 and P2 decoders have same number of layers, shallow NAR decoder has less number of layers than deep decoders. The discriminators are simply a series of ConvLayer-BatchNorm-LeakyReLU-ConvLayer-LinearLayer-Sigmoid. The first ConvLayer has the output dimension of 16, stride of 2 and kernel size of 3, while the second has dimension of 1, stride of 1 and kernel size of 3.

The selected best hyperparameters for AMLNet are listed in Table II and used for the evaluation of the test set.

Following [8], we report the standard $\rho$0.5 and $\rho$0.9-quantile losses. The quantile loss function is applied to predict quantiles, and quantile $\rho$ is the value below which a fraction of observations in a group falls. Given the ground truth $y$ and $\rho$-quantile of the predicted distribution $\hat{y}$, the $\rho$-quantile loss is given by $\mathrm{QL}_{\rho}(y,\hat{y})$:

Table III shows the $\rho$0.5 and $\rho$0.9 losses of all models, including the three AMLNet versions which use different decoders (AMLNet-P1, AMLNet-P2, AMLNet-S). As N-BEATS and Persistence produce point forecasts, only the $\rho$0.5-loss is reported for them. AMLNet is the most accurate method - it outperforms the other methods on all data sets except for $\rho$0.5 on Electricity where the Logsparse Transformer is the best model. For AMLNet, S decoder successfully inherits the abilities of P1 and P2 decoders with fewer layers and has the best performance on Hanergy and Solar sets. Overall, all decoders of AMLNet outperform their backbone model Informer except for Solar and Electricity sets where P1 underperforms, indicating the AMLNet is beneficial. Both NAR branches (P2 and S) exhibit clear improvement over Informer, suggesting the design for overcoming the disadvantages of AR and NAR models is successful for improving forecasting accuracy.

AST [5] employs adversarial training to improve the contiguous and fidelity at the sequence level. AST is not compared directly because it generates quantile forecasts and minimises quantile loss, while we consider probabilistic forecasting with different objective functions in Eq. (6). To compare the effectiveness of AMLNet with AST, we apply the adversarial training of AST to AMLNet and its backbone - Informer, and the results are shown in Table IV. The adversarial training improves the performance of Informer, while AMLNet still exhibits advantages. The design of AMLNet is compatible with the AST adversarial training, combining both techniques achieves further performance improvement on the S decoder.

To study the capabilities of AMLNet in addressing error accumulation and modeling output space interdependence, we conduct a comparative analysis with two benchmark models: classic AR model DeepAR and NAR model Informer. The evaluation is performed on both the Sanyo and Hanergy datasets, providing insights into AMLNet’s performance across different scenarios.

Fig. 3 illustrates the $\rho$0.5-loss of various models across different forecasting horizons, using a fixed pasting history. The loss of all models tends to increase as the forecasting horizon expands. However, it is evident that the performance of AR models, such as DeepAR, deteriorates more significantly compared to NAR models. Remarkably, AMLNet’s P1 decoder consistently outperforms DeepAR across different horizons, demonstrating its capability to mitigate the adverse effects of error accumulation. Conversely, NAR models, including Informer and AMLNet, exhibit relatively stable performance over varying forecasting horizons. This observation indicates that AMLNet’s design effectively addresses the issue of error accumulation in its P1 decoder.

Referencing Fig. 2 (c), it becomes evident that AMLNet’s S decoder exhibits lower cosine distances in hidden states compared to its backbone counterpart shown in Fig. 2 (b). This distinction is particularly pronounced when observing the lighter color in Fig. 2 (c) as compared to Fig. 2 (b). Additionally, the average cosine distances of the hidden states in the P2 and S decoders on both the Sanyo and Hanergy datasets are significantly lower, by 28% and 23% respectively, compared to the backbone model. Furthermore, the average DTW distances of the P2 and S decoder predictions exhibit a reduction of 18% and 17%, respectively. These findings underline the efficacy of our designed approach in learning and leveraging output space interdependence. This enables the model’s hidden states to exhibit greater similarity to neighboring states and subsequently generates more realistic and coherent prediction trajectories.

We conducted an evaluation of the inference time for different configurations of AMLNet, as well as the NAR backbone Informer and the AR baseline LogTrans. The results are summarized in Table V. All experiments were conducted on the same computer configuration, and the reported values represent the average elapsed time in milliseconds along with the standard deviation from 10 runs. Notably, the NAR models exhibit faster inference times compared to the P1 decoder, primarily due to their inherent parallelizability. Informer and AMLNet-P2 demonstrate similar inference speeds, which is consistent with their comparable architectural characteristics. AMLNet-S, designed with fewer layers, stands out as the fastest among the models evaluated.

To assess the effectiveness of our proposed methods, we conducted an ablation study, focusing on the $\rho$0.5-loss metric for various model configurations. The results are presented in Table VI. In this table, $\mathcal{L}_{o}$ corresponds to the classic online KD, $\mathcal{L}_{wo}$ represents our outcome-driven KD, $\mathcal{L}_{GAN}$ indicates adversarial KD applied to the last hidden layer, and $\mathcal{L}_{hGAN}$ refers to our hint-driven KD.

Among the key findings from this ablation analysis:

• •

  AMLNet, when combined with our proposed KD methods, emerges as the most effective model configuration, attaining the highest accuracy.

• •

  Both outcome-driven KD and hint-driven KD lead to improved accuracy when incorporated into the frameworks, underscoring the efficacy of both design approaches.

• •

  The Backbone+$\mathcal{L}_{wo}$+$\mathcal{L}_{GAN}$ outperforms Backbone+$\mathcal{L}_{o}$+$\mathcal{L}_{hGAN}$ substantially, suggesting outcome-driven KD exerts a more pronounced impact on accuracy enhancement compared to the hint-driven KD.

• •

  S decoders tend to outperform P1 and P2 decoders, supporting the notion that our design of online knowledge distillation from P1 and S is a beneficial strategy.