Reinforced Decoder: Towards Training Recurrent Neural Networks for Time Series Forecasting

In the general multi-step-ahead time series forecasting problem, a dataset $D$ contains input–target pairs $(X,\ Y)$, where $X=(X_{1},\ \dots,\ X_{T})\in\mathbb{R}^{T}$ denotes the historical observations with $T$ time lags, and $Y=(y_{T+1},\ \dots,\ y_{T+H})\in\mathbb{R}^{H}$ denotes the corresponding observations in the future over the horizon $H$. The goal of time series prediction is to learn a target function, $\mathcal{F}(\cdot)$, on the dataset that maps the input $X$ to its corresponding target sequence. That is, $\hat{Y}=\mathcal{F}(X)$. As shown in Fig. 1, the RNN-based S2S model generally formulates this function using an encoder–decoder architecture [33]. The encoder introduces a hidden state, $h_{j}$, at each time step $j$, to capture the temporal relationships within the historical time series:

where $f_{e}(\cdot)$ denotes the non-linear activation function inside the encoder, and $X_{j}$ contains the historical observations of the target variate $y_{j}$ and the corresponding covariates. The last $h_{T}$, which summarizes the whole sequence of historical observations, is usually used as the context vector c that is passed to the decoder [5, 17]. Next, in the decoding stage, a decoder is utilized to generate the predictions auto-regressively. Specifically, $\hat{y}_{T+k}$ can be obtained according to the following:

where $1\leq k\leq H$, $f_{d}(\cdot)$ denotes the non-linear activation function inside the recurrent units of the decoder, $g(\cdot)$ is the function inside the output layer, and $\hat{y}_{T+k}$ represents the model’s prediction for its corresponding true observation ${y}_{T+k}$ at time step $T+k$.

Common choices for $f_{e}(\cdot)$ and $f_{d}(\cdot)$ include the Elman RNN (ERNN) [34], long short-term memory (LSTM) [35], and gated recurrent unit (GRU) [36]. In this study, we use LSTM to illustrate the mechanism of the proposed reinforced decoder (Fig. 3) [10, 5]. In addition, a state-of-the-art variant of LSTM with embedded hybrid spiking neurons (i.e., ESLSTM [16]) is also used to test the general ability of our method in relation to different recurrent units.

Backpropagation is the fundamental algorithm for training neural networks via gradient descent [37]. In an RNN, calculating gradients requires backpropagation through time (BPTT)  [38], due to the sequential nature of forward propagation, which incurs high memory and computational costs. The teacher forcing (TF) strategy was first proposed to avoid BPTT for simple recurrent networks lacking hidden-to-hidden connections [37, 18]. In the TF strategy, the ground truth values are fed as decoder inputs instead of self-generated predictions, decoupling the time steps to allow for parallelized training. Subsequent work extended the TF strategy to more complex recurrent networks containing both hidden-to-hidden and output-to-hidden recurrence [15, 39, 37], such as the decoder shown in Eq. (2)–(3). In these cases, the TF strategy is often combined with BPTT, where BPTT primarily handles hidden-to-hidden recurrence, and TF is used for output-to-hidden recurrence. This combination enables the model to capture long-term dependencies while reducing the computational costs associated with full BPTT. However, as true values are unavailable when extrapolating, the TF strategy induces exposure bias [19], which can lead to poor performance [15].

In order to mitigate the exposure bias of TF, Lamb et al. [21] proposed the professor forcing (PF) method, which used generative adversarial networks (GANs) to align the hidden states between the training and extrapolation stages over multiple steps. Subsequently, many variants of the PF method have been derived [22, 23]. However, these methods suffer from mode collapse, which is inherent to GANs and hinders effective training [24, 25]. Hence, more general methods adjust the exposure level of the ground truth values in TF [13, 10]. For example, Taigman et al. [26] proposed a variant of the TF technique, which uses the average of true values and predictions with added noise as the decoder input. Bengio et al. [27] designed a scheduled sampling approach, gradually replacing true values with self-generated outputs based on probability. However, existing adjustment schemes either introduce new sources of noise or lack adaptability to the dynamic training process. Furthermore, previous research has mainly focused on machine translation and vocal synthesis, with little attention being paid to time series forecasting. Thus, this study introduces auxiliary models and the reinforcement learning method to enable dynamic and adaptive guidance (i.e., input) control of the decoder in the context of multi-step-ahead time series prediction.

Inspired by the principles of behaviorist psychology, reinforcement learning (RL) aims to empower computational agents to learn optimal behaviors through trial-and-error interactions with their environment [32]. With the core objective of maximizing cumulative rewards over time, the RL framework has been successfully applied to many sequential decision-making problems, ranging from game playing to resource management [30, 31, 32]. Specifically, RL lies in the concept of the agent–environment interaction, where, the agent represents the action-maker, and the environment encompasses information external to the agent within the context of the considered problem.

To effectively apply RL algorithms, the problem is usually required to be formulated as a Markov decision process (MDP), which comprises a state space $\mathcal{S}$, an action space $\mathcal{A}$, a state transition function $\mathcal{P}\left(\cdot\right)$ satisfying the Markov property, and a reward calculation function $\mathcal{R}\left(\cdot\right)$. At each step $k$ of the sequential decision problem, the agent perceives the current state $s_{k}$ of the environment and selects an action $a^{\star}_{k}$, according to a mapping from states to actions called the policy. The environment then updates its state based on the transition function and provides a feedback reward to the agent.

Within the framework of RL, the goal of sequential decision-making is shifted from finding an optimal set over all possible sequences of actions to focusing only on the optimal actions under the states, i.e., learning the optimal policy. Therefore, it can lead to more efficient and effective decision-making in complex, dynamic problems, where traditional methods may be limited. Among the categories of RL algorithms, variants of policy gradient (PG) algorithms, which model the policy directly with a parameterized machine learning model, have consistently produced state-of-the-art results [40]. In general, the policy $\pi_{\theta}$ (with parameters $\theta$) is a probability distribution over the actions $a^{j}\in\mathcal{A}$, conditioned on the state $s_{k}$ at step $k$. In the case that a stochastic policy is used, where the action space is discrete, such as the input selection problem for the decoder in this study, the policy can be defined as follows:

In this way, the optimal policy is obtained by optimizing the parameters $\theta$ through gradient ascent with a set of training samples. Among the PG algorithms derived from the policy gradient theorem, REINFORCE is one of the most representative algorithms [29, 30, 31, 32]. In this study, we follow REINFORCE to provide the calculation formula for the gradient.

As shown in Fig. 3, in contrast to the standard RNN-based S2S model containing solely an encoder and a decoder [33], the proposed reinforced decoder additionally incorporates the agent and a model pool. For the sake of simplicity but without loss of generality, the LSTM unit is used here to compose the S2S model to illustrate the mechanism of the proposed reinforced decoder, for which the forward propagation process is detailed as follows.

Encoder. This reads the historical observations $X_{1:T}$ sequentially and builds the hidden state according to Eq. (1). The last state, $h_{T}\in\mathbb{R}^{N_{e}}$, is used as the context vector c and is passed to the decoder [5, 17].

Reinforced Decoder. This outputs the predictions based on the context vector and the inputs generated by the models in the pool. Specifically, $f_{d}(\cdot)$ and $g(\cdot)$ in Eq. (2)–(3) are summarized as follows:

where $s_{k},c_{k},\tilde{c}_{k}\in\mathbb{R}^{N_{d}}$ are the hidden state, the cell state, and candidate cell state of the decoder at step $k\in\left\{1,2,...,H\right\}$, respectively; $N$ is the cell dimension; $U_{\psi},U_{i},U_{o},U_{c}\in\mathbb{R}^{N_{d}\times\left(N_{d}+1+N_{e}\right)}$ denote the weight matrices of the forget gate, input gate, output gate, and cell state, respectively, and $b_{\psi},b_{i},b_{o},b_{c}\in\mathbb{R}^{N_{d}}$ denote the corresponding bias vectors; $\psi_{k},i_{k},o_{k}\in\mathbb{R}^{N_{d}}$ are the forget, input, and output gate vectors; $V\in\mathbb{R}^{1\times N_{d}},b\in\mathbb{R}^{1}$ denote the weight matrix and bias vector of the output layer. Note that $\varphi_{T+k}\in\mathbb{R}^{1}$ is the input to the decoder at time step $T+k$, which is denoted as follows:

where $a^{\star}_{k-1}$ denotes the model chosen from the model pool to provide the decoder inputs, and $\hat{y}_{T+k-1}^{a^{\star}_{k-1}}$ represents its forecast of $y_{T+k-1}$. The model pool consists of the decoder itself and auxiliary models.

In this context, the auxiliary models are expected to have the capacity to generate multi-step-ahead predictions without any specific constraints on their internal structure. Consequently, a variety of options is available for auxiliary models, ranging from simple statistical models [6] to more complex machine learning models [7, 8, 9]. In order to strike a balance between computational efficiency and the ability to capture diverse temporal relationships, it is advisable to consider non-autoregressive models with lower computational requirements. The specific auxiliary model chosen can be tailored to the requirements and characteristics of the considered tasks.

Note that we consider two auxiliary models in this study, i.e., well-trained multiple-output support vector regression (MSVR)[8] and multi-layer perception (MLP), in view of their high efficiency and wide application in multi-step time series prediction [41, 2, 1, 42, 3]. In addition, preliminary experimentation revealed that they exhibited superior performance in terms of both computational efficiency and predictive accuracy, compared to other multi-step time series forecasting approaches such as ARIMA [6] and random forest [9].

Given the inherent sequential nature of discerning the optimal model $a^{\star}_{k-1}$, where each selection influences the subsequent predictions of the decoder and choices, the model selection procedure exhibits the characteristics of a sequential decision-making problem [43]. Such sequential decision characteristics, coupled with the inherent uncertainty of an optimal input-providing model, render manual experience or conventional optimization techniques ill-suited for this purpose. Formulating the problem as a framework for sequential decision-making under uncertainty provides a principled approach to address this complexity.

Inspired by the remarkable progress of RL in sequential decision-making problems, the REINFORCE algorithm [29] was introduced to solve the decoder input selection problem. As illustrated in Fig. 3, at each step $k\in\left\{1,2,\dots,H-1\right\}$, the agent observes the current decoding situation and selects a model for which the prediction is fed into the decoder. The decoder then transits to the next state and gives a reward to the agent. The specific mathematical model, MDP $\left\langle\mathcal{S},\ \mathcal{A},\ \mathcal{P},\ \mathcal{R}\right\rangle$, is summarized as follows:

State space. Each state is defined as the current hidden state of the decoder (i.e., $s_{k}$), as it stores the decoding status that retains all relevant information for the agent to select actions.

Action space. Each element $a^{j}$ within the action space $\mathcal{A}$ describes the candidate behavioral characteristics of the agent. Here, the action space $\mathcal{A}$ is a set containing the candidate models for generating the predictions (inputs) for the decoder (i.e., the decoder itself and the auxiliary models).

Transition function. As the state is defined as the hidden state of the decoder, its transitions follow Eq. (5)–(12), which can be simplified to $s_{k+1}=\mathcal{F}_{\vartheta}(s_{k},\ a^{\star}_{k})$. $\mathcal{F}_{\vartheta}\left(\cdot\right)$ consists of the encoder LSTM and decoder LSTM, in which $\vartheta$ includes the weight matrices and bias vectors related to the forget gates, input gates, output gates, and the output layer.

Reward function. The reward, $r_{k}$, is a numerical value representing the instant reward to the agent affiliated with executing the action $a^{\star}_{k}$ in state $s_{k}$. As the agent learns through trial and error, the design of the reward function dictates the optimal policy. In this study, it is desired that the decoder generates accurate predictions for the next time step after being fed with the predictions from the model selected by the agent. Therefore, $r_{k}$ is defined as follows [28]:

where $\alpha$ is a hyperparameter that balances the two types of rewards, $\text{\emph{Rank}}_{k}$ is designed to motivate the agent to select the model with the highest accuracy at step $k$, and $\text{\emph{Accuracy}}_{k}$ evaluates the performance of the decoder at step $k+1$ using the chosen prediction of the model as the input. These terms are defined as follows:

where $N_{a}$ is the number of actions in the action space $\mathcal{A}$, which is also the number of models in the model pool. The models are rank-ordered from 1 (the best) to $N_{a}$ (the worst),, according to the prediction accuracy. Furthermore, $rank\left(a^{\star}_{k}\right)$ indicates the ordinal number of the model selected by the agent at step $k$.

where $\beta$ is a hyperparameter that converts penalties to reward-based metrics, and $\hat{y}_{T+k+1}$ is the prediction generated by the decoder at step $k+1$ after being fed into the selected input by the agent.

In addition, the policy $\pi_{\theta}(\cdot)$ of the agent in Eq. (4) is modeled using a multi-layer perception (MLP) with a single hidden layer and a softmax activation function [44], as defined below. The parameters $\theta$ encompass the weight matrices and bias vectors in the MLP.

where $W_{1}\in\mathbb{R}^{N_{p}\times N_{d}},\ W_{2}\in\mathbb{R}^{N_{a}\times N_{p}},\ b_{1}\in\mathbb{R}^{N_{p}},b_{2}\in\mathbb{R}^{N_{a}}$ denote the weight matrices and bias vectors of the hidden layer and output layer, respectively, and $N_{p}$ is the number of nodes in the hidden layer. Then, the action (model) taken by the agent at step $k$ is obtained by:

The environment faced by the agent would be time-varying if the encoder and the decoder were updated concurrently with the agent’s learning process. In order to ensure the environment remains stationary that is a crucial condition for convergence guarantees of policy gradient methods [45], we adopt an asynchronous updating strategy for the agent and the encoder–decoder RNNs (Fig. 3). Specifically, the parameters of the policy network ($\pi_{\theta}(\cdot)$) within the agent are held constant when training RNNs ($\mathcal{F}_{\vartheta}(\cdot)$) comprising the encoder and decoder. Conversely, the parameters of the RNNs are fixed during the policy network training. Both $\theta$ and $\vartheta$ are updated in the direction of the performance gradient. Algorithm 1 outlines the training process in detail.

Agent. Let $\tau=\left\{s_{1},\ a^{\star}_{1},\ r_{1},\ \ldots,\ s_{H-1},\ a^{\star}_{H-1},\ r_{H-1}\right\}$ be a trajectory denoting the interactions between the agent and the environment in one complete decoding process. The agent’s goal is to obtain a policy that maximizes the expected total reward along $\tau$, defined as follows:

where $R(\tau)=\sum_{k=1}^{H-1}\gamma^{k}r_{k}$ is the cumulative discounted reward from the start state, $\gamma$ is a discount factor so that $0\leq\gamma<1$. $P\left(\tau\right)$ denotes the probability of producing $\tau$ given the policy $\pi_{\theta}$ and encoder-decoder $\mathcal{F}_{\vartheta}$. That is, $P\left(\tau\right)=P_{\mathcal{F}_{\vartheta}}\left(s_{1}\right)\prod_{k=1}^{H-1}\pi_{\theta}\left(a^{\star}_{k}|s_{k}\right)P_{\mathcal{F}_{\vartheta}}\left(s_{k+1}|s_{k},a^{\star}_{k}\right)$. The gradient of the training object for the agent is formulated as follows:

As asynchronous training is used, the RNNs remain unchanged when the policy is updated, which is independent of $\theta$. Both $\bigtriangledown_{\theta}\text{log}P_{\mathcal{F}_{\vartheta}}(s_{1})$ and $\bigtriangledown_{\theta}\text{log}P_{\mathcal{F}_{\vartheta}}\left(s_{k+1}|s_{k},a^{\star}_{k}\right)$ are equal to 0. Then, Eq. (19) can be transformed as follows:

where $N$ denotes the number of samples in the training set. $a^{\star}_{k,n}$, $s_{k,n}$, and $\tau_{n}$ represent the corresponding action, state, and trajectory for the sample $n$, respectively. Moreover, we use the $\epsilon$-greedy exploration strategy to execute the actions to jump out of local optimal point during the training process, where the agent executes an action randomly with a certain probability, $\epsilon<1$, instead of the policy-based action. That is,

The agent does not explore but acts exclusively according to the policy as in Eq. (17), when $\epsilon=0$.

RNNs. The parameters of the encoder and the decoder (i.e., RNNs), are trained to minimize any deviations in the ground truth sequence from the prediction sequence; that is,

where $\mathcal{L}(\hat{Y})=\frac{1}{H}\sum_{k=1}^{H}(y_{T+k}-\hat{y}_{T+k})^{2}$ is the Mean Square Error. Then, the gradient of the training object for $\mathcal{F}_{\vartheta}$ is calculated based on the backpropagation-through-time method [38], which is simply formulated as follows:

where $E_{k,n}=\hat{y}_{T+k,n}-y_{T+k,n}$. $X_{1:T,\ n}$ and $\varphi_{T+1:T+k,\ n}$ represent the historical observations and the decoder inputs for the sample $n$, respectively. Similarly, $\hat{y}_{T+k,n}$ and $y_{T+k,n}$ denote the corresponding predicted and true values for the sample $n$, respectively. Due to space limitations, more details on the gradient flow are shown in Appendix I.

In this study, we evaluated the performance of the proposed RD approach on five datasets that cover both simulated and real-world time series, which have different time granularities and are either univariate or multivariate. Brief descriptions of each are provided below, with the prediction tasks summarized in Table I.

MG. The Mackey-Glass time series has been commonly recognized as the benchmark in a series of studies related to time series forecasting [41]. Each data point is generated by the non-linear delay differential equation as follows:

where $\phi_{t}$ was initialized at 1.2, referring to the settings in [41], and 7000 observations were simulated.

SML. The SML time series ¹¹1https://archive.ics.uci.edu/ml/datasets/sml2010 was collected from an indoor temperature monitoring system installed in a domotic house over approximately 40 days. The data were sampled every minute and smoothed with 15-minute means [46].

PM. This daily time series ²²2https://archive.ics.uci.edu/ml/datasets/PM2.5+Data+of+Five+Chinese+Cities contains the $\text{PM}_{2.5}$ data from Chengdu from 1 January 2010 to 31 December 2015, obtained by averaging the hourly data in a publicly available dataset.

ILI. This time series of weekly influenza-like illness rates ³³3https://github.com/XinzeZhang/TimeSeriesForecasting-torch/tree/master/data/real/ili was collected from the Influenza Weekly Report issued on the website of the Chinese National Influenza Center. It contains 629 values observed in southern China from the first week of 2010 to the third week of 2022.

ETTh. The reliable forecasting of oil temperature in power transformers is an essential issue for ensuring safe and efficient system operation [47]. The ETTh time series at the hourly level contains the operating data of the electricity transformer from 1 January 2018 to 26 June 2018 ⁴⁴4https://github.com/zhouhaoyi/ETDataset/blob/main/ETT-small/ETTh1.csv. Each data point consists of the target variable “oil temperature” and six different types of external power load features.

Given the $H$-step-ahead prediction sequence $\left\{\hat{y}_{T+1},\ldots,\ \hat{y}_{T+H}\right\}$ and the corresponding target sequence $\left\{y_{T+1},\ldots,\ y_{T+H}\right\}$, two prediction accuracy measures [48], i.e., the root mean square error (RMSE) and mean absolute percentage error (MAPE), were considered. The definitions are as follows:

Baselines. Sequence-to-sequence (S2S) models constructed by LSTM unit and ESLSTM unit were selected for experimentation, termed as LSTM and HSN-LSTM [16] respectively. Moreover, the performance of the proposed reinforced decoder (RD) was compared against five decoding and training strategies to validate its effectiveness. The comparison strategies are summarized as follows:

• •

  Free Running (FR) [15, 10]. It is a fundamental training approach, where the predictions generated early are fed back as decoder inputs for the predictions at later steps during the training and extrapolation stages.

• •

  Teacher Forcing (TF) [18, 33]. It is a popular training approach, where the ground truth values are used as the decoder inputs during the training stage.

• •

  Scheduled Sampling (SS) [27]. It is a curriculum learning strategy that gradually converts the training process from a fully guided scheme using ground truth values to a lesser one that mostly uses generated predictions. At every time step, a coin is flipped to decide whether to use the previous ground truth values, according to probability $p$. This probability decays following the linear schedule referring to [27, 49].

• •

  Professor Forcing (PF) [21]. It is a typical improvement on TF that mitigates exposure bias through applying adversarial training techniques. Two shared-weight S2S models serve as generators, with the decoders producing outputs fed by true values and self-generated predictions, respectively. In addition, another distinct RNN is introduced as a discriminator. The discriminator is trained to accurately distinguish the distributions of the hidden states derived from these two generators, and the S2S models are trained to fool the discriminator while minimizing the prediction error.

• •

  Non-autoregressive (NAR) [16, 10, 17]. It is based on the non-autoregressive decoding structure, which generates multiple outputs directly, rather than in a step-by-step manner. For HSN-LSTM, it follows the original structure in [16], where the predictions are generated using two attention modules (variable attention and fusion attention) based on hidden states in the encoder constructed by ESLSTM units. Correspondingly, for LSTM, multiple predictions are projected from the last hidden state of the encoder through a dense layer [10].

Hyperparameter settings. As normalization is a standard requirement for time series modeling and prediction [41], we scaled each series by the min–max method and divided it into training, validation, and testing sets with a ratio of 0.64, 0.16, and 0.20, respectively [48]. To ensure the consistency of the initial untrained S2S network allowing for a fair comparison, the general hyperparameters of the fundamental models (e.g., the dimensions of the ESLSTM units in HSN-LSTM) were determined according to some rules of thumb, and were kept the same across all strategies. The specific hyperparameters for comparative training strategies were tuned using the random search method [50], given that the distribution of data on different datasets is not the same.

For our proposed RD strategy, the coefficient balancing the two types of rewards was chosen from $\{0,0.1,...,1\}$, and the number of nodes in the hidden layer of the policy network was chosen from $\{32,64,96,...,512\}$. All models were trained by the Adam optimizer with the maximum training epoch set to 100, and the early-stopping strategy was used to save the best model for all strategies. The experiments were repeated 10 times and ran on an NVIDIA RTX4060 16GB GPU. A comprehensive overview of the hyperparameter settings and search space is provided in Appendix III. The codes and full details of the experiments are also provided on Github⁵⁵5https://github.com/QiSima/TimeSeriesForecasting-RD..

The prediction performance of all benchmark models was examined on the five datasets regarding two accuracy measures, i.e., RMSE and MAPE. The averaged results of the S2S models (LSTM and HSN-LSTM) under different strategies are summarized in Table II and Table III, respectively, with the best results highlighted in bold. The results lead to the following conclusions:

• •

  The proposed reinforced decoder outperforms competing training strategies (the win counts are in the last column) for both univariate and multivariate tasks across different prediction horizons. This demonstrates the effectiveness of the proposed strategy in enhancing the prediction capacity of S2S models applied to time series forecasting.

• •

  The non-autoregressive strategy, NAR, exhibits superior performance compared to FR, illustrating the negative impact of error accumulation. It is conceivable that one reason for the superiority of the RD is its ability to rectify inputs with more accurate ones. Additionally, the better performance of RD compared to NAR substantiates the effectiveness of exploiting valuable information from auxiliary models while preserving temporal dependencies in the decoding process [51, 52].

• •

  Despite the popularity of TF in the multi-step-ahead prediction literature [18, 3, 39], its performance does not consistently surpass that of FR due to exposure bias, e.g., on the MG series, SML series, and ETTh series, which is in agreement with [15]. This is another conceivable reason for the superiority of the proposed approach, which mitigates exposure bias through feeding decoder inputs generated from the same mechanism during both the training and extrapolation stages.

• •

  At least one of the SS and PF strategies has better predictive performance than the TF method in most prediction tasks. These two approaches are improvements over TF and aim to alleviate exposure bias [27, 21, 39], demonstrating the negative effect of exposure bias on predictive performance from another perspective.

In addition, to examine the predictive reliability, the results of 10 repeated experiments are shown as box plots (Fig. 4). Compared with other strategies, the proposed RD strategy not only achieves lower RMSE levels but also has higher prediction stability over repeated trials.

Furthermore, we also conducted input noise experiments on the MG dataset to demonstrate the robustness of the RD strategy [53]. After being trained on noise-free data, the models underwent evaluation using the samples from the test dataset with added noise. The noise was sampled from a Gaussian distribution with the mean equal to 0 and the standard deviation ranging from 1e-4 to 2.5e-3. As shown in Fig. 5, the degradation in model accuracy differs noticeably across training strategies when exposed to noise. Regardless of the underlying S2S model (LSTM or HSN-LSTM), TF exhibits the most pronounced degradation with increasing noise levels, as it uses noise-free true values as input during the training process. The prediction performance of RD remains better than that of the other strategies as the noise level increases. This is due to the mechanism of selecting inputs from a member in the model pool, which implicitly ensembles multiple models. As previous research has demonstrated, ensemble learning can improve the robustness of predictions and enhance generalization ability [2, 28]. Our findings suggest that RD confers advantages in noisy real-world settings due to the introduction of the model pool and adaptive decoder input selection.

To further assess the effectiveness of reinforcement learning, the ablation studies of the LSTM-based S2S models were conducted on five datasets. Table IV reports comparative results on the SML series. More results on other datasets are given in Appendix IV.

• •

  AM. This serves as a reference, representing the well-trained auxiliary models (MLP and MSVR).

• •

  LSTM+FR. This S2S model, constructed by LSTM units, always takes self-generated predictions as the decoder inputs without rectification (i.e., trained by FR).

• •

  LSTM+AM. There is no input selection for the decoder. The decoder always takes the predictions from a fixed candidate model (MLP or MSVR) as inputs during both the training and the extrapolation stages.

• •

  LSTM+RD. This is the reinforced decoder approach, where reinforcement learning is used to dynamically select the input-providing model to rectify the decoder inputs.

Compared with LSTM+FR, which relies solely on self-generated predictions, the two LSTM+AM models have better predictive performance, demonstrating the utility of introducing auxiliary models. Moreover, the smaller the prediction error of the auxiliary model, the better the prediction performance of LSTM+AM with its predictions as decoder inputs. This demonstrates that feeding those inputs that are closer to true values helps the decoder reduce error accumulation and improve prediction performance.

In addition, as the prediction performance of the auxiliary models is much better than the results of the decoder itself, LSTM+RD with only one auxiliary model surpasses LSTM+FR and approximates the corresponding LSTM+AM models. This further confirms that the superiority of RD stems from effectively utilizing predictive information from auxiliary models via the REINFORCE method.

It is worth noting that although the well-trained MLP outperforms the LSTM+FR and MSVR, LSTM+AM, which directly takes the predictions of the MLP as decoder inputs, fails to achieve the best performance. In contrast, LSTM+RD, which includes the well-trained MLP and MSVR within its model pool, achieves the best results in this task. To gain further insights, we performed slice analysis on the training process of the agent in the LSTM+RD (MSVR, MLP) model when applied to the 48-step-ahead task.

Fig. 7 compares the agent’s actions on the training and validation sets throughout the training process. The samples in the validation set are not involved in updating the agent’s policy. It can be observed that the auxiliary models have differing predictive abilities at different time steps, and the proportion of choices corresponding to the better-performing model significantly increases after training. While MLP has better predictions overall, MSVR excels at certain time steps. This helps to explain why LSTM+RD (MSVR, MLP) outperforms LSTM+AM (MLP). At the same time, it also confirms the need for input selection, as well as the effectiveness of the reinforcement learning method.

In addition, it is noteworthy that, in contrast to its behavior on the training set, the agent chooses the predictions of the well-trained MLP as decoder inputs at time step $T+1$ for 86% of the samples in the validation set. This suggests that the agent is not simply learning to execute a fixed set of actions, but is learning to discern the predictive capabilities of each model for different samples. The insight gleaned from training enables it to select the superior model to provide inputs for the decoder, even in the absence of knowledge regarding actual prediction performances on new samples. This characteristic sets reinforcement learning apart from traditional optimization methods and makes it well-suited with respect to the dynamics inherent in the decoder input selection problem.

Furthermore, we investigated how the dynamic, adaptive features unique to reinforcement learning take effect when rectifying the decoder inputs. Fig. 6 delineates the actions taken by the trained agent at different steps on the test dataset. The results substantiate that, first, different models have varying prediction performances at different time (decoding) steps [28], justifying the necessity of input selection. Second, the input selection scheme developed with reinforcement learning (the trained agent) is not only sample adaptive consistent with the conclusions drawn from Fig. 7, but also adapts to varying decoding steps. This further illustrates the suitability and superiority of reinforcement learning in the context of this problem.

In addition to recurrent neural networks, the S2S models based on the attention mechanism have also recently exhibited promising performance in machine translation and speech recognition [54, 47, 55]. To verify the generality of the RD strategy, we compare it with the above-mentioned strategies (NAR, FR, TF, SS, and PF), based on the well-known self-attention-based model, Informer [47].

In particular, Informer (+NAR) predicts outputs by one forward procedure, i.e., generative inference, rather than auto-regressive decoding. For the Informer+RD model, the environment states observed by the agent are defined as the outputs generated by the self-attention modules within the decoder. These outputs serve as the contextual information that the agent uses to make decisions regarding input selection during the dynamic decoding process. The hyperparameters of Informer followed [47] to ensure a fair comparison and the details are provided in Appendix III. Table V reports the results, with the smallest value highlighted in bold. The results show that the RD strategy still achieves the best performance among various training strategies, demonstrating its architecture-agnostic utility for optimizing S2S models.

As Informer is well-suited to long-range dependencies, we further examined the NAR, FR, and RD strategies on the ETTh dataset with longer prediction horizons (Fig. 8). Overall, the prediction performance of the FR strategy decreases rapidly as the forecasting horizon increases due to the accumulation of errors. In contrast, RD mitigates this by rectifying decoder inputs through the introduction of multiple auxiliary models and adaptive selection (by the agent), sustaining strong performance even for longer horizons. Furthermore, RD also outperforms the NAR strategy thanks to external valuable information from auxiliary models.