Addressing Concept Shift in Online Time Series Forecasting: Detect-then-Adapt

Unlike traditional out-of-distribution (OOD) detection or anomaly detection methods, which aim to determine whether a given data instance deviates significantly from the training or normal distribution, our focus is on assessing whether the current online testing distribution substantially differs from the training distribution or exceeds the model’s capability. Our detector operates on the hypothesis that a statistically significant increase in the error rate triggers a drift alarm.

Depicted in Figure 3b, our online monitoring strategy involves tracking testing losses, assuming a normal distribution $\mathcal{N}(\mu,\sigma)$, and subsequently subjecting them to a statistical test. Specifically, employing a test window size $l_{w}$, we designate the latest $l_{w}$ losses as the test target. The mean value of losses within the test window, denoted as $\tilde{\mu}$, is computed, along with the corresponding z-score $\frac{\tilde{\mu}-\mu}{\sigma/\sqrt{|B|}}$. We then assess the confidence level at which the score of new data instances deviates from the overall distribution of test losses. Notably, in the context of a normal distribution, both significantly smaller and larger scores should trigger the drift alarm. However, in our scenario, a smaller loss signifies commendable model performance under new data instances. Consequently, the alarm is activated exclusively when $\tilde{\mu}>\mu$. In addition, we implement a detector reset every $m_{t}$ steps, where $m_{t}\geq l_{w}$. This design offers two distinct advantages: firstly, it prevents a scenario where outliers with high losses persist, leading to an inflated $\mu$ that does not accurately reflect the true loss distribution; secondly, this flexible approach empowers users to determine the restart time based on their preferences or the model’s idleness. The corresponding pseudocode is presented in Algorithm 1.

Simply fully fine-tuning the model $f_{\theta}$ on new data points in $l_{w}$ may lead to limited performance. This is because (a) it could result in catastrophic forgetting of knowledge learned from historical data if we let the optimization procedure reach its optimum, and (b) the limited number of training examples in $l_{w}$ makes it impossible to learn an optimal model. It then prompts a crucial question: Can we leverage historical data to enhance our model adaptation? Given historical examples stem from different distributions compared to test instances, directly using historical data for adaption could result in sub-optimal performance. We propose a data augmentation method aimed at alleviating the gap in feature distribution. It augments existing historical data with synthetically generated examples. These synthetic examples are crafted by introducing new instances-dependent Gaussian noise to historical instances. Below, we justify this approach by an analysis of a linear model, with all proofs available in appendix VI.

Concept drift detection and adaptation play a crucial role in addressing real-world needs, especially in the context of on-device continual learning and inference.

Real-World Relevance: The paradigm of concept drift detection aligns with the practical demands of on-device continual learning and inference, as highlighted in the work by [12]. For instance, VR/AR headsets can leverage continual learning to define play boundaries and recognize locations in the physical world for virtual overlays. Similarly, home robots, smart appliances, and smartphones need to adapt and learn about their environments and user preferences. In these scenarios, online learning is essential, but there are also significant periods of time during which offline memory consolidation is feasible, such as when a mobile device is being charged or when its owner is asleep.

Adaptation Amidst Device Busyness: Even during busy device operation, the need for full-model updates becomes imperative when a substantial concept drift occurs. As depicted in Figure 3, our model’s performance under a new concept might deteriorate significantly. In such cases, predictions become highly unreliable and may lead to adverse consequences, especially in high-risk tasks. Therefore, prioritizing model adjustments before deployment becomes essential, ensuring that the model is aligned with the current concept and avoiding potential negative impacts. This not only enhances the reliability of predictions but also ensures the safety and performance of intelligent systems across various applications.

Datasets We investigate a diverse set of datasets for time series forecasting. ETT [37]³³3https://github.com/zhouhaoyi/ETDataset logs the target variable of the ”oil temperature” and six features of the power load over a two-year period. We also analyze the hourly recorded observations of ETTh2 and the 15-minute intervals of ETTm1 benchmarks. Additionally, we study ECL⁴⁴4https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014 (Electricity Consuming Load), which gathers electricity consumption data from 321 clients between 2012 and 2014. The WTH⁵⁵5https://www.ncei.noaa.gov/data/local-climatological-data/ dataset contains hourly records of 11 climate features from almost 1,600 locations across the United States. The Weather⁶⁶6https://www.bgc-jena.mpg.de/wetter/ is recorded every 10 minutes for 2020 whole year, which contains 21 meteorological indicators, such as air temperature, humidity. The Traffic⁷⁷7http://pems.dot.ca.gov/ is a collection of hourly data from California Department of Transportation, which describes the road occupancy rates measured by different sensors on San Francisco Bay area freeways.

Implementation Details For all benchmarks, we set the look-back window length at 60 and vary the forecast horizon from $H={1,24,48}$. We split the data into two phases: warm-up and online training, with a ratio of 25:75. We follow the optimization details outlined in [37] and utilize the AdamW optimizer [20] to minimize the mean squared error (MSE) loss. To ensure a fair comparison, we set the epoch and batch sizes to one, which is consistent with the online learning setting. We make sure that all baseline models based on the TCN backbone use the same total memory budget as FSNet, which includes three times the network sizes: one working model and two exponential moving averages (EMAs) of its gradient. For ER, MIR, and DER++, we allocate an episodic memory to store previous samples to meet this budget. For transformer backbones, we find that a large number of parameters do not benefit the generalization results and always select the hyperparameters such that the number of parameters for transformer baselines is fewer than that for FSNet. In the warm-up phase, we calculate the mean and standard deviation to normalize the online training samples and perform hyperparameter cross-validation. For different structures, we use the optimal hyperparameters that are reported in the corresponding paper.

Baselines of adaptation methods We evaluate several baselines for our experiments, including methods for continual learning, time series forecasting, and online learning. Our first baseline is OnlineTCN [39], which continuously trains the model without any specific strategy. The second baseline is Experience Replay (ER) [9], where previous data is stored in a buffer and interleaved with newer samples during learning. Additionally, we consider three advanced variants of ER: TFCL [4], which uses a task-boundary detection mechanism and a knowledge consolidation strategy; MIR [3], which selects samples that cause the most forgetting; and DER++ [8], which incorporates a knowledge distillation strategy. It is worth noting that ER and its variants are strong baselines in the online setting, as we leverage mini-batches during training to reduce noise from single samples and achieve faster and better convergence. Finally, we compare our method to FSNet [22] and OneNet [36], which are the previous state-of-the-art online adaptation model and strategy respectively. For OneNet, we use the average version for clarity, namely the combination strategy of two branches is just averaging.

Metrics Because learning occurs over a sequence of rounds. At each round, the model receives a look-back window and predicts the forecast window. All models are commonly evaluated by their accumulated mean-squared errors (MSE) and mean-absolute errors (MAE); namely, the model is evaluated based on its accumulated errors over the entire learning.

License. All the assets (i.e., datasets and the codes for baselines) we use include an MIT license containing a copyright notice and this permission notice shall be included in all copies or substantial portions of the software.

Environment. We conduct all the experiments on a machine with an Intel R Xeon (R) Platinum 8163 CPU @ 2.50GHZ, 32G RAM, and four Tesla-V100 (32G) instances. All experiments are repeated $3$ times with different seeds.

We present a brief overview of the baselines employed in our experiments.

First, OnlineTCN adopts a conventional TCN backbone [39] consisting of ten hidden layers, each layer containing two stacks of residual convolution filters.

Secondly, ER [9] expands on the OnlineTCN baseline by adding an episodic memory that stores previous samples and interleaves them during the learning process with newer ones.

Third, MIR [3] replaces the random sampling technique of ER with its MIR sampling approach, which selects the samples in the memory that cause the highest forgetting and applies ER to them.

Fourthly, DER++ [8] enhances the standard ER method by incorporating a knowledge distillation loss on the previous logits.

Finally, TFCL [4] is a task-free, online continual learning method that starts with an ER process and includes a task-free MAS-styled [2] regularization.

All the ER-based techniques utilize a reservoir sampling buffer, which is identical to that used in [22].

Hyper-parameters For the hyper-parameters of FSNet and the baselines mentioned in Section IV-B, we follow the setting in [22]. Besides, we cross-validate the hyper-parameters on the ETTh2 dataset and use them for the remaining ones. In particular, we use the following configuration:

• •

  Learning rate $3e-3$ on Traffic and ECL and $1e-3$ for other datasets. Learning rate $1e-2$ for the EGD algorithm and $1e-3$ for the offline reinforcement learning block, where the selection scope is $\{1e-3,3e-3,1e-2,3e-2\}$.

• •

  Number of hidden layers $10$ for both cross-time and cross-variable branches, where the selection scope is $\{6,8,10,12\}$.

• •

  Adapter’s EMA coefficient $0.9$, Gradient EMA for triggering the memory interaction $0.3$, where the selection scope is $\{0.1,0.2,\dots,1.0\}$.

• •

  Memory triggering threshold $0.75$, where the selection scope is $\{0.6,0.65,0.7\dots,0.9\}$.

• •

  Episodic memory size: 5000 (for ER, MIR, and DER++), 50 (for TFCL).

Cumulative Performance: This section explores the cumulative performance of various baselines, as illustrated in Table I, focusing on mean-squared errors (MSE) and mean-absolute errors (MAE). Notably, the D³A-OneNet, a novel approach that deploys OneNet with our D³A strategy, outperforms the majority of competing baselines across diverse forecasting horizons. Moreover, the flexibility of D³A enables seamless integration with advanced online forecasting methods and representation learning structures, enhancing the model’s overall robustness. Interestingly, in datasets with higher forecasting error rates, D³A demonstrates superior performance. For instance, when applied to the challenging ECL dataset with a forecasting horizon of 48, D³A achieves a remarkable improvement of $52.2\%$ over TCN and a substantial improvement of $46.2\%$ over FSNet. The average MSE and MAE values of D³A consistently outshine those obtained using traditional online learning strategies in isolation. This highlights the pivotal role played by D³A in incorporating concept drift detection and bias correlation during adaptation, underscoring its significance in achieving superior predictive accuracy.

Explore more efficient adaptation strategies. As discussed in subsection III-C, D³A plays a crucial role in addressing significant concept drift even when the model is occupied. However, we aim to enhance the efficiency of this process. Initially, we propose that, during the adaptation phase, only the linear regressor undergoes full fine-tuning until convergence, reducing backward computation time and avoiding the need for gradient storage from the resource-intensive encoder. However, a simple retraining approach proves ineffective, leading to prolonged and challenging convergence. To expedite the learning process, we employ a large learning rate ($1e-3$ by default) for warm-up, decaying it by a factor of $3$ when the average training losses in one epoch do not decrease. This strategy, termed D³A ^∗, achieves a favorable balance between effectiveness and efficiency. As depicted in Figure 4, on two representative datasets, the inference cost of D³A ^∗ is significantly lower than that of D³A and comparable to the baseline models. Simultaneously, the performance of D³A ^∗ does not compromise much compared to D³A.

Forecasting results visualization In contrast to baseline models that encounter challenges in adapting to new concepts and yield subpar forecasting outcomes, D³A adeptly captures intricate patterns within time series data. As illustrated in Figure 5, the baseline model struggles to dynamically adjust to evolving data patterns, leading to prolonged overfitting to the existing trend. For instance, when FSNet encounters a sequence of data instances depicting an increasing trend, it tends to persistently retain this trend for an extended duration. Conversely, D³A actively monitors forecasting losses, prompting timely updates to the model parameters whenever they are deemed inadequate for the current data pattern. This dynamic adaptation capability enables D³A to navigate shifting trends more effectively, resulting in improved forecasting accuracy.

This paper adheres to the experimental setup introduced in FSNet [22] and OneNet [36], where the true values of each time step are disclosed to enhance the model’s performance in subsequent rounds. Specifically, at time step $t$, with a look-back window $L$: $(\mathbf{x}_{i})_{i=t}^{L+t}$ and forecasting window $H$: $(\mathbf{x}_{i})_{i=L+1+t}^{L+H+t}$. Post-evaluation, the true values of $(\mathbf{x}_{i})_{i=L+1+t}^{L+H+t}$ are observed and utilized for training the model. Subsequently, at time step $t+1$, the input becomes $(\mathbf{x}_{i})_{i=t+1}^{L+t+1}$ and the task is to forecast $(\mathbf{x}_{i})_{i=L+2+t}^{L+H+t+1}$. Notably, there is a significant overlap in the forecasting target, with only one new point added. It is crucial to highlight that in real-world scenarios, the true values for the forecast horizon $H$ may not be accessible until $H$ rounds later, introducing the concept of online forecasting with delayed feedback. This scenario presents increased challenges and efficiency constraints, as the model cannot undergo retraining at each round, and training is restricted to every $H$ rounds. Tables II presents cumulative performance metrics, considering MSE and MAE. As expected, all methods exhibit reduced performance under delayed feedback compared to the traditional online forecasting setting. Notably, the SOTA method FSNet shows sensitivity to delayed feedback, particularly when $H=48$, falling behind a simple TCN baseline on specific datasets. Our proposed method, D³A, proves advantageous across all baselines, particularly on the challenging datasets like ETTH2, Weather, and ECL. Specifically, concerning the average MSE on the challenging subset, D³A reduces TCN’s MSE by $\mathbf{33\%}$, while achieving performance gains of $\mathbf{34.2\%}$ and $\mathbf{23.7\%}$ for FSNet and OneNet, respectively. Additionally, D³A outperforms existing online learning baselines by a significant margin.

Ablation studies of different hyper-parameters. We present ablation studies investigating the impact of different hyperparameters, specifically the loss weight $\lambda$ during adaptation and the memory size $l_{w}$. As depicted in Figure 6a, various datasets exhibit preferences for distinct choices. Overall, datasets with a greater number of variables tend to benefit more from the proposed augmentation strategy. This is intuitively explained by the challenging nature of learning the ECL dataset, and our augmentation strategy effectively utilizes debiased historical data to expedite the learning process. For the ETTh2 dataset, a smaller weight of $0.1$ yields optimal performance. This suggests that only a modest regularization is necessary, and training the model solely on the drifted instances allows for quick adaptation to new concepts in these datasets. Consequently, for datasets with a substantial number of variables, we default to $\lambda=2.0$, and $0.1$ otherwise. Regarding memory size, we observe a tradeoff, as a small memory size proves sensitive to concept drift and noise, while a large memory may not effectively signal when concept drift occurs. For simplicity, we choose $l_{w}=16$ as the default memory size for all datasets.

The confidence level is represented by the threshold $\alpha_{t}$ rather than the direct use of $\alpha$. Given a specific value for $\alpha_{t}$, we calculate the corresponding threshold $\alpha$ such that the area under the normal curve beyond this threshold aligns with the specified significance level ($\alpha_{t}$). Utilizing the factor $\alpha=1-\alpha_{t}/2$ is essential for a two-tailed test. In our ablation studies, it becomes evident that as the confidence level decreases, we impose a tighter constraint on drift detection. Only when the shift in the loss distribution is significant enough does the detector trigger an alarm. Consequently, the occurrence of drift alarms and the total re-training instances are reduced. Regarding model performance, our observations indicate that more frequent retraining does not consistently enhance performance. In contrast, optimal model performance is achieved when retraining occurs judiciously based on proper data and timing considerations.

Detecting Concept Drift Events: When Does the Concept Detector Trigger an Alarm? We explore the scenarios in which our concept detector initiates a concept drift alarm by visualizing data instances and losses. The model is trained on the ECL dataset with $H$ set to $48$. As illustrated in Figure 7, in cases 1 and 2, discernible patterns emerge simultaneously in both data distribution and losses, prompting the detector to raise an alarm—an intuitive response. Intriguingly, in case 3, data patterns may not be as straightforward for human observers, yet our detector can discern a significant increase on the forecasting error, prompting the model to adapt to the new concept.