Enabling Time-series Foundation Model for Building Energy Forecasting via Contrastive Curriculum Learning

The design of the difficulty measurer is usually based on model performance or data pattern analysis. Considering that the curriculum is for adapting an existing TSFM $\mathcal{M}$, which has been pre-trained with various knowledge and patterns, we can directly make inference with the TSFM and use the performance as the difficulty score of samples in the real-world dataset ${D}_{train}$ (Eq. 1). Here, $\mathcal{L}(\cdot,\cdot)$ denotes the prediction error of the TSFM in terms of CV-RMSE.

For the simulated dataset $D^{\prime}_{train}$, the key challenge is that the difficulty measurer for $u$ is not suitable for $u^{\prime}$ because $\mathcal{L}(\mathcal{M}(x^{\prime}),y^{\prime})$ introduces bias. We then leverage contrastive learning to predict the TSFM comprehension on the representation of $u^{\prime}$ and hence to determine the difficulty. We first define TSFM comprehension and contrastive pairs, based on which we introduce the contrastive model and how to estimate the difficulty of $u^{\prime}$.

Note that, contrastive pairs construction relies solely on real $\{u\}$ and we set $\delta$ to 0.01 based on our experimental analysis. It is a typical phenomenon that two dissimilar samples, which appear dissimilar in terms of time-series patterns (e.g., as measured by DTW similarity), may correspond to a similar TSFM comprehension, indicated by a low value of $C_{\mathcal{M}}(\cdot,\cdot)$. This is related to the uncertain knowledge encapsulated by the TSFM.

The right part of Figure 2 shows the design of our contrastive learning model $f$. Considering the huge amount of negative pairs, we leverage the classical memory bank structure and adopt temporal convolutional network (TCN) as the encoder since it can be trained efficiently and shows superior performance in capturing daily and weekly seasonality, which are major temporal patterns in building energy time-series. As shown in Eq. 2, we apply InfoNCE loss (Oord et al., 2018) and introduce the value of $C_{\mathcal{M}}(u,u_{k})$ as the weight $\omega_{k}$ of negative pairs. Here, $u_{j}$, $u_{k}$ are positive and negative samples of $u$, $sim(\cdot,\cdot)$ calculates the cosine similarity between each pair of data embeddings, and $\tau$ is a scaling parameter. After training $f$, we obtain the difficulty of a simulated sample $u^{\prime}$ through Eq. 3.

After measuring the difficulty of samples, we leverage a linear continuous scheduler to select training samples at each epoch. Specifically, samples are first sorted by their difficulty. Then, a function $\lambda(t)=\text{min}(1,\lambda_{0}+(1-\lambda_{0})\cdot t/T_{\text{grow}})$ decides the percentage of the easiest samples to be used at the $t$-th epoch, where $\lambda_{0}$ denotes the initial percentage of the easiest samples for training and $T_{\text{grow}}$ is the epoch when $\lambda(t)$ grows to 1. Then, let $\mathcal{D}=\{v_{i}\}_{i=1}^{n}$, the training set at the $t$-th epoch is given by $\mathcal{D}_{t}=\{v_{i}\}_{i=1}^{n\cdot\lambda(t)}$. In order to let TSFMs fully explore patterns in building energy data, we set the number of training epochs larger than $T_{\text{grow}}$, which means that TSFMs will be trained using the whole training set.

TSFMs. We adopt two product-level TSFMs, which are Tiny Time Mixer (TTM) (Ekambaram et al., 2024) from IBM and Chronos (Ansari et al., 2024) from Amazon, for adapting to BEF tasks³³3We use the variants with the largest number of parameters for both TSFMs, which are TTM-5M and Chronos-710M..

Datasets. Simulated and real-world public building energy datasets are used for experiments: (1) Buildings-900K (Emami et al., 2023). This dataset contains hourly energy consumption time-series from 900k simulated buildings over two years. (2) Building Data Genome Project (BDG) (Miller et al., 2020). This project aggregates 19 real-world building energy datasets from different locations around the world (totaling 1,636 buildings), where hourly energy meter data over a two-year period are collected for each building. (3) UCI Electricity (Trindade, 2015). This dataset collects electricity consumption data from 370 houses for four years, sampled at 15-minute interval.

Baselines & Metrics. To explore improvement achieved by our method, we compare the TSFMs fine-tuned with our method (denoted as TSFM+CCL-FT) against: (1) the original pre-trained TSFMs (denoted as TSFM); and (2) the TSFMs directly fine-tuned without our method (denoted as TSFM+FT). Besides, we adopt three state-of-the-art time-series forecasting models adopted in BEF field for comparisons: LSTM (Chitalia et al., 2020) (a classical RNN architecture for handling sequential data), Autoformer (Jiang et al., 2022) and Temporal Fusion Transformer (TFT) (Giacomazzi et al., 2023) (two transformer-based models tailored for time-series forecasting). For performance evaluation, we consider a standard metric in BEF field: the Coefficient of Variation of the Root Mean Square Error (CV-RMSE): $\frac{\sqrt{\sum_{i=1}^{n}(y_{i}-\hat{y})^{2}/n}}{\sum_{i=1}^{n}y_{i}/n}$.

Setup. We select five datasets from BDG together with the UCI dataset as the evaluation set since they cover most building types and climate conditions. In zero-shot setting, all data from the target building are used for testing. In few-shot setting, 10% of data are used for training and the remaining 90% of data are used for testing. The other 15 datasets from BDG and the simulated dataset Buildings-900K are used for TSFMs fine-tuning. The number of fine-tuning steps is set to 1000. The look-back window length and forecast horizon is set by default values of TSFMs during fine-tuning.⁴⁴4512-96 for TTM and 512-64 for Chronos. For evaluation, we set three forecast horizons, i.e., 24, 96, 192, as TSFMs can adapt to different horizons. The experiments are conducted on a Linux server with two NVIDIA GeForce RTX 4090 24GB GPUs.

Overall performance. We evaluate our method and baselines in zero-shot and few-shot settings under three forecast horizons in Table 2. Overall, TSFM+CCL-FT consistently outperforms TSFM and TSFM+FT in zero-shot setting, with average improvements in CV-RMSE at 18.3%, 11.3% for TTM and 12.6%, 8.5% for Chronos. Similar results are observed in few-shot setting where our method surpasses baselines by 14.9%, 11.9% for TTM and 12.7%, 8.9% for Chronos. Besides, as CV-RMSE ¡ 0.3 is an industrial requirement defined by ASHRAE (ASHRAE, 2002) for deployable forecasting models, we observe that on each dataset, there are cases where our method successfully reduces the error of pre-trained TSFMs to less than 0.3. Moreover, we find that although our CCL method enhances the performance of both TSFMs over their pre-trained version, the improvement ratio is higher on TTM. This is likely due to that TTM has a smaller model size and there are less energy datasets in its pre-training set such that the effect of our method is significant. Additionally, we notice that for Chronos, TSFM+FT only slightly outperforms TSFM by 4.3% which is consistent with our preliminary experiment result on TTM.

Next, we compare TSFM+CCL-FT with three state-of-the-art baseline forecasting models. Here, TSFM+CCL-FT is evaluated under few-shot setting while the baselines are first trained using the first 50% of data from each test building and then tested on the remaining 50% of data. As shown in Table 3, we observe that Chronos fine-tuned with the CCL strategy outperforms the best of baselines on almost all datasets, with an improvement of 14.6% on average. For TTM, although its performance is improved with our method, it still lags behind the best baseline, particularly on the BDG-Rat and UCI datasets.

Ablation Study. To take a closer look at the contribution of the designed contrastive-aware difficulty measurer in our method, we implement a variant of our method named TSFM+CL-FT, which simply uses the performance of TSFMs as difficulty for both real and simulated samples. The zero-shot performance of our method and this variant is compared in Figure 4. We observe that TSFM+CCL-FT outperforms TSFM+CL-FT by 7.4% and 7.7% for TTM and Chronos, respectively. With further analysis, we find that TSFM+CL-FT is still better than TSFM-FT under the same experiment setting, which validates the effectiveness of curriculum learning in enhancing TSFMs adaptation.

Next, we study the effect of the size of fine-tuning set on the performance of TSFMs. In Figure 4, we evaluate the corresponding versions of fine-tuned TSFMs under varying proportion of fine-tuning set. The performance of the original pre-trained TSFMs is included for reference. The results indicate the superior performance of our method on each setting. Specifically, we observe that the improvement over baseline increases along with the size of fine-tuning set from 2.9% to 13.2%. This implies the capability of our method in handling a larger and more complicated dataset.