Channel-aware Contrastive Conditional Diffusion for Multivariate Probabilistic Time Series Forecasting

In this paper, we look into the task of multivariate probabilistic time series forecasting. Given the past observation $\mathbf{x}\in\mathbb{R}^{L\times D}$ as conditioning time series, the goal is to generate a group of $S$ plausible forecasts $\{\hat{\mathbf{y}}_{0}^{(s)}\in\mathbb{R}^{H\times D}\}_{s=1}^{S}$ from the learned conditional predictive distribution $p_{\theta}(\mathbf{y}_{0}|\mathbf{x})$. Here, $D$ is the number of channels, $L$ and $H$ indicate the lookback window length and prediction horizon respectively. $\theta$ stands for the parameters of a conditional diffusion forecaster which represents the real predictive distribution $q(\mathbf{y}_{0}|\mathbf{x})$. We allocate diverse values to horizon $H$ and channel number $D$ to construct a holistic benchmark which can completely evaluate the capability of different conditional diffusion models on various forecasting scenarios.

Conditional diffusion models have exhibited impressive capability on a wide variety of controllable multi-modal synthesis tasks (Chen et al., 2024b). It dictates a bi-directional distribution transport process between raw data $\mathbf{y}_{0}$ and prior Gaussian noise $\mathbf{y}_{K}\in\mathcal{N}(\mathbf{0},\mathbf{I})$ via $K$ diffusion steps. The forward process gradually degrades clean $\mathbf{y}_{0}$ to fully noisy $\mathbf{y}_{K}$ and can be fixed as a Markov chain: $q(\mathbf{y}_{0:K})=q(\mathbf{y}_{0})\textstyle\prod_{k=1}^{K}q(\mathbf{y}_{k}|\mathbf{y}_{k-1})$, where $q(\mathbf{y}_{k}|\mathbf{y}_{k-1}):=\mathcal{N}(\mathbf{y}_{k};\sqrt{1-\beta_{k}}\mathbf{y}_{k-1},\beta_{k}\mathbf{I})$ and $\beta_{k}$ is the degree of imposed step-wise Gaussian noise. We can accelerate the forward sampling procedure and obtain closed-form latent state $\mathbf{y}_{k}$ at arbitrary step $k$ by a noteworthy property (Ho et al., 2020): $\mathbf{y}_{k}=\sqrt{\bar{\alpha}_{k}}\mathbf{y}_{0}+\sqrt{1-\bar{\alpha}_{k}}\bm{\epsilon}$, where $\bar{\alpha}_{k}:=\textstyle\prod_{s=1}^{k}(1-\beta_{s})$ and $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. The reverse generation process converts known Gaussian to realistic prediction data $\mathbf{y}_{0}$ given input conditions $\mathbf{x}$, which can be cast as a parameterized Markov chain: $p_{\theta}(\mathbf{y}_{0:K}|\mathbf{x})=p(\mathbf{y}_{K})\textstyle\prod_{k=K}^{1}p_{\theta}(\mathbf{y}_{k-1}|\mathbf{y}_{k},\mathbf{x})$. The overall training objective can be simplified as minimizing the step-wise denoising loss below:

A potential issue for current conditional diffusion models lies in forging an effective conditioning mechanism that can enhance the alignment between given conditions $\mathbf{x}$ and produced data $\mathbf{y}_{0}$, like the coherent semantics between textual descriptions and visual renderings (Esser et al., 2024), or the conformity of generated vehicle motions to scenario constraints (Jiang et al., 2023). However, such data consistency is hard to represent for temporal conditional probability modeling. We thus explicitly learn to amplify the prediction-related temporal information conveyed from past conditioning time series to generated trajectories. Such predictive mutual information can reflect underlying temporal properties in historical sequences, to which the produced forecasts should comply.

As discussed above, to more efficiently represent the useful predictive modes involved in conditioning time series, we choose to explicitly maximize the prediction-oriented mutual information when learning the conditional diffusion forecaster. Learning to maximize mutual information is effective to boost the consistency between two associated variables (Song & Ermon, 2019), which has been actively applied to self-supervised learning (Liang et al., 2024b) and multi-modal alignment (Liang et al., 2024a). Regarding conditional diffusion learning, there also exist several related works (Wang et al., 2023; Zhu et al., 2022) which explicitly employ mutual information maximization to enhance high-level semantic coherence between input prompts and generated samples. While we propose a complementary way to equip the conditional diffusion forecaster with this tool to bolster the utilization of informative temporal patterns. Besides, we provide a distinct composite loss design and more concrete interpretations on the benefits of the contrastive scheme to ordinary conditional diffusion.

Among the two practical methods to maximize the intractable mutual information (Tsai et al., 2020), contrastive learning aids to strengthen the association by discriminating intra-class from inter-class samples. Contrastive predictive coding (Oord et al., 2018) realizes such objective by optimizing the contrastive lower bound with low variance via the prevalent InfoNCE loss:

During each iteration, we create a set of $N$ negative samples via the negative construction operation $q^{n}(\mathbf{y}_{0})$ on positive data $\mathbf{y}_{0}$. $f(\mathbf{y}_{0},\mathbf{x})$ accounts for the density ratio $\frac{q(\mathbf{y}_{0}|\mathbf{x})}{q(\mathbf{y}_{0})}$ and can be any types of positive real functions. This flexible form of the density ratio function offers a natural initiative of the following denoising-based contrastive conditional diffusion.

Forward predictive learning is another way to boost the inter-dependency by fully reconstructing target $\mathbf{y}_{0}$ conditioned on given $\mathbf{x}$. This reconstruction learning can be realized by learning a deterministic mapping or conditional generative model from $\mathbf{x}$ to $\mathbf{y}_{0}$. As $I(\mathbf{y}_{0};\mathbf{x})=H(\mathbf{y}_{0})-H(\mathbf{y}_{0}|\mathbf{x})$, and $H(\mathbf{y}_{0})$ is irrelevant to discovering the entanglement between $\mathbf{x}$ and $\mathbf{y}_{0}$, thereby maximizing $I(\mathbf{x};\mathbf{y}_{0})$ boils down to optimizing the predictive lower bound $-H(\mathbf{y}_{0}|\mathbf{x})=\mathbb{E}_{q(\mathbf{x},\mathbf{y}_{0})}[\log{p_{\theta}(\mathbf{y}_{0}|\mathbf{x})}]$, which is aligned with the likelihood-based objective of naive conditional diffusion learning. (Tsai et al., 2020) claims that combining both predictive and contrastive learning tactics can significantly raise the quality of obtained task-related features. Accordingly, we equip vanilla conditional time series diffusion with a denoising-based InfoNCE contrastive loss to further boost the temporal predictive information between past conditions and future forecasts. A concise motivation of this information-theoretic contrastive diffusion forecasting is depicted in Fig. 1.

Recent progress on multivariate prediction methods (Liu et al., 2023; Ilbert et al., 2024) show that proper integration of channel management strategies in time series backbones is critical to discover univariate dynamics and cross-variate correlations. But this problem has not been well explored in multivariate diffusion forecasting and previous conditional denoiser structures do not obviously distinguish such heterogeneous channel-centric temporal properties. To this end, we design a channel-aware conditional denoising network which incorporates composite channel manipulation modules, i.e. channel-independent dense encoders and channel-mixing diffusion transformers. This architecture can efficiently represent intra-variate and inter-variate temporal correlations in past conditioning $\mathbf{x}$ and future predicted $\mathbf{y}_{0}$ under different noise levels, as well as being robust to diverse prediction horizons and channel numbers.

Channel-independent dense encoders. We develop two channel-independent MLP encoders to extract unique temporal variations in each individual channel of observed condition $\mathbf{x}$ and corrupted latent state $\mathbf{y}_{k}$ at each diffusion step. The core ingredient in latent and condition encoders is the channel-independent dense module (CiDM) borrowed from TiDE (Das et al., 2023a), which stands out as a potent MLP building-block for universal time series analysis models (Das et al., 2023b). A salient element in CiDM is the skip-connecting MLP residual block which can improve temporal pattern expressivity. The $D$ linear layers in parallel are shared and used for separate channel feature embedding. We stack $n_{enc}$ CiDM modules of $e_{hid}$ hidden dimension to transform both $\mathbf{x}$ and $\mathbf{y}_{k}$ into $e_{hid}\times D$ size. These two input encoders can be easily adjusted to accommodate different context windows and hidden feature dimensions.

Channel-wise diffusion transformers. To regress step-wise Gaussian noise $\bm{\epsilon}_{k}$ on raw $\mathbf{y}_{0}$ more precisely, we should fully exploit implicit temporal information in pure conditioning $\mathbf{x}$ and polluted target $\mathbf{y}_{k}$. We concatenate the latent encoding of $\mathbf{x}$ and $\mathbf{y}_{k}$ along the channel axis and then leverage $n_{att}$-depth channel-wise diffusion transformer (DiT) blocks to aggregate heterogeneous temporal modes from various channels. DiT is an emergent diffusion backbone for open-ended text-to-image synthesis which merits eminent efficiency, scalability and robustness (Peebles & Xie, 2023; Esser et al., 2024). Two critical components in DiT are multi-head self-attention for feature fusion and adaptive layer norm (adaLN) layers to absorb other conditioning items (e.g. diffusion step embedding, text labels) as learnable scale and shift parameters. Although DiT has been adopted by few works (Cao et al., 2024; Feng et al., 2024) for generative time series modeling, they do not adapt it in a fully channel-centric angle. We repurpose DiT to model multivariate predictive distribution by replacing naive point-wise attention to a channel-wise version, which can blend correlated temporal information from different variates in $\mathbf{x}$ and $\mathbf{y}_{k}$. Afterwards, we develop a output decoder with $n_{dec}$ CiDMs plus a last adaLN to yield the prediction of imposed noise $\bm{\epsilon}_{k}$ given $\mathbf{x}$ and $\mathbf{y}_{k}$.

Unlike previous empirically designed temporal conditioning schemes to make better exploitation of past predictive information, we instead propose to explicitly maximize the prediction-related mutual information $I(\mathbf{y}_{0};\mathbf{x})$ between past observations $\mathbf{x}$ and future forecasts $\mathbf{y}_{0}$ via an adapted denoising-based contrastive strategy. We will employ the learnable denoising network $\bm{\epsilon}_{\theta}(\cdot)$ to represent the contrastive lower bound of $I(\mathbf{y}_{0};\mathbf{x})$ presented by Eq. 2, and exhibit this information-theoretic contrastive refinement is complementary and aligned with original conditional denoising diffusion optimization, which is actually another forward predictive method to maximize $I(\mathbf{y}_{0};\mathbf{x})$.

To improve the diffusion forecasting capacity more essentially, the developed contrastive learning item is wished to directly benefit naive step-wise denoising-based training procedure, i.e. regularizing noise elimination behaviors of the conditional denoiser $\bm{\epsilon}_{\theta}(\cdot)$. Since the density ratio function $f(\mathbf{y}_{0},\mathbf{x})$ constituting the contrastive mutual information lower bound in Eq. 2 can be any positive-valued forms, this flexibility naturally motivates us to prescribe $f(\cdot)$ using the step-wise denoising objective in Eq. 1, for both positive sample $\mathbf{y}_{0}$ and a group of negative samples $\mathbf{y}_{0}^{(n)}$:

where $\tau$ is the temperature coefficient in the softmax-form contrastive loss. The negative sample set $\{\mathbf{y}_{0}^{(n)}\}_{n=1}^{N}$ is constructed by a hybrid time series augmentation method which alters both temporal variations and point magnitudes (See Appendix A.3 for details.). Then, we can derive the following contrastive refinement loss which is coincident with vanilla step-wise denoising diffusion training:

Apparently, the devised denoising-based temporal contrastive learning can not only seamlessly coordinate with standard diffusion training at each step $k$, but also improve the conditional denoiser behaviors in out-of-distribution (OOD) regions. These OOD areas are constituted by the low-density diffusion paths of negative samples, which are not touched by merely executing denoising learning along the high-density probability paths of positive samples.

The naive denoising diffusion model trained by log-likelihood maximization (Ho et al., 2020) totally owns $K$-step valid training items. To align with this step-wise denoising distribution learning, we can amortize the contrastive regularization in Eq. 4 to each training step, and derive the overall learning objective below:

where $\log p_{\theta}(\mathbf{y}_{0}|\mathbf{x})$ can be decomposed as $\textstyle\sum_{k=1}^{K}\mathcal{L}_{k}^{denoise}$ and indicates the predictive distribution learning. Whilst $\max_{\theta}I_{\theta}(\mathbf{y}_{0};\mathbf{x})$ governs the information-theoretic contrastive learning. Then, the practical training loss of the devised CCDM at each diffusion step can be presented as:

So far, we obtain the overall step-wise training procedure for CCDM, which is a $\lambda$-weighted combination of the vanilla denoising term in Eq. 1 and auxiliary contrastive item in Eq. 4. The whole training algorithm is clarified in Appendix A.4, which is efficient, end-to-end and seamlessly coupled with original simplified denoising diffusion.

Theoretical insights. Beyond the concrete method described above, we offer two-fold interpretations on how conditional diffusion forecasters can gain from auxiliary temporal contrastive learning. From the neural mutual information perspective, Eq. 1 and Eq. 4 train the parameterized conditional denoiser $\bm{\epsilon}_{\theta}(\cdot)$ to simultaneously optimize two complementary lower bounds (i.e. predictive and contrastive) of the prediction-related mutual information $I_{\theta}(\mathbf{y}_{0};\mathbf{x})$ between future forecasts and past conditioning time series. According to (Tsai et al., 2020), this composite learning method can enhance the representation efficiency to distill task-related information from accessible conditions. The contrastive scheme assists $\bm{\epsilon}_{\theta}(\cdot)$ to learn helpful negative instances, which can gain useful discriminative temporal patterns for accurate multivariate predictive distribution recovery and mitigate over-fitting on historical training time series. From the distribution generalization perspective, explicitly optimizing the probabilities of unexpected negative samples can render $\bm{\epsilon}_{\theta}(\cdot)$ see more OOD regions that pure denoising fashion on positive in-distribution samples do not encompass. In time series forecasting, there always exists distribution shift between unforeseen testing data and historical training data. The contrastive term in Eq. 4 intuitively minimizes the possibility $\log p_{\theta}(\mathbf{y}_{0}^{(n)})$ of undesirable spurious forecasts by directly impeding $\bm{\epsilon}_{\theta}(\cdot)$ from correctly removing the noise over negative $\mathbf{y}_{0}^{(n)}$. This contrastive enforcement helps $\bm{\epsilon}_{\theta}(\cdot)$ avoid low-density areas and undergo more OOD areas during in-distribution training. In light of the arguments in (Wu et al., 2024), promoting the denoiser robustness in OOD regions in testing stage is crucial to sample plausible forecasts.

Moreover, we reveal the upper bound of forecasting error on testing data for conditional diffusion models in Proposition 1. It obviously reflects that the efficacy of conditional diffusion forecasters is inextricably intertwined with the step-wise noise regression accuracy of trained $\bm{\epsilon}_{\theta}(\cdot)$ on unknown test time series. In this regard, resorting to temporal contrastive refinement or other auxiliary training regimes is sensible to boost conditional denoiser behaviors and final prediction outcomes.

Proposition 1. Let $q^{te}(\mathbf{y}_{0}|\mathbf{x})$ be the ground truth distribution of test time series, and $p^{te}_{\theta}(\mathbf{y}_{0}|\mathbf{x})$ be the approximated predictive distribution by the developed conditional diffusion model. Let the KL-divergence between $q^{te}(\mathbf{y}_{0}|\mathbf{x})$ and $p^{te}_{\theta}(\mathbf{y}_{0}|\mathbf{x})$ represent the resulting probabilistic forecasting error. Then the denoising diffusion-induced forecasting error is upper-bounded:

Such upper bound is determined by the denoising behaviors of learned $\bm{\epsilon}_{\theta}(\cdot)$ on unknown test time series. $A_{k}$ is a step-wise constant related to noise schedule, and $C$ is a constant depending on test data quantities. See Appendix A.1 for the proof.

Datasets. We choose six multivariate time series datasets, i.e. ETTh1, Exchange, Weather, Appliance, Electricity, Traffic, which cover a wide range of temporal dynamics and channel number $D$ to completely gauge the probabilistic forecasting performance. We manually establish a more comprehensive benchmark with diverse values of lookback window $L$ and prediction horizon $H$, distinct from previous models which merely attest their generative forecasting capacity on a single short-term setup. Refer to Appendix. A.5 for more details on datasets.

Evaluation metrics. We adopt two standard metrics to assess the quality of both probabilistic and deterministic forecasts resulting from the generated prediction intervals. CRPS (Continuous Ranked Probability Score) is used to assess the reliability of the estimated predictive distribution, and MSE (Mean Squared Error) is used to quantify the accuracy of calculated point forecasts. See Appendix A.6 for more details on metrics.

Baselines. We select five currently remarkable denoising diffusion-based generative forecasters for comparisons, including TimeGrad (Rasul et al., 2021), CSDI (Tashiro et al., 2021), SSSD (Alcaraz & Strodthoff, 2022), TimeDiff (Shen & Kwok, 2023), TMDM (Li et al., 2023). Since these models do not shed light on outcomes on long-term probabilistic forecasting scenarios, we fully reproduce them on the newly constructed benchmark.

Implementation details. We normally execute the end-to-end contrastive diffusion training in Eq. 6 using 200 epochs. To reduce the contrastive learning costs on those cases which consume enormous computational resources, we also employ a cost-efficient two-stage training strategy. Concretely, we firstly pretrain a low-cost naive diffusion forecaster by Eq. 1 and fine-tune it by the total contrastive manner in Eq. 6 with only 30 epochs. We keep the temperature coefficient $\tau=0.1$ and randomly generate $S=100$ mulivariate profiles to compose prediction intervals. See Appendix A.7 for more details on network architecture and contrastive training configurations in different forecasting setups. All experiments are conducted on a single NVIDIA A100 40GB GPU.

We demonstrate the devised CCDM model can outperform existing diffusion forecasters on most of the generative forecasting cases in Table 1. Concretely, CCDM can attain the best outcomes on $16/24$ deterministic and $20/24$ probabilistic evaluations, with $9.10\%$ and $15.66\%$ average improvement of MSE and CRPS on these cases. Especially on two most difficult datasets Electricity and Traffic, CCDM garners notable progress of $8.94\%$, $10.59\%$ on MSE and $19.73\%$, $19.10\%$ on CRPS. These prominent increases reflect the devised channel-centric structure and contrastive refinement on the diffusion forecaster can enhance its representation efficiency of implicit predictive information on diverse prediction scenarios. The second-best model CSDI also manifests excellent forecasting ability especially on Weather, which has complex multivariate temporal correlations. The hybrid attention module in CSDI can well capture these relations but it entices high computational overhead and over-fitting to other datasets. TMDM and TimeDiff also attain small MSE on few cases due to their extra deterministic pre-training operations on conditioning encoders. Note that we completely replicate TimeGrad on the whole benchmark for the first time even with severe inference costs, and validate it can actually realize reasonable forecasting results. In Fig. 4, we depict different diffusion produced prediction intervals on one case. We can clearly see that CCDM’s interval is much more faithful, while TimeDiff’s area is sharper but loses diversity and accuracy. See Appendix A.11 for more forecasting result showcases and Appendix A.8 on time cost analysis.

Below, we empirically investigate the efficacy of the devised denoising-based temporal contrastive refinement, including three vital factors for contrastive learning practice and its generality on other existing diffusion forecasters.

Influence of contrastive weight $\lambda$. The complementary step-wise denoising-based contrastive loss in 4 can enhance the alignment between diffusion generated forecasts and given temporal predictive information. To elucidate the impact of contrastive refinement in different degrees on original diffusion optimization, we escalate the contrastive weight $\lambda$ in Eq. 5 from 0.0001 to 0.01 and display corresponding forecasting outcomes in Fig. 5. We can totally find that imposing contrastive regime on denoising diffusion training can indeed promote the generative forecasting capacity, and the gain margin moderately fluctuates among various weights and datasets. Roughly, a modest weight between 0.0005 and 0.005 can lead to better improvement. We also empirically observe that larger $\lambda$ can accelerate the diffusion training convergence. See Appendix A.10 for more detailed analysis on the influence of negative number $N$ and temperature $\tau$.

Generality of contrastive training. We add the step-wise denoising contrastive training presented in 4 to two existing diffusion forecasters to validate its generality on conditional time series diffusion learning. From the results shown in Table 3, it is obvious that CSDI’s generative forecasting ability can be further enhanced by contrastive diffusion training. Its hybrid attention network can represent complex temporal patterns more properly by handling more OOD negative samples. While for TimeDiff which owns extra pre-trained auto-regressive conditioning encoders, CRPS values constantly decrease but some unexpected increases appear on MSE. It may stem from the side effect of redundant contrastive procedure conveyed to the well-behaved deterministic pre-training strategy.