Skillful Nowcasting of Convective Clouds With a Cascade Diffusion Model

Previous studies have employed multiple deep learning models, each tailored to specific lead time intervals, to mitigate the accumulation of prediction errors [8, 9]. In this study, constrained by computational resources, we train only two models within a cascade framework. SATcast-phase 1 predicts cloud evolution for lead times from T+1 to T+4, while SATcast-phase 2 utilizes outputs from SATcast-phase 1 to generate predictions for lead times from T+5 to T+8. For lead times beyond T+8, we evaluate model performance by iteratively applying SATcast-phase 1 to generate satellite image forecasts.

Figure 1 compares the root mean square error (RMSE), correlation coefficient, multi-scale structure similarity (MSSSIM) [32], and critical success index (CSI) between SATcast, the persistence model, and the optical flow method. The CSI is calculated using a threshold of 240 K. Results are spatially averaged over the region of interest (86^∘ to 150^∘ E in longitude and 1^∘ to 41^∘ N in latitude), based on 512 testing sequences, each spanning 32 hours (the first 8 hours serve as input, and the following 24 hours as the target), from September to December 2022. The red shading in SATcast denotes the one standard deviation calculated across lead time ranges for samples with different initialization times.

As shown in Figure 1, all metrics for SATcast and the persistence model degrade sharply up to T+8 but stabilize after T+12. In contrast, the performance of the optical flow method decreases consistently. Notably, the persistence model exhibits slight performance improvements between T+20 and T+24, likely due to the influence of the diurnal cycle [33]. Overall, SATcast consistently outperforms both the persistence model and the optical flow method across all metrics. The correlation coefficient for SATcast stabilizes above 0.7, significantly surpassing the persistence baseline, which drops below 0.5 after T+12. Similarly, SATcast achieves a CSI of around 0.5, indicating reliable detection of convective clouds, while the persistence model’s CSI falls below 0.3. The MSSSIM for SATcast remains approximately 0.6, suggesting that the quality of predicted cloud fields remains high even as the lead time increases. In contrast, the MSSSIM for the persistence model is about 0.45, due to variations in weather systems. These results demonstrate that the cascade framework with single-round fine-tuning effectively reduces cumulative errors, resulting in minimal degradation of predictive performance beyond T+8. The stability of SATcast’s metrics beyond T+12 further underscores its reliability for longer-term cloud predictions.

Figure 2 demonstrates the SATcast’s 24-hour forecasting capability using a typhoon case. The spatial distribution of brightness temperature from the optical flow method becomes very smooth after several time steps. We focus on SATcast’s performance, with the optical flow results presented in Supplementary Figure S12. SATcast successfully captures the spatiotemporal evolution of a large-scale convection system moving from northern China to Japan. It also accurately predicts two tropical disturbances east of the Philippines, which latter intensified into Typhoon Sonca and Typhoon Nesat. The model reproduces the complex intensity fluctuations, showing an initial weakening followed by intensification, due to interactions with cold eddies, within the 24-hour period. Predicted typhoon intensity and track closely align with observations, as shown in Supplementary Figure S1, with additional cases provided in Supplementary Figures S2–S3. Notably, SATcast achieves these high-fidelity predictions with a single round of fine-tuning, effectively mitigating forecast error accumulation. This continuous forecasting capability circumvents prediction discontinuities, suggesting SATcast as a reliable framework for convective cloud forecasting.

In addition, SATcast demonstrates strong generalization capability. Despite being trained exclusively on channel 12 data, it effectively predicts the brightness temperature on channel 9 by incorporating FuXi data. Relevant metrics and examples are provided in Supplementary Figures S9-S10.

In this subsection, we compare three variants of SATcast (see details in Table 1) for forecast lead times up to 8 hours, using 1,024 sequences, each spanning 16 hours (with the first 8 hours as input and the subsequent 8 hours as the target). The sequences, collected from September to December 2022, are used to evaluate the effectiveness of various modules within SATcast.

Figure 3 presents the metrics, spatially averaged over the region of interest (86^∘ to 150^∘ E in longitude and 1^∘ to 41^∘ N in latitude), for the persistence model, SATcast, and the three SATcast variants. CSI is still computed using a threshold of 240 K. Among all the variants, SATcast-NoC, which directly predicts the entire 8-hour sequence, performs the worst for lead times T+1 to T+3. This is likely due to the backward propagation of errors from frames beyond T+4, which hinders its optimization for earlier predictions. However, SATcast-NoC outperforms SATcast-NoF, which excludes FuXi forecasts for lead times beyond T+4, suggesting that FuXi forecasts enhance performance, as satellite imagery alone are insufficient for predicting longer lead times.

SATcast-NoF performs well up to T+3 but deteriorates rapidly thereafter, likely due to its inability to capture the physical processes underlying convective cloud evolution. To further examine the impact of fine-tuning on SATcast predictions beyond the initial four hours, a variant without fine-tuning, SATcast-NoT, performs better than other models except SATcast. Fine-tuning is essential in aligning SATcast’s predictions with the data distribution of satellite imagery, as SATcast-NoT often overestimates values beyond T+4 (Supplementary Figure S8).

Although the error differences among models appear subtle, Figure 4 and Supplementary Figures S4-S7 reveal significant variations in their ability to predict the spatial distribution of convection, with SATcast showing the best skill. Figure 4 also shows SATcast’s performance in predicting the intensification of Typhoon Nanmadol and the dissipation of Typhoon Muifa on September 13, 2022. Muifa, one of the most powerful typhoons on record to strike Shanghai, and, Nanmadol, one of the strongest typhoons of 2022, caused substantial damage, undersocring the importance of accurate forecasts.

Starting at T+1 (2022-09-14-07:00 UTC), SATcast and its variants capture the textural features and central position of Nanmadol, characterized by a low cloud-top temperature and distinct core region. However, significant differences emerge by T+3 (2022-09-14-09:00 UTC). Without FuXi forecasts, SATcast-NoF demonstrates significant degradation in its ability to resolve the typhoon’s structure, and SATcast-NoC produces only a loosely organized system. In contrast, SATcast maintains robust skill in delineating both the eye of the typhoon and its structural integrity (further details are provided in Supplementary Figure S7). By T+8 (2022-09-14-14:00 UTC), these differences become more pronounced. SATcast-NoC and SATcast-NoF predict only fragmented and sporadic convection patterns, while SATcast successfully captures the intensification and movement of the tropical cyclone. Although SATcast exhibits some discrepancies in the detailed organization of convection compared to observations, it remains the most reliable model in maintaining the typhoon’s overall structure and dynamics.

As Typhoon Muifa approached the east coast of China, it gradually weakened and split into two cloud clusters by T+3. SATcast demonstrated superior skill in accurately capturing both the spatial evolution and intensity variations during this process. By T+8, SATcast’s predictions exhibit a slight southward bias in the location of convective clouds north of Shanghai compared to observations. Despite this, SATcast still accurately forecasts the variations in convection intensity and shape. In contrast, although SATcast-NoC and SATcast-NoF also predict a weakening system, their representations of convective cloud organization show significant morphological inaccuracies.

Notably, satellite observations indicate that scattered convective clouds over Southeast Asia and the South China Sea intensified over time. SATcast accurately predicts this strengthening at T+3, and its forecast at T+8 remains acceptable. In comparison, both SATcast-NoC and SATcast-NoF significantly underestimate the intensity of these convective clouds at T+3 and T+8. Additional cases, including those with and without tropical cyclones, are detailed in Supplementary Figures S4-S7. In these cases, SATcast consistently outperforms its variants in predicting changes in the intensity and organization of convective clouds, while the variants often erroneously predict weakening and dispersion of convective systems.

Permutation feature importance is a valuable technique for interpreting multimodal deep learning models, especially in atmospheric studies [34, 31]. This method quantifies the importance of each feature by selectively shuffling specific feature dimensions and measuring the resulting performance degradation. It helps identify key physical parameters for predicting satellite imagery and refining the model. To address the computational cost of repeated shuffling, we propose a threshold-based permutation strategy (detailed in Supplementary Information). This strategy enables effective shuffling with only one shuffle per batch, significantly reducing computational costs.

Figure 5 presents heatmaps of the normalized mean squared error (N-MSE) ratios between permuted and baseline predictions ($\frac{MSE_{f}}{MSE_{m}}$). Higher ratios correspond to greater feature importance, highlighting the most influential features for model predictions.

Figure 5a shows the feature importance, measured by N-MSE variations, for the eight hours of historical satellite imagery and FuXi predicted variables. For T+1 to T+3, the N-MSE values from shuffled satellite imagery are significantly higher than those from the FuXi data, indicating the essential role of satellite observations during this period. However, as the lead time increases, the importance of satellite imagery gradually diminishes, while the significance of FuXi data grows, ultimately surpassing that of satellite data at T+8. This shift highlights the increasing relevance of FuXi data, which provides insights into future atmospheric patterns, for predictions over longer lead times. It also explains why SATcast-NoC outperforms SATcast-NoF after T+3.

Figure 5b examines the importance of individual FuXi variables, all of which exhibit relatively low N-MSE values (below 1.1), with several parameters emerging as particularly impactful. Variables across different pressure levels, including high (250 hPa), middle (500 and 700 hPa), and low (850 hPa), are selected to ensure comprehensive representation. Notably, the importance of these variables varies with forecast lead time. Key features generating N-MSE about 1.05 include geopotential at 250 and 850 hPa (z250 and z850), specific humidity at 250 and 500 hPa (q250 and q500), near-surface temperature (t2m), and mean sea-level pressure (msl). z250 and z850 influences atmospheric motions, such as convergence and divergence patterns, through geostrophic balance and adjustment, thereby guiding cloud movement and intensity changes. q250 and q500, which are closely tied to clouds, exhibit a more pronounced influence from T+5 to T+8, whereas lower-level specific humidity (q850) has less impact. Surface temperature and pressure contribute thermodynamic and dynamic forcing to the atmosphere, affecting prediction accuracy. Furthermore, msl and geopotential (z250 and z850) serve as practical indicators of convection centers and tropical cyclones. These findings demonstrate that incorporating critical atmospheric fields significantly enhances the performance of the SATcast models.

The results obtained by SATcast on channel 9 (Supplementary Figures S9-S10) further demonstrate the model’s capacity to interpret physical information from a different perspective. Brightness temperature on Channel 9, which represent high-level water vapor, exhibit greater stability, leading to superior performance across various metrics compared to those from channel 12. This distinction can be attributed to the fact that, for shorter lead times, SATcast operates more like a video prediction model, heavily relying on satellite imagery for prediction. However, for longer lead times, the model’s performance declines rapidly due to inconsistencies between the targets detected by channels 9 (high-level water vapor) and 12 (cloud). This mismatch hinders the model’s ability to accurately interpret the influence of physical variables on the target, resulting in a significant reduction in skill for channel 9 predictions.

We obtained the sequences by incorporating 13 physical variables (detailed in Supplementary Table 1) from FuXi’s hourly predictions, aligned with the spatiotemporal coordinates of the FY-4A satellite data.

Our analysis utilizes FY-4A AGRI Level 1 data from channel 12, which has a central wavelength of 10.7 µm, a spatial resolution of 4 km, and a temporal resolution of 1 hour, covering the period from 2019 to 2022. This channel primarily captures cloud top temperature and surface temperature. The data is then interpolated to a $0.25^{\circ}$ resolution to reduce computational costs while maintaining resolution consistent with FuXi data.

These sequences are combined with the 13 physical variables predicted by FuXi every 12 hours, aligned with the latitude, longitude, and time information from the satellite data. The resulting dataset is organized in a T, C, H, W format (12, 14, 160, 256), where T represents time steps, C denotes channels (including both satellite data and physical variables), and H and W correspond to the spatial dimensions in the latitude and longitude directions. The training dataset for SATcast and SATcast-NoF comprises 19,904 sequences, each spanning 12 hours. Additionally, a separate dataset for training SATcast-NoC includes 17,408 sequences, each with a duration of 16 hours. To evaluate the performance of SATcast and assess the impact of its modules, distinct validation sequences lasting 16 and 32 hours were also prepared.

In the forward process of the diffusion model, noise is incrementally added to the target. Let $P(x_{0})$ represent the sample distribution. The following discrete-time Gaussian process is defined:

where $x_{t}$ denotes the noise added at time step $t$ to $x_{0}$, and is calculated as: $x_{t}=\sqrt{\bar{\alpha_{t}}}x_{0}+\sqrt{1-\bar{\alpha_{t}}}\epsilon$, with $\epsilon$ being the noise sampled from $\mathcal{N}(0\sim 1)$. Here, $\alpha_{t}$ is a fixed schedule over $t$, and $\beta_{t}=1-\alpha_{t}$. The network is trained to learn the noise injected to the data. During sampling, the diffusion model reverses the forward diffusion process to recover $x_{t-1}$ from $x_{t}$, given by:

where $\tilde{\mu_{t}}(x_{t},x_{0}=\frac{1}{\sqrt{\alpha_{t}}}(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha_{t}}}}\epsilon_{\theta})$, $\bar{\alpha_{t}}=\prod_{1}^{t}\alpha_{t}$, and $\epsilon_{\theta}$ is the noise predicted by the network. Additionally, $\tilde{\beta_{t}}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$. Thus, $x_{t-1}$ can be sampled from the mean and variance [26, 37].

Consider a sequence of satellite image time series, where the nowcasting task predicts future $y$ frames satellite imageries based on a past $x$ frames satellite imageries and FuXi variables from time steps $\mathrm{T}-x-1$ to $\mathrm{T}+y$. During training, noise is added to the future $y$ frames at level $t$, while the past frames and FuXi variables serve as the conditions. Therefore, the reverse process is modified as:

The network training objective is simplified as follows:

where $\epsilon_{\theta}$ represents the noise predicted by the network.

The basic model, SATcast, is a cascade model that incorporates FuXi data, and undergoes a two-phase training, as illustrated in Figure 6b. In the first phase, the model learns to predict frames from T+1 to T+4 using past satellite frames from T$-$7 to T+0 and the corresponding FuXi predictions from T$-$7 to T+4. The second phase extends the prediction to frames T+5 to T+8, utilizing both past satellite frames from T$-$3 to T+0, the previously generated frames from T+1 to T+4, and the corresponding FuXi predictions. To optimize computational efficiency, SATcast-phase 2 is initialized with weights from SATcast-phase 1. We also present results from SATcast-NoT, where SATcast did not undergo fine-tuning in phase 2. The second variant, SATcast-NoC, adopts a direct prediction approach, predicting frames T+1 to T+8 simultaneously using the same input data as SATcast-phase 1. The third variant, SATcast-NoF excludes FuXi predictions as conditions while maintaining autoregressive prediction for all eight frames.

The model is developed using Pytorch [39]. Training the model takes approximately 20 hours on a cluster of 8 Nvidia H800 GPUs. The model is trained for 300 epochs using a batch size of 16 per GPU. The AdamW [40] optimizer is used with parameters $\beta_{1}=0.9$ and $\beta_{2}=0.99$, and the learning rate is initialized at $5.0\times 10^{-5}$ with a warm-up phase followed by cosine annealing. An exponential moving average is also applied during training [41]. The smooth L1 loss is used as the loss function.

To improve the model’s ability to predict severe cases, sequences containing category three typhoons are resampled once. Classifier-free guidance is employed during both training and sampling to improve image quality [42]. Additionally, offset noise is applied during sampling to reduce mismatches in data distribution [43].

For target regions spanning a wide range of latitude, we apply latitude-weighted RMSE, correlation coefficient and CSI, which are calculated as follows:

where $\tau$ is the forecast lead time, $\mathrm{T_{0}}$ is the initial time, and H and W correspond to the numbers of grid points in longitude and latitude, respectively. $a_{i}$ is the latitude weighting factor for the ith latitude index [44]:

CSI is commonly used for the evaluation of severe weather forecasts. TP refers to the number of correctly predicted positive pixels; FP is the number of negative pixels incorrectly predicted as positive; FN is the number of positive pixels incorrectly predicted as negative; and TN is the number of correctly predicted negative pixels.

MSSSIM combines Structural Similarity Index (SSIM) at multiple scales to yield more robust results. Latitude-weighting is not applied here, as the image is treated as a whole. It is calculated by:

The comparisons about luminance, contrast and structure are denoted by the three terms in equation 9, the $M$ means difference scales of the images. The three exponents adjust the relative importance of different components. More details about the calculation can be found in Wang el al. (2020) [32].