MMINR: Multi-frame-to-Multi-frame Inference with Noise Resistance for Precipitation Nowcasting with Radar

In precipitation nowcasting, ConvRNN based methods achieve an excellent effect. ConvLSTM [3] is the first ConvRNN based method for precipitation nowcasting. It combines convolutional operation with LSTM to extract spatial and temporal information. PredRNN [5] enhances the spatial information extraction ability of the model through extra spatial memory units. PredRNN++ [6] proposed a Causal LSTM unit and gradient highway unit to alleviate the gradient propagation difficulties. MIM [7] decomposes the spatial-temporal information into stationary and non-stationary features and then uses the corresponding unit to predict the features. Although the ConvRNN based methods are designed for spatiotemporal forecasting problem, it suffers from an unavoidable problem that they cannot be computed in parallel, resulting in more extended calculation and training time.

FCN based models can be solved by parallel computation but lack the sensitivity to temporal information. Many FCN based methods adopt a recurrent strategy. It inputs multi-frame to the model and outputs single frame as prediction result, then it feeds the result into the model again to predict the next frame result and finally multi-frame prediction results can be got. RainNet [8] follows the model of Unet [9], which adopts multi-frame-to-single-frame inference in the training phase and recurrent strategy in the testing phase to achieve multiple frames prediction as output. SmaAt-UNet [10] aims at lightweight. It reduces the number of convolutions in Unet and uses the attention module to compensate for missing semantic information, drastically reducing the model size. However, SmaAt-UNet only predicts one future frame and does not have the capability of continuous prediction. RDCNN [11] contains a recurrent structure based on CNN that is different from LSTM. It auto generates convolution kernel from input data. But RDCNN still does not avoid the accumulation of prediction errors. Therefore, recurrent prediction leading to prediction error accumulation is an urgent problem to be solved.

From the perspective of computer vision, we define the precipitation nowcasting problem as a sequence-to-sequence problem. In this task, the input data and prediction result are the sequences of radar echo maps. $X$ and $\hat{Y}$ denote the input data and the prediction result respectively. And $\hat{Y}$ has the same size as $X$. $X=\{x_{1},x_{2},...,x_{N}\}$ is a collection of $N$ radar echo maps $x_{i}$, each with the same size. $\hat{Y}=\{\hat{y_{1}},\hat{y_{2}},...,\hat{y_{M}}\}$ is a collection of $M$ prediction frames $\hat{y_{i}}$. So the prediction process can be defined as the following:

$$\hat{Y}=\Gamma(X).$$

(1)

where $\Gamma$ represents the precipitation nowcasting model.

ConvRNN based models and FCN based models contain different cyclic structure in precipitation nowcasting, as shown in Fig. LABEL:fig:differentModel. FCN based model receives multiple frames as input and outputs a single frame as a prediction result. Then it feed the prediction result back into the model. Such process can be defined as:

$$\hat{y}_{n+1}=F(x_{0},x_{1},...,x_{n}),$$

(2)

$$\hat{y}_{n+2}=F(x_{1},...,x_{n},\hat{y}_{n+1}).$$

(3)

where $x_{i}$ and $\hat{y}_{j}$ represent the i-th input frame and the j-th prediction result, respectively. $F$ means FCN based model. Many FCN based models will predict a single frame in the training phase, parallelize computation, and use the recurrent prediction method mentioned above in the testing phase.

ConvRNN based models receive only one frame as input at each time step and uses memory units to transfer features between time steps, as shown in (a) of Fig. LABEL:fig:differentModel. Both FCNs or ConvRNN based methods have the same characteristic. In the testing phase, they take the output from the previous time step as input of the next step. Due to the output at the previous time step must contain the prediction error, the next time step will accumulate error in the recurrent process.

As shown in (c) of the Fig. LABEL:fig:differentModel, we propose the Multi-frame-to-Multi-frame Inference (MMI) model to avoid error accumulation. It takes a sequence (i.e., multiple frames) as input and directly outputs a sequence. The total process can be written as:

$$\hat{y}_{n+1},\hat{y}_{n+2},...,\hat{y}_{n+m}=\phi(x_{0},x_{1},...,x_{n}).$$

(4)

in which $\phi$ means MMI. $n$ and $m$ represent the number of input and output frames. In general, the last layer in other FCN based model are a single convolution operation with one output channel. But in our method, the last convolution layer has $m$ output channels.

Due to the time interval between two continuous radar frames usually up to several minutes, some rainfall areas vanish, arise, or deform suddenly between frames. These areas are almost unpredictable and should be seen as precipitation noises. Those noises will be transmitted to deeper layers, resulting in the model performance degradation in the classical FCN based model, e.g., RainNet [8]. To address this problem, we design a novel Noise Resistance (NR) strategy, which is composed of NDM in encoder and SRM in decoder.

Our model is trained and tested in a precipitation nowcasting benchmark database proposed by Koninklijk Nederlands Meteorologisch Instituut (KNMI). This database has 420,000 radar echo maps of Netherlands in 5-min intervals and [10] converts the radar maps to rainfall maps. It contains 5734 frame sequences in the training set, 1557 sequences in the test set. We chose randomly 5000 sequences from the default training set as the training set and the rest of 734 as the validation set. Every sequence includes 18 frames, with the size of 288$\times$288 pixels. As shown in Table II, we divide the rainfall maps by four thresholds, 0.5 mm/h, 2 mm/h, 5 mm/h, 10 mm/h. They represent light rain, light to moderate rain, moderate rain, and moderate to heavy rain respectively. We train all methods in our experiments on training set. The results of all methods are obtained on testing set. In order to avoid sudden rainfall intensity that harms the training process, we assign 19 mm/h to any rainfall intensity bigger than 19mm/h. For calculation convenience, we deal data with formula as

$$y=\frac{ln(e,x+1)}{1.5}-1$$

(7)

$x$ represents raw precipitation data, $y$ represents the processed data. This formula keeps the data between -1 and 1 and allocates a suitable interval for high rainfall intensity.

For comprehensive evaluation, we adopt two binary metrics and two non-binary metrics. Binary metrics binarize the prediction results and focus on the accuracy of prediction results on different rainfall intensity predictions. Non-binary metrics concentrate on the similarity of the prediction results and GT. Critical Success Index (CSI) and Heidke Skill Score (HSS) [14] are binary metrics. CSI measures the fraction of observed and/or forecast events that are correctly predicted and HSS measures the ratio of the odds of making a hit to the odds of making a false alarm. All binary evaluation metrics except correlation are based on true negative (TN), false positive (FP), false negative (FN), and true positive (TP). For binary rainfall map, we define 0.5mm/h, 2mm/h, 5mm/h, 10mm/h as the specific threshold. Due to the extreme imbalance of the precipitation dataset, [15] proposes Balance-MSE (B-MSE) and Balance-MAE (B-MAE) which assign weights to rainfall areas according to rainfall intensities to mitigate the negative impact of data imbalance.

PyTorch implements MMINR with a station equipped with NVIDIA GeForce RTX 2080Ti GPU. We set 0.0001 as the initial learning rate and use Adam optimizer for stochastic gradient descent method. We use a mini-batch of 16 sequences. B-MAE is used as the validation loss function. When the validation loss is no longer decreasing during the training phase, the model with the smallest validation loss is selected as model well trained for prediction.

For a comprehensive comparison between MMINR and other SOTAs, as shown in Tab. III, we classify all methods into four categories based on the training and testing strategies used in the original papers. For fair comparison and to explore the model performance, we also provide variants of our models for comparison with other FCN based methods.

As shown in Fig. 5, we provide a visual comparison of the proposed MMINR with other SOTAs. Our method is more accurate than other ConvRNN models in predicting the direction and the trend of rainfall movement. Besides, our model has better robustness in predicting moderate to heavy rainfall (rainfall intensity greater than 10 mm/h). Although RainNet consistently predicts heavy rainfall, most predictions are wrong, and the predicted rainfall area has severe deformation, which is mainly due to error accumulation.

To investigate the importance of NR, we compare MMINR without NR (denoted by MMINR-NDM-SRM) and with NDM (denoted by MMINR-SRM), as shown in Tab. I. From this table, we can see that MMINR-SRM outperforms MMINR-SRM-NDM in all indicators, especially at rainfall intensity above 10 mm/h, with CSI and HSS improving by 43% ($\frac{0.0364-0.0253}{0.0253}=0.4387$) and 41%, respectively. This also shows that NDM does reduce the model’s attention to noise and improves the ability to mine semantic information.

In addition, comparing MMINR with MMINR-SRM illustrates the importance of SRM. The data show that all metrics results of MMINR are better than without SRM, and all of them have a substantial improvement, which indicates that SRM does play a complementary semantic role.

Finally, comparing MMINR with MMIR without NR (MMINR-NDM-SRM) shows the importance of the proposed NR strategy, in which MMIMR outperforms MMINR-NDM-SRM in all metrics.