Unsupervised Cumulative Domain Adaptation for Foggy Scene Optical Flow

Depth Association. Given consecutive stereo images ${[\textbf{\emph{I}}_{t}^{l},\textbf{\emph{I}}_{t+1}^{l},\textbf{\emph{I}}_{t}^{r},\textbf{\emph{I}}_{t+1}^{r}]}$, we first take RAFT [36] to estimate optical flow F with optical flow photometric loss [44] in an unsupervised manner as follow,

where warp is the warping operator, $\psi$ is a sparse $L_{p}$ norm ($p=0.4$). $O_{f}$ and $O_{b}$ are the forward and backward occlusion mask by checking forward-backward consistency, and $\odot$ is a matrix element-wise multiplication. Similar to optical flow, stereo depths ${[\textbf{\emph{D}}_{t}^{l},\textbf{\emph{D}}_{t+1}^{l}]}$ can be obtained by DispNet [39] via photometric loss and smooth loss,

Motion Knowledge Transfer. To associate depth with fog, we synthesize the foggy images ${[\textbf{\emph{J}}_{t}^{s},\textbf{\emph{J}}_{t+1}^{s}]}$ corresponding to the clean images using atmospheric scattering model [26],

where A denotes the predefined atmospheric light. ${\emph{t}(\cdot)}$ is a decay function related depth. We then encode the synthetic foggy images into motion features ${[{f}^{s}_{t},{f}^{s}_{t+1}]}$, and compute the temporal cost volume ${\emph{cv}_{temp}=({f}^{s}_{t})^{T}\cdot w({f}_{t+1}^{s})}$, where T denotes transpose operator and w is the warp operator. Note that, in order to enable the flow model to have a suitable receptive field for smooth constraint of motion feature, we employ a spatial context attention (SCA) module with a non-local strategy [22]. Specifically, we devise a sliding window with a learnable kernel on the motion feature ${f}^{s}_{t}$ to match the non-local similar feature ${{f}_{sim}}$, and generate k similar features corresponding to the cropped features from the motion feature ${f}^{s}_{t}$ during sliding searching, as ${[{f}_{sim}^{1},{f}_{sim}^{2},...,{f}_{sim}^{k}]}$. And then we compute the spatial attention cost volume,

Correlation Distribution Alignment (CDA). We begin with two encoder ${\textbf{\emph{E}}_{s}},{\textbf{\emph{E}}_{r}}$ for the synthetic foggy images ${[\textbf{\emph{J}}_{t}^{s},\textbf{\emph{J}}_{t+1}^{s}]}$ and the real foggy images ${[\textbf{\emph{J}}_{t}^{r},\textbf{\emph{J}}_{t+1}^{r}]}$, respectively. We encode them to obtain the cost volume ${\emph{cv}_{s},\emph{cv}_{r}}$ with the warp operator and the SCA module. Furthermore, we randomly sample N correlation values in the cost volumes ${\emph{cv}_{s},\emph{cv}_{r}}$ to represent the entire correlation distribution of cost volume normalized into [0, 1] for both the domains. According to the range of correlation, we choose threshold values ${[\delta_{1},\delta_{2},...,\delta_{k-1}]}$ to label the sampled correlation into k classes, such as high correlation, and low correlation. Then, the correlation distribution p is estimated by,

where n is the number of the sampled correlation of one category. Note that we add an offset 1 to each category sampled correlation of Eq. 8 to ensure at least a single instance could be present in the real foggy domain. Thus, to align the features of the synthetic and real foggy domains, we minimize the correlation distribution distance between the two domains by enforcing Kullback-Leibler divergence,

where ${\emph{p}_{s,i}}$, ${\emph{p}_{r,i}}$ denote the i category sampled correlation distributions of the synthetic foggy domain and the real foggy domain, respectively. The aligned correlation distributions represent that both the domains could have similar optical flow estimation capabilities. Finally, we decode the aligned cost volume to predict optical flow for real foggy images.

Self-Supervised Training Strategy. To improve the robustness of knowledge transfer, we present a self-supervised training strategy that attempts to transfer motion knowledge from the synthetic foggy domain to the real foggy domain at the optical flow field level. We feed the real foggy images ${[\textbf{\emph{J}}_{t}^{r},\textbf{\emph{J}}_{t+1}^{r}]}$ to the flow network of the synthetic foggy domain, which outputs the optical flow as the pseudo-labels ${\textbf{\emph{F}}_{pseudo}}$ (seeing the red arrow in Fig. 2). We then impose a self-supervised loss on the optical flow ${\textbf{\emph{F}}_{real}}$ estimated by the flow network of the real foggy domain,

Dataset. We take the KITTI2015 [25] dataset as the representative clean scene. We validate the performance of optical flow on one synthetic and three real foggy datasets.

$\bullet$ Fog-KITTI2015. We construct a synthetic foggy KITTI dataset with different densities of fog (e.g., dense fog and light fog) onto images of KITTI2015 [25] via atmospheric scattering model [26]. We select 8400 images of Fog-KITTI2015 dataset for training and 400 images for testing.

$\bullet$ Fog-GOF. GOF [14] is a dataset containing four different scenes with synchronized gyro readings, such as regular scenes and adverse weather scenes. We choose foggy images of GOF to compose a new foggy dataset, namely Fog-GOF, of which 1000 images for training and 105 images for testing.

$\bullet$ DenseFog. We seek the real foggy dataset collected by DenseFogFlow [40], namely DenseFog, of which 2346 images for training and 100 images for testing.

$\bullet$ Real-Fog World. We collect degraded videos under real foggy scenes from Youtube, with 1200 and 240 images for training and testing, respectively.

Comparison Methods. We choose three competing methods GyroFlow [14], DenseFogFlow [40] and RobustFlow [15] which are designed for adverse weather optical flow. Moreover, we select several state-of-the-art supervised (GMA [9]) and unsupervised (SMURF [32], UFlow [11] and Selflow [19]) optical flow approaches designed for clean scenes. The unsupervised methods are first trained on KITTI2015 for initialization and re-trained on the target degraded dataset. The supervised method is first trained on the synthetic dataset, and then trained on target real datasets via self-supervised learning [32], denoted with ‘ssl’ in Table. 5. As for the comparison on Fog-KITTI2015, we design two different training strategies for competing methods. The first is that we directly train the comparison methods on foggy images. The second is to perform the defogging first via defog approaches (e.g., FFA-Net [27] and AECR-Net [38]), and then we train the comparison methods on the defogging results (named as FFA-Net+ / AECR-Net+).

Evaluation Metrics. We choose average endpoint error (EPE [4]) and the lowest percentage of flow outliers (F1-all [25]) as evaluation metrics for the quantitative evaluation. The smaller the index is, the better the predicted result is.

Effectiveness of Cumulative Adaptation Architecture. To illustrate the effectiveness of cumulative DAMA-CAMA architecture, in Fig. 7, we show the optical flow estimation of different adaptation strategies and visualize their low-dimensional distributions via t-SNE. In Fig. 7 (b), we can observe that there exist artifacts in the motion boundary without domain adaptation. With DAMA only in Fig. 7 (c), most of the outliers caused by degradation are removed. with both DAMA and CAMA in Fig. 7 (d), the motion boundary is clearer. Moreover, we visualize their corresponding t-SNE distribution in Fig. 7 (e). The blue, red, and yellow diamonds denote the distributions without domain adaptation, with DAMA only and with DAMA-CAMA, respectively. The blue distribution is scattered, the red distribution is gradually focused, and the yellow distribution is most concentrated, illustrating that the cumulative domain adaptation could progressively improve real foggy scene optical flow.

Effectiveness of Adaptation Losses. We study how the adaptation losses of the proposed method contribute to the final result as shown in Table 3. $\mathcal{L}^{geo}_{flow}$ aim to enforce the optical flow in rigid regions. $\mathcal{L}^{consis}_{flow}$ is to transfer motion knowledge from the clean domain to the synthetic foggy domain. The goal of $\mathcal{L}^{self}_{flow}$ and $\mathcal{L}^{kl}_{corr}$ is to distill motion knowledge of the synthetic foggy domain to the real foggy domain. We can observe that the motion consistency loss $\mathcal{L}^{consis}_{flow}$ make a major contribution to the optical flow result, and the correlation distribution alignment loss $\mathcal{L}^{kl}_{corr}$ can further improve the optical flow under real foggy scenes.

How dose the Depth Improve Optical Flow? We study the importance of depth in transferring motion knowledge from clean domain to synthetic foggy domain in Fig. 8. Without depth association in Fig. 8 (b), the rigid motion boundaries are blurry. With depth association in Fig. 8 (c), the optical flow is global-smooth with sharp boundaries. Besides, we visualize their training iteration process in Fig. 8 (d). We can observe that depth association can further improve the optimal value that the proposed model converges to. Therefore, the depth geometry association can enhance the rigid motion boundary for synthetic foggy images.

Importance of Correlation Distribution Alignment. In Table 4, we show the effect of different modules on the optical flow of real foggy domain. EMA is to prevent the weights of the network from falling into the local optimum at the training stage. SCA aims to enhance the motion saliency in the cost volume. CDA is to transfer motion knowledge from the synthetic foggy domain to the real foggy domain by aligning the correlation distributions of both the domains. As shown in Table 4, EMA and SCA contribute a small improvement on the optical flow, while the CDA plays a key role in improving the optical flow of real foggy domain.

Why Associate Depth with Fog? We also study the effect of different foggy image synthesis strategies on the optical flow in Table 5. The GAN-based strategy cannot perform well. The reason is that GAN may erratically produce some new artifacts during the image translation, but instead exacerbate the synthetic-to-real foggy domain gap. On the contrary, since fog is a non-uniform degradation related to depth, it is reasonable that we use depth to synthesize foggy images. Note that, the depth estimated by monocular is not accurate enough due to the weak constraints. We also upsample the sparse depth in KITTI dataset into the dense depth to synthesize the foggy images (pseudo-GT strategy), while this strategy is slightly inferior to our stereo-based strategy. The proposed depth association motion adaptation could make a positive contribution to motion knowledge transfer.

Limitation. In Fig. 9, we discuss the limitation of the proposed UCDA-Flow. Compared with the SOTA optical flow method GMA [9], UCDA-Flow obtains the clearer motion boundary in the too-distantons, but fails for the too distant moving objects under foggy scenes. There are two reasons for this problem. First, our framework requires depth to enhance the optical flow in rigid regions, but it is difficult for the stereo strategy to obtain accurate depth in distant regions. Second, degradation is so severe that the details of the distant moving object are lost. In the future, we attempt to employ lidar for detecting distant objects.